Setting up a FullMesh network on Mikrotik via EoIP tunnels
The initial situation is this: there are 8 offices in different parts of the country, they need to be brought together into a single network so that the availability of each office is maximum in any disasters. As routers in all offices are Mikrotik. On the main site - CCR CCR1036-12G, on the rest - 1100 AHx2
In order to avoid problems with the Internet, 2 channels were stretched from different providers, food was also reserved and they came to the question “what kind of network to build?”. As the title of the article shows, they finally decided to build FullMesh.
This scheme fully satisfies the requirements of the management - if any Internet channel or even any office fails, the network remains connected. There was only a question with routing. Of the options was a universal bridge with RSTP, OSPF, and static routes. Naturally, I ended up choosing OSPF - less problems than static and less load for routers than with RSTP.
')
The setting itself and the finished config under the cut.
I decided to connect routers using EoIP tunnels, 2 for each pair - from the main provider to the primary and from the backup to the backup. I will describe the configuration for one pair, since the others are configured identically.
On the first router we create 2 tunnels:
Raise the tunnels from the second side:
We configure OSPF, we will exchange routes through the backbone zone.
On the first router:
On the second router:
Finally, add the addresses for the created tunnels:
On the first router:
On the second router:
We get 2 routers that exchange routes to their OSPF zones. Repeat this procedure for all pairs of routers.
As a result, we get just such a FullMesh network (I apologize in advance for the quality of the scheme - I didn’t find a way to adequately draw the network scheme on Linux, so I used the Gliffy online drawing app):
All routers are included in the common backbone area with id 0.0.0.0 +, each of them is borderline for its own zone with an ID equal to the local IP of the router.
Immediately after the routers discover each other, they exchange the routes they know, and based on the cost of the link, they choose the best ones. In this configuration, the best route will always be the most direct (more precisely, 2 direct routes - the main and backup), in which case the traffic going through the failed route will be delivered through all the routers remaining available.
Thus, we were no longer scared by the technical work of the provider, as well as periodic problems with the patency of GRE packets in certain areas - for the full connectivity of the network, less than half of the existing tunnels are sufficient. Well, as a bonus, we received balancing of traffic between the tunnels - since the costs of the main and backup tunnels are the same, OSPF automatically sends traffic to both tunnels, balancing the load between them roughly equally.
If you have any questions or suggestions for optimizing this configuration, welcome to comments.
As promised, the ready config of one of the routers (all names and IP are changed by agreement with the security service):
UPD: This config is merged with the test labs, and not with the sale. From the moment of writing this article, in the run-up from this scheme, they managed to refuse in favor of L2VPN from providers with OSPF for routing.
In order to avoid problems with the Internet, 2 channels were stretched from different providers, food was also reserved and they came to the question “what kind of network to build?”. As the title of the article shows, they finally decided to build FullMesh.
This scheme fully satisfies the requirements of the management - if any Internet channel or even any office fails, the network remains connected. There was only a question with routing. Of the options was a universal bridge with RSTP, OSPF, and static routes. Naturally, I ended up choosing OSPF - less problems than static and less load for routers than with RSTP.
')
The setting itself and the finished config under the cut.
I decided to connect routers using EoIP tunnels, 2 for each pair - from the main provider to the primary and from the backup to the backup. I will describe the configuration for one pair, since the others are configured identically.
On the first router we create 2 tunnels:
/interface eoip ladd keepalive=1s,3 local-address=xxx.xxx.xxx.xxx name=FIRST remote-address=yyy.yyy.yyy.yyy tunnel-id=1 add keepalive=1s,3 local-address=zzz.zzz.zzz.zzz name=FIRST_BAK remote-address=www.www.www.www tunnel-id=2 Raise the tunnels from the second side:
/interface eoip ladd keepalive=1s,3 local-address=yyy.yyy.yyy.yyy name=FIRST remote-address=xxx.xxx.xxx.xxx tunnel-id=1 add keepalive=1s,3 local-address=www.www.www.www name=FIRST_BAK remote-address=zzz.zzz.zzz.zzz tunnel-id=2 We configure OSPF, we will exchange routes through the backbone zone.
On the first router:
/routing ospf area add area-id=192.168.0.1 name=FIRST /routing ospf interface add cost=10 dead-interval=5s hello-interval=1s interface=FIRST \ network-type=point-to-point use-bfd=yes add cost=10 dead-interval=5s hello-interval=1s interface=FIRST_BAK \ network-type=point-to-point use-bfd=yes /routing ospf network add area=FIRST network=192.168.0.0/24 add area=backbone network=10.0.0.0/22 On the second router:
/routing ospf area add area-id=192.168.1.1 name=SECOND /routing ospf interface add cost=10 dead-interval=5s hello-interval=1s interface=FIRST \ network-type=point-to-point use-bfd=yes add cost=10 dead-interval=5s hello-interval=1s interface=FIRST_BAK \ network-type=point-to-point use-bfd=yes /routing ospf network add area=SECOND network=192.168.1.0/24 add area=backbone network=10.0.0.0/22 Finally, add the addresses for the created tunnels:
On the first router:
ip address add address=10.0.1.1/30 interface=FIRST network=10.0.1.0 ip address add address=10.0.1.5/30 interface=FIRST_BAK network=10.0.1.4 On the second router:
ip address add address=10.0.1.2/30 interface=FIRST network=10.0.1.0 ip address add address=10.0.1.6/30 interface=FIRST_BAK network=10.0.1.4 We get 2 routers that exchange routes to their OSPF zones. Repeat this procedure for all pairs of routers.
As a result, we get just such a FullMesh network (I apologize in advance for the quality of the scheme - I didn’t find a way to adequately draw the network scheme on Linux, so I used the Gliffy online drawing app):
All routers are included in the common backbone area with id 0.0.0.0 +, each of them is borderline for its own zone with an ID equal to the local IP of the router.
Immediately after the routers discover each other, they exchange the routes they know, and based on the cost of the link, they choose the best ones. In this configuration, the best route will always be the most direct (more precisely, 2 direct routes - the main and backup), in which case the traffic going through the failed route will be delivered through all the routers remaining available.
Thus, we were no longer scared by the technical work of the provider, as well as periodic problems with the patency of GRE packets in certain areas - for the full connectivity of the network, less than half of the existing tunnels are sufficient. Well, as a bonus, we received balancing of traffic between the tunnels - since the costs of the main and backup tunnels are the same, OSPF automatically sends traffic to both tunnels, balancing the load between them roughly equally.
If you have any questions or suggestions for optimizing this configuration, welcome to comments.
As promised, the ready config of one of the routers (all names and IP are changed by agreement with the security service):
Config
# aug/23/2015 19:15:28 by RouterOS 6.30.2 # software id = 4RCZ-RTPX # /interface ethernet set [ find default-name=ether10 ] mac-address=4C:5E:0C:5A:64:22 name=\ ISP2 set [ find default-name=ether1 ] mac-address=4C:5E:0C:5A:64:19 set [ find default-name=ether2 ] mac-address=4C:5E:0C:5A:64:1A master-port=\ ether1 set [ find default-name=ether3 ] mac-address=4C:5E:0C:5A:64:1B master-port=\ ether1 set [ find default-name=ether4 ] mac-address=4C:5E:0C:5A:64:1C master-port=\ ether1 set [ find default-name=ether5 ] mac-address=4C:5E:0C:5A:64:1D master-port=\ ether1 set [ find default-name=ether6 ] mac-address=4C:5E:0C:5A:64:1E set [ find default-name=ether7 ] mac-address=4C:5E:0C:5A:64:1F set [ find default-name=ether8 ] mac-address=4C:5E:0C:5A:64:20 set [ find default-name=ISP1 ] mac-address=4C:5E:0C:5A:64:21 name=ISP1 set [ find default-name=ether11 ] mac-address=4C:5E:0C:5A:64:23 set [ find default-name=ether12 ] mac-address=4C:5E:0C:5A:64:24 set [ find default-name=ether13 ] mac-address=4C:5E:0C:5A:64:25 /interface eoip add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:74:DC:6B:70:C1 \ name=FIRST remote-address=xxx.xxx.xxx.xxx tunnel-id=1 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:74:DC:6B:70:C1 \ name=FIRST_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=2 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \ name=SECOND remote-address=xxx.xxx.xxx.xxx tunnel-id=3 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \ name=SECOND_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=4 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \ name=THIRD remote-address=xxx.xxx.xxx.xxx tunnel-id=5 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \ name=THIRD_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=6 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \ name=FOURTH remote-address=xxx.xxx.xxx.xxx tunnel-id=7 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \ name=FOURTH_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=8 add keepalive=1s,3 local-address=xxx.xx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \ name=FIFTH remote-address=xxx.xxx.xxx.xxx tunnel-id=9 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \ name=FIFTH_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=10 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \ name=SIX remote-address=xxx.xxx.xxx.xxx tunnel-id=11 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \ name=SIX_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=12 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \ name=SEVENTH remote-address=xxx.xxx.xxx.xxx tunnel-id=13 add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \ name=SEVENTH_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=14 /routing ospf area add area-id=192.168.0.1 name=LOCAL /snmp community set [ find default=yes ] addresses=192.168.0.0/16 /system logging action set 0 memory-lines=100 set 1 disk-lines-per-file=100 /tool user-manager customer set admin access=\ own-routers,own-users,own-profiles,own-limits,config-payment-gw /user group set read policy="read,test,winbox,sniff,sensitive,!local,!telnet,!ssh,!ftp,!re\ boot,!write,!policy,!password,!web,!api" /ip firewall connection tracking set generic-timeout=1m tcp-close-timeout=5s tcp-close-wait-timeout=5s \ tcp-established-timeout=1m tcp-fin-wait-timeout=5s tcp-last-ack-timeout=\ 5s tcp-time-wait-timeout=5s udp-stream-timeout=1m /ip address add address=192.168.0.1/24 interface=ether1 network=192.168.0.0 add address=xxx.xxx.xxx.xxx/xx interface=ISP2 network=xxx.xxx.xxx.xxx add address=xxx.xxx.xxx.xxx/xx interface=ISP1 network=xxx.xxx.xxx.xxx add address=10.0.1.18/30 interface=FIRST_BAK network=10.0.1.16 add address=10.0.1.46/30 interface=SECOND network=10.0.1.44 add address=10.0.1.50/30 interface=SECOND_BAK network=10.0.1.48 add address=10.0.1.73/30 interface=THIRD network=10.0.1.72 add address=10.0.1.77/30 interface=THIRD_BAK network=10.0.1.76 add address=10.0.1.86/30 interface=FOURTH network=10.0.1.84 add address=10.0.1.90/30 interface=FOURTH_BAK network=10.0.1.88 add address=10.0.1.94/30 interface=FIFTH network=10.0.1.92 add address=10.0.1.98/30 interface=FIFTH_BAK network=10.0.1.96 add address=10.0.1.102/30 interface=FIRST network=10.0.1.100 add address=10.0.1.217/30 interface=SIX network=10.0.1.216 add address=10.0.1.221/30 interface=SIX_BAK network=10.0.1.220 add address=10.0.1.225/30 interface=SEVENTH network=10.0.1.224 add address=10.0.1.229/30 interface=SEVENTH_BAK network=10.0.1.228 /ip dns set servers=8.8.4.4 /ip firewall filter add action=drop chain=input comment="Drop Invalid" connection-state=invalid add action=drop chain=forward connection-state=invalid add chain=input comment=Established connection-state=established add chain=input comment=Tunnels protocol=gre add chain=input comment=WinBox dst-port=8291 protocol=tcp add chain=input comment=NTP dst-port=123 protocol=udp add chain=input comment="From Local" src-address=192.168.0.0/16 add chain=input comment=Ping protocol=icmp add action=drop chain=input comment="DONT MOVE - DROP" in-interface=\ ISP2 add action=drop chain=input comment="DONT MOVE - DROP" in-interface=\ ISP1 /ip firewall mangle add action=change-mss chain=forward new-mss=clamp-to-pmtu protocol=tcp \ tcp-flags=syn tcp-mss=1460-65535 /ip firewall nat add action=masquerade chain=srcnat out-interface=\ ISP2 add action=masquerade chain=srcnat out-interface=ISP1 /ip firewall service-port set ftp disabled=yes set tftp disabled=yes set irc disabled=yes set h323 disabled=yes set sip disabled=yes set pptp disabled=yes /ip route add check-gateway=ping distance=1 gateway=8.8.8.8 add distance=2 gateway=10.10.20.1 add check-gateway=ping distance=1 dst-address=8.8.8.8/32 gateway=10.10.10.1 scope=10 /ip service set telnet disabled=yes set ftp disabled=yes set www disabled=yes set ssh disabled=yes set api disabled=yes /routing ospf interface add interface=FIRST network-type=point-to-point add interface=FIRST_BAK network-type=point-to-point add interface=SECOND network-type=point-to-point add interface=SECOND_BAK network-type=point-to-point add interface=THIRD network-type=point-to-point add interface=THIRD_BAK network-type=point-to-point add interface=FOURTH network-type=point-to-point add interface=FOURTH_BAK network-type=point-to-point add interface=FIFTH network-type=point-to-point add interface=FIFTH_BAK network-type=point-to-point add interface=SIX network-type=point-to-point add interface=SIX_BAK network-type=point-to-point add interface=SEVENTH network-type=point-to-point add interface=SEVENTH_BAK network-type=point-to-point add interface=ether1 network-type=broadcast passive=yes /routing ospf network add area=backbone network=10.0.0.0/22 add area=FIRST network=192.168.0.0/24 /system clock set time-zone-autodetect=no time-zone-name=Europe/Kiev /system identity set name=MikroTik /system resource irq rps set ether1 disabled=yes set ether2 disabled=yes set ether3 disabled=yes set ether4 disabled=yes set ether5 disabled=yes set ether6 disabled=yes set ether7 disabled=yes set ether8 disabled=yes set ISP1 disabled=yes set ISP2 disabled=yes set ether11 disabled=yes /system scheduler add interval=6h name=schedule1 on-event=BackupToMail policy=\ ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=\ oct/28/2014 start-time=01:00:00 add interval=6h name=schedule2 on-event=BackupToFTP policy=\ ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=\ oct/28/2014 start-time=01:00:00 /system script add name=BackupToMail owner=root policy=\ ftp,reboot,read,write,policy,test,password,sniff,sensitive source="{\r\ \n:log info \"Starting Backup Script...\";\r\ \n:local sysname [/system identity get name];\r\ \n:local sysver [/system package get system version];\r\ \n:log info \"Flushing DNS cache...\";\r\ \n/ip dns cache flush;\r\ \n:delay 2;\r\ \n:log info \"Deleting last Backups...\";\r\ \n:foreach i in=[/file find] do={:if ([:typeof [:find [/file get \$i name]\ \_\\\r\ \n\"\$sysname-backup-\"]]!=\"nil\") do={/file remove \$i}};\r\ \n:delay 2;\r\ \n:local smtpserv [:resolve \"smtp.gmail.com\"];\r\ \n:local Eaccount \"xxxxxx@gmail.com\";\r\ \n:local TOaccount \"xxxxxx@gmail.com\";\r\ \n:local pass \"xxxxxx\";\r\ \n:local backupfile (\"\$sysname-backup-\" . \\\r\ \n[:pick [/system clock get date] 7 11] . [:pick [/system \\\r\ \nclock get date] 0 3] . [:pick [/system clock get date] 4 6] . \".backup\ \");\r\ \n:log info \"Creating new Full Backup file...\";\r\ \n/system backup save name=\$backupfile;\r\ \n:delay 2;\r\ \n:log info \"Sending Full Backup file via E-mail...\";\r\ \n/tool e-mail send from=\"<\$Eaccount>\" to=\$TOaccount server=\$smtpserv\ \_\\\r\ \nport=587 user=\$Eaccount password=\$pass tls=yes file=\$backupfile \\\r\ \nsubject=(\"\$sysname Full Backup (\" . [/system clock get date] . \")\")\ \_\\\r\ \nbody=(\"\$sysname full Backup file see in attachment.\\nRouterOS version\ : \\\r\ \n\$sysver\\nTime and Date stamp: \" . [/system clock get time] . \" \" . \ \\\r\ \n[/system clock get date]);\r\ \n:delay 5;\r\ \n:local exportfile (\"\$sysname-backup-\" . \\\r\ \n[:pick [/system clock get date] 7 11] . [:pick [/system \\\r\ \nclock get date] 0 3] . [:pick [/system clock get date] 4 6] . \".rsc\");\ \r\ \n:log info \"Creating new Setup Script file...\";\r\ \n/export file=\$exportfile;\r\ \n:delay 2;\r\ \n:log info \"Sending Setup Script file via E-mail...\";\r\ \n/tool e-mail send from=\"<\$Eaccount>\" to=\$TOaccount server=\$smtpserv\ \_\\\r\ \nport=587 user=\$Eaccount password=\$pass tls=yes file=\$exportfile \\\r\ \nsubject=(\"\$sysname Setup Script Backup (\" . [/system clock get date] \ . \\\r\ \n\")\") body=(\"\$sysname Setup Script file see in attachment.\\nRouterOS\ \_\\\r\ \nversion: \$sysver\\nTime and Date stamp: \" . [/system clock get time] .\ \_\" \\\r\ \n\" . [/system clock get date]);\r\ \n:delay 5;\r\ \n:log info \"All System Backups emailed successfully.\\nBackuping complet\ ed.\";\r\ \n}" add name=BackupToFTP owner=antony policy=\ ftp,reboot,read,write,policy,test,password,sniff,sensitive source="# Set l\ ocal variables. Change the value in \"\" to reflect your environment.\r\ \n\r\ \n:local hostname [/system identity get name];\r\ \n:local password \"xxxxxx\"\r\ \n:local username \"xxxxxx\"\r\ \n:local ftpserver \"xxx.xxx.xxx.xxx\"\r\ \n\r\ \n# Set Filename variables. Do not change this unless you want to edit the\ \_format of the filename.\r\ \n\r\ \n:local time [/system clock get time];\r\ \n:local date ([:pick [/system clock get date] 0 3] \\\r\ \n. [:pick [/system clock get date] 4 6] \\\r\ \n. [:pick [/system clock get date] 7 11]);\r\ \n:local filename \"\$hostname-\$date-\$time\";\r\ \n\r\ \n# Create backup file and export the config.\r\ \n\r\ \nexport compact file=\"\$filename\"\r\ \n/system backup save name=\"\$filename\"\r\ \n\r\ \n:log info \"Backup Created Successfully\"\r\ \n\r\ \n# Upload config file to FTP server.\r\ \n\r\ \n/tool fetch address=\$ftpserver src-path=\"\$filename.rsc\" \\\r\ \nuser=\$username mode=ftp password=\$password \\\r\ \ndst-path=\"\$filename.rsc\" upload=yes\r\ \n\r\ \n# Upload backup file to FTP server.\r\ \n\r\ \n/tool fetch address=\$ftpserver src-path=\"\$filename.backup\" \\\r\ \nuser=\$username mode=ftp password=\$password \\\r\ \ndst-path=\"\$filename.backup\" upload=yes\r\ \n\r\ \n:log info \"Backup Uploaded Successfully\"\r\ \n\r\ \n# Delete created backup files once they have been uploaded\r\ \n# so they don't accumulate and fill up storage space on the router.\r\ \n\r\ \n/file remove \"\$filename.rsc\"\r\ \n/file remove \"\$filename.backup\"\r\ \n\r\ \n:log info \"Local Backup Files Deleted Successfully\"" /system watchdog set automatic-supout=no watchdog-timer=no /tool e-mail set address=64.233.161.109 from=<mikrotik> password=xxxxxx port=587 \ user=xxxxxx@gmail.com UPD: This config is merged with the test labs, and not with the sale. From the moment of writing this article, in the run-up from this scheme, they managed to refuse in favor of L2VPN from providers with OSPF for routing.
Source: https://habr.com/ru/post/315102/
All Articles