public:blog

SD-WAN multi-site L2VPN service with RouterOS 7.x (Part 2)

Hello all,

In the previous post I showed you how to create a SD-WAN multi-site L2VPN service using RouterOS7.x, with Wireguard and EoIP. However, I mentioned that there are other technologies/protocols that allow to create the same services. One example is VxLAN (Virtual Extensible Local Area Network), another protocol that allows to encapsulate a layer-2 broadcast domain on top of a layer-3 network (i.e., IP), effectively creating an overlay network. In VxLANs each tunnel is identified by a VNI (VxLAN Network Identifier) and the encapsulation/decapsulation is performed by a VTEP (VxLAN Tunnel End-Point).

To allow comparing both configuration we will use the same topolgoy as in the previous post, as depicted next. As you can see, we have three customer sites (i.e., pink, blue, green boxes) and the SD-WAN core (i.e., grey box). Eeach site has a Mikrotik router running RouterOS 7.x and is connected to the SD-WAN core, which is also running a router with RouterOS 7.x. For the IP addressing we use the CG-NAT address space (i.e., 100.64.0.0/8), but the same would work with a public address space; all is assumed is that the customer sites have IP connectivity over the Internet and that the SD-WAN core and the customer sites can be reached (i.e., required UDP ports on the firewall are open on both ends).

2022/05/04 08:52 · pere

SD-WAN multi-site L2VPN service with RouterOS 7.x

Hello all,

In this post I will show you how to create a SD-WAN multi-site L2VPN service using RouterOS 7.x. For those who don't know, a L2VPN service enables to inter-connect various remote sites over the Internet and integrate them into a single layer-2 broadcast domain, allowing them to interact as if they were connected to the same switch. In particular, we will create a L2VPN service with a star topology where the service controller is hosted in the cloud (i.e., in the SD-WAN core) and all the customer sites are connected to it. Other L2VPN topologies and technologies will be covered in future posts, so stay tuned.

To create the SD-WAN multi-site L2VPN service I will use RouterOS 7.x and various of its new features: Wireguard for establishing a site-to-site VPN between each remote site and the SD-WAN core, and EoIP tunnels to transport Ethernet frames on top of IP and UDP packets. To avoid the cable hassle I will also be using eve-ng, a web-based network emulator that supports both Mikrotik RouterOS 6.x and RouterOS 7.x using the CHR (Cloud Hosted Router) images.

Allright, so let's start by showing the network topology, with the three customer sites (i.e., pink, blue, green boxes) and the SD-WAN core (i.e., grey box), as depicted next. Eeach site has a Mikrotik router running RouterOS 7.x and is connected to the SD-WAN core, which is also running a router with RouterOS 7.x. For the IP addressing we use the CG-NAT address space (i.e., 100.64.0.0/8), but the same would work with a public address space; all is assumed is that the customer sites have IP connectivity over the Internet and that the SD-WAN core and the customer sites can be reached (i.e., required UDP ports on the firewall are open on both ends).

Now that the topology is clear, we start by showing the basic configuration for the SD-WAN core router and one of the customer sites routers that allows to have connectivity over the Internet. The configuration for the remaining customer site routers will be similar, but the Wireguard interfaces and EoIP tunnels will need to be configured propoerly. This is left as an exercise to the reader.

As it can be seen, the SD-WAN core basic configuration includes creating the IP pools to be delivered using DHCP on each public interface (i.e., 100.64.x.0/30, where x={1,2,3}), assigning the IP addresses on each public interface (i.e., 100.64.1.1/30 on ether2) and configuring the DHCP leases to provide the default gateway. The SD-WAN core router also has a DHCP client assigned to ether1 to provide connectivity to the Internet. This would allow the SD-WAN core router to act as the gatway for the whole customer network, allowing to provide a centralized firewall service, among others.

  1. /ip pool
  2. add name=dhcp_pool0 ranges=100.64.1.2
  3. add name=dhcp_pool1 ranges=100.64.2.2
  4. add name=dhcp_pool2 ranges=100.64.3.2
  5. add address-pool=dhcp_pool0 interface=ether2 name=dhcp1
  6. add address-pool=dhcp_pool1 interface=ether3 name=dhcp2
  7. add address-pool=dhcp_pool2 interface=ether4 name=dhcp3
  8. /ip address
  9. add address=100.64.1.1/30 interface=ether2 network=100.64.1.0
  10. add address=100.64.2.1/30 interface=ether3 network=100.64.2.0
  11. add address=100.64.3.1/30 interface=ether4 network=100.64.3.0
  12. /ip dhcp-client
  13. add interface=ether1
  14. /ip dhcp-server network
  15. add address=100.64.1.0/30 gateway=100.64.1.1
  16. add address=100.64.2.0/30 gateway=100.64.2.1
  17. add address=100.64.3.0/30 gateway=100.64.3.1

The basic configuration for the customer router is very simple too, as it only requires a DHCP client to obtain its public IP address from the Internet provider using the ether1 interface.

  1. /ip dhcp-client
  2. add interface=ether1

With such basic configuration all customer routers are able to ping the SD-WAN core router, allowing to start building the L2VPN service using Wireguard and EoIP. The first step is creating the Wireguard interfaces and peers on the SD-WAN core router, as well as on the customer site routers.

As it can be seen, the configuration on the SD-WAN core router is quite straightforward. We create the Wireguard interfaces and its remote peers, and then assign IP addresses to each. Notice that for the Wireguard interfaces we are using the 10.0.x.0/30 address space, where x=1,2,3 is assigned for each customer site.

  1. /interface wireguard
  2. add listen-port=625 mtu=1420 name=wg-cust1-site1
  3. add listen-port=60235 mtu=1420 name=wg-cust1-site2
  4. add listen-port=4981 mtu=1420 name=wg-cust1-site3
  5. /interface wireguard peers
  6. add allowed-address=0.0.0.0/0 endpoint-address=100.64.1.2 endpoint-port=62822 interface=wg-cust1-site1 public-key="DTBYLO9STZNLKtarao22fpNfYgS9rbJffTw6Jjf8aWw="
  7. add allowed-address=0.0.0.0/0 endpoint-address=100.64.2.2 endpoint-port=1536 interface=wg-cust1-site2 public-key="4YGgWzqUruH6YpCN0khF8wig+flAunK+r3BFSqhVxjQ="
  8. add allowed-address=0.0.0.0/0 endpoint-address=100.64.3.2 endpoint-port=11027 interface=wg-cust1-site3 public-key="B7UQZf6sNWApXRco8x79vjpHGiDO7csgRvDE1LMezRc="
  9. /ip address
  10. add address=10.0.1.1/30 interface=wg-cust1-site1 network=10.0.1.0
  11. add address=10.0.2.1/30 interface=wg-cust1-site2 network=10.0.2.0
  12. add address=10.0.3.1/30 interface=wg-cust1-site3 network=10.0.3.0

Similarly, on the customer router the configuration is as simple as creating the Wireguard interface and peer, and assigning the corresponding IP address. Notice how the endpoint-address, endpoint-port and public-key parameters have to match each other for the tunnels to be established. Also, we allow all traffic to flow through the tunnel (i.e., allowed-address=0.0.0.0/0), but that could be restriced if necessary.

  1. /interface wireguard
  2. add listen-port=62822 mtu=1420 name=wg-cust1-site1
  3. /interface wireguard peers
  4. add allowed-address=0.0.0.0/0 endpoint-address=100.64.1.1 endpoint-port=625 interface=wg-cust1-site1 public-key="s1L9bKuPTTXrAV8Dlk9rnDdRxFP7tZeaVOl3CkZdZDY="
  5. /ip address
  6. add address=10.0.1.2/30 interface=wg-cust1-site1 network=10.0.1.0

With such configuration now each customer router is able to reach the internal Wireguard interface (i.e., 10.0.x.1/30) on SD-WAN core router over the Internet connection (i.e., 100.64.0.0/8) using a secure (i.e., encyrpted) connection. This is an essential step, as setting up the EoIP tunnels requires IP connectivity between both routers, as shown next.

For the SD-WAN core router we need to create three EoIP interfaces, one for each customer site. Notice how each EoIP tunnel is configured to have the remote-address set to the IP address of the Wireguard interface on the other side (i.e., the customer router), as well a unique tunnel-id parameter that is set to 11 for the first customer site. For the other customer sites we will use a tunnel-id parameter set to 12 and 13, respectively. After that, we can create a bridge interface (i.e., bridge-cust1) and add each EoIP interface (i.e., eoip-cust1-site1) to it, allowing layer-2 traffic to flow between them.

  1. /interface eoip
  2. add mac-address=00:00:5E:80:00:01 name=eoip-cust1-site1 remote-address=10.0.1.2 tunnel-id=11
  3. add mac-address=00:00:5E:80:00:02 name=eoip-cust1-site2 remote-address=10.0.2.2 tunnel-id=12
  4. add mac-address=00:00:5E:80:00:03 name=eoip-cust1-site3 remote-address=10.0.3.2 tunnel-id=13
  5. /interface bridge
  6. add name=bridge-cust1 protocol-mode=none
  7. /interface bridge port
  8. add bridge=bridge-cust1 interface=eoip-cust1-site1
  9. add bridge=bridge-cust1 interface=eoip-cust1-site2
  10. add bridge=bridge-cust1 interface=eoip-cust1-site3

For the customer site router the configuration is similar. We create the EoIP tunnel and configure the remote-address and tunnel-id parameters to match the configuration on the SD-WAN core router. After that, we create a bridge interface and add both the EoIP interface (i.e., eoip-cust1-site1) and the Ethernet port (i.e., ether2) where the L2VPN service is provided.

  1. /interface eoip
  2. add mac-address=00:00:5E:80:01:01 name=eoip-cust1-site1 remote-address=10.0.1.1 tunnel-id=11
  3. /interface bridge
  4. add name=bridge-eoip-cust1 protocol-mode=none
  5. /interface bridge port
  6. add bridge=bridge-eoip-cust1 interface=ether2
  7. add bridge=bridge-eoip-cust1 interface=eoip-cust1-site1

At this point the EoIP tunnel should be up and running on both sites, so it is time to test it. To do that, we create a DHCP address pool and server on the bridge interface of the SD-WAN core router, as described next. As shown, we are using the 192.168.1.0/24 network address, and we use the 192.168.1.1 address for the gateway and give out the 192.168.1.2-192.168.1.254 range using DHCP.

  1. /ip pool
  2. add name=dhcp-pool-cust1 ranges=192.168.1.2-192.168.1.254
  3. /ip dhcp-server
  4. add address-pool=dhcp-pool-cust1 interface=bridge-customer1 name=dhcp-cust1
  5. /ip address
  6. add address=192.168.1.1/24 interface=bridge-cust1 network=192.168.1.0
  7. /ip dhcp-server network
  8. add address=192.168.1.0/24 gateway=192.168.1.1

On the customer site we have don't have anything else to do, just connect a computer (i.e., Mikrotik5) to the Ethernet interface of the router that is inside the bridge (i.e., ether2) and check that it can acquire an IP address from the DHCP server that we have setup at the bridge of the SD-WAN core router. As it can be observed, we can successfully obtain an IP address of the 192.168.1.0/24 range (i.e., 192.168.1.250) and ping the remote gateway (i.e., 192.168.1.1).

  1. [admin@MikroTik5] > /ip dhcp-client/print
  2. Columns: INTERFACE, USE-PEER-DNS, ADD-DEFAULT-ROUTE, STATUS, ADDRESS
  3. # INTERFACE USE-PEER-DNS ADD-DEFAULT-ROUTE STATUS ADDRESS
  4. 0 ether1 yes yes bound 192.168.1.250/24
  5. [admin@MikroTik5] > ping 192.168.1.1
  6. SEQ HOST SIZE TTL TIME STATUS
  7. 0 192.168.1.1 56 64 3ms594us
  8. 1 192.168.1.1 56 64 6ms154us
  9. 2 192.168.1.1 56 64 3ms467us
  10. 3 192.168.1.1 56 64 7ms115us
  11. sent=4 received=4 packet-loss=0% min-rtt=3ms467us avg-rtt=5ms82us max-rtt=7ms115us

The multi-site L2VPN service is now running, but there are no limits to how much bandwidth each site can use, which is not acceptable from the point of view of the service provider. For that we can use the simple queue mechanism, allowing to set a maximum download and upload limit to each Wireguard interface. This is implemented on the SD-WAN core router as shown next.

  1. /queue simple
  2. add max-limit=250k/250k name=queue1 target=wg-cust1-site1
  3. add max-limit=250k/250k name=queue2 target=wg-cust1-site2
  4. add max-limit=250k/250k name=queue3 target=wg-cust1-site3

With such modification we can now test the available bandwidth using the /tool/bandwidth-test tool, as shown next. As it can be observed, running a bi-directional test using TCP gives to following results.

  1. [admin@MikroTik5] > /tool/bandwidth-test address=192.168.1.1 direction=both protocol=tcp
  2. status: running
  3. duration: 4m54s
  4. tx-current: 253.3kbps
  5. tx-10-second-average: 215.3kbps
  6. tx-total-average: 200.0kbps
  7. rx-current: 253.7kbps
  8. rx-10-second-average: 211.2kbps
  9. rx-total-average: 213.2kbps
  10. random-data: no
  11. direction: both
  12. connection-count: 20
  13. local-cpu-load: 5%
  14. remote-cpu-load: 12%
2022/04/20 21:12 · pere

Hello World!

Hello world!

This is a first post just to check that the blog plugin for dokuwiki is working properly.

In the next posts I will start adding content related to switching and routing using Mikrotik equipment and RouterOS.

Stay tuned!

Pere

2022/04/20 10:46 · pere
  • public/blog.txt
  • Last modified: 2022/04/20 10:46
  • by pere