Docker: flexible network without NAT for all occasions

• default bridge network;
• host network;
• user defined network.

• we have a map where services are hosted on servers;
• we have a map of ports for each service and its type (or types, if there are many);
• we have an agreement that the ports should be unique.

• if we run several services on the same machine with --net = host , then we will not get the intersection of the ports: everything will start and will work;
• if one of the eth-interfaces becomes not enough for us, then we will physically connect one more and by means of, for example, DNS, spread the load between them.

• service is critical;
• lack of sufficient experience to offer the most transparent and quick service transfer to the container;
• You can "add something else to your taste."

• it works on the same core (yes, it happens);
• filling the first space, it is worth noting that you can start several service instances and use taskset / - cpuset-cpus ;
• the service “heavily” uses the network, and it also requires a large number of ports for outgoing connections.

• on the machine where the service was planned to raise, it was necessary to add an additional IP address (or even several) - ip a add (here you can immediately point out the many disadvantages of this approach that we know about);
• it is worth remembering the above, in order not to get, for example, 2 identical addresses on different machines;
• in the configuration of the daemon, it was worth pointing out at what address it works, just in order not to “eat” all the ports of a neighbor or host system.

• leave everything as it is, but wrap in a container;
• bring up all the same additional IP addresses on dockerhost;
• Bind the application to a specific address.

• there is no restriction in 1 MAC address from one link on the active network equipment;
• the number of containers on the host will not be so large that it can lead to mac capacity exceeding. Over time, this moment in our country can, of course, change;
• L3 is not needed at this stage.

So let's go!

# docker network create -d macvlan --subnet=1.1.1.0/24 --gateway=1.1.1.1 -o parent=eth0 cppbig_vlan Error response from daemon: plugin not found 

 docker[2012]: time="2016-08-04T11:44:44.095241242Z" level=warning msg="Unable to locate plugin: macvlan, retrying in 1s" docker[2012]: time="2016-08-04T11:44:45.095489283Z" level=warning msg="Unable to locate plugin: macvlan, retrying in 2s" docker[2012]: time="2016-08-04T11:44:47.095750785Z" level=warning msg="Unable to locate plugin: macvlan, retrying in 4s" docker[2012]: time="2016-08-04T11:44:51.095970433Z" level=warning msg="Unable to locate plugin: macvlan, retrying in 8s" docker[2012]: time="2016-08-04T11:44:59.096197565Z" level=error msg="Handler for POST /v1.23/networks/create returned error: plugin not found" 

• --net = host ;
• --net = macvlan .

 # docker run -it --net=host --name=iperf_w_host_net --entrypoint=/bin/bash dockerio.badoo.com/itops/sle_12_base:latest # iperf3 -s -p 12345 ----------------------------------------------------------- Server listening on 12345 ----------------------------------------------------------- 

 # iperf3 -c 1.1.1.2 -p 12345 -t 30 

 # iperf3 -c 1.1.1.2 -p 12345 -t 30 -P 8 

 # docker run -it --net=cppbig_vlan --name=iperf_w_macvlan_net --ip=1.1.1.202 --entrypoint=/bin/bash dockerio.badoo.com/itops/sle_12_base:latest # iperf3 -s -p 12345 ----------------------------------------------------------- Server listening on 12345 ----------------------------------------------------------- 

 # iperf3 -c 1.1.1.202 -p 12345 -t 30 

 # iperf3 -c 1.1.1.202 -p 12345 -t 30 -P 8 

• a part of the service availability checks is checked by Zabbix “helpers” that are performed on the host where the service is running;
• there is a need to use caching DNS, which is located on the host system. In our case, this is Unbound;
• there is a need to use access to some other services running on the host system.
• These are just some of the reasons why we need access to the “host <==> container”. It is impossible to take and remake the architecture of such nodes overnight.

1. Use two or more physical links on the machine. This allows interaction through a neighboring interface. For example, take eth1 and give it specifically for MACVLAN, and continue to use eth0 on the host system. The option is certainly not bad, but it entails the need to keep the same number of links on all the machines where we plan to launch such services. Implement it is expensive, not fast and not always possible.
   
   
2. Use another additional IP address on the host system, hang it on the virtual MACVLAN interface, which you need to raise on the host system. This is about as difficult from the point of view of support (“not to forget / not to forget”), as the previous sentence is time. And, since earlier I said that our service itself requires an additional address, then in the end, to start such a service, we need:
   
   
   • the address for the host host's main interface (1);
   • address for the service (2);
   • address for the virtual interface through which we will interact with the service (3).
   
   In this case, it turns out that we need too many IP addresses, which, by and large, will be used a little. In addition to the excessive expenditure of IP addresses, it is also worth remembering that we will need to support static routes through this very virtual interface to the container. This is not an insurmountable complexity, but the complication of the system as a whole is a fact.
   
   The attentive reader will ask the question: “Why the address on the main interface and on the MACVLAN interface, if you can give the address of the main interface to the virtual one?” In this case, we will leave our system without addresses on the real interfaces, but I’m not ready for that step yet.
   
   In the previous two variants it was assumed that the addresses of all interfaces belong to the same network. As it is easy to imagine, even with 100 servers in such a subnet, if we have three addresses, then we will not get into / 24 anymore.
   
   
3. Service IP. The idea is that we make a separate subnet for services. What it looks like:
   
   
   • we start to send “tagged” traffic to the server;
   • native VLAN is left as the main network for dockerhost (eth0);
   • we lifted the virtual interface with 802q, without the IP address on a host to the system;
   • use for the service IP-address from the service network.

• in order to send "tagged" traffic to the interface, who do we need? That's right, networkers! We ask them to switch the access port to the port to which we are sending 2 VLANs;
  
  
• raise an additional interface on the host:
  
  

   # cat /etc/sysconfig/network/ifcfg-vlan8 BOOTPROTO='static' STARTMODE='auto' VLAN='yes' ETHERDEVICE='eth0' # ip -d link show vlan8 31: vlan8@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether e4:11:5b:ea:b6:30 brd ff:ff:ff:ff:ff:ff promiscuity 1 vlan protocol 802.1Q id 8 <REORDER_HDR> 

  
  
• get a MACVLAN network in Docker
  
  

   # docker network create -d macvlan --subnet=1.1.2.0/24 --gateway=1.1.2.1 -o parent=vlan8 c_services 

  
  
• make sure the Docker network appears:
  
  

   # docker network ls | grep c_services a791089219e0 c_services macvlan 

 -t nat -A OUTPUT -d 127.0.0.11 -p udp --dport 53 -j DNAT --to-destination 127.0.0.11:35373 -t nat -A POSTROUTING -s 127.0.0.11 -p udp --sport 35373 -j SNAT --to-source :53 -t nat -A OUTPUT -d 127.0.0.11 -p tcp --dport 53 -j DNAT --to-destination 127.0.0.11:41214 -t nat -A POSTROUTING -s 127.0.0.11 -p tcp --sport 41214 -j SNAT --to-source :53 

 # modinfo xt_nat filename: /lib/modules/4.4.0-3.1-default/kernel/net/netfilter/xt_nat.ko alias: ip6t_DNAT alias: ip6t_SNAT alias: ipt_DNAT alias: ipt_SNAT author: Patrick McHardy <kaber@trash.net> license: GPL srcversion: 9982FF46CE7467C8F2361B5 depends: x_tables,nf_nat intree: Y vermagic: 4.4.0-3.1-default SMP preempt mod_unload modversions 

• on Docker issues;
• in source code.

  if out, err := exec.Command("modprobe", "-va", "nf_nat").CombinedOutput(); err != nil { logrus.Warnf("Running modprobe nf_nat failed with message: `%s`, error: %v", strings.TrimSpace(string(out)), err) } if out, err := exec.Command("modprobe", "-va", "xt_conntrack").CombinedOutput(); err != nil { logrus.Warnf("Running modprobe xt_conntrack failed with message: `%s`, error: %v", strings.TrimSpace(string(out)), err) } 

  if err := r.setupIPTable(); err != nil { return fmt.Errorf("setting up IP table rules failed: %v", err) } 

• (1 and 6) the mobile client establishes a connection with a certain URL, followed by a balancer;
• (2) the balancer selects the required instance of our service and allows you to establish a client-service connection;
• (3 and 4) then our service proxies requests from the client to the cluster with the code, but also through the balancer in the form of nginx. This is where we returned to our request that nginx should be on the same machine as the service. At the moment there is also a limitation in that it should be on the host, and not in the container (this, by the way, would solve the problem immediately). We will not discuss the reasons for this requirement in this article, but accept it as a condition.
• (5) each instance of our service has a specific id that the code needs to understand through which instance to respond to the client.

 # ip r default via 1.1.2.254 dev eth0 10.0.0.0/8 via 1.1.2.1 dev eth0 1.1.2.0/24 dev eth0 proto kernel scope link src 1.1.2.14 192.168.0.0/16 via 1.1.2.1 dev eth0 

 # ip r default via 1.1.2.1 dev eth0 1.1.2.0/24 dev eth0 proto kernel scope link src 1.1.2.14 

• do it with your hands, using the container namespace;
• take pipework https://github.com/jpetazzo/pipework and do the same, but with it.

1. A network that has only 1 default gw and no external balancer.
   
   
   
   
   
2. A network with more than one GW: for example, an external request balancer. The difficulty is that we do not drive internal traffic through it.
   
   

• they are not ready to take responsibility and monitor all networks in which the routing will be exactly like that;
• we, for our part, are not ready to support static routes for all such networks on servers

• ;
• IP ;
• ;
• ( , ).

 FROM dockerio.badoo.com/itops/sle_12_base:latest MAINTAINER #MAINTEINER# RUN /usr/bin/zypper -q -n in iproute2 RUN groupadd -g 1001 wwwaccess RUN mkdir -p /local/SERVICE/{var,conf} COPY get_configs.sh /local/SERVICE/ COPY config.cfg /local/SERVICE/ ADD SERVICE-CERTS/ /local/SERVICE-CERTS/ ADD SERVICE/bin/SERVICE-BINARY-${DVERSION} /local/SERVICE/bin/ ADD SERVICE/conf/ /local/SERVICE/conf/ COPY routes.sh /etc/cont-init.d/00-routes.sh COPY env.sh /etc/cont-init.d/01-env.sh COPY finish.sh /etc/cont-finish.d/00-finish.sh COPY run /etc/services.d/SERVICE/ COPY finish /etc/services.d/SERVICE/ RUN touch /tmp/fresh_container ENTRYPOINT ["/init"] 

• s6 overlay supervisor ;
• iproute, ;
• , ( /etc/cont-init.d/), , , , (/etc/cont-finish.d/);
• /tmp/fresh_container , , . , ;

1. get_configs.sh — , , , , , , . Docker Meetup ;
   
   
2. routes.sh — , :
   
   

    #!/usr/bin/with-contenv sh if [ ! -x /usr/sbin/ip ];then echo -e "\e[31mCan't execute /usr/sbin/ip\e[0m"; [ $(pgrep s6-svscan) ] && s6-svscanctl -t /var/run/s6/services exit 1; else LTMGW=$(/usr/sbin/ip r show | /usr/bin/grep default | /usr/bin/awk {'print $3'} | /usr/bin/awk -F \. {'print $1"."$2"."$3".254"'}) DEFGW=$(/usr/sbin/ip r show | /usr/bin/grep default | /usr/bin/awk {'print $3'} | /usr/bin/awk -F \. {'print $1"."$2"."$3".1"'}) /usr/sbin/ip r replace default via ${LTMGW} /usr/sbin/ip r add 192.168.0.0/16 via 10.10.8.1 dev eth0 /usr/sbin/ip r add 10.0.0.0/8 via 10.10.8.1 dev eth0 echo -e "\e[32mAll job with routes done:\e[0m\n$(/usr/sbin/ip r show)" fi 

   
   
3. env.sh — , ; 1 :
   
   

    #!/usr/bin/with-contenv sh if [ ! -z "${ISTEST}" ];then exit 0;fi if [ ! -n "${SERVICETYPE}" ];then echo -e "\e[31mPlease set SERVICE type\e[0m"; [ $(pgrep s6-svscan) ] && s6-svscanctl -t /var/run/s6/services exit 1; fi bash /local/SERVICE/get_configs.sh || exit 1 echo -e "\e[32mSERVICE ${SERVICETYPE} is running\e[0m" 

   
   
4. finish.sh — , pid- . ( ), , , pid- :)
   
   
5. run — , :
   
   

    #!/usr/bin/with-contenv bash exec /local/SERVICE/bin/SERVICE-${DVERSION} -l /local/SERVICE/var/mobile-${SERVICETYPE}.log -P /local/SERVICE/var/mobile-${SERVICETYPE}.pid -c /local/SERVICE/conf/SERVICE.conf -v ${VERBOSITY} 

   
   
6. finish — , , :
   
   

    #!/bin/sh [ $(pgrep s6-svscan) ] && s6-svscanctl -t /var/run/s6/services 

   
   

 docker run -d --net=c_services --ip=1.1.2.17 --name=SERVICE-INSTANCE16 -h SERVICE-INSTANCE16.local --cap-add=NET_ADMIN --add-host='nginx.localhost:1.1.1.17' -e SERVICETYPE=INSTANCE16_eu1 -e HOST_IP=1.1.1.17 --volumes-from=badoo_loop dockerio.badoo.com/cteam/SERVICE:2.30.0_994