Geekly Articles each Day
📜 ⬆️ ⬇️

Docker and crutches in production



Inspired by the publication "Understanding Docker," a small example of crutches around a docker for running web applications.

I tried different strapping technologies, but some (fig) look a bit clumsy to use, and some (kubernetis, mesos) look too abstract and complex.
')
There are several machines in my configuration, various web applications run on the machines, some of them require local storage. As a basic scheme, we take a configuration of two frontends and one backend, ceph (FS) provides data roaming for the backend where it is needed.

The machines have a private network interface. Frontends are also public.

On configuration day, I use a bunch of etcd + skydns (service discovery), runit (container status monitoring) and ansible (configuration). Here is the module code ansible, which I will discuss:

a lot of code
#!/usr/bin/env python import os, sys from string import Template def on_error(msg): def wrap(f): def wrapped(self, module): try: return f(self, module) except Exception, e: module.fail_json(msg="%s %s: %s" % (msg, self.name, str(e))) return wrapped return wrap class Service: SERVICE_PREFIX = 'docker-' SERVICES_DIR = '/etc/sv' RUNNING_SERVICES_DIR = '/etc/service' def __init__(self, name, image, args, announce, announce_as, port): self.name = name self.image = image if args is not None: self.args = args else: self.args = '' self.announce = announce self.announce_as = announce_as self.port = port def _needs_etcd(self): return self.announce is not None def _service_name(self): return self.SERVICE_PREFIX + self.name def _root_service_dir(self): return os.path.join(self.SERVICES_DIR, self._service_name()) def _announced_service_dir(self): return os.path.join(self._root_service_dir(), 'services', 'service') def _etcd_service_dir(self): return os.path.join(self._root_service_dir(), 'services', 'announce') def _run_service_link(self): return os.path.join(self.RUNNING_SERVICES_DIR, self._service_name()) def _root_run_file(self): return os.path.join(self._root_service_dir(), 'run') def _announced_service_run_file(self): return os.path.join(self._announced_service_dir(), 'run') def _etcd_run_file(self): return os.path.join(self._etcd_service_dir(), 'run') def exists(self): return os.path.isdir(self._root_service_dir()) def scheduled_to_run(self): return os.path.exists(self._run_service_link()) @on_error("Error starting service") def start(self, module): if self._needs_update(module): self.install(module) if self.scheduled_to_run(): return False os.symlink(self._root_service_dir(), self._run_service_link()) return True @on_error("Error stopping service") def stop(self, module): if not self.scheduled_to_run(): return False os.unlink(self._run_service_link()) return True @on_error("Error installing service") def install(self, module): if self._needs_update(module): self.stop(module) self.remove(module) self._create_service(module) return True else: return False @on_error("Error creating service") def _create_service(self, module): self._create_service_dirs(module) self._write_run_file(self._root_run_file(), self._render_root_run()) if self._needs_etcd(): self._write_run_file(self._announced_service_run_file(), self._render_service_run()) self._write_run_file(self._etcd_run_file(), self._render_etcd_run()) def _write_run_file(self, name, content): f = open(name, 'w') f.write(content) os.fchmod(f.fileno(), 0755) f.close() @on_error("Error verifying service existence") def _needs_update(self, module): if self.exists(): if os.path.exists(self._root_run_file()): root_run = self._render_root_run() curr_run = open(self._root_run_file()).read() if root_run != curr_run: return True if self._needs_etcd(): if os.path.exists(self._announced_service_run_file()): service_run = self._render_service_run() curr_run = open(self._announced_service_run_file()).read() if service_run != curr_run: return True if os.path.exists(self._etcd_run_file()): etcd_run = self._render_etcd_run() curr_run = open(self._etcd_run_file()).read() if etcd_run != curr_run: return True else: return True else: return True else: return True else: return True return False @on_error("Error creating service directory") def _create_service_dirs(self, module): os.mkdir(self._root_service_dir(), 0755) if self._needs_etcd(): os.mkdir(os.path.join(self._root_service_dir(), 'services'), 0755) os.mkdir(self._announced_service_dir(), 0755) os.mkdir(self._etcd_service_dir(), 0755) @on_error("Error removing service") def remove(self, module): if not self.exists(): return False if self.scheduled_to_run(): self.stop(module) from shutil import rmtree rmtree(self._root_service_dir()) return True def _render_root_run(self): if self._needs_etcd(): return self._render_runsv_run() else: return self._render_service_run() def _render_service_run(self): args = self.args if self.announce: if self.port is not None: port = self.port else: port = self.announce if self.announce_as != 'container': args += " -p $ANNOUNCE_IP:" + self.announce + ":" + port return Template("""#!/bin/bash CONTAINER_NAME=$name ifconfig eth1 >/dev/null 2>&1 if [[ $$? -eq 0 ]]; then PUBILC_IF=eth0 PRIVATE_IF=eth1 else PUBILC_IF=eth0 PRIVATE_IF=eth0 fi case "$announce_as" in public) ANNOUNCE_IP="`ifconfig $$PUBILC_IF | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\\2/p'`" ;; private) ANNOUNCE_IP="`ifconfig $$PRIVATE_IF | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\\2/p'`" ;; *) ANNOUNCE_IP="" ;; esac docker inspect $$CONTAINER_NAME|grep State >/dev/null 2>&1 if [ $$? -eq 0 ]; then docker rm $$CONTAINER_NAME || { echo "cannot remove container $$CONTAINER_NAME"; exit 1; } fi docker pull $image exec docker run \ -i --rm \ --name $$CONTAINER_NAME \ --hostname "`hostname`-$name" \ $args \ $image """).substitute(name=self.name, image=self.image, args=args, announce_as=self.announce_as) def _render_runsv_run(self): return """#!/bin/bash runsvdir -P services & RUNSVPID=$! trap "{ sv stop `pwd`/services/*; sv wait `pwd`/services/*; kill -HUP $RUNSVPID ; exit 0; }" SIGINT SIGTERM wait """ def _render_etcd_run(self): return Template("""#!/bin/bash ETCD="http://192.0.2.1:4001" DOMAIN="com/example/prod/s/$name/`hostname`" ifconfig eth1 >/dev/null 2>&1 if [[ $$? -eq 0 ]]; then PUBILC_IF=eth0 PRIVATE_IF=eth1 else PUBILC_IF=eth0 PRIVATE_IF=eth0 fi case "$announce_as" in public) ANNOUNCE_IP="`ifconfig $$PUBILC_IF | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\\2/p'`" ;; private) ANNOUNCE_IP="`ifconfig $$PRIVATE_IF | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\\2/p'`" ;; *) ANNOUNCE_IP="" ;; esac enable -f /usr/lib/sleep.bash sleep trap "{ curl -L "$$ETCD/v2/keys/skydns/$$DOMAIN" -XDELETE ; exit 0; }" SIGINT SIGTERM while true; do if [[ "$announce_as" == "container" ]]; then ANNOUNCE_IP="`docker inspect --format '{{ .NetworkSettings.IPAddress }}' $name`" fi curl -L "$$ETCD/v2/keys/skydns/$$DOMAIN" -XPUT -d value="{\\"host\\": \\"$$ANNOUNCE_IP\\", \\"port\\": $port}" -d ttl=60 >/dev/null 2>&1 sleep 45 done""").substitute(name=self.name, port=self.announce, announce_as=self.announce_as) def main(): module = AnsibleModule( argument_spec = dict( state = dict(required=True, choices=['present', 'absent', 'enabled', 'disabled']), name = dict(required=True), image = dict(required=True), args = dict(default=None), announce = dict(default=None), announce_as = dict(default='private', choices=['public', 'private', 'container']), port = dict(default=None) ) ) state = module.params['state'] name = module.params['name'] image = module.params['image'] args = module.params['args'] announce = module.params['announce'] announce_as = module.params['announce_as'] port = module.params['port'] svc = Service(name, image, args, announce, announce_as, port) if state == 'present': module.exit_json(changed=svc.install(module)) if state == 'absent': module.exit_json(changed=svc.remove(module)) if state == 'enabled': module.exit_json(changed=svc.start(module)) if state == 'disabled': module.exit_json(changed=svc.stop(module)) module.fail_json(msg='Unexpected position reached') sys.exit(0) from ansible.module_utils.basic import * main()


Let's see what happens when we launch a new service; for example, run influxdb:

 ansible -i hosts node-back-1 -s -m rundock -a 'state=enabled name=influxdb image="registry.s.prod.example.com:5000/influxdb:latest" args="--volumes-from data.influxdb -p $PRIVATE_IP:8083:8083" announce=8086 port=8086'

Ansible adds to the machine a new task for runit, which contains two subtasks, a container and an announcement:

 $ cat /etc/sv/docker-influxdb/services/service/run #!/bin/bash CONTAINER_NAME=influxdb INTERFACE=eth0 PRIVATE_IP="`ifconfig $INTERFACE | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'`" docker inspect $CONTAINER_NAME|grep State >/dev/null 2>&1 if [ $? -eq 0 ]; then docker rm $CONTAINER_NAME || { echo "cannot remove container $CONTAINER_NAME"; exit 1; } fi docker pull registry.s.prod.example.com:5000/influxdb:latest exec docker run -i --rm --name $CONTAINER_NAME --hostname "`hostname`-influxdb" --volumes-from data.influxdb -p $PRIVATE_IP:8083:8083 -p $PRIVATE_IP:8086:8086 registry.s.prod.example.com:5000/influxdb:latest

runit will kill the old container, if it was, download the new image and launch the docker interactively. If the container dies - runit will restart it. In the data.influxdb container data.influxdb a mapping is made on the way to the file system, where the influx will store its data.

Second service:

 $ cat /etc/sv/docker-influxdb/services/announce/run #!/bin/bash ETCD="http://192.0.2.1:4001" DOMAIN="com/example/prod/s/influxdb/`hostname`" INTERFACE=eth0 enable -f /usr/lib/sleep.bash sleep trap "{ curl -L "$ETCD/v2/keys/skydns/$DOMAIN" -XDELETE ; exit 0; }" SIGINT SIGTERM while true; do PRIVATE_IP="`ifconfig $INTERFACE | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'`" curl -L "$ETCD/v2/keys/skydns/$DOMAIN" -XPUT -d value="{\"host\": \"$PRIVATE_IP\", \"port\": 8086}" -d ttl=60 >/dev/null 2>&1 sleep 45

The module for bash adds sleep as a built-in command, now bash will update the record for the domain, and influxdb will be available at node-back-1.influxdb.s.prod.example.com.

Crutch : in a good way, the announcement should be made from inside the container, since the announcement will be alive even if the container went into a crash-loop.

Now fix the grafana for the frontend:

 ansible -i hosts node-back-1 -s -m rundock -a 'state=enabled name=grafana image="tutum/grafana:latest" args="-e INFLUXDB_HOST=influxdb.s.prod.example.com -e INFLUXDB_PORT=8086 -e INFLUXDB_NAME=metrics -e INFLUXDB_USER=metrics -e INFLUXDB_PASS=metrics -e HTTP_PASS=metrics -e INFLUXDB_IS_GRAFANADB=true" announce=8087 port=80'

Here the port and announce are different, since the standard container gives grafana on port 80, and we give it out to 8087.

And finally upstream in nginx:

 upstream docker_grafana { server grafana.s.prod.example.com:8087; keepalive 512; }

crutch : ports nailed by hand. In an amicable way, something like this can teach nginx to use SRV records.

Talk about solution stability?


Frontend. If the frontend dies, you need to update the DNS records. Some time we lie and be sad.

Detection. etcd / skydns is generally difficult to kill if they are adequately assembled into a consensus.

Backend service. We rezolvim service without the name of the machine, so that you can run multiple backends; skydns will balance the load or quickly replace dead services.

File system. In an ideal world, we have a completely unchanging state, but in life everything is sadder. Databases that understand replication can have storage on a local disk or in the usual --volume . Where it is necessary to distribute something between containers, ceph works (paxos, for good, is also difficult to kill).

Source: https://habr.com/ru/post/253999/

More articles:


All Articles