How to make an LXC container on a router running LEDE

From time to time people are tempted to do something strange. Why - in fact it does not matter, some kind of argument that it is necessary to do this way - will be. But the main thing is the desire to do something strange. Well, once decided to do something strange, then why not describe this process.

So, how to run an LXC container with Debian, not somewhere out there, but on a piece of hardware running under LEDE (also known as OpenWRT fork)?

(it is assumed that the reader himself will figure out why he needs LXC on the router and is already familiar with this container technology)

Attempt optimistic

For the desired frauds we take a piece of metal (or better, while x86 virtuals for the test) with LEDE installed and see:

root@lede:~# opkg find lxc* lxc - 1.1.5-3 - LXC is the userspace control package for Linux Containers, a lightweight virtual system mechanism sometimes described as "chroot on steroids". lxc-attach - 1.1.5-3 - Utility lxc-attach from the LXC userspace tools lxc-auto - 1.1.5-3 - LXC is the userspace control package for Linux Containers, a lightweight virtual system mechanism sometimes described as "chroot on steroids". This package adds and initscript for starting and stopping the containers on boot and shutdown. lxc-autostart - 1.1.5-3 - Utility lxc-autostart from the LXC userspace tools lxc-cgroup - 1.1.5-3 - Utility lxc-cgroup from the LXC userspace tools lxc-checkconfig - 1.1.5-3 - Utility lxc-checkconfig from the LXC userspace tools lxc-clone - 1.1.5-3 - Utility lxc-clone from the LXC userspace tools lxc-common - 1.1.5-3 - LXC common files lxc-config - 1.1.5-3 - Utility lxc-config from the LXC userspace tools lxc-configs - 1.1.5-3 - LXC virtual machine common config files lxc-console - 1.1.5-3 - Utility lxc-console from the LXC userspace tools lxc-create - 1.1.5-3 - Utility lxc-create from the LXC userspace tools lxc-destroy - 1.1.5-3 - Utility lxc-destroy from the LXC userspace tools lxc-device - 1.1.5-3 - Utility lxc-device from the LXC userspace tools lxc-execute - 1.1.5-3 - Utility lxc-execute from the LXC userspace tools lxc-freeze - 1.1.5-3 - Utility lxc-freeze from the LXC userspace tools lxc-hooks - 1.1.5-3 - LXC virtual machine hooks lxc-info - 1.1.5-3 - Utility lxc-info from the LXC userspace tools lxc-init - 1.1.5-3 - LXC Lua bindings lxc-ls - 1.1.5-3 - Utility lxc-ls from the LXC userspace tools lxc-lua - 1.1.5-3 - LXC Lua bindings lxc-monitor - 1.1.5-3 - Utility lxc-monitor from the LXC userspace tools lxc-monitord - 1.1.5-3 - Utility lxc-monitord from the LXC userspace tools lxc-snapshot - 1.1.5-3 - Utility lxc-snapshot from the LXC userspace tools lxc-start - 1.1.5-3 - Utility lxc-start from the LXC userspace tools lxc-stop - 1.1.5-3 - Utility lxc-stop from the LXC userspace tools lxc-templates - 1.1.5-3 - LXC virtual machine templates lxc-unfreeze - 1.1.5-3 - Utility lxc-unfreeze from the LXC userspace tools lxc-unshare - 1.1.5-3 - Utility lxc-unshare from the LXC userspace tools lxc-user-nic - 1.1.5-3 - Utility lxc-user-nic from the LXC userspace tools lxc-usernsexec - 1.1.5-3 - Utility lxc-usernsexec from the LXC userspace tools lxc-wait - 1.1.5-3 - Utility lxc-wait from the LXC userspace tools 

Hm This seems easier than expected. True, everything is broken into separate utilities, but oh well. We first lxc-checkconfig and see what it thinks of our core.

 root@lede:~# lxc-checkconfig Kernel configuration not found at /proc/config.gz; searching... lxc-checkconfig: unable to retrieve kernel configuration Try recompiling with IKCONFIG_PROC, installing the kernel headers, or specifying the kernel configuration path with: CONFIG=<path> lxc-checkconfig 

Oops. Thinking nothing. Some time later, it turns out that the default LEDE core is built without LXC support, and the packages in the repository are for manual manipulations only. But there are packages, and the kernel is quite fresh - it means you can rebuild the kernel and use it. So do.

Understanding the LEDE Build

Let's go to collect a brief guide assembly LEDE . We put the necessary packages for your OS. I recommend to start with the configuration with the default configuration to verify that errors are caused by not including LXC support and you need to dig the cause separately from this note.

 git clone https://git.lede-project.org/source.git build cd build git checkout v17.01.4 ./scripts/feeds update -a ./scripts/feeds install -a 

You can build LEDE using the official image settings for the desired hardware. To do this, go to https://downloads.lede-project.org/releases/ select the current release (at the time of writing it is 01/17/4), then in the targets/ our architecture and find the file config.seed next to the LEDE image. For example, for x86, the address is https://downloads.lede-project.org/releases/17.01.4/targets/x86/generic/config.seed

Download this file to the build directory under the name .config and use make defconfig expand it into a full build file.

 wget -O .config https://downloads.lede-project.org/releases/17.01.4/targets/x86/generic/config.seed make defconfig time make 

I’ve had about one and a half hours, to look for ready-made images in bin/targets/ .

In general, make can be called in several threads, but I was impressed with the warning in the build documentation. But this might expose bugs! and decided not to hurry anywhere.

And pay attention to the disk space was enough. Take 20GB is quite simple.

Add LXC

A little familiar with the build, now it's time to add kernel options to support LXC. Many thanks to almost the only article on building OpenWRT with LXC for the starting point. But the article is old, and the options for the current LXC is not all. Add to our .config package of options:

 CONFIG_KERNEL_AIO=y CONFIG_KERNEL_BLK_CGROUP=y CONFIG_KERNEL_BLK_DEV_BSG=y CONFIG_KERNEL_CGROUPS=y CONFIG_KERNEL_CGROUP_CPUACCT=y CONFIG_KERNEL_CGROUP_DEVICE=y CONFIG_KERNEL_CGROUP_FREEZER=y CONFIG_KERNEL_CGROUP_PIDS=y CONFIG_KERNEL_CGROUP_SCHED=y CONFIG_KERNEL_CPUSETS=y # CONFIG_KERNEL_DEBUG_FS is not set # CONFIG_KERNEL_DEBUG_INFO is not set # CONFIG_KERNEL_DEBUG_KERNEL is not set CONFIG_KERNEL_DEVPTS_MULTIPLE_INSTANCES=y CONFIG_KERNEL_DEVTMPFS=y CONFIG_KERNEL_DEVTMPFS_MOUNT=y CONFIG_KERNEL_DIRECT_IO=y CONFIG_KERNEL_FANOTIFY=y CONFIG_KERNEL_FHANDLE=y CONFIG_KERNEL_FREEZER=y CONFIG_KERNEL_IPC_NS=y # CONFIG_KERNEL_KALLSYMS is not set CONFIG_KERNEL_LXC_MISC=y CONFIG_KERNEL_MEMCG=y CONFIG_KERNEL_MEMCG_SWAP=y CONFIG_KERNEL_MM_OWNER=y CONFIG_KERNEL_NAMESPACES=y CONFIG_KERNEL_NETPRIO_CGROUP=y CONFIG_KERNEL_NET_CLS_CGROUP=y CONFIG_KERNEL_NET_NS=y CONFIG_KERNEL_PID_NS=y CONFIG_KERNEL_POSIX_MQUEUE=y CONFIG_KERNEL_PROC_PID_CPUSET=y CONFIG_KERNEL_RESOURCE_COUNTERS=y CONFIG_KERNEL_USER_NS=y CONFIG_KERNEL_UTS_NS=y CONFIG_PACKAGE_ip-tiny=y CONFIG_PACKAGE_ipset=y CONFIG_PACKAGE_iptables-mod-conntrack-extra=y CONFIG_PACKAGE_iptables-mod-ipopt=y CONFIG_PACKAGE_kmod-8021q=y CONFIG_PACKAGE_kmod-fuse=y CONFIG_PACKAGE_kmod-ip6tables-extra=y CONFIG_PACKAGE_kmod-ipt-conntrack-extra=y CONFIG_PACKAGE_kmod-ipt-ipopt=y CONFIG_PACKAGE_kmod-ipt-ipset=y CONFIG_PACKAGE_kmod-ipt-nat-extra=m CONFIG_PACKAGE_kmod-ipt-nat6=m CONFIG_PACKAGE_kmod-macvlan=y CONFIG_PACKAGE_kmod-nf-nat6=m CONFIG_PACKAGE_kmod-nfnetlink=y CONFIG_PACKAGE_kmod-veth=y CONFIG_PACKAGE_terminfo=y 

In principle, all this can be found in the usual make menuconfig . Now you can call it, these parameters will not be lost and you can customize the assembly for yourself.

But, unfortunately, this is not all options. Some are not in make menuconfig and need to change the configuration of the kernel itself. In theory, the right way is to

 make kernel_menuconfig 

And putting down all the corresponding marks there. But somehow it is a chore to look for them, so the path is wrong: we re-run the updated .config

 make defconfig #   -   menuconfig   ,    .  make prepare 

We find somewhere around build_dir/target-i386_pentium4_musl-1.1.16/linux-x86_generic/linux-4.4.92/.config generated kernel configuration file (the path, as you can see, differs from the heap of parameters). And add to the end of another pack of settings

 CONFIG_CHECKPOINT_RESTORE=y CONFIG_CGROUP_FREEZER=y CONFIG_FREEZER=y CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m CONFIG_NETLINK_DIAG=y CONFIG_PACKET_DIAG=y CONFIG_INET_DIAG=y CONFIG_UNIX_DIAG=y CONFIG_FHANDLE=y CONFIG_INET_UDP_DIAG=m # CONFIG_PROC_STRIPPED is not set CONFIG_CFQ_GROUP_IOSCHED=y 

CONFIG_IKCONFIG_PROC and CONFIG_IKCONFIG are just the thing that lxc-checkconfig could not say. And CONFIG_PROC_STRIPPED are OpenWRT patches that CONFIG_PROC_STRIPPED some files from the /proc pseudo /proc . The setting should reduce the amount of memory consumed, but the varings were obtained with lxc-start . But LXC with a lack of memory and is not needed.

After such an intervention (and it must be repeated after make prepare every time) we do

 time make 

Now a lot of everything has already been compiled and will not be recompiled, therefore it works much faster. For 15 minutes, I had an update.

We load the resulting image onto a piece of iron or a virtual machine and try to execute lxc-checkconfig . Must see a neat picture like this:

 root@lede:~# lxc-checkconfig --- Namespaces --- Namespaces: enabled Utsname namespace: enabled Ipc namespace: enabled Pid namespace: enabled User namespace: enabled Network namespace: enabled Multiple /dev/pts instances: enabled --- Control groups --- Cgroup: enabled Cgroup clone_children flag: enabled Cgroup device: enabled Cgroup sched: enabled Cgroup cpu account: enabled Cgroup memory controller: enabled --- Misc --- Veth pair device: enabled Macvlan: enabled Vlan: enabled Bridges: enabled Advanced netfilter: enabled CONFIG_NF_NAT_IPV4: enabled CONFIG_NF_NAT_IPV6: enabled CONFIG_IP_NF_TARGET_MASQUERADE: enabled CONFIG_IP6_NF_TARGET_MASQUERADE: enabled CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled --- Checkpoint/Restore --- checkpoint restore: enabled CONFIG_FHANDLE: enabled CONFIG_EVENTFD: enabled CONFIG_EPOLL: enabled CONFIG_UNIX_DIAG: enabled CONFIG_INET_DIAG: enabled CONFIG_PACKET_DIAG: enabled CONFIG_NETLINK_DIAG: enabled File capabilities: enabled Note : Before booting a new kernel, you can check its configuration usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig 

Ok, here is the core and ready.

Container preparation

Now we put a small packet of packages (in fact, they can be rolled up immediately into an image that is going to make , but I didn’t understand this)

 opkg install lxc-start lxc-attach lxc-stop lxc-ls lxc-monitor lxc-monitord lxc-checkconfig 

I prefer not to use templates from lxc-templates - my scripts have been available since openvz, and the LXC config is pretty typical. But I checked the debian build in LEDE for the article, more packages are needed:

 opkg install lxc-create lxc-templates debootstrap bash getopt rsync 

Then a surprise, out of the box still does not work. /var/cache exists in LEDE as nodev and debootstrap cannot go there. Therefore, instead of /var/cache we are doing a symlink somewhere.

 mkdir /mnt/cache ln -s /mnt/cache /var/cache lxc-create -n testlxc -t debian -- -r jessie 

So, container rootfs is ready. Let's try to run:

 lxc-start -n testlxc -d 

The task of connecting to the container was in trouble. lxc-console complains about the bad file descriptor . But the container itself is running and is active. Well, they wanted a strange, strange received. We do:

 lxc-attach -n testlxc -- bash 

And here we are with console access inside the container. What is wrong with lxc-console could not be lxc-console out, but uncritically, I left it like this.

It is possible to place a directory with containers by and large anywhere, as long as the path to this directory is specified in /etc/lxc/lxc.conf .

Containers autostart

The last detail remains: configure the autostart of the container (s) with the system. Install the lxc-auto package

 opkg install lxc-auto 

And we get the init-script and the reference configuration file /etc/config/lxc-auto . We bring it to the desired form from the required number of container name transfers:

 config container option name testlxc 

And send the system to reboot for verification. The init script unfortunately turns out to be rather silly: it calls lxc-stop & for all containers from the config and goes to sleep by default for 300 seconds. So even if the containers are turned off quickly - the script will still sleep.

PS: at first I tried to run LEDE in an LXC container on Debian. After some time, the trials with gdb even managed this, but on the task it was normal to start the firewall from the regular configs stalled and went from the reverse.