Run / debug Python scripts in LXC / LXD containers from under VS Code
Hi, Habr!
In this article, I would like to share with you the setting up of a development environment for developing scripts in Python and launching them in an isolated environment, almost a couple of clicks in the VS Code editor.
As an isolated environment, the primary configuration and creation of a container based on LXC running on Ubuntu 17.10 OS will be considered. Why not a docker? It is difficult to answer, give some arguments. In part, the concept of OS virtualization is subjectively closer in spirit than process virtualization, but the principle that will be set forth here, with some amendments, will approach the launch of code in Docker containers.
')
Before installing LXD, you need to decide what type of backend storage for containers will be used. It will not be possible to change later. Several options are available: ZFS, btrfs, LVM, or the host file system. For comparison, a comparison table is available in the official documentation. I suggest to choose ZFS which will allow in the future to easily create COW (copy-on-write) snapshots of containers. Install the ZFS utility package in the system:
There are no difficulties here, the package is included in the standard delivery of the system, so we command in the console:
After installing LXD, the current user will be added to the lxd system group, but in order to continue setting up with the necessary privileges in the system, you must now re-activate the user session . Just log out and log back in to your Desktop Meneger session or reboot completely. You can check whether the current session of a user is in the lxd group using the id command:
If you see the lxd group, then everything is fine, you can continue on.
The initialization of LXD is done interactively on behalf of the current user who is in the lxd group. Run the command and follow the instructions and comments provided below:
If the network is not set up now, but to start creating containers, then when they are created, we will receive a warning that the container is not connected to any of the networks. This will not be an error, just to them there will be no network access through the interfaces. At the beginning, you can create the required number of containers, and then configure the network and attach containers to it using the lxc network attach command. Now we need to determine what type of network to build. There are several options for building LXC networks, we will choose the bridge option (a simple soft switch between the host and the containers) and build a network based on it.
When creating a network bridge, we can immediately set the properties after specifying its name or rely on the autoconfiguration that will set the IPv4 and IPv6 subnets to a random choice. By default, the parameter is ipv4.nat = false and for some reason transit traffic packets from containers will not be routed further than the host to other local network segments, even when IP forwarding is explicitly enabled in the host core. Apparently NAT in LXD is more than translating addresses from local network segments into global ones, since when this parameter is activated, IP forwarding on the host is also activated, so we will explicitly enable it:
The configuration of the container is conventionally divided into two levels, global and local. The global configuration is set in a configuration profile that is attached to a specific container or group of containers, and the local configuration is applied only to a specific container. That is, all settings in the profile are inherited by the container, and local settings take precedence over the profile. Multiple profiles can be attached to one container.
We will create an LXD profile so that in the newly created containers by default there is a nic interface that will be linked to the host (network bridge) lxdbr0 switch :
For example, create a container and name it pytest , which will be accessible from the host by that name. Run the launch command which loads the image from the ubuntu repository : version 17.10 and the amd64 architecture and from this image will create a container. Additionally, with the option we specify that this container needs to be configured from the network0 profile:
Next will download from the network image. Let's wait a bit, and a little later we can see the available containers in the system and their status:
Here we will install the necessary software in the container. Launch the package manager inside the container that updates the system:
Next, install the pip package manager (Python package installer), as well as the ptvsd python module (Visual Studio remote debugging server for Python). The ptvsd module must be installed necessarily version 3.0.0, on more recent versions (current 3.2.1.0) I received an error, which is indicated in the issue on github.com . The author of the Python extension to VS Code that interacts with ptvsd as a temporary solution recommends using version 3.0.0.
As part of LXD, we have completed the configuration.
Installation options for the application are described on the official page at code.visualstudio.com/docs/setup/linux , we modify the commands a bit:
After execution, we will have the VS Code of the latest version from the Microsoft repository.
PS At the time of writing, a new version 1.18.0 was released in which, in my opinion, one of the most anticipated (2 years) community “Multi-root workspaces” and “Git indicators in the Explorer” was included in my opinion. All changes can be read on this page .
Let's start installing extensions. If you install VS Code for the first time, for example, after using Atom (as the author), I recommend paying attention to the extensions in the list below, and to achieve the goal of this article it will be enough to install only Python and Code Runner. Extensions are installed from the marketplace on the left side of the editor.
The following settings are optional, they do not directly relate to running Python scripts in containers, but this will bring you closer to if you want it to be visually as in Atom.
In the open editor, open the editor's settings: file-> Preferences-> Settings. In the window that opens, on the right, in USER_SETTINGS make the following changes and save (Ctrl + S):
We have already done a great job and it is time for the final part. Now we have almost everything ready to run the scripts in the container.
Create a projects directory, for example, at / home / user1 / projects . It will house our future python projects. We will link this directory with the container, i.e., create a disk device in the container and link this directory with the internal / mnt / hostfs . We execute the following commands:
If we locally create a directory or file in the / home / user1 / projects directory, it will immediately be reflected in the container at / mnt / hostfs . You can verify this by running the BASH shell in the container online and running the ls command:
You will see that the files and directories in the container will be read-only and will have the UID: GID equal to nobody: nogroup . This is enough to run the scripts, but if you need to make the files and directories in the container available for writing, read the " Direct user / group mapping " chapter in this article .
And so, now we can return to our editor, if you closed it, then reopen it. Click on the menu File -> New File . This will create a new tab in which you need to enter the following python-code, which we will run in the container. Save it in the / home / user1 / projects / pytest / test.py directory , and then the file (tab) can be closed.
VS Code "will say" that you have not installed the pylint module, but we do not need it now, you can install it later, on your own, through the pip module manager. Now you need to open the workspace in VS Code. To do this, click the Open Folder button on the left in the File Explorer and select the / home / user1 / projects / pytest / directory :
VS Code is reinitialized, all windows will close completely and a workspace will open in which the test.py file will be visible. Open it by double clicking on the name on the left in File Explorer VS Code.
Call up the workspace settings by clicking on the menu File -> Preferences -> Setting, then in the window on the right above, instead of USER_SETTINGS, select from the drop-down list WORKSPACE_SETTINGS:
Make and save the following settings so that you can do it like this:
In the file explorer, you can find the settings.json file in the .vscode directory, where the settings of the workspace (the current project) are saved.
Return to the open test.py file and launch it for execution via the command from the context menu of the Run code :
In the output of the integrated terminal you will see information in which you can make sure that the code was executed on the remote machine (container). Now, whenever you need to run a project in a container, it is enough to make settings in WORKSPACE_SETTINGS or create a .vscode / settings.json file for each project and then slightly correct the paths in the settings.
Uncomment the lines in the test.py file and save this file:
Whenever there is a need for remote debugging, these lines of code must be inserted into the file that is launched for execution. The ptvsd module on the container side opens port 3000 and waits for an external debugger to connect. If you wish, you can configure ptvsd in your own way.
Run the code by calling the Run Code context menu. In the integrated terminal you will see a message that the script is waiting for a connection on port 3000. Go to the debugger and create a new configuration for this project:
You will have the launch.json file for editing in which there will already be presets. Add the following code block to it:
Save it, it will appear in the .vscode / launch.json directory next to the setting.json file. Then, select Python LXC in the bottom list on the left; this is the debugger configuration that we just entered:
Click Start Debugging. In the window with the test.py file , you will see a working debugger and confirmation that the code has been launched remotely:
That's all. Setup is complete. All that is needed in the future for the new python project is to configure two config files .vscode / settings.json and .vscode / launche.json . If you have questions on the topic, please ask in the comments. Thank you for your attention, successful projects for you!
UPD from 11/10/2017 21:35 MSK.
In the comments, the user SirEdvin asked about the possibility of autocompletion (or IntelliSense in Microsoft terms) in the editor of those python modules that are installed only in the container, i.e., there are no such modules on the local machine that runs VS Code.
In the article this moment was missed, so I gladly supplement with a small chapter:
For example, your project uses a module that does not come in the default python box, and you only installed this module in a container. How to make VS Code “understand”, autocomplete, display hints in the editor without installing the module on the host, but getting it from the module in the container?
To resolve this issue, open the WORKSPACE_SETTINGS settings at: YOUR_PUT_Project / .vscode / settings.json and add the python.autoComplete.extraPaths parameter, and specify the list of container directories containing python modules as keys.
For example, our container is available in the host file system at: / var / lib / lxd / containers / pytest / rootfs / . This is the root file system of the pytest container. Knowing where Python installs external libraries, we can easily add this path to get the full path to external libraries. For modules like numpy, the full path will be: /var/lib/lxd/containers/pytest/rootfs/usr/local/lib/python3.6/dist-packages . We will add this path to the python.autoComplete.extraPaths setting, like this:
Save the settings and return to your project. Try importing the numpy module and triggering completion with CTRL + SPACE. You should get it like on the screen:
If it does not work right away, do not be discouraged, VS Code caches the results and it may not work right away, try to close-open the editor, play around with the CTRL + SPACE keys after the keyword from ... import.
That's all :)
In this article, I would like to share with you the setting up of a development environment for developing scripts in Python and launching them in an isolated environment, almost a couple of clicks in the VS Code editor.
As an isolated environment, the primary configuration and creation of a container based on LXC running on Ubuntu 17.10 OS will be considered. Why not a docker? It is difficult to answer, give some arguments. In part, the concept of OS virtualization is subjectively closer in spirit than process virtualization, but the principle that will be set forth here, with some amendments, will approach the launch of code in Docker containers.
')
What will you get?
- Learn to launch python scripts in a container in one click, as well as remote debug from VS Code
- Learn how to install LXD, set up a network and launch the first container
- Familiarize yourself with installing and configuring the VS Code source editor, a set of extensions that allow you to comfortably write and debug code.
What will not be here?
- There will not be a full review of LXC / LXD, you will have to find additional information about Linux containers yourself.
- The python module virtualenv for creating a virtual environment on the developer’s host as an alternative to containers will not be considered.
- There will be no comparison in the spirit of Docker vs LXD, you will draw your own conclusions based on your experience.
Content:
- Select backend storage
- Installing the LXD package
- LXD Initialization
- LXC network configuration
- Creating a container configuration profile
- Container creation
- Installing software in a container
- Installing VS Code and Extensions
- Configuring VS Code Configuration
- Linking of directories Host <-> Container
- Configure VS Code to run scripts in a container
- Remote Debug in VS Code
- IntelliSense for external Python libraries
Select backend storage
Before installing LXD, you need to decide what type of backend storage for containers will be used. It will not be possible to change later. Several options are available: ZFS, btrfs, LVM, or the host file system. For comparison, a comparison table is available in the official documentation. I suggest to choose ZFS which will allow in the future to easily create COW (copy-on-write) snapshots of containers. Install the ZFS utility package in the system:
$ sudo apt install zfsutils-linux Installing the LXD package
There are no difficulties here, the package is included in the standard delivery of the system, so we command in the console:
$ sudo apt install lxd
After installing LXD, the current user will be added to the lxd system group, but in order to continue setting up with the necessary privileges in the system, you must now re-activate the user session . Just log out and log back in to your Desktop Meneger session or reboot completely. You can check whether the current session of a user is in the lxd group using the id command:
$ id uid=1000(user1) gid=1000(user1) groups=1000(user1),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),118(lpadmin),128(sambashare),129(lxd)
If you see the lxd group, then everything is fine, you can continue on.
LXD Initialization
The initialization of LXD is done interactively on behalf of the current user who is in the lxd group. Run the command and follow the instructions and comments provided below:
$ lxd init # , ? ZFS! # <yes>, <ENTER> Do you want to configure a new storage pool (yes/no) [default=yes]? # <ENTER> Name of the new storage pool [default=default]: lxdpool # , ZFS Name of the storage backend to use (dir, lvm, zfs) [default=zfs]: zfs # ZFS ? .. lxdpool ZFS . # <ENTER> Create a new ZFS pool (yes/no) [default=yes]? # , ? # , dd # . # <no>, <ENTER> Would you like to use an existing block device (yes/no) [default=no]? # loop- (zfs pool)? # GB, 1GB Size in GB of the new loop device (1GB minimum) [default=15GB]: 5 # LXD ? # LXD , - unix /var/lib/lxd/unix.socket # <no>, <ENTER>: Would you like LXD to be available over the network (yes/no) [default=no]? # ? Would you like stale cached images to be updated automatically (yes/no) [default=yes]? no # ? <no>, Would you like to create a new network bridge (yes/no) [default=yes]? no # LXD has been successfully configured.
LXC network configuration
If the network is not set up now, but to start creating containers, then when they are created, we will receive a warning that the container is not connected to any of the networks. This will not be an error, just to them there will be no network access through the interfaces. At the beginning, you can create the required number of containers, and then configure the network and attach containers to it using the lxc network attach command. Now we need to determine what type of network to build. There are several options for building LXC networks, we will choose the bridge option (a simple soft switch between the host and the containers) and build a network based on it.
When creating a network bridge, we can immediately set the properties after specifying its name or rely on the autoconfiguration that will set the IPv4 and IPv6 subnets to a random choice. By default, the parameter is ipv4.nat = false and for some reason transit traffic packets from containers will not be routed further than the host to other local network segments, even when IP forwarding is explicitly enabled in the host core. Apparently NAT in LXD is more than translating addresses from local network segments into global ones, since when this parameter is activated, IP forwarding on the host is also activated, so we will explicitly enable it:
$ lxc network create lxdbr0 ipv6.address=none ipv4.address=10.0.3.1/24 ipv4.nat=true Network lxdbr0 created $ lxc network list +--------+----------+---------+-------------+---------+ | NAME | TYPE | MANAGED | DESCRIPTION | USED BY | +--------+----------+---------+-------------+---------+ | ens33 | physical | NO | | 0 | +--------+----------+---------+-------------+---------+ | lxdbr0 | bridge | YES | | 0 | +--------+----------+---------+-------------+---------+
Creating a container configuration profile
The configuration of the container is conventionally divided into two levels, global and local. The global configuration is set in a configuration profile that is attached to a specific container or group of containers, and the local configuration is applied only to a specific container. That is, all settings in the profile are inherited by the container, and local settings take precedence over the profile. Multiple profiles can be attached to one container.
We will create an LXD profile so that in the newly created containers by default there is a nic interface that will be linked to the host (network bridge) lxdbr0 switch :
$ lxc profile copy default network0 $ lxc profile device add network0 eth0 nic parent=lxdbr0 nictype=bridged
Container creation
For example, create a container and name it pytest , which will be accessible from the host by that name. Run the launch command which loads the image from the ubuntu repository : version 17.10 and the amd64 architecture and from this image will create a container. Additionally, with the option we specify that this container needs to be configured from the network0 profile:
$ lxc launch ubuntu:17.10/amd64 --profile network0 pytest
Next will download from the network image. Let's wait a bit, and a little later we can see the available containers in the system and their status:
$ lxc list +--------+---------+------------------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +--------+---------+------------------+------+------------+-----------+ | pytest | RUNNING | 10.0.3.13 (eth0) | | PERSISTENT | 0 | +--------+---------+------------------+------+------------+-----------+
Installing software in a container
Here we will install the necessary software in the container. Launch the package manager inside the container that updates the system:
$ lxc exec pytest -- /bin/bash -c 'apt update && apt upgrade -y'
Next, install the pip package manager (Python package installer), as well as the ptvsd python module (Visual Studio remote debugging server for Python). The ptvsd module must be installed necessarily version 3.0.0, on more recent versions (current 3.2.1.0) I received an error, which is indicated in the issue on github.com . The author of the Python extension to VS Code that interacts with ptvsd as a temporary solution recommends using version 3.0.0.
$ lxc exec pytest -- /bin/bash -c 'apt install python3-pip -y' $ lxc exec pytest -- /bin/bash -c 'pip3 install ptvsd==3.0.0'
As part of LXD, we have completed the configuration.
Installing VS Code and Extensions
Installation options for the application are described on the official page at code.visualstudio.com/docs/setup/linux , we modify the commands a bit:
$ cd ~ $ wget -c https://packages.microsoft.com/keys/microsoft.asc $ cat microsoft.asc | gpg --dearmor > microsoft.gpg $ sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg $ sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main" > /etc/apt/sources.list.d/vscode.list' $ sudo apt update $ sudo apt install code
After execution, we will have the VS Code of the latest version from the Microsoft repository.
PS At the time of writing, a new version 1.18.0 was released in which, in my opinion, one of the most anticipated (2 years) community “Multi-root workspaces” and “Git indicators in the Explorer” was included in my opinion. All changes can be read on this page .
Let's start installing extensions. If you install VS Code for the first time, for example, after using Atom (as the author), I recommend paying attention to the extensions in the list below, and to achieve the goal of this article it will be enough to install only Python and Code Runner. Extensions are installed from the marketplace on the left side of the editor.
- IntelliSense for CSS class names
- Python
- Jinja
- Django template
- file-icons
- One dark pro
- Subtle Match Brackets
- TODO Highlight
- Code runner
Configuring VS Code Configuration
The following settings are optional, they do not directly relate to running Python scripts in containers, but this will bring you closer to if you want it to be visually as in Atom.
In the open editor, open the editor's settings: file-> Preferences-> Settings. In the window that opens, on the right, in USER_SETTINGS make the following changes and save (Ctrl + S):
{ // "telemetry.enableTelemetry": false, "editor.fontFamily": "Menlo, Consolas, 'DejaVu Sans Mono', monospace", "editor.lineHeight": 21, "editor.minimap.enabled": true, "editor.minimap.showSlider": "always", "editor.minimap.renderCharacters": false, "files.insertFinalNewline": true, "files.trimFinalNewlines": true, "explorer.openEditors.visible": 0, // : "terminal.integrated.lineHeight": 1.25, "terminal.integrated.scrollback": 2000, // : "workbench.iconTheme": "file-icons", "workbench.colorTheme": "One Dark Pro", "workbench.colorCustomizations": { // https://code.visualstudio.com/docs/getstarted/theme-color-reference "editor.lineHighlightBackground": "#2C323C" }, // "editor.matchBrackets": false, "subtleBrackets.styles": { "global": { "color": "#61afef", "borderColor": "#61afef", "borderWidth": "1px", "borderStyle": "none none solid none" } }, // Python "python.pythonPath": "python3", // -: "code-runner.executorMap": { "python": "python3 $fileName" } }
Linking of directories Host <-> Container
We have already done a great job and it is time for the final part. Now we have almost everything ready to run the scripts in the container.
Create a projects directory, for example, at / home / user1 / projects . It will house our future python projects. We will link this directory with the container, i.e., create a disk device in the container and link this directory with the internal / mnt / hostfs . We execute the following commands:
$ mkdir ~/projects $ lxc config device add pytest hostfs disk path=/mnt/hostfs source=/home/user1/projects/
If we locally create a directory or file in the / home / user1 / projects directory, it will immediately be reflected in the container at / mnt / hostfs . You can verify this by running the BASH shell in the container online and running the ls command:
$ lxc exec pytest -- /bin/bash $ ls -l /mnt/hostfs/
You will see that the files and directories in the container will be read-only and will have the UID: GID equal to nobody: nogroup . This is enough to run the scripts, but if you need to make the files and directories in the container available for writing, read the " Direct user / group mapping " chapter in this article .
Configure VS Code to run scripts in a container
And so, now we can return to our editor, if you closed it, then reopen it. Click on the menu File -> New File . This will create a new tab in which you need to enter the following python-code, which we will run in the container. Save it in the / home / user1 / projects / pytest / test.py directory , and then the file (tab) can be closed.
import os #### BEGIN. Visual Studio remote debugging server for Python # import ptvsd # print('Waiting connection on port 3000') # ptvsd.enable_attach('my_secret', address=('0.0.0.0', 3000)) # ptvsd.wait_for_attach() # ptvsd.break_into_debugger() #### END. Visual Studio remote debugging server for Python with open('/etc/lsb-release') as f: lsb_release = f.readlines() info = { 'HOSTNAME': os.uname().nodename, 'DISTRIB': lsb_release[3][21:-2], 'USER': os.environ['USER'], 'LOGNAME': os.environ['LOGNAME'], 'HOME': os.environ['HOME'], 'PWD': os.environ['PWD'] } print(info)
VS Code "will say" that you have not installed the pylint module, but we do not need it now, you can install it later, on your own, through the pip module manager. Now you need to open the workspace in VS Code. To do this, click the Open Folder button on the left in the File Explorer and select the / home / user1 / projects / pytest / directory :
VS Code is reinitialized, all windows will close completely and a workspace will open in which the test.py file will be visible. Open it by double clicking on the name on the left in File Explorer VS Code.
Call up the workspace settings by clicking on the menu File -> Preferences -> Setting, then in the window on the right above, instead of USER_SETTINGS, select from the drop-down list WORKSPACE_SETTINGS:
Make and save the following settings so that you can do it like this:
{ "code-runner.executorMap": { // root: // "python": "lxc exec pytest -- /bin/bash -c 'cd /mnt/hostfs/pytest && python3 $fileName'" // ubuntu: "python": "lxc exec pytest -- /bin/bash -c 'su -l ubuntu -c \"cd /mnt/hostfs/pytest && python3 $fileName\"'" }, "code-runner.runInTerminal": true }
In the file explorer, you can find the settings.json file in the .vscode directory, where the settings of the workspace (the current project) are saved.
Return to the open test.py file and launch it for execution via the command from the context menu of the Run code :
In the output of the integrated terminal you will see information in which you can make sure that the code was executed on the remote machine (container). Now, whenever you need to run a project in a container, it is enough to make settings in WORKSPACE_SETTINGS or create a .vscode / settings.json file for each project and then slightly correct the paths in the settings.
Remote Debug in VS Code
Uncomment the lines in the test.py file and save this file:
#### BEGIN. Visual Studio remote debugging server for Python import ptvsd print('Waiting connection on port 3000') ptvsd.enable_attach('my_secret', address=('0.0.0.0', 3000)) ptvsd.wait_for_attach() ptvsd.break_into_debugger() #### END. Visual Studio remote debugging server for Python
Whenever there is a need for remote debugging, these lines of code must be inserted into the file that is launched for execution. The ptvsd module on the container side opens port 3000 and waits for an external debugger to connect. If you wish, you can configure ptvsd in your own way.
Run the code by calling the Run Code context menu. In the integrated terminal you will see a message that the script is waiting for a connection on port 3000. Go to the debugger and create a new configuration for this project:
You will have the launch.json file for editing in which there will already be presets. Add the following code block to it:
{ "name": "Python LXC", "type": "python", "request": "attach", "localRoot": "${workspaceRoot}", "remoteRoot": "/mnt/hostfs/pytest", "host": "pytest", "port": 3000, "secret": "my_secret" }
Save it, it will appear in the .vscode / launch.json directory next to the setting.json file. Then, select Python LXC in the bottom list on the left; this is the debugger configuration that we just entered:
Click Start Debugging. In the window with the test.py file , you will see a working debugger and confirmation that the code has been launched remotely:
That's all. Setup is complete. All that is needed in the future for the new python project is to configure two config files .vscode / settings.json and .vscode / launche.json . If you have questions on the topic, please ask in the comments. Thank you for your attention, successful projects for you!
UPD from 11/10/2017 21:35 MSK.
In the comments, the user SirEdvin asked about the possibility of autocompletion (or IntelliSense in Microsoft terms) in the editor of those python modules that are installed only in the container, i.e., there are no such modules on the local machine that runs VS Code.
In the article this moment was missed, so I gladly supplement with a small chapter:
Setting auto-completion (IntelliSense) for external python libraries
For example, your project uses a module that does not come in the default python box, and you only installed this module in a container. How to make VS Code “understand”, autocomplete, display hints in the editor without installing the module on the host, but getting it from the module in the container?
To resolve this issue, open the WORKSPACE_SETTINGS settings at: YOUR_PUT_Project / .vscode / settings.json and add the python.autoComplete.extraPaths parameter, and specify the list of container directories containing python modules as keys.
For example, our container is available in the host file system at: / var / lib / lxd / containers / pytest / rootfs / . This is the root file system of the pytest container. Knowing where Python installs external libraries, we can easily add this path to get the full path to external libraries. For modules like numpy, the full path will be: /var/lib/lxd/containers/pytest/rootfs/usr/local/lib/python3.6/dist-packages . We will add this path to the python.autoComplete.extraPaths setting, like this:
Save the settings and return to your project. Try importing the numpy module and triggering completion with CTRL + SPACE. You should get it like on the screen:
If it does not work right away, do not be discouraged, VS Code caches the results and it may not work right away, try to close-open the editor, play around with the CTRL + SPACE keys after the keyword from ... import.
That's all :)
Source: https://habr.com/ru/post/342080/
All Articles