On the development of a single desktop application in Python

Hello. In this article I want to talk about the development of an open source program for offline storage of notes - OutWiker. I do this project in my spare time, the first version of the program was released back in 2010, and OutWiker still continues to evolve. So historically, I do the code almost alone (although occasionally I get useful individual patches), but users are actively involved in translating the program into their native language and sometimes send design styles for pages that I then include in the assembly. And so about how many interesting ideas they send, and say no.

What is OutWiker

I once wrote about this program in Habré, but it was so long ago that before talking about the features of the internal structure and development process, I must say what the program is from the user's point of view. All references related to the project are given at the end of the article. So, OutWiker is a program for storing notes in the form of a tree, in the English-language Internet such software is usually called outliner (therefore, the program has such a name). Among the more well-known "colleagues" OutWiker with similar functionality are Zim, WikidPad, CherryTree, and many others (well, and, of course, org mode for Emacs). The logical question from the user's point of view is how OutWiker differs from other representatives of the tree-like notebooks. At the moment, after so much time has passed since the start of development, I’m not ready to compare all this software in full. In ancient times, I tried dozens of outliners, I used about five for quite a long time, but everywhere I missed something, I wanted to take one opportunity from WikidPad, another from a wikidvodka, which can work offline, etc. Therefore, as an answer to this question, I will simply list the main features that OutWiker possesses.

All notes are stored as folders on the disk. This is done for two reasons - for reliability, so that, for example, with the hard dying of the hard, all the notes do not go to the next world with one file. And besides, it allows you to view and edit notes without OutWiker. Of course, in this method of storage has its drawbacks, but nothing is given just like that.
An arbitrary number of files can be attached to each note (in fact, folders, but this feature is not particularly advertised). One of the purposes of attachments (mostly pictures) is to use them in the text of notes. Thus, changing the attached picture, we immediately see a new picture in the text of the page.
OutWiker doesn’t have a visual editor (I think it will appear sometime, but so far no hands reach it), but there are several types of pages - HTML pages, wiki pages and Markdown pages (after installing the appropriate plug-in). In OutWiker, the emphasis is on wikipages. Wikinote reminds pmWiki (not a very common site engine), but with some differences, I’ll tell you about the features of wikination and its implementation below.
As you already understood from the previous paragraph, the program can work with plugins that are currently more than 20. For example, using plugins for wikiscases you can add code coloring for different programming languages, support for formulas in LaTeX format, the ability to insert different counters (for example, drawing numbers), graphs and charts for data, you can add templates for frequently entered phrases, create pages, download them from the Internet and much more.
Each note can be marked with an arbitrary number of tags. Tag cloud is shown on the side of the program window.
The program is cross-platform, there are builds for Windows and Linux (with Mac OS X, I still do not add up).
The source code is open and distributed under the GPL 3 license.

Something about domestic kitchen development

Language and main libraries

Now I will tell you how OutWiker works from the inside, as well as what problems it had to face during development.

The entire project is written in Python, and in Python 2.7, the transition to Python 3.x is planned, but not in the near future (why, I will say a little later). To create an interface, use the wxPython library. At the inception stage of the project, I chose between wxPython and PyQt as an interface library, the rest of these libraries were rejected, because they create an interface that looks alien either under Windows or under Linux, and maybe both there and there. As a result, I stopped at wxPython because the size of the program was smaller. I will not mention the exact size figures anymore, and the project has already greatly expanded since then. I am pleased with the choice of wxPython, although some unpleasant moments were associated with it (I’ll also talk about them later).

I chose Python because everything is great with cross-platform. It was originally planned to add the ability to create plugins to the program, and since plugins are regular Python scripts, there is no need to assemble them separately for different operating systems. In general, I have no regrets about the choice of language, and there were no brakes that would arise precisely because of the language. There are sections of the program that do not work very quickly, but there are other reasons, mainly in having to read a large number of files from the disk.

As for cross-platform, the program is not so many places, whose work depends on the operating system, and they are all related to the interface. Perhaps the biggest difference lies in the way HTML pages are displayed. Under Windows, the Internet Explorer engine is used, and under Linux - WebKit. Previously, OutWiker worked under wxPython 2.8 and the Internet Explorer and WebKit engines were used with hacks, which were no longer relevant with the release of wxPython 3.0, now the classes for working with these engines are built into the library. True, I continue to use Internet Explorer through a hack with a call to a COM object, because such use gives more possibilities for adjusting the behavior of the engine. Theoretically, the class wx.html2.WebView , which should work on Windows through Internet Explorer, and on Linux via WebKit, has the GetNativeBackend () method, which should return a pointer to the “real” engine, but for some reason I have always returned this method None .

I must admit that the project was greatly influenced by the WikidPad program. I myself used it for a long time, it was also written in Python + wxPython, and sometimes I even peeked at its sources to see how some of the points were made there. I must say that such spying saved a lot of time.

Wiki

There are three types of pages for pages with design: HTML pages, Markdown pages (you need to add the appropriate plug-in) and WikPages. The fact that the program should be a booklet I decided from the inception of the project. Unlike HTML, wikinations are more concise and easy to expand by adding new commands. Another advantage of wikinocations is the ability to convert an embedded text description into a graphic object. For example, in OutWiker there is a command to insert formulas in the LaTeX format, insert graphs that are built either from data from a text file or from data that is inserted directly into the page code, there is a plugin that creates a description of diagrams from the text description, etc. . In addition, since the wiki page is eventually converted to HTML for display to the user, this can also be used, for example, when preparing articles for a site, if the site engine is waiting for input in HTML format. For example, now I am writing this article in the form of a wiknotation, and at the end I will switch to the HTML tab and get ready HTML code that will only need to be customized for the particular site.

Wikotascript:

Result:

HTML:

As a wikination, I focus on the pmWiki engine, it is not very common, but I already had experience with it and I like the thoughtfulness of its notation. At some points in the implementation of the parser, I moved away from the original notation, but in general I try to stick to it. It is very easy to add your commands in this format (:: command_name:) ... (:: command_nameend :).

Sometimes people ask why I didn’t initially use Markdown, but in my opinion this is too limited a language, besides pmWiki is more logical. For example, in Markdown I am always confused with the order of the arguments for the link: [text] (http://example.com/) - two types of brackets, their order is also important. In the wikination used, the link is in the form: [[text -> example.com ]]. The arrow indicates the direction of the link. In fact, there is also a second link format, but personally I usually use this.

By the way, in OutWiker it is not necessary to memorize a wikinacation; all design elements can be inserted through the menu or using the toolbar. For HTML and Markdown pages, such buttons and menu items are also available.

To create a wikiparser uses a very handy library pyparsing . It is convenient, first of all, because it is just one pyparsing.py file, which can be put into the source code and do not pull an extra dependency in the project. With the help of this library, all used tokens of the wikination are described, and then they are converted into HTML. About pyparsing there is detailed documentation with many examples and even a small book. In addition, the author of the library actively participates in the forum on the site and is ready to help with the use of the library.

Localization

Some users are actively working on translating OutWiker into other languages, for which many thanks to them. For co-transfer is used service crowdin.com . This is a fairly convenient service where users can offer their own translation options for phrases, discuss them.

At the moment, the program supports English, Russian, Ukrainian and Swedish. Unfortunately, support for some languages has stopped. So, for example, we need translators who will complete the Italian localization (the Italian guy who once actively supported this localization, disappeared somewhere) and the German localization (now part of the German translation was done by German, and part of my wife as a practice of German, which she teaches. It is necessary that someone check and approve the second part of the translation. In addition, plug-ins were not translated into German). There are several languages in which someone started to do translations, but they are not brought to mind (Bulgarian and Portuguese Brazilian). Well, of course, any other translations are welcome.

For example, what Ukrainian and Swedish localization looks like.

To use localizations inside OutWiker, the standard gettext module is used.

Assembly features

Since Python is an interpreted programming language, in order for a user to run a program, theoretically, Python and all necessary libraries should be installed. Of course, forcing ordinary users to install Python, wxPython and other required libraries, and even the necessary versions, is inhumane. The user must download the program, unpack it, install it using the installer and the "Next - Next - Next" method, then launch the icon on the desktop. Fortunately, for programs written in Python, this is all done relatively simply. If we talk about Windows, the Python interpreter without standard libraries fits into a dll-size of slightly more than 3 MB. There are several utilities that make executable files (exe-shniki, if we talk about Windows) from Python-scripts. The most famous of them are cx_Freeze and pyInstaller . The possibilities are approximately equivalent (although pyInstaller allows you to create a single exe-file from the Python script, but cx_Freeze does not have this possibility), but they have different approaches to where to store the * .pyc library files, and how to set build options

For creating exe-shnik OutWiker for a long time used cx_Freeze. There were no problems with it until it came out cx_Freeze 5.0, in this version the author rewrote the inside of the utility. It can be seen because of this, something went wrong. The OutWiker build created with cx_Freeze 5.0 started to hang at the start, moreover, judging by the results of debugging, on level ground, where no problems are expected (most likely this was somehow connected with multithreading). After cx_Freeze 5.0.2 was released, this problem disappeared, but OutWiker stopped unloading from memory when the program was closed. This problem could be solved by explicitly calling sys.exit (0) at the end of the program, but this is a crutch that I did not want to add. In addition, who knows what other problems could arise when using cx_Freeze 5.0.x. It would be possible to stay on the time-tested cx_Freeze 4.x, but I do not like to use outdated libraries. Then it was decided to switch to pyInstaller. The move was fast enough, literally in one day. There were no problems with the new build. Therefore now pyInstaller is used for assembly. From the point of view of users, nothing should change.

The same cx_Freeze / pyInstaller can also be used to create binary builds for Linux, so that the resulting build can run on different Linux distributions. Under Windows, an installer is created from the resulting assembly using Inno Setup, and under Linux, deb packages are created from a similar assembly (on virtual machines, as will be described later).

Another feature of the assembly is how different assembly targets are launched. It all started with a small Makefile, which gradually grew, and there were more and more problems with it. They were mainly related to cross-platform - it was hard to maintain a single Makefile under Windows and Linux. At one time I got sick of it and I began to look for an alternative, preferably, it should be written in Python. The alternative was found rather quickly in the form of the Fabric program. Now all the former Makefile targets have been rewritten into regular Python functions, and indeed the build system has grown enormously. Here, for example, is the Fabric task list now.

 $ fab -l
 Available commands:

     apiversion Print current OutWiker API versions
     apiversions Print current OutWiker API versions
     build Create artefacts for current version.
     clear Remove artefacts after all assemblies
     create_tree Create wiki tree for the tests
     deb Assemble the deb packages
     deb_binary Create binary deb package
     deb_binary_clear Remove binary deb package
     deb_clear Remove the deb packages
     deb_install assembly deb package for current Ubuntu release
     Ubuntu release
     deb_sources_included create files for uploading in PPA (including sources)
     deploy Upload unstable version to site
     doc Build documentation
     linux_binary Assemble binary builds for Linux
     linux_clear Remove binary builds for Linux
     locale Update the localization file (outwiker.pot)
     locale_plugin Create or update the localization file for pluginname plug-in
     outwiker_changelog Generate OutWiker's changelog for the site
     plugin_changelog Generate plugin's changelog for the site
     plugin_locale plug-in for or pluginname
     plugins Create an archive with plugins (7z required)
     plugins_clear Remove an archive with plugins (7z required)
     plugins_list Print plugins list for th site
     prepare_virtual Prepare virtual machine
     run run outwiker from sources
     site_versions Compare current outWiker and plugins
     sources Create the sources archives
     sources_clear Remove the sources archives.
     test Run the unit tests
     test_build Run the build unit tests
     upload_binary Upload unstable version to site
     upload_plugin Upload plugin to site
     upload_plugins_pack Upload archive with all plugins to site
     vm_halt Stop virtual machines for build
     vm_linux_binary Create 32- and 64-bit assembly on virtual machines
     vm_prepare Prepare virtual machines for build
     vm_remove_keys Remove local SSH keys for remote virual machines
     vm_run Run virtual machines for build
     vm_stop Stop virtual machines for build
     vm_update Update the virtual machines
     win Build OutWiker for Windows with cx_Freeze
     win_clear Remove assemblies under Windows

As you can see, there are a lot of commands, they are all described in the documentation . This includes commands for building under Windows and Linux, running tests, working with virtual machines for building, updating localizations, posting new versions on the site and others. Despite the fact that, judging by the documentation, Fabric is more focused on working with remote servers (conditional analogue of Ansible), but as a replacement for the Makefile, I really like it.

Ubuntu Wars

Several other unpleasant moments were associated with wxPython, which, fortunately, were quickly resolved. All these problems arose under Ubuntu, which I primarily focus on when developing (due to the fact that I have the main operating system). The first unpleasant moment was associated with the transition from wxPython 2.8 to 3.0 (not to be confused with Python 3.x). The fact is that during the transition from version 2.8 to 3.0 there were some problems with backward compatibility, but, fortunately, these two versions could be installed simultaneously and from the python script you can choose the desired version using wxversion. (with the release of wxPython 4.0, this option will no longer be available). Since Python is an interpreted language, and it is always present in Ubuntu, there is no need to create binary assemblies for running under this operating system, but when installing a deb-package, the source code is scattered in the right folders and started using the python command runoutwiker.py . All necessary libraries for wxPython and WebKit are listed as dependencies. However, with the release of Ubuntu 16.04 LTS, two problems arose. The first one was related to the fact that in this version Ubuntu removed wxPython 2.8, so I had to urgently migrate to wxPython 3.0 and refuse to support another long-lived version of Ubuntu 14.04 LTS. But that's not all, it turned out that in Ubuntu 16.04 (it was fixed in the next versions) wxPython was somehow incorrectly compiled, and the WebKit engine does not work in it without shamanism. Fortunately, OutWiker advanced users helped, who suggested that the problem was bypassed using LD_PRELOAD and an indication of the path to one wxPython related library. As I said, the need for this crutch in future versions of Ubuntu has disappeared, but in order to support Ubuntu 16.04, it has to be used so far.

In order to protect myself and users from similar changes in Ubuntu in the future, I recently started making binary builds for Linux. Such assemblies are made for 32-bit and 64-bit systems. In the future, this will allow you to compile wxPython yourself with the necessary parameters and not to depend on the libraries in the Ubuntu repositories. To create binary assemblies (and deb-packages based on them), Ansible, Vagrant and VirtualBox are used. With Vagrant, two virtual machines are started (32 and 64 bits), program sources are downloaded via SSH using Ansible, binary builds and deb packages are created there, and downloaded back to the host computer. But recently and here the developers of Ubuntu have planted a pig. The fact is that virtual machines are created based on Ubuntu 17.04, and in the future I was going to transfer virtual machines to Ubuntu 17.10. I would have been happy to use the Ubuntu 16.04 LTS version for building, but, as I said, there are problems with wxPython. But suddenly, the Ubuntu developers decided to abandon the support of 32-bit systems. Perhaps in the future to build on virtual machines will have to switch to another distribution. Of course, one could also refuse to support 32-bit systems, but at the moment such support doesn’t bother me as a developer, and may be useful to users.

Other used libraries

In addition to the wxPython and pyparsing libraries already mentioned, other libraries are used in the project. Briefly list them.

PyEnchant . Used to check spelling.
Pygments . Used in Source plugin for coloring source code pieces. At the entrance of the library we submit the source code, at the output we get beautifully designed HTML. Pygments understands a huge number of programming languages, including exotic and rarely used, and also allows you to choose a design style (more than a dozen styles are attached to the library).
Python Markdown . As the name implies, it is intended for parsing Markdown notation.
KaTeX . Used in TeXEquation plugin. JavaScript library for rendering formulas in LaTeX format. Previously, the mimeTeX program was used, but the formulas created with it looked not very aesthetically pleasing and there were problems with displaying formulas on pages with a dark background, because mimeTeX creates a picture with black text.
Beautiful Soup . Used in the WebPage plugin for creating pages from pages on the Internet (all images, CSS styles and JavaScript scripts are downloaded). The library is used to parse and edit the downloaded HTML code. For example, to replace the original links to pictures in the HTML code on the way to the downloaded pictures.
Chardet . Used in the same WebPage plugin to determine the encoding of the downloaded page.
Jinja2 . Used in the Snippets plugin, which is designed to insert template text into notes. Templates are created in Jinja format (with some simplifications).
Blockdiag . It is used in the Diagrammer plugin, which, according to the text description, builds a scheme of cubes interconnected by arrows. The main work in the plugin is performed by the Diagrammer library. It can be viewed as a simplified version of dot and Graphviz .

Plans for further development

The OutWiker project is actively developing, proof of which is the commit statistics on github:

')
Now the source takes about 90 MB, which includes almost 4,300 files, of which 1,772 are python scripts.

There are a lot of plans and ideas (the number of issues on github is now slightly less than 350). Any wishes of users get there, even if in the near future their hands will not reach their realization. Since I deal with the code almost alone, new opportunities do not appear as quickly as I would like. Recently, the stable versions of OutWiker are released about once a year, but every month I post unstable versions. In fact, their instability is a philosophical question, because I don’t post frankly non-working versions, and sometimes unstable versions are “more stable” stable in the sense that after they are released, they fix some non-critical flaws from the last release. But in unstable versions, I can afford to not have time to grind something, and maybe not to notice something after adding a new feature.

At the moment, the work is going in two directions - to make a more modern look of the program and reduce the number of files created for each note. As for the interface, I want to get rid of the available toolbars and make them Delphi style. But most likely this global remake will leave until the next version 2.2.

In the light of the ever-present problems with Ubuntu described above, I would like to try making a snap-package or / and a flatpak package under Linux.

It is necessary to write documentation (I mean technical, with help so far everything is in order). In the documentation, I describe the internal structure of the program, especially the assembly, etc. While almost all the documentation is written in Russian, I hope that by the release I will take myself in hand and translate at least part of it into English.

In the longer term, you will need to upgrade to wxPython 4.0, which, I hope, will soon receive release status, and Python 3.x. It is logical to first switch to wxPython 4.0, which supports both versions of Python, and then switch to Python 3.