Creating a web application on Go in 2017

A couple of weeks ago, I started developing another web application, only now on Go . Being mainly a backend developer, I didn’t write the whole web application often, so each such case was more like a challenge. At such times, I wanted someone to write a guide to web development for people who do not have the opportunity to go all day into the subtleties of great design, but who just need to create a working, functional website without too much fuss.

I decided to take this opportunity and make such a guide in which to start a web application from scratch in the way it should be done in 2017 (in my understanding). I spent many hours digging into things that I used to avoid before, just so that at least once in many years I could say with confidence that I have my own opinion and experience in this matter, and get a working recipe for myself which may be useful not only for me.

The article begins a short series highlighting what I learned in the process. This first post is a general introduction describing the current state of affairs, problems and why I think Go is a good choice. Subsequent articles will be more detailed and contain more code. I am curious how my experience correlates with yours; maybe I’m wrong about something, so feel free to comment.

If you are only interested in the code, it is here .

Introduction

Previously, my basic knowledge of HTML, CSS, and JavaScript was enough for my modest needs in site-building. Most of the applications that I have ever created have been made using mod_python , directly using the handler publishing mechanism (note: per.: You can see an example here ). It's funny that being an early follower of Python, I also worked a lot with Rails . Over the past few years, I have focused on the infrastructure of (big) data, which is not at all a web development, although the need for web interfaces is not uncommon here. In fact, the application I am working on is an application for working with data, but it is not an open source and what it does is irrelevant for this article. In general, this should make it clear which way I look at it all.

Python and Ruby

A year ago, I would recommend Python or Ruby as a web application environment. There may be other similar languages, but from my point of view, Python and Ruby dominate the world.

Most of the time, the main task of the web application was to construct the web pages using the layout of the final HTML on the server side. Both Python and Ruby are very well suited for extracting data from a database and turning it into a bunch of HTML code using templates. There are many frameworks / tools to choose from, for example, Rails, Django, Sinatra, Flask, etc. etc.

And although these languages ​​have certain significant limitations, such as GIL , the ease with which they solve the problem of generating HTML is much more valuable than the compromises that have to be made.

GIL

GIL (Global Interpreter Lock) is a special mention. Of course, this is the biggest limitation of any solution in Python or Ruby, but this is a very slippery topic, people often prefer to pretend that there is no problem. And if we’re talking about it, emotions usually go over the edge, endless discussions on the topic of GIL take place in the Ruby and Python communities.

For those unfamiliar with this problem, GIL allows only one thing to be executed at a time. When you create threads and they “look” like running in parallel, in fact the interpreter still executes the instructions sequentially. This means that one process can use only one CPU.

There are alternative implementations, for example, based on JVM, but they are rarely used. I don’t know exactly why, maybe they are not fully compatible, or they probably don’t support C-extensions correctly, and they may still have GIL. Not sure, but as far as I can tell, the implementation in C is usually used. To make an interpreter without GIL, you have to rewrite it completely, and this can already change the behavior of the language (in my naive understanding), and therefore it seems to me that GIL will remain.

Web applications of any significant scale necessarily require the ability to service requests in parallel, using the capabilities of each CPU on the machine. So far, the only possible solution is to launch several instances of the application as separate processes.

This is usually done using additional software, such as Unicorn / Gunicorn, with each process listening to its own port and running behind some kind of connection balancer, such as Nginx and / or Haproxy. Alternatively, this can be done through Apache and its modules (such as mod_python or mod_wsgi), in any case it is difficult. Such applications usually rely on the database server as an arbiter for any competitively sensitive tasks. When implementing caching, in order not to store multiple copies of the same on the same server, shared storage is required, such as Memcached or Redis, and usually both. Also, these applications can not do background processing, for this there is a separate set of tools, such as Resque. And then all of these components require monitoring to be sure that it all works. Logs must be consolidated, and they have their own additional tools. Given the inevitable complexity of this setup, a configuration manager, such as Chef or Puppet, is also required. Nevertheless, these kits are usually not able to support a large number of long-term compounds - a problem known as C10K .

As a result, a simple web application with a database requires a whole bunch of parts before it can serve the Hello World! Page. And almost all of this is due to GIL.

Appearance of one-page applications

Farther and farther into the past is the generation of HTML on the server. The latest (and correct) trend is to build the user interface and render completely on the client side, using JavaScript. Applications whose user interface is completely controlled by JS are sometimes called a one-page application and, in my opinion, the future is with them, whether we like it or not. In such applications, the server only serves the data, usually in the form of JSON, without generating HTML code. In this case, that enormous complexity, introduced primarily for the possibility of using the popular scripting language [to create a web application], turns out to be unnecessary. Especially considering that Python or Ruby brings little benefit when all output is JSON.

Golang look

Go is gradually undermining the established world of web applications. It natively supports parallel execution, which eliminates the need for almost all components commonly used to work with GIL constraints.

Go programs are binary programs that are natively launched, so no language-specific installation is required on the server. The problem of ensuring the correct version of the execution environment required by the application disappears; There is no separate execution environment - it is built into the binary. Go programs can easily and elegantly run tasks in the background, so there is no need for tools like Resque. These programs run as a single process, so caching becomes trivial, which means that Memcached or Redis are not needed. Go can control an unlimited number of parallel connections, leveling the need for front-end protection, such as Nginx.

With Go, a high layered tower from Python, Ruby, Bundler, Virtualenv, Unicorn, WSGI, Resque, Memcached, Redis, etc., etc. reduced to just one binary. The only third-party component that is usually still needed is the database (I would advise PostgreSQL). It is important to note here that all these tools can still be used, but you can do without them with Go.

The launch time of such a Go program will most likely be an order of magnitude greater than any Python / Ruby application, requiring less memory and lines of code.

Well, is there a popular framework?

The short answer is: the framework is optional and not recommended. There are many projects that claim to be a great framework, but I think it's better to do without them. This is not only my personal opinion, I find this opinion fairly common in the Go community.

It is necessary to understand why frameworks were created at all. In the world of Python / Ruby, this happened because these languages ​​were not originally designed to serve web pages, and many external components were needed to solve this problem. The same can be said about Java, which, like Python and Ruby, is as old as the web, as we know it, or even a bit older.

As far as I remember, early versions of Python "out of the box" did not provide anything for working with the database, there were no templates, HTTP support was confusing, work with the network was nontrivial, even encryption was then illegal and in general, a lot of things were missing. The framework provided all these necessary pieces and set language-specific development rules for all common variants of web applications.

Go, on the other hand, was created by people who already had experience and were versed in web development. It includes almost everything you need. One or two external packages may be needed to solve some specific problems, such as OAuth, but in no case this pair of packages is a “framework”.

If all of the above with regards to frameworks does not sound convincing enough, it is useful to consider the learning curve for frameworks and risks. It took me about two years to build a relationship with Rails. Frameworks can become abandoned and obsolete, and it is hard and sometimes impossible to port an application to a new framework. Considering how quickly everything changes in information technology, the framework certainly should not be chosen lightly.

I would like to highlight tools and frameworks that attempt to mimic idioms common to Python, Ruby, or JavaScript environments. Everything that looks, or feels, or claims the role of “Rails for Go”, including techniques such as injections, dynamic publication of methods, etc., which are highly dependent on reflection, does not fit Go ideology, therefore it’s better to stay away.

Undoubtedly, frameworks make some things easier, especially in the typical world of CRUD applications for business, where applications have many pages with a large number of fields, manipulate data in complex and constantly changing database schemas. I'm not sure that in such an environment Go is a good choice, especially if performance and scalability are not in priority.

Another problem common to frameworks is that they abstract the low-level mechanisms from the developer so that over time they become so mysterious that it is literally impossible to understand what they really have going on there. What begins with a lexical pseudonym for a single line of JavaScript becomes a layer in layers of transpilers, minimizers, on top of helpers hidden somewhere in subdependencies. Once something breaks, and it is impossible to understand where to look for the problem. It's nice when you know exactly what is happening, and Go is very good at it.

How about a database and ORM?

Similar to frameworks, ORMs in Go are not very common. To begin with, Go does not support objects - what is indicated by O in the abbreviation ORM.

I know if instead of using the convenient User.find(:all).filter... , which is provided with something like ActiveRecord, writing SQL manually is something unheard of in some communities, but I still think that this attitude must change. SQL is a great language. Dealing with SQL directly is not so difficult, and in return we get more freedom and opportunities. Perhaps the most tedious part of such direct work is copying data from the database cursor to the structures, but the sqlx project is very useful here .

Conclusion

In my opinion, the article describes in some detail the current situation on the server side. I think the client part is better to be allocated in a separate post, so today - everything. To summarize, we build an application with approximately the following requirements:

• Minimal dependency on third party packages.
• No web framework.
• PostgreSQL as a database.
• Single page application.

Continuation