How we manage configurations in pics.io

At the end of 2012, the guys and I were going to do a crazy thing - put a RAW converter in Photoshop browser, add photoshop and add the ability to collaborate on photos. On the technical side, we practically checked everything: the technologies that allowed all this to be realized were raw, sometimes they needed to be included in the browser settings ... but they were. We called it Pics.io and started.

We had a stack of prototypes / proof-of-concept, sheets of code with a huge amount of hardcodes. We decided that we need to combine all this into one big product that will completely cover the workflow of the photographer: we invented architecture, wrote a framework, and began to make functionality.

It's time to let the first users. It was quickly discovered that they want our idea, but do not want to use what we have programmed. We had no illusions about this — it’s impossible to attack Photoshop for six months. For a while, we tried to add new features to Pics.io. Users came, registered, but returned only to see how we were doing. It became clear that even early adopters need a product that they can use.

We decided to try to make a separate service that could be used right after the release and which would be beneficial. This is how live.pics.io , a service for voice discussion of photos online, appeared. We have already written an article on technical details here on Habré. Despite the large number of problems with the relatively new technology of WebRTC, the experiment appealed both to us and our users.
')
Then we thought that we could split Pics.io into three simpler parts and work on them separately:

• Edit.pics.io image editing
• Raw.pics.io to convert RAW formats
• Live.pics.io for discussion and collaboration

Then, by linking these services with each other using the DAM (Digital Assets Management) tools and adding user accounts, we got “big” (as we call it between ourselves) Pics.io. Which, in turn, will allow us to achieve our goal - to become a universal package for processing photos in the browser. In total, we received four independent products, each of which is designed to fulfill the tasks of a specific user segment.

Further more. We always wanted to be where our users are. At first it seemed to us that our users are on the Internet, so we will build everything in the browser. Further analysis showed that the Internet is too broad a concept, and it also needs to be segmented. So we found our users on Facebook , Chrome OS, mobile and of course on the web. So, four products on four platforms. It turns out sixteen possible configurations. Not weak, to say the least. Of course, we have some experience in the team, but this is a really cool task that we really wanted to solve.

Lead sixteen different products in parallel is clearly an impossible task for a team of seven people. And what if these are not separate products, but simply configurations of deployment of a set of components? Aha Here it is all the power of configuration management!

So, what we have is:

• team of seven
• sixteen products
• a bunch of custom and open-source code for all occasions (starting from logging and ending with parallel processing of raw formats)

This whole story led us to a rather curious application architecture and interesting solutions in configuration management. It seems to me that the experience that we are going to share will be of interest to everyone who develops complex web applications.


The most important thing that we wanted to achieve is to make all parts of the project make the most of the same code base. Fixes and changes should immediately fall into the micro-services, and become part of a large application. The deployment procedure should be as simple as possible, and if you wish, we should be able to add our pieces to any third-party services. How, then, from a pile of disparate pieces and scripts to assemble a complete infrastructure, within which it is easy to work?

The first thing we did was to bring all the sources into one repository (initially we tried several) from which any of our products is built. All the main development is carried out in one branch, and each product has its own release branch, from which it rolls out to production. Although now we are thinking about leaving only one release branch.

Then we broke our code canvases into minimal modules, the dependencies between which are resolved using RequireJS . On the one hand, this led to a long work on refactoring most of the source code and the emergence of new modules, on the other hand, it allowed us to remove dependencies that should not exist and reduce coherence within the project to a large extent.

Naturally, this approach required a special interaction between the modules, and most of them have a number of limitations (in order to be used in any product). For example, we have absolutely no possibility to use Backbone-models in some modules, since they can be embedded, for example, in raw.pics.io, which does not use Backbone at all. But such restrictions do not really bother us, but on the contrary separate high-level UI modules from the common core and do not litter it with unnecessary dependencies.

RequireJS helps us to assemble modules. For each service there is a configuration file that describes all the dependencies between the modules. Then with the help of the r.js builder, Grunt generates the necessary builds and the only thing we need is to keep our modules “clean” and assemble the new product from the finished pieces using configs. Thus, when developing, we do not think about which product the feature will fall into, since it will fall into all products at once.

Naturally, such an approach required more complex testing - before each release we recheck several products instead of testing only one, but the use of common sources greatly simplified development, although it complicated the configuration and testing. Although check your products once again never hurts. ;)

The scripts for the build and deployment are written, but we are still deploying manually. Our next step is CI, for which we have chosen Strider-CD , which will launch the deployment to the test / production server and / or tell us when something went wrong.

Starting the project, though we presented, we didn’t really understand how we would be able to conduct configurations. Our entire infrastructure has greatly changed during this time, but this is due to the fact that the project itself, its vision and requirements have changed. I am sure that further processes will also change in order to make it convenient and pleasant for us to work.

I think you should not try to immediately come up with an ideal system, it still will not work, because in two months you will have to change something. Instead, it’s better to use the convenient tools that you need right now and help you and your team in the best possible way.