Remember 2016: our best posts of the year

New Year is getting closer, only a few days left. It's time to take stock. In total for this year we published almost 200 posts (this one is 198th) with a total rating of 6912, received 6,245 comments, 21,835 favorites and almost 3 million views! We sat down with tenderness and nostalgia remembered the best posts in our blog for 2016. Join now!

Dangerous video: how I found a vulnerability in video hosting and did not die after 7 days from cdump

In this post, Mail.Ru Cloud’s programmer told how he found and reported one interesting vulnerability, for which he received a solid gesheft.

Ivan Grigorov: “For top baghunters $ 25K a month is not a problem” from Penny_2_Lane

How to start looking for bugs? Could this be the only source of income? Which Bug Bounty to participate in? How much do baghunters earn? And why is it especially beneficial to deal with the search for vulnerabilities in a crisis? Read the answers to these and other questions in our interview with one of the best bug hunters in the HackerOne platform rating.
')

Bear, dismemberment and February 14 from puelle

By February 14, we decided to assemble a robotic bear that would not only be able to communicate and give people happy hugs, but also analyze human actions. After all, hugs, both sluggish and intense, are able to give true feelings and reveal hidden emotions. Suddenly there are two people in the company who are hugging exactly the same? What if PHP programmers prefer long-term touches, and barbels are limited to dry-fingered? With the help of robomed we managed to collect the most unusual data from the life of IT people.

How to turn from a server-side programmer into a client-side from nikiasi in 7 days

The story of a freshly-minded mobile developer about how he abruptly changed his specialization, having left the post of C / C ++ group leader - Mail.Ru Mail.

Some thoughts and tips on optimizing C ++ code from nekipelov

As Cap suggests, in this post one of our programmers shares tips on optimizing C ++ code so that the speed of its work does not fall below the baseboard Python / Ruby.

Markhor malloc experiments

All malloc POSIX implementations in one way or another run into a number of specific functions. Compared with the naive selection and release of pages, rounding the required size in a big way, malloc has many advantages. In this article, three experiments were conducted with malloc on POSIX-compatible operating systems - Linux and OS X.

Tarantool vs Microsoft Azure competitors by rvncerr

We decided to check how well Tarantool works in Microsoft Azure in comparison with other similar offers - Azure Redis Cache, Bitnami Memcached, Aerospike and VoltDB. The word “good” is understood as “fast”, that is, we will compare the number of requests processed per second (Throughput, RPS).

(Why) Mail.Ru Mail includes strict DMARC from z3apa3a

In this article, we explained some technical details on the inclusion of a strict DMARC policy on all domains owned by Mail.Ru, and gave recommendations to owners of services, mail servers and mailing lists.

Best Go practices, six years in business from FZambia

Extensive material on the best practices of Go, time-tested. Which ones are outdated or become ineffective? What's new for the last year?

Infected Redux Tutorial

This tutorial covers building a full-stack application from scratch using Redux and Immutable-js. Applying the TDD approach, you will go through all the stages of constructing a Node + Redux backend and React + Redux frontend applications. In addition, the use of tools such as ES6, Babel, Socket.io, Webpack, and Mocha has been reviewed. The set is very curious, and you instantly master it!

How to set up an extensible system for regression testing on phones: the Mail.Ru mobile Mail experience from p_alexey

The article describes how we built from scratch a flexible and extensible system for performing autotests on Android smartphones. Now we have about 60 devices used for regression testing of Mail.Ru Mail mobile application. On average, they test about 20 application builds daily. About 600 UI tests and more than 3,500 unit tests are performed for each build.

Stylization of images using neural networks: no mysticism, just mathan from mephistopheies

Surely, you have noticed that the theme of the stylization of photographs for various artistic styles is actively discussed on these web sites of yours. Reading all these popular articles, you might think that magic is going on under the hood of these applications, and the neural network really fantasizes and redraws the image from scratch. It just so happened that our team was faced with a similar task: as part of the intra-corporate hackathon, we made video styling, because application for photos already. In this article, we'll figure out how this network "redraws" images, and consider the articles that made this possible.

Features of the file systems that we encountered when developing the ask.ogorev Mail.Ru Cloud synchronization mechanism

One of the main functions of the Mail.Ru Cloud client's desktop client is data synchronization. Its goal is to bring the folder on the PC and its presentation in the Cloud to the same state. In developing this mechanism, we met with some, at first glance, fairly obvious features of various file and operating systems. However, if you don’t know about them, you can face some rather unpleasant consequences (you won’t be able to download or delete the file). In this article, we have collected features, the knowledge of which will allow you to work correctly with the data on the disks and, possibly, will save you from the need for an urgent hotfix.

Arrays in PHP 7: AloneCoder hash tables

The article discusses in detail how the hash tables are arranged in PHP 7, how to work with them from the point of view of the C language and how to manage them using PHP tools (using structures called arrays). Do not forget that we use hash tables everywhere (usually in the role of dictionaries), therefore, we need to design them so that they are quickly processed by the processor and consume little memory. These structures have a decisive impact on the overall performance of PHP, since local arrays are not the only place where hash tables are used.

We break garbage collection and deserialization in PHP from AloneCoder

A story about two use-after-free vulnerabilities in the garbage collection algorithm in PHP. One is present in all versions of PHP 5 ≥ 5.3 (fixed in PHP 5.6.23). The second is in all versions of PHP ≥ 5.3, including versions of PHP 7 (fixed in PHP 5.6.23 and PHP 7.0.8). Vulnerabilities can be remotely applied through the PHP de-serialization function. Using them, the authors of the article found RCE on pornhub.com, for which they received a prize of $ 20,000, plus $ 1,000 for each of the two vulnerabilities from the Internet Bug Bounty committee on Hackerone.

PHP: the wrong path from AloneCoder

In the world of PHP programming, there is a set of trends. Some people actively promote them (in books and on websites) as “modern PHP”, while others put them as outdated, stupid, or simply wrong. It seems that all these people tirelessly try to get everyone to program as they see fit. This article is written to share a pragmatic view of PHP programming. A look dictated by experience and practical implications, not popular trends, theories or academic dogmas.

Frontend: Development and support (+ voting) by RubaXa

Let's imagine that you were transferred to a new project. Or you changed the job and only heard about the project. Here you sit at the workplace, the manager comes to you, shakes hands and ... right off immediately opens the project page, pokes a finger at the monitor and asks to insert a "informer about the upcoming event X". On this you part ... What to do? Where to begin? How to create an "informer"? Where to find the right template? And a sea of ​​other issues. This is a story about how we try to organize these processes, what tools we create for the preparation of SPA. In addition, we will talk about the technical details of the implementation of Live Coding / Hot Reload and a bit about VirtualDom and React with Angular.

ExcelArt - isometry "for free". Draw a pseudo-volume phone without 3D and Photoshop from flash_b

Is it possible to get a pseudo-volumetric image without 3D programs? “Of course,” you say, “you take, strain, and draw from scratch.” And if without "you take, you strain and draw"? And if at all without any special knowledge? Is it possible to get isometric objects without spending time on render? Is there any digital illustration out of the usual graphics programs? These and other questions are answered in the article devoted to the new method of creating digital illustrations - ExcelArt.

Difficult quest for habravchan: 25 levels from gag_fenix

This post presents a puzzle designed specifically for Habr users. Challenge yourself, can you solve it?

Overview of the OPCache extension for PHP from AloneCoder

PHP is a scripting language that compiles the files you need to run by default. During compilation, it extracts opcodes, executes them, and then immediately destroys them. PHP was designed this way: when it goes to execute the R request, it forgets everything that was performed during the R-1 request. It is very unlikely that on production servers the PHP code will change between the execution of several requests. So we can assume that during compilations the same source code is always read, and therefore the opcode will be exactly the same. And if you extract it for each script, it turns out a waste of time and resources.

The book "How to survive the full end of the dinner, or security in PHP." Part 1 of sunsai

On the Internet you can find many articles and safety guides. This book seemed to us quite detailed, with a concise and understandable. We hope it helps you learn something new and make your sites safer and more secure.

Springboard Calling Magic Functions in AloneCoder PHP 7

This article discusses in detail the optimization in a virtual machine in PHP 7 (Zend virtual machine). First, let's touch on the theory of springboards function calls, and then find out how they work in PHP 7. If you want to fully understand everything, then it is better to have a good understanding of the work of the Zend virtual machine. For starters, you can read how the VM is arranged in PHP 5, and here we will talk about the PHP 7 VM. Although it has been reworked, it works in much the same way as PHP 7. Therefore, if you figure it out in the PHP 5 VM, you will understand with VM PHP 7 will not be any difficulty.

Lectures of the Technosphere. 1 semester Introduction to Data Analysis (Spring 2016) by Olga_ol

A new compilation of lectures of Mail.Ru Technosphere, this time we present the spring course "Introduction to Data Analysis" in open access, where students are introduced to the field of data analysis, the basic tools, tasks and methods that any data researcher encounters in his work. The course is taught by Evgeny Zavyalov (Mail.Ru Search project analyst, who extracts useful business knowledge from data generated by the search engine and desktop applications), Mikhail Grishin (research programmer from the data analysis department) and Sergey Rybalkin (senior programmer from the Allods Team studio) .

Pitfalls Bash by AloneCoder

In this article we’ll talk about errors made by bash programmers. In all the examples there are some flaws. You will be able to avoid many of the errors described below if you always use quotes and never use wordsplitting! Word breaking is a flawed legacy practice inherited from the Bourne shell. It is applied by default if you do not enclose quotes. In general, the overwhelming majority of pitfalls are somehow connected with the substitution without quotation marks, which leads to word splitting and globbing of the resulting result.

Overview of deep convolutional neural network topologies from mephistopheies

An article with a talking title, do not add, do not subtract. Material for trained readers: it is implied that you are already familiar with the error back-propagation algorithm and understand how the basic building blocks of convolutional neural networks work: convolution and pooling.

Not a single gap: how we created a wireless network for 3000 devices from znoom

The Mail.Ru Group office is a 26-storey building, where a lot of people work. This is a story about how our Wi-Fi network has changed over the past 11 years, and what it is today.

Stories of old mails from yeah_boss

Last-minute servers, a catastrophic fall in services, the heroic rescue of a portal with the help of a wife left at the computer So a fantastic technotriller could begin, but in fact the story of the Russian Internet began, of which we became a part. In honor of the 18th anniversary of the company, we first talked about the real history of the Mail.Ru Group. For the first time in years of life, we’ll share what used to be at the level of myths and legends, even within the company itself. We talked to people who had worked in the company for many years and were directly involved in all the events.

What happened when we were tired of looking at graphics of 5,000 servers in monitoring (and when there were more than 10,000 servers) from Sharapoff

The story of how Odnoklassniki zadolbalsya manually monitor 5000 servers and made a new monitoring system. Now working with 10 thousand servers is spent 1-2 hours a week for processing alerts.

Parsing JSON is a minefield from AloneCoder

JSON is the de facto standard when it comes to (de) serialization, network data exchange and mobile development. But how well are you familiar with JSON? We all read specifications and write tests, test popular JSON libraries for our needs. The article shows that JSON is an idealized format, and not an ideal one, which many consider it. There are not even two libraries behaving the same way. Moreover, extreme cases and harmful payload can lead to bugs, crashes and DoS, mainly because JSON libraries are based on specifications that evolve over time, which leaves many things poorly or not documented at all.

We invite you to Russian AI Cup 2016 from sat2707

This post details the task that was proposed for the participants of the Russian AI Cup 2016 championship.

50 shades of Go: traps, pitfalls and common mistakes beginners from 3vilhamst3r

Go is a simple and fun language. But in it, as well as in any other languages, there are reefs. And in many of them Go itself is not to blame. Some are a natural consequence of the arrival of programmers from other languages, others arise from misconceptions and lack of details. If you find time and read the official specifications, wikis, mailing lists, blog posts and source code, many of the pitfalls will become obvious to you. But not everybody starts like that, and that's fine. If you are new to Go, the article will help you save a lot of hours that you would spend on debugging code. We will consider versions of Go 1.5 and below.

ICQ: 20 years is not the limit from Dimitryopho

More recently, we recalled how from release to release, ICQ was subjected to plastic surgery and facelift. And on the day of the anniversary - ICQ was 20 years old - we decided to look in retrospect, which technologies were used in ICQ for two decades.

Math CSS Gateways by AloneCoder

CSS-lock (CSS-lock) is a technique from responsive web design, which allows not to jump from one value to another, but to move smoothly, depending on the current size of the viewport. The idea and one of the implementations suggested by Tim Brown in the article Flexible typography with CSS locks. The article describes the method itself, its limitations and the underlying mathematics. Do not worry: there is basically one addition and subtraction. In addition, everything is divided into stages and decorated with graphs.

Monitoring and configuring the Linux network stack: receiving data from AloneCoder

This article discusses how packets are received on computers running the Linux kernel, and also discusses the issues of monitoring and configuring each component of the network stack as packets move from the network to user space applications. Here you will find a lot of source code, because without a deep understanding of the processes you will not be able to configure and monitor the Linux network stack.

Puzzles TCP by AloneCoder

They say that you can not fully understand the system until you understand its failures. In this article, TCP errors are presented as puzzles, in the style of Car Talk or old Java puzzles. Like any other good puzzles, they are very easy to reproduce, but solutions are usually surprising. And instead of focusing our attention on mysterious details, these puzzles help to explore some of the underlying principles of how TCP works.

Lectures of the Technosphere. Preparatory course "Algorithms and data structures" (spring 2016) from Olga_ol

The purpose of this course is to introduce students to the basic algorithms used for software development. You will learn how to choose the appropriate data structures and algorithms for the implementation of emerging problems, and learn how to use C / C ++ languages ​​for the implementation of algorithms.

Efficient storage: as we have done from 50 PB, 32 PB from AndrewSumin

Changes in the ruble exchange rate two years ago made us think about ways to reduce the cost of iron for Mail.Ru Mail. We needed to reduce the amount of iron purchased and the price for hosting. What does mail consist of: indexes and letter bodies 15% of the volume, files - 85%. The place for optimizations should be searched for in files (attachments in letters). At that time, we did not implement file deduplication; according to our estimates, it can save up to 36% of the total mail volume: many users receive the same emails (social networks with pictures, stores with price lists, etc.). This post is about the implementation of such a system, made under the guidance of PSIAlt.

About C language and performance from AloneCoder

If a programmer is familiar only with high-level languages, for example, PHP, then it is not so easy for him to master some ideas typical of low-level languages ​​and critical for understanding the capabilities of information and computing processes. For the most part, the reason is that in low and high level languages ​​we solve different problems. But how can you consider yourself a professional in any (high-level) language, if you don’t even know how a processor works, how it performs calculations, in an effective way? Today, automatic memory management is becoming a major problem in most high-level languages, and many programmers approach it without a sufficient theoretical basis. Knowledge of low-level processes greatly helps in the development of effective high-level programs.

On the eve of the birthday of the first female programmer: my story from 6thSence

“You are a girl, why do you need this?” Or “In women, the brain is not adapted to programming”. "We will deal with the work here, and you - the decoration of the team." Surely every female programmer has heard this in her address many times. This is the story of our employee about how she came to the profession of a programmer.

Debugging your OS: AloneCoder memory allocation tutorial

The story of how, at first glance, an unremarkable bug report led to a whole investigation into the Linux memory allocation system.

The truth about traditional JavaScript benchmarks from AloneCoder

For many of those who have studied programming languages, compilers and virtual machines, it’s still surprising that, for all its elegance in terms of the structure of the JavaScript language, we are not very good at compiling and cannot boast a great standard library. Depending on who your interlocutor is, you can list the flaws in JavaScript for weeks and still find out some oddity about which you have not heard. But despite the obvious shortcomings, today JavaScript is a key technology on the web, it is moving toward dominance in the server / cloud realm (thanks to Node.js), and also penetrates the Internet of things.


Thanks to everyone who read us, commented, put the pros and cons. We hope that in the coming year the posts will be even more interesting. Holiday greetings!