Python 3 to facebook
Hello!
We launched the anniversary fifth stream of the Python Developer course. Before the start of each stream, they often ask what version we use in training (both in general and not only, no matter how strange it may sound, considering what the course is called, Python) and the different nuances of migration from one version to another. Today we want to share an article about the migration experience from 2 to 3 on Facebook, which was told on PyConf.
Go.
')
The transition to Python 3 has become much more popular in recent years, but the process is far from complete. In the infrastructure of many large companies using Python, large blocks of Python 2.7 code remain, and Facebook is no exception. Jason Fried visited PyCon 2018 to talk about the changes in the company over the past 4 years - at the very beginning of Python 3 was virtually absent, but eventually became the main version of Python in the company. Jason helped to achieve this goal, and his report [video on YouTube] is an excellent source of ideas for other organizations that want to migrate.
Fried started working on Facebook in 2011 and quickly realized that he needed to learn Python if he wanted to get a review of the code faster. A little later, he realized that he had become the main driving force behind switching to Python 3 on Facebook. This was never part of his plans, and it happened by itself as he worked with Python.
Fried began by taking an active part in the internal Python group, and was often the first to be ready to answer any questions that might arise. As a result, he became famous (ârather acquired ill fameâ) among Pythonists on Facebook, by fixing the code on his own, without asking permission if he saw that the language is being used incorrectly. This is possible on Facebook, due to the lack of a vertical hierarchy of control; everyone has as many rights to roll back your changes as you have to make these changes initially. Over time, these edits helped to strengthen Freidâs authority in the Python Facebook community, which will come in handy during the migration process.
According to Frayd, changing something like a Python version of the scale of Facebook required some time and a lot of diplomacy. He wanted to tell âa story about how I and a couple of other engineers in my spare time and without any authority made Python 3 the main version on Facebookâ.
In 2013, there was a rudimentary support for Python 3.3 on Facebook. It appeared as part of the task of adding Python 3 support to the build system. But this task was blocked by support for Python 3 in the libraries of Facebook, which in turn was blocked by the lack of support for Python 3 in the build system. The situation as from âTrick-22â : Python 3 was âavailableâ, but nothing in the Facebook environment supported it.
In addition, in 2013, Python 3 on Facebook was surrounded by negative moods. Almost everyone thought that the company would simply remain in Python 2.7 forever. There was also talk about switching to a completely different language. Even Fried himself said (in the inner group) that the transition to Python 3 will never happen on Facebook. Only one person challenged this statement and offered to do something about it; at that moment, Fried ignored the call to action but continued to think about the idea.
Ray of hope
But, according to Freid, hope glimmered. In January 2013, the linter used four imports from
On Facebook, a âuniversally usedâ framework for serialization and remote Apache Thrift procedure calls is used. And its exceptional compatibility with Python 2 has become the main blocker. But in a survey on potentially interesting innovations conducted by the Thrift group on Facebook, adding support for Python 3 was a popular option. Fried voted for him, but not because he wanted to promote Python 3; he simply thought that the Python 2 interface was similar to something from Java, and therefore required refactoring.
His thought process began to change after the Guido van Rossum lecture in Yelp, San Francisco on the topic âTulipâ, which turned out to be a module asyncio . Fried was always a fan of asynchronous programming in Python, but considered it fragmented due to the differences between the frameworks (for example, Twisted , gevent ) that provided it. Tulip looked like it could make asynchronous I / O interoperable. The lecture had not yet finished, and Frayd had already spoken to the Thrift Facebook team, proposing to introduce Tulip support for Python 3, in addition to waiting for the Twisted port, gevent and others. A few days later Trift published a roadmap showing upcoming support for Python 3 and Tulip.
This happened in early 2014, but nothing has changed for another six months; users did not appear and did not plan to do it, in fact, they did not know about any changes.
New project
In August 2014, Frayd began a project on rewriting the inherited service. Initially, for these purposes, he planned to use Python 2 and gevent, but then he realized that by the time the work was completed, it was already outdated. Someone must be the first to make a change; for Facebook and Python 3, it was Fried. âYou must be the person for Python 3 in your organization.â
So he started his project using Python 3 and âeverything was brokenâ; no wonder nobody used Python 3. The build system could not even collect its code, and all third-party wheel packages were available only for Python 2. And when he did fix enough things to build his service, he instantly crashed on launch - because of something deep in the code that set up the entry point on Facebook.
Therefore, in order to make the code work, Frayd was forced to fix everything else; He rebuilt hundreds of third-party wheels to work with both versions of Python, updated all internal libraries to make them compatible with 2/3. However, every day someone commits his dependency changes only for Python 2. Not surprisingly, Fried is tired of fixing regressions. The only solution is to enforce compliance with the requirements of Python 3 within the organization, but this is not possible on Facebook. However, if you start to act as if you have the authority to do this, people will start to believe that you really have them.
Fried, using his social influence, added Pyflakes linter to the build process. Given the presence of PEP 8 , he justified the addition of a new one in that it would solve another category of code problems; Plus, Pyflakes had fewer false positives, so it was less annoying for developers. Frayd set everything up so that Pyflakes runs on all the code sent in the review, first for Python 2 and then Python 3. This helped distribute the work on compliance with Python 3 among all developers, which made it possible to make progress in the Fried project.
At first, he had to explain to people that the linter was not broken, and it makes sense to modify the code to be compatible with Python 3. If the developers believed that the transition to Python 3 was too complicated, they would have returned to the idea of âââstay with Python 2 foreverâ. With the help of Frayd, making the code compatible with Python 3 has become much easier. âPlug in the linter,â and therefore Freid himself, has become easier than complaining about the need to edit the code, so everyone did.
Training
All this helped to stop the âbleedingâ, but did not help to achieve significant progress in the implementation of Python 3 on Facebook. Fried joined the team that taught Python to new employees. The linters already gave errors when the code was not compatible with 2 or 3, but Fried wanted to reach the point where the code compatible with 2/3 was written only for projects, and the new code was written only in Python 3. Once again Fried took situation in their hands: to make this statement, in 2015, he changed the slides in the Python class for new employees. The idea was that at some unknown time in the future, Facebook would want to switch to Python 3, so there is no point in writing code for Python 2 - someday it will have to be rewritten. He explained to the newbies that all this should work within the Facebook infrastructure and build systems, and if this was not possible, you had to send a bug, or fix everything yourself. âOddly enough, it worked.â
In January 2015, Fried âfinally releasedâ his project. He spent the rest of the year telling people about his success and the need to switch to Python 3. During the year, new allies appeared in switching to Python 3 on Facebook.
One of them was Luka Langa (Ĺukasz Langa), who âsomehow convinced Instagram to switch to Python 3â. In 2016, Lang formed a new team in Facebook responsible for the control of Python, which was called the âMinistry of Stupid Walkingâ . Since they were the âPython teamâ, the âfictitious authorityâ mentioned above worked; people thought they could make decisions about Python on Facebook.
In 2016, there was a slow but steady increase in the use of Python 3 in the company. He was talked about at meetings, used in new projects. The opinion changed, although Python 3 was still not considered the default version, and the projects voluntarily chose to use it. In May 2016, Fried announced his intention to switch the build system to Python 3 by default. The idea was fully supported, so after a few days the switch was carried out - without negative consequences.
At the end of 2016, the project team released a report with the results of switching to Python 3. The developers simply launched 2to3 in the code and fixed all the points for which he swore. When the resulting code was launched, it was discovered that it is 40% faster and uses only half of the memory. This pointed to a myth that Frayd often encountered: Python 3 is slower than Python 2. This could be true for earlier versions of Python 3, but this is not relevant now.
Good things
In early 2017, Instagram migrated to Python 3 and Facebook reaped the benefits of âa glorious future, where you can enjoy good things.â Upgrading the Python version was not such a scary process and opened up the possibility of using new features. Facebook developers can now concentrate on issues such as new static typing features or migrating services to use asyncio. âPython on Facebook has become fun again.â
A new problem - people ask when you can abandon support for Python 2. When regressions appear in support of Python 2 libraries or modules, developers often ask if they can simply switch to Python 3. The problem is the opposite of the one that was several years ago. "Oh, how beautiful the world in which I live."
During the lecture, Fried showed a graph of entry points to the Python service of Facebook, starting from Q3 2015 - at that time there were a total of 4 Python 3 points. At the time of the transition to Python 3, in mid-2016, 4% of all points were already Python 3. In March 2018, their number exceeded 50%; in mid-May, at the time of the lecture, there were 55% âof the tens of thousands of Facebook entry pointsâ. According to Freid, the company is now ashamed to write code that only works on Python 2.
Fried then analyzed the process. He noted that more needs to be done than simply creating something new; you need to lead developers to this, âbeing the change you want to see.â It is necessary to attract people, even if they do not understand that they are helping. Linters and unit tests contribute to this. It is very important to teach new employees what you are looking for. When you achieve the result, celebrate while enjoying the âgood thingsâ: write âcool Python code 3â. Seeing how new features can be used, the rest will also want to switch.
Fried answered a few questions from the audience. One question was about how to make this change in a more traditional, hierarchical organization. Fried suggested that it could be even simpler, since there is no need to convince thousands of developers, but only a chain of managers, starting with someone who sees the benefits of the transition. But on the other hand, it can be more difficult if the company's culture is conservative. This will help focus on improving the quality of the code. Another question concerned an indivisible code that is not split into multiple input points. For such a situation, Fried offered to see an Instagram presentation (video on YouTube) from PyCon 2017.
Other organizations can learn a lot from this lecture, but it is clear that it is important to have a strong supporter who is ready to control and coordinate the whole process. Companies planning a transition will need someone like Fried.
THE END
We are waiting for a question and comments that you can leave here or ask Stas directly in his open lesson .
We launched the anniversary fifth stream of the Python Developer course. Before the start of each stream, they often ask what version we use in training (both in general and not only, no matter how strange it may sound, considering what the course is called, Python) and the different nuances of migration from one version to another. Today we want to share an article about the migration experience from 2 to 3 on Facebook, which was told on PyConf.
Go.
')
The transition to Python 3 has become much more popular in recent years, but the process is far from complete. In the infrastructure of many large companies using Python, large blocks of Python 2.7 code remain, and Facebook is no exception. Jason Fried visited PyCon 2018 to talk about the changes in the company over the past 4 years - at the very beginning of Python 3 was virtually absent, but eventually became the main version of Python in the company. Jason helped to achieve this goal, and his report [video on YouTube] is an excellent source of ideas for other organizations that want to migrate.
Fried started working on Facebook in 2011 and quickly realized that he needed to learn Python if he wanted to get a review of the code faster. A little later, he realized that he had become the main driving force behind switching to Python 3 on Facebook. This was never part of his plans, and it happened by itself as he worked with Python.
Fried began by taking an active part in the internal Python group, and was often the first to be ready to answer any questions that might arise. As a result, he became famous (ârather acquired ill fameâ) among Pythonists on Facebook, by fixing the code on his own, without asking permission if he saw that the language is being used incorrectly. This is possible on Facebook, due to the lack of a vertical hierarchy of control; everyone has as many rights to roll back your changes as you have to make these changes initially. Over time, these edits helped to strengthen Freidâs authority in the Python Facebook community, which will come in handy during the migration process.
According to Frayd, changing something like a Python version of the scale of Facebook required some time and a lot of diplomacy. He wanted to tell âa story about how I and a couple of other engineers in my spare time and without any authority made Python 3 the main version on Facebookâ.
In 2013, there was a rudimentary support for Python 3.3 on Facebook. It appeared as part of the task of adding Python 3 support to the build system. But this task was blocked by support for Python 3 in the libraries of Facebook, which in turn was blocked by the lack of support for Python 3 in the build system. The situation as from âTrick-22â : Python 3 was âavailableâ, but nothing in the Facebook environment supported it.
In addition, in 2013, Python 3 on Facebook was surrounded by negative moods. Almost everyone thought that the company would simply remain in Python 2.7 forever. There was also talk about switching to a completely different language. Even Fried himself said (in the inner group) that the transition to Python 3 will never happen on Facebook. Only one person challenged this statement and offered to do something about it; at that moment, Fried ignored the call to action but continued to think about the idea.
Ray of hope
But, according to Freid, hope glimmered. In January 2013, the linter used four imports from
__future__ (print_function, division, absolute_imports, andunicode_literals) . Initially, they appeared to extend the life of the Python 2 codebase, and were added everywhere so that the linter stopped swearing. This ultimately made it easier to convert modules to Python 3.On Facebook, a âuniversally usedâ framework for serialization and remote Apache Thrift procedure calls is used. And its exceptional compatibility with Python 2 has become the main blocker. But in a survey on potentially interesting innovations conducted by the Thrift group on Facebook, adding support for Python 3 was a popular option. Fried voted for him, but not because he wanted to promote Python 3; he simply thought that the Python 2 interface was similar to something from Java, and therefore required refactoring.
His thought process began to change after the Guido van Rossum lecture in Yelp, San Francisco on the topic âTulipâ, which turned out to be a module asyncio . Fried was always a fan of asynchronous programming in Python, but considered it fragmented due to the differences between the frameworks (for example, Twisted , gevent ) that provided it. Tulip looked like it could make asynchronous I / O interoperable. The lecture had not yet finished, and Frayd had already spoken to the Thrift Facebook team, proposing to introduce Tulip support for Python 3, in addition to waiting for the Twisted port, gevent and others. A few days later Trift published a roadmap showing upcoming support for Python 3 and Tulip.
This happened in early 2014, but nothing has changed for another six months; users did not appear and did not plan to do it, in fact, they did not know about any changes.
New project
In August 2014, Frayd began a project on rewriting the inherited service. Initially, for these purposes, he planned to use Python 2 and gevent, but then he realized that by the time the work was completed, it was already outdated. Someone must be the first to make a change; for Facebook and Python 3, it was Fried. âYou must be the person for Python 3 in your organization.â
So he started his project using Python 3 and âeverything was brokenâ; no wonder nobody used Python 3. The build system could not even collect its code, and all third-party wheel packages were available only for Python 2. And when he did fix enough things to build his service, he instantly crashed on launch - because of something deep in the code that set up the entry point on Facebook.
Therefore, in order to make the code work, Frayd was forced to fix everything else; He rebuilt hundreds of third-party wheels to work with both versions of Python, updated all internal libraries to make them compatible with 2/3. However, every day someone commits his dependency changes only for Python 2. Not surprisingly, Fried is tired of fixing regressions. The only solution is to enforce compliance with the requirements of Python 3 within the organization, but this is not possible on Facebook. However, if you start to act as if you have the authority to do this, people will start to believe that you really have them.
Fried, using his social influence, added Pyflakes linter to the build process. Given the presence of PEP 8 , he justified the addition of a new one in that it would solve another category of code problems; Plus, Pyflakes had fewer false positives, so it was less annoying for developers. Frayd set everything up so that Pyflakes runs on all the code sent in the review, first for Python 2 and then Python 3. This helped distribute the work on compliance with Python 3 among all developers, which made it possible to make progress in the Fried project.
At first, he had to explain to people that the linter was not broken, and it makes sense to modify the code to be compatible with Python 3. If the developers believed that the transition to Python 3 was too complicated, they would have returned to the idea of âââstay with Python 2 foreverâ. With the help of Frayd, making the code compatible with Python 3 has become much easier. âPlug in the linter,â and therefore Freid himself, has become easier than complaining about the need to edit the code, so everyone did.
Training
All this helped to stop the âbleedingâ, but did not help to achieve significant progress in the implementation of Python 3 on Facebook. Fried joined the team that taught Python to new employees. The linters already gave errors when the code was not compatible with 2 or 3, but Fried wanted to reach the point where the code compatible with 2/3 was written only for projects, and the new code was written only in Python 3. Once again Fried took situation in their hands: to make this statement, in 2015, he changed the slides in the Python class for new employees. The idea was that at some unknown time in the future, Facebook would want to switch to Python 3, so there is no point in writing code for Python 2 - someday it will have to be rewritten. He explained to the newbies that all this should work within the Facebook infrastructure and build systems, and if this was not possible, you had to send a bug, or fix everything yourself. âOddly enough, it worked.â
In January 2015, Fried âfinally releasedâ his project. He spent the rest of the year telling people about his success and the need to switch to Python 3. During the year, new allies appeared in switching to Python 3 on Facebook.
One of them was Luka Langa (Ĺukasz Langa), who âsomehow convinced Instagram to switch to Python 3â. In 2016, Lang formed a new team in Facebook responsible for the control of Python, which was called the âMinistry of Stupid Walkingâ . Since they were the âPython teamâ, the âfictitious authorityâ mentioned above worked; people thought they could make decisions about Python on Facebook.
In 2016, there was a slow but steady increase in the use of Python 3 in the company. He was talked about at meetings, used in new projects. The opinion changed, although Python 3 was still not considered the default version, and the projects voluntarily chose to use it. In May 2016, Fried announced his intention to switch the build system to Python 3 by default. The idea was fully supported, so after a few days the switch was carried out - without negative consequences.
At the end of 2016, the project team released a report with the results of switching to Python 3. The developers simply launched 2to3 in the code and fixed all the points for which he swore. When the resulting code was launched, it was discovered that it is 40% faster and uses only half of the memory. This pointed to a myth that Frayd often encountered: Python 3 is slower than Python 2. This could be true for earlier versions of Python 3, but this is not relevant now.
Good things
In early 2017, Instagram migrated to Python 3 and Facebook reaped the benefits of âa glorious future, where you can enjoy good things.â Upgrading the Python version was not such a scary process and opened up the possibility of using new features. Facebook developers can now concentrate on issues such as new static typing features or migrating services to use asyncio. âPython on Facebook has become fun again.â
A new problem - people ask when you can abandon support for Python 2. When regressions appear in support of Python 2 libraries or modules, developers often ask if they can simply switch to Python 3. The problem is the opposite of the one that was several years ago. "Oh, how beautiful the world in which I live."
During the lecture, Fried showed a graph of entry points to the Python service of Facebook, starting from Q3 2015 - at that time there were a total of 4 Python 3 points. At the time of the transition to Python 3, in mid-2016, 4% of all points were already Python 3. In March 2018, their number exceeded 50%; in mid-May, at the time of the lecture, there were 55% âof the tens of thousands of Facebook entry pointsâ. According to Freid, the company is now ashamed to write code that only works on Python 2.
Fried then analyzed the process. He noted that more needs to be done than simply creating something new; you need to lead developers to this, âbeing the change you want to see.â It is necessary to attract people, even if they do not understand that they are helping. Linters and unit tests contribute to this. It is very important to teach new employees what you are looking for. When you achieve the result, celebrate while enjoying the âgood thingsâ: write âcool Python code 3â. Seeing how new features can be used, the rest will also want to switch.
Fried answered a few questions from the audience. One question was about how to make this change in a more traditional, hierarchical organization. Fried suggested that it could be even simpler, since there is no need to convince thousands of developers, but only a chain of managers, starting with someone who sees the benefits of the transition. But on the other hand, it can be more difficult if the company's culture is conservative. This will help focus on improving the quality of the code. Another question concerned an indivisible code that is not split into multiple input points. For such a situation, Fried offered to see an Instagram presentation (video on YouTube) from PyCon 2017.
Other organizations can learn a lot from this lecture, but it is clear that it is important to have a strong supporter who is ready to control and coordinate the whole process. Companies planning a transition will need someone like Fried.
THE END
We are waiting for a question and comments that you can leave here or ask Stas directly in his open lesson .
Source: https://habr.com/ru/post/425233/
All Articles