What a programmer who is switching to Python needs to remember

Once upon a time, in my student years, I was bitten by a python, although the incubation period dragged on and it turned out that I became a programmer on a pearl.

However, at some point, the pearl exhausted itself and I decided to go in for python, at first I just did something and understood what was needed for this task, and then I realized that we need some systematic knowledge and read several books:

• Bill Lyubanovich “Simple Python. Modern programming style "
• Dan Bader "Pure Python. Subtleties programming for pros
• Brett Slatkin, Python Secrets: 59 Recommendations for Writing Effective Code

Which seemed to me quite suitable for understanding the main subtleties of the language, although I don’t remember what they mentioned about slots , but I’m not sure that this is a really necessary feature - if it’s already pressed from memory, then most likely this method alone will not be enough it all depends on the situation.

As a result, I have accumulated some notes about the features of python, which, I think, can be useful to anyone who wants to migrate to it from other languages.

I noticed that during interviews on python quite often they ask questions about things that are not related to real development, such as what can be a dictionary key (or about what x = yield y means), well, dudes, in real life it can be only a number or string, in those unique cases where it is not, you can read the documentation and figure out why to ask this? To find what the interviewee does not know? So in the end, everyone will remember the answer to this question and it will stop working.

I consider as relevant the python versions above 3.5 ( it’s time to forget about the second python) because This is the version in stable debian, which means more recent versions in all other places)

Since I am not a python guru at all, I hope that they will correct me in the comments if I suddenly blinked out some nonsense.

Typing

Python is a dynamically typed language i.e. it checks type conformance during execution, for example:

 cat type.py a=5 b='5' print(a+b) 

perform:

 python3 type.py ... TypeError: unsupported operand type(s) for +: 'int' and 'str' 

However, if your project has matured to the need for static typing, then the python provides such an opportunity by using the mypy static analyzer:

 mypy type.py type.py:3: error: Unsupported operand types for + ("int" and "str") 

True, not all errors are caught like this:

 cat type2.py def greeting(name): return 'Hello ' + name greeting(5) 

mypy does not swear here, and an error will occur when it is executed, so the current versions of python support a special syntax for specifying the types of function arguments:

 cat type3.py def greeting(name: str) -> str: return 'Hello ' + name greeting(5) 

and now:

 mypy type3.py type3.py:4: error: Argument 1 to "greeting" has incompatible type "int"; expected "str" 

Variables and data

Variables in python do not store data, but only refer to them, and the data is mutable (immutable) and immutable (immutable).
This leads to different behavior depending on the type of data in almost identical situations, for example, the following code:

 x = 1 y = x x = 2 print(y) 

leads to the fact that the variables x and y refer to different data, and this:

 x = [1, 2, 3] y = x x[0] = 7 print(y) 

No, x and y remain links to the same list (although, as noted in the comments, the example is not very successful, but I have not yet thought of a better idea) that, by the way, in python, you can check with the is operator (I am sure that the creator of Java has always lost good sleep from shame when I found out about this operator in python).

Although the strings look like a list are an immutable data type, this means that the string itself cannot be changed, you can only generate a new one, but you can assign a different value to the variable, although the source data will not change:

 >>> mystr = 'sss' >>> str = mystr #       >>> mystr[0] = 'a' ... TypeError: 'str' object does not support item assignment >>> mystr = 'ssa' #    >>> str #         'sss' 

Speaking of strings, because of their immobility, concatenation of a very large list of strings by adding or append in a loop may not be very effective (depends on translation in a particular compiler / version), usually for such cases it is recommended to use the join method, which behaves a little unexpected:

 >>> str_list = ['ss', 'dd', 'gg'] >>> 'XXX'.join(str_list) 'ssXXXddXXXgg' >>> str = 'hello' >>> 'XXX'.join(str) 'hXXXeXXXlXXXlXXXo' 

Firstly, the string in which the method is called becomes the separator, and not the beginning of a new line as one might think, and secondly, you need to transfer a list (an iterated object), not a separate line, because such is also an iterable object and will be joined by character .

Since the variables are references, it’s quite normal to want to make a copy of the object so as not to break the original object, however there is a pitfall - the copy function copies only one level, which is clearly not what is expected from a function with that name, so you use deepcopy .

A similar problem with copying can occur when a collection is multiplied by a scalar, as was recently discussed here .

Area of ​​visibility

The scope of variables in python is limited to the module / function in which it is defined and nested functions, but there is subtlety — the variable is available by default for reading in nested namespaces, but the modification requires the use of special keywords nonlocal and global to modify variables one level higher or global visibility respectively.

For example, the following code:

 x = 7 print(id(x)) def func(): print(id(x)) return x print(func()) 

It works with one global variable, and this:

 x = 7 print(id(x)) def func(): x = 1 print(id(x)) return x print(func()) print(x) 

already generates local.
From my point of view, this is not very good, in theory any use of non-local variables in a function is part of the function's public interface, its signature, and therefore must be declared explicitly and apparently at the beginning of the function.

Function arguments

Python provides simply awesome options for specifying function arguments — positional, named arguments, and their combinations.

But you need to understand how the transfer of arguments is carried out - In Python, all variables are data references, you can guess that the transmission is carried out by reference, but there is a feature here - the link itself is passed by value i.e. You can modify the mutable value at the link:

 def add_element(mylist): mylist.append(3) mylist = [1,2] add_element(mylist) print(mylist) 

perform:

 python3 arg_modify.py [1, 2, 3] 

however, you cannot overwrite the original link in the function:

 def try_del(mylist): mylist = [] return mylist mylist = [1,2] try_del(mylist) print(mylist) 

The original link is alive and working:

 python3 arg_kill.py [1, 2] 

Also, you can set default values ​​for arguments, but there is one unclear thing to remember: default values ​​are calculated once when defining a function, it does not create any problems if you pass unchanged data as default values, and if you pass variable data or dynamic value, the result will be a bit unexpected:

changeable data:

 cat arg_list.py def func(arg = []): arg.append('x') return arg print(func()) print(func()) print(func()) 

result:

 python3 arg_list.py ['x'] ['x', 'x'] ['x', 'x', 'x'] 

dynamic value:

 cat arg_now.py from datetime import datetime def func(arg = datetime.now()): return arg print(func()) print(func()) print(func()) 

we get:

 python3 arg_now.py 2018-09-28 10:28:40.771879 2018-09-28 10:28:40.771879 2018-09-28 10:28:40.771879 

OOP

OOP in python is done quite interestingly (some properties are worth something) and this is a big topic, but sapiens familiar with OOP can easily google everything (or find it on Habré ) what he wants, so there’s no point in repeating, the only minus of the standard classes is the template code all sorts of dunder methods , I personally like the attrs library, it is much more pythonic.
It is worth mentioning that in Python, all objects, including functions and classes, the classes can be created dynamically (without using eval ) using the type function.
Also it is worth reading about metaclasses ( on habr ) and descriptors ( habr ).
A feature to remember is that the attributes of a class and an object are not the same; in the case of immutable attributes, this does not cause problems since the attributes are “shadowed” (shadowing) —there are automatically created attributes of an object with the same name, but in the case of variable attributes you can get not exactly what was expected:

 cat class_attr.py class MyClass: storage = [7,] def __init__(self, number): self.number = number obj = MyClass(1) obj2 = MyClass(2) obj.number = 5 obj.storage.append(8) print(obj2.storage, obj2.number) 

we get:

 python3 class_attr.py [7, 8] 2 

as you can see, obj changed, and storage changed in obj2 as well. this attribute (as opposed to number ) does not belong to the instance, but to the class.

.sort () vs sorted ()

In python, there are two ways to sort a list. The first is the .sort() method which changes the source list and returns nothing (None) i.e. I can't do it like this:

 my_list = my_list.sort() 

The second is the sorted() function which spawns a new list and is able to work with all objects being iterated. Who wants more information should start with SO .

Standard library

Typically, the standard python library includes excellent solutions to typical problems, but it is worthwhile to approach critically, for there are plenty of oddities. True, it also happens that what seems strange at first glance turns out to be the best solution, you just need to know all the conditions (see further about range), but there are still oddities.

For example, the bundled unit test module unittest has nothing to do with python and strongly smells of java, therefore, as the python author says : "Eveybody is using py.test ...". Although it is quite interesting, though not always the appropriate doctest module comes as standard.

The same story with the module for parsing command line parameters - the bundled argparse is a demonstration of the OOP of the brain, and the docopt module seems just a smart solution - the ultimate self-documenting! Although, according to rumors, despite the docopt and for click there is a niche.

With the debugger as well - as I understood the bundled pdb very few people use, there are many alternatives, but it seems the majority of developers use ipdb , which, from my point of view, is most convenient to use via the debug wrapper module.
Instead of import ipdb;ipdb.set_trace() it allows you to simply write an import debug , it also adds a see module for convenient inspection of objects.

To replace the standard serialization module pickle do dill , then by the way it is worth remembering that these modules are not suitable for data exchange in external systems because to restore arbitrary objects obtained from an uncontrolled source is unsafe, for such cases there is json (for REST) ​​and gRPC (for RPC).

To replace the standard module, the processing of regular expressions re is done by the regex module with all sorts of additional buns, like the \p{Cyrillic} character classes.
By the way, something was missed for a python fun debugger for regexes similar to pearl barley .

Here is another example - a person made his own in-place module to fix the curvature and incompleteness of the API of the standard fileinput module in the in-place part of file editing.

Well, I think there are many such cases, since even I didn’t get one, so be vigilant and don’t forget to look at all kinds of awesome useful lists, I think that a good pythonic has a chuik for a measure of pythonism, this is by the way a topic for a separate conversation - according to my feelings (of course, there are no statistics on this topic and apparently there cannot be) in the python-world the level of specialists is above average, because often good software will be written in python, write in the comments what you think about this.

Parallelism and Competitiveness

Python provides ample opportunities for both parallel and competitive programming, but not without its features.

If you need parallelism, and this happens when your tasks require calculations, then you should pay attention to the multiprocessing module.

And if your tasks have a lot of IO waiting, then the python provides a lot of options to choose from, from threads and gevent , to asyncio .
All of these options look quite suitable for use (although threads require much more resources), but there is a feeling that asyncio is squeezing the rest slowly, including thanks to all kinds of buns like uvloop .

If someone didn’t notice - in the thread Python it’s not about parallelism, I’m not competent enough to tell well about GIL , but there are enough materials on this topic, so there’s no such need, the main thing to remember is that the threads in python (more precisely in CPython) behave differently from other programming languages ​​- they are executed only on one core, which means they are not suitable for cases when you need real parallelism, however, execution of threads is suspended while waiting for input / output, so you can use them for competitiveness.

Other oddities

In python, a = a + b not always equivalent to a += b :

 a = [1] a = a + (2,3) TypeError: can only concatenate list (not "tuple") to list a += (2,3) a [1, 2, 3] 

For details, I send it to SO, for I have not yet found the time to figure out why it is so, in the sense of the reasons why they did it, like this again about mutability.

Oddities that are not weird

At first glance it seemed to me strange that the range type does not include the right border, but then a kind person suggested that I should not learn where I need to learn and it turned out that everything is quite logical.

A separate large topic is rounding (although this is a problem common to almost all programming languages), besides the fact that rounding is used in any manner other than what everyone has learned in the school course of mathematics, this also imposes problems of floating point numbers, referring to detailed article .
Roughly speaking, instead of the usual, in the school course of mathematics, the round-up algorithm for half-up uses the algorithm half to even , which reduces the likelihood of distortion in statistical analysis and is therefore recommended by the IEEE 754 standard.

Also, I could not understand why -22//10=-3 , and then another kind person indicated that this inevitably follows from the mathematical definition itself, according to which the remainder cannot be negative, which leads to such unusual behavior for negative numbers.
ACHTUNG! Now this is again a strange thing and I do not understand anything, see this thread .