Selection @pythonetc, April 2019

This is the tenth collection of tips on Python and programming from my author’s @pythonetc channel.

Previous selections .

')

---

Storing and sending objects over the network in the form of bytes is a very big topic. For these purposes, Python usually uses a number of tools, let's discuss their advantages and disadvantages.

As an example, I will try to serialize a Cities object that contains City objects in a specific order. Four approaches can be used:

1. JSON. Human-readable, easy to use, but consumes a lot of memory. The same is true of the YAML and XML formats.

class City: def to_dict(self): return dict( name=self._name, country=self._country, lon=self._lon, lat=self._lat, ) class Cities: def __init__(self, cities): self._cities = cities def to_json(self): return json.dumps([ c.to_dict() for c in self._cities ]).encode('utf8') 

2. Pickle. This native Python tool, customizable, consumes less memory than JSON. Disadvantage: you need to use Python to extract data.

 class Cities: def pickle(self): return pickle.dumps(self) 

3. Protobuf (and other binary serializers, for example, msgpack). It consumes even less memory, can be used from any programming language, but requires writing an explicit scheme:

 syntax = "proto2"; message City { required string name = 1; required string country = 2; required float lon = 3; required float lat = 4; } message Cities { repeated City cities = 1; } class City: def to_protobuf(self): result = city_pb2.City() result.name = self._name result.country = self._country result.lon = self._lon result.lat = self._lat return result class Cities: def to_protobuf(self): result = city_pb2.Cities() result.cities.extend([ c.to_protobuf() for c in self._cities ]) return result 

4. Manually. You can manually package and unpack data using a struct module. This way you can achieve the lowest possible memory consumption, but sometimes it is better to use protobuf , because it supports versioning and explicit schemes.

 class City: def to_bytes(self): name_encoded = self._name.encode('utf8') name_length = len(name_encoded) country_encoded = self._country.encode('utf8') country_length = len(country_encoded) return struct.pack( 'BsBsff', name_length, name_encoded, country_length, country_encoded, self._lon, self._lat, ) class Cities: def to_bytes(self): return b''.join( c.to_bytes() for c in self._cities ) 

---

If the function argument has a default value of None and is annotated as T , then mypy will automatically consider it Optional[T] (that is, Union[T, None] ).

With other types, this doesn't work, so you can't write something like f(x: A = B()) . Also this trick does not work with the assignment of a variable: a: A = None will result in an error.

 def f(x: int = None): reveal_type(x) def g(y: int = 'x'): reveal_type(y) z: int = None reveal_type(z) $ mypy test.py test.py:2: error: Revealed type is 'Union[builtins.int, None]' test.py:4: error: Incompatible default for argument "y" (default has type "str", argument has type "int") test.py:5: error: Revealed type is 'builtins.int' test.py:7: error: Incompatible types in assignment (expression has type "None", variable has type "int") test.py:8: error: Revealed type is 'builtins.int' 

***

In Python 3, when exiting from the except block, variables that store caught exceptions are removed from locals() , even if they already existed:

 >>> e = 2 >>> try: ... 1/0 ... except Exception as e: ... pass ... >>> e Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'e' is not defined 

If you want to keep the reference to the exception, you need to use another variable:

 >>> error = None >>> try: ... 1/0 ... except Exception as e: ... error = e ... >>> error ZeroDivisionError('division by zero',) 

In Python 2, however, this does not happen.

---

You can easily make your own pypi repository. It allows you to release packages inside your project and install them with pip , as if they were regular packages.

It is important to note that you do not need to install any special software, you can use a regular HTTP server. This is how it works for me.

Take the primitive pythonetc package.

 setup.py: from setuptools import setup, find_packages setup( name='pythonetc', version='1.0', packages=find_packages(), ) pythonetc.py: def ping(): return 'pong' 

Let's release it to the ~/pypi directory:

 $ python setup.py sdist bdist_wheel … $ mv dist ~/pypi/pythonetc 

And we will start to provide the package from the pypi.pushtaev.ru domain using nginx:

 $ cat /etc/nginx/sites-enabled/pypi server { listen 80; server_name pypi.pushtaev.ru; root /home/vadim/pypi; index index.html index.htm index.nginx-debian.html; location / { autoindex on; try_files $uri $uri/ =404; } } 

Now the package can be installed:

 $ pip install -i http://pypi.pushtaev.ru --trusted-host pypi.pushtaev.ru pythonetc … Collecting pythonetc Downloading http://pypi.pushtaev.ru/pythonetc/pythonetc-1.0-py3-none-any.whl Installing collected packages: pythonetc Successfully installed pythonetc-1.0 $ python Python 3.7.0+ (heads/3.7:0964aac, Mar 29 2019, 00:40:55) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pythonetc >>> pythonetc.ping() 'pong' 

---

Often you need to declare a dictionary with keys of the same name as local variables. For example:

 dict( context=context, mode=mode, action_type=action_type, ) 

In ECMAScript for such cases there is even a special form of the object literal (called the Object Literal Property Value Shorthand):

 > var a = 1; < undefined > var b = 2; < undefined > {a, b} < {a: 1, b: 2} 

You can create the same helper in Python (alas, it is not at all as good as ECMAScript notation):

 def shorthand_dict(lcls, names): return {k: lcls[k] for k in names} context = dict(user_id=42, user_ip='1.2.3.4') mode = 'force' action_type = 7 shorthand_dict(locals(), [ 'context', 'mode', 'action_type', ]) 

You may ask, why pass locals() as a parameter? Can one get the caller's locals in the callee? You can, but you have to use the inspect module:

 import inspect def shorthand_dict(names): lcls = inspect.currentframe().f_back.f_locals return {k: lcls[k] for k in names} context = dict(user_id=42, user_ip='1.2.3.4') mode = 'force' action_type = 7 shorthand_dict([ 'context', 'mode', 'action_type', ]) 

You can go even further and apply this solution - https://github.com/alexmojaki/sorcery :

 from sorcery import dict_of dict_of(context, mode, action_type)