How to add dynamism in Python 2.7?

Have you ever wanted to add a field to the dict class? Do you dream to write action.name.len() instead of len(action.name) ? Do you want to add flexibility to your favorite Python? Are you told that this is impossible? Then let's get into some of the details of the Python object model!

In Python 2.7, all built-in classes and classes written in C are immutable. That is, you cannot remove / add / replace a method or field in any built-in type. But at the same time, classes created in pure Python can be easily changed at runtime.
Note: in this article we will talk about new-style classes. How new-style differs from old-style can be found in the official documentation: www.python.org/doc/newstyle

Example:

 class foo(object): def getA(self): return "A" x = foo() print x.getA() # “A” def getB(obj): return "B" foo.getA = getB #  print x.getA() # “B” 

But doing this trick with the list or dict class is no longer possible.
')
>>> list.length = len
TypeError: can't set attributes of built-in/extension type 'list'


This behavior is not accidental. Although at first glance, list and foo are instances of the same type metaclass. But the Python interpreter distinguishes between these two types, and provides different behavior when trying to change the list of class members.

Why not?

There is an official version: here (last paragraph), the opinion of Guido van Rossum here or here . In a nutshell, problems will arise with multiple Python interpreters in the same address space.
One of the obstacles is also the problem of replacing any built-in method. If you, for example, replace the string.__len__ with your own implementation, then this change will in no way be reflected on the Python modules written in C. From an API point of view, the PyString_Size (...) function will remain unchanged. Such dissonance can lead to subtle bugs and undefined behavior.

What to do?

If you can not, but really want to, then ... Take the source code of Python 2.7 (http://hg.python.org/cpython/ just switch to the "2.7" branch). Finding the code that throws an exception is very simple; just search for the text “can't set attributes of built-in / extension type” . The required lines are in the typeobject.c file in the "type_setattro" function. This function is called when the Python script tries to add or change a property of a class. The function is readable as type.__setattr__ . To remove the restriction that prevents us, we need to replace this method with our own more loyal implementation.
It cannot be done from the Python script. Any attempt to override type.__setattr__ results in an already familiar exception:

TypeError: can't set attributes of built-in/extension type 'type'

But if you write a C-th module and access the type object, then instead of a pointer to the function "type_setattro" you can substitute a pointer to your own version of the __setattr__ method.

Let's get started!

I hope that you already know how to write Python modules in C. Standard documentation describes very well how this is done (http://docs.python.org/extending/extending.html). Our module will not have any functions, classes or fields. All magic will occur at the moment of import of the module by the interpreter.

 #include <Python.h> static setattrofunc original_setattr_func = NULL; PyMODINIT_FUNC inittypehack(void) { PyObject *m; m = Py_InitModule("typehack", NULL); if (m == NULL) return; apply_patch(); } void apply_patch() { original_setattr_func = PyType_Type.tp_setattro; //   __setattr__ PyType_Type.tp_setattro = new_setattr_func; //  __setattr__  } 

PyType_Type is a structure that stores all information about the metaclass type : name, size of the object in memory, flags. In particular, it stores pointers to functions that implement certain metaclass methods.

That's all. It remains to come up with the implementation of new_setattr_func . I will not give all the code here. Just describe the logic of the work.

1. You can not change existing fields and methods. You can only add your own.
2. When a new attribute is added to the class, the __dyn_attrs__ field is __dyn_attrs__ , in which the strings with the names of all the added attributes are stored. In the future, it will be possible to replace only the attributes from this list. This is such a foolproof protection, which does not give a 100% guarantee, but helps to keep the original attributes intact.
3. When trying to replace a class attribute, a check is made that the name of the attribute being modified is in the __dyn_attrs__ list. Otherwise, an exception is thrown.
4. After changing the list of class attributes, it is necessary to reset the cache by calling the PyType_Modified(type) function.

The source code of the project in Google Code is available by reference .
(I don’t attach the build scripts as such, since everything was done on the knee. I hope you know how to compile a * .c file in your OS)

Profit?

Now you can do these miracles:

>>> import typehack #god mode on
>>> def custom_len(text):
... return len(txt)
...
>>> list.size = custom_len # "size"
>>> ['Tinker', 'Tailor', 'Solder', 'Spy'].size()
4
>>> str.len = property(custom_len) # "len"
>>> "Hello".len
5

Conclusion

And the conclusion is that Python is so dynamic programming language that its behavior can be changed on the fly. The Python object model allows you to do all this without creating your own version of the interpreter, but using a small plug-in. The principle of open kimono plays into our hands.

Good luck in mastering the magic of Python

PS I did not implement the removal of class attributes, and did not conduct a full test to identify all possible problems. This is just a small hack, Proof Of Concept. I suspect that it is not much harder to add attribute deletion. Also, porting to Python 3 should not cause serious complications: their object model is similar.