Thunderargs: practice of use. Part 2

History of creation
Part 1

Good day. In brief, let me remind you that thunderargs is a library that allows you to use annotations for processing incoming arguments.

In addition, it makes it possible to simply throw a gate, which will pull these same arguments for the function from somewhere else. For example, from the request object in a flake. And in the end, instead of
')

@app.route('/ugly_calc', dont_wrap=True) def ugly_calc(): x, y = int(request.args['x']), int(request.args['y']) op_key = request.args.get('op') if not op_key: op_key = '+' op = OPERATION.get(op_key) return str(op(x, y)) 

doing

 @app.route('/calc') def calc(x:Arg(int), y:Arg(int), op:Arg(str, default='+')): return str(OPERATION[op](x, y)) 

I think everyone at least roughly understood what it would be about in the article. All that it describes is reflections on the future of the project and the approximate arrangement of the Milestone on it. Well, of course, the first sketches of all sorts of different features.

In this part

• Consider the structural changes in the project and a couple of critical errors in the initial structure.
• We will understand how validators work and how you can customize the errors they generate.
• Create a rudiment of specialized arguments (IntArg, StrArg, ListArg, and so on)
• Prepare a class that will automatically pull an object out of the database by id received in the request
• We will generate entry points for the model class.
• Implement the listeners and see how you can make a validator for several arguments.
• Make sure that the information about the structure of the arguments can be safely transferred to the database, and nothing for this we will not
• And, finally, we speculate about the small interesting, and not implemented in the framework of these experiments

Structural changes, or why I need to kick

Well, now briefly about the important events in the fate of the project. Firstly, I finally read how Armin Ronasher recommends making modules to a flag , and brought his “pet” to the right kind. To do this, I completely separated the main library functionality (this library and turnip remained under the name thunderargs) from the functionality that allows using it as a supplement to Flask (now this crap can be put under the name flask-thunderargs, as you can guess). Yes, in fact, this is just a separation of the interface from the core, which is viable without this interface. And this should have been done from the very beginning. For my hindsight, I paid almost five hours spent on reorganization.
In general, I will briefly describe what has changed and what it means:

Now we have two - the core and the interface to the fusion

The main library, as I said, can be used without any external interfaces. And, of course, it can be used to create your own interfaces. For example, to other web frameworks. Or to argparse. Or to gabber bot. Yes, in general, to anything.
In fact, from this point the project is only in the black.

flask-thunderargs is now a full-featured flask module

The only trouble is that the interface itself is just tiny. In fact, it is all enclosed in this file. If someone decides to write their own interface to another lib, you can safely rely on it.
And the endpoint initialization process has changed, of course. Now the minimal application looks like this:

 from flask import Flask from flask.ext.thunderargs import ThunderargsProxy from thunderargs import Arg app = Flask(__name__) ThunderargsProxy(app) @app.route('/max') def find_max(x: Arg(int, multiple=True)): return str(max(x)) if __name__ == '__main__': app.run() 

So it goes.

Making mistakes

In the last part, we already figured out how to create our own validators. And made sure that it is quite simple. I recall:

 def less_than_21(x): return x < 21 @app.route('/step5_alt') def step5_1(offset: Arg(int, default=0, validators=[lambda x: x >= 0 and x < len(elements)]), limit: Arg(int, default=20, validators=[less_than_21])): return str(elements[offset:offset+limit]) 

As we see, there are two options for creating them. One - inline, with the help of lambda. The second is fully weighted. Now I want to show why the full-weight version is preferable.

The person who tested the experiments of the last part could have noticed that the validators created by the factory throw pretty beautiful and understandable errors:

 thunderargs.errors.ValidationError: Value of `limit` must be less than 21 

But our example gives us incomprehensible and speaking errors:

 thunderargs.errors.ValidationError: Argument limit failed at validator #0.Given value: 23 

To handle this is quite simple. Moreover, our mistake will be even better than the original one:

 experiments.custom_error.LimitError: limit must be less than 21 and more than 0. Given: 23 

For this result, we need the following code:

 class LimitError(ValidationError): pass 

 from thunderargs.errors import customize_error from experiments.custom_error import LimitError message = "{arg_name} must be less than 21 and more than 0. Given: {value}" @customize_error(message=message, error_class=LimitError) def limit_validator(x): return x < 21 and x>0 @app.route('/step5_alt2') def step5_2(offset: Arg(int, default=0, validators=[lambda x: x >= 0 and x < len(elements)]), limit: Arg(int, default=20, validators=[limit_validator])): return str(elements[offset:offset+limit]) 

In general, to customize an error, you simply need to attach a customize_error decorator to the validator function. The following variables are always passed to the error text:

• error_code - error number to display; intrasystem crap for lovers of systematization;
• arg_name is the name of the argument that corresponds to the name assigned to the argument in the function declaration; In our case, for example, limit;
• value - the value obtained by the validator; in the case of flask-thunderargs, this is most often string, since all but reques.json and reques.files give it exactly;
• validator_no — validator sequence number; I strongly doubt that it will be useful in correctly composed validators;

In addition, you can pass to customize_error any named parameters that the error class will eat under the appropriate names. This is convenient, for example, if we need to transfer some data specified in the config as a notification to the end user. This also applies if you are writing an error generator. As an example, consider the classic decorator factory from validfarm:

 def val_in(x): @customize_error("Value of `{arg_name}` must be in {possible_values}", possible_values=x) def validator(value): return value in x return validator 

The possible_values ​​in this example are taken from x, a variable that will be passed to the factory by the programmer, and will be obtained during the application launch.
Estimated version: 0.4

Inherited Variable Classes

Obviously, reducing the level of abstraction is useful for the end user of the library. And the first step in this direction will be specialized classes. Here is an example:

 class IntArg(Arg): def __init__(self, max_val=None, min_val=None, **kwargs): kwargs['p_type'] = int if not 'validators' in kwargs or kwargs['validators'] is None: kwargs['validators'] = [] if min_val is not None: if not isinstance(min_val, int): raise TypeError("Minimal value must be int") kwargs['validators'].append(val_gt(min_val-1)) if max_val is not None: if not isinstance(max_val, int): raise TypeError("Maximal value must be int") kwargs['validators'].append(val_lt(max_val+1)) if min_val is not None and max_val is not None: if max_val < min_val: raise ValueError("max_val is greater than min_val") super().__init__(**kwargs) 

But the application of this class:

 from experiments.inherited_args import IntArg @app.route('/step7') def step7(x: IntArg(default=0, max_val=100, min_val=0)): return str(x) 

The main feature of such classes is that there is no need to manually describe some parameters of the input argument. In addition, there is no need to manually describe some validators. And it becomes possible to specify their meaning in the code, which is very important for readability.
Estimated version: 0.4

Inherited classes for ORM

Suppose that we have a class of documents, made through mongoengine:

 class Note(Document): title = StringField(max_length=40) text = StringField(min_length=3, required=True) created = DateTimeField(default=datetime.now) 

We must have a getter who must return a specific document. Let's make an independent class for this task:

 class ItemArg(Arg): def __init__(self, collection, **kwargs): kwargs['p_type'] = kwargs.get('p_type') or ObjectId kwargs['expander'] = lambda x: collection.objects.get(pk=x) super().__init__(**kwargs) 

All he does is change the input arguments. Simply expands them to the required set. And even such a minimalistic version allows us to do this:

 @app.route('/step9/get') def step9_2(note: ItemArg(Note)): return str(note.text) 

Pretty nice, right?

Presumptive version: it makes sense to bring in an independent library

We generate flock getters

Imagine that we have a class in the model whose getters do not perform any special actions. It is necessary to write a getter, which will give the user information in the same form in which it is stored in the database. In this case, we will not interfere with the generator of getters. Let's do it:

 def make_default_serializable_getlist(cls, name="default_getter_name"): @Endpoint def get(offset: IntArg(min_val=0, default=0), limit: IntArg(min_val=1, max_val=50, default=20)): return list(map(lambda x: x.get_serializable_dict(), cls.objects.skip(offset).limit(limit))) get.__name__ = name return get 

This function should create a getter for the MongoEngine collection. The only additional condition is that the collection class must have a get_serializable_dict method. But I think no one will have any particular problems with this. And here is one of the options for applying this thing:

 getter = make_default_serializable_getlist(Note, name='step11_getter') app.route('/step11_alt3')(json_resp(getter)) 

Here, the auxiliary function json_resp is json_resp , but in fact it does not do anything interesting, it simply wraps the controller's response into flask.jsonify (if it can). In addition, in this example, I used the decorator without applying the classical syntax. In my opinion, this is justified, otherwise I would have to do a transport wrapper that does not perform any useful activity.

Estimated version: similar to the previous one

Call logging and something else

Let's log each user movement that fits into the rules we describe. To do this, we propose a simple decorator, which will take a callback function:

 def listen_with(listener): def decorator(victim): @wraps(victim) def wrapper(**kwargs): listener(func=victim, **kwargs) return victim(**kwargs) return wrapper return decorator 

and callback itself:

 def logger(func, **kwargs): print(func.__name__) print(kwargs) 

This callback simply displays all the received arguments on the screen. Now consider a more useful example:

 def denied_for_john_doe(func, firstname, lastname): if firstname == 'John' and lastname == 'Doe': raise ValueError("Sorry, John, but you are banned") @app.route('/step13') @listen_with(denied_for_john_doe) def step13(firstname: Arg(str, required=True), lastname: Arg(str, required=True)): return "greeting you, {} {}".format(firstname, lastname) 

Here, as we see, there is a test of the possibility of using a combination of values. Generally, formally, such a construction is not a licenser, and should be separated from them, listeners. But for now, as part of the experiment, let's leave it like this. Here is a more architecturally correct example:

 def mail_sender(func, email): if func.__name__ == 'step14': #   ,    #  ,      :( pass @app.route('/step14') @listen_with(mail_sender) def step14(email: Arg(str, required=True)): """   ,    ,     :( """ return "ok" 

Okay, not an example, but its preparation.

Estimated version: 0.5

The structure of the arguments in the database

And now we proceed to the dessert. Today, we have a “delicious” storage of the structure of incoming arguments in the database.
The fact is that such an architecture reduces the code responsible for receiving and processing data, in fact, to the data. And we can take this data from anywhere. From the config file, for example. Or from a DB. And really, if you think about the difference between these two data sources? Let's get started

First we need to create a table of correspondences of the objects of the program currently being executed with data imported from the database. In the example we will use only one type already described by us above. Therefore, so far only he will be here:

 TYPES = {'IntArg': IntArg} 

Now we need to describe the model, which, in fact, will store and display information about the incoming arguments of the entry points.

 class DBArg(Document): name = StringField(max_length=30, min_length=1, required=True) arg_type = StringField(default="IntArg") params = DictField() def get_arg(self): arg = TYPES[self.arg_type](**self.params) arg.db_entity = self return arg 

Here, as we can see, the name of the argument, its type and additional parameters are specified, which will be passed to the constructor of this type. In our case, this is IntArg, and we can have max_val, min_val, required, default, and all the others that are correctly processed by the ORM.
The get_arg function get_arg intended to get an Arg instance with a configuration stored in the database. Now we need the same balalaika for the structures that we usually add to the functions, describing the individual arguments through annotations. Yes, yes, all of this merges into a specific construction, which is then fed to the argument parser.

 class DBStruct(Document): args = ListField(ReferenceField(DBArg)) def get_structure(self): return {x.name: x.get_arg() for x in self.args} 

It is much simpler, and it is hardly worth describing separately. Perhaps it is worthwhile to clarify for people who are not “communicating” with mongoengine that the ListField(ReferenceField(DBArg)) construction means only that in the database in this field we will store a list of the elements of the DBArg class.

And we also need a piece that will link the above into something integral and concrete. Let's just say, apply it all to living tasks. And there is such a task. Let's assume that you and I have a store or auction. Sometimes it happens that for those. task in the admin panel, among other things, should be able to create categories of goods, each of which will have its own parameters, unique to it. We will attach ourselves to this task.

 class Category(Document): name = StringField(primary_key=True) label = StringField() parent = ReferenceField('self') arg_structure = ReferenceField(DBStruct) def get_creator(self): @Endpoint @annotate(**self.arg_structure.get_structure()) def creator(**kwargs): return Item(data=kwargs).save() creator.__name__ = "create_" + self.name return creator def get_getter(self): pass 

Here we have described the category model. It will have a system name, which is necessary for naming functions and endpoints, a display name, which means nothing at all to us, and a parent (yeah, we will prepare for inheritance in advance). In addition, the data structure used for this category is indicated. Finally, a function is described that will automatically create a creator function for this category. It would be nice to screw the cache and other goodies, but for now, as part of the experiment, we ignore it.

And finally, we need a model for storing user data, through which end users will fill in the information about products. Here, as in all previous examples, this will be presented in a simplified form:

 class Item(Document): data = DictField() category = ReferenceField(Category) 

I think there is no need for special explanations at all.

Well, now let's create the first category of products:

 >>> weight = DBArg(name="weight", params={'max_val': 500, 'min_val':0, 'required': True}).save() >>> height = DBArg(name="height", params={'max_val': 290}).save() >>> human_argstructure = DBStruct(args=[weight, height]).save() >>> human = Category(name="human", arg_structure=human_argstructure).save() 

Yes, I know that selling people is not very ethical, but it just so happened :)

Now we need a wrapper with which we will create the names of the goods:

 @app.route('/step15_abstract') def abstract_add_item(category: ItemArg(Category, required=True, p_type=str)): creator = category.get_creator() wrapped_creator = app._arg_taker(creator) return str(wrapped_creator().id) 

Now it looks very ugly. This is due to another error in the architecture. However, much less significant than the previous one. Anyway. Now I will explain what is happening here.

First we get the category instance in a way that has already been described above (see the example with the Note model). Accordingly, if the user tries to add a product to a non-existent category, he will receive DoesNotExist. The primary key in this category is its system name, and it is the user who must pass it as an identifier. In our case, this is human . Accordingly, the entire request should look like this:
localhost:5000/step15_abstract?category=human&weight=100&height=200
The remainder is intended for the constructor to be called to get other parameters. app._arg_taker is a decorator that allows endpoint to “get” the missing arguments from source. In our case, this is request.args, but, in principle, the source can be anything. Actually, in this fragment, my architectural error lies. In an amicable way, the need to wrap nested endpoints in such a decorator should not arise.

Estimated version: never, it's just an experience

Conclusion and future

Well, perhaps this will finish today. Now you can speculate on lengthy topics. First of all, I would like to thank all those who responded to the first posts. Even despite the fact that no one has made a single constructive proposal, you helped me a lot in moral terms :)

And now briefly about intentions and desires.
The main focus of the coming months will be commenting the code, refactoring and test coverage. Yes, I myself know that in this area my code is simply disgusting, it would be foolish to deny it.
In addition, I would like to write another couple of gates, like flaskovy, to other frameworks. In general, I would like to find such places where my library would be useful. For now, only accept tornado and argparse.

As for the library itself, here I think it is important to focus on reverse reporting. Let's say we use thunderargs to write the restful interface. It would be cool if he could give information to the final library, which would allow to create some kind of json-rpc, so that the client can find out at OPTIONS request which parameters which method he accepts and which errors in their endpoints can occur.

Later I will write another final article. She will already be tightly tied to "real life". I believe that there will be a description of the coding process of some service. Now I have only one idea, and it is connected with the tag system on one interesting site (with a sad panda). But I will be glad to hear other suggestions. Microblogging, Q & A forums, whatever. I do not care about platitudes or anything like that. It is important that the example of this code could show as many aspects of my "pet" as possible. Among other things, it will allow to check it in, and, perhaps, to find a couple of bugs or architectural flaws.

Thanks for attention. As always, I am glad to any criticism and any wishes.

main turnip
flag-gate (the code of all experiments from the article is here)