When MVC is not enough
One of the main advantages of frameworks is their predefined architecture. You open an unfamiliar project and immediately know where and how to search for a connection code with a database, or HTML, or a url scheme. In addition, it allows the developer not to think about the storage structure of the code and at the same time be sure that the project will look more or less adequate. But I want to talk about the case when the implementation of MVC in Django, namely, the distribution of logic in the files models, forms, views, templates was inconvenient and what alternative was built based on it.
Our task was to make an engine for statistical reporting on Django. We created selectors for getting data from Oracle and widgets for displaying this data in the form of tables or graphs (using HighChart). But these are all purely technological solutions, without much magic. If you are interested, I will tell in a separate post. And now I would like to draw attention to a more unusual part of the project. To provide report compilers with a convenient way to compile these reports.
Here are a few points:
There was an option to store the report settings in the database. But tracking changes is much easier in a version control system than in a database. In addition, it was clear in advance that the engine will develop, and changing the data scheme is perhaps the most painful for any system.
Means files. Which engine will read and do something based on them. The format was assumed to be different. And JSON, and ini, and invent some kind of your own. XML was flagged immediately as hard to read. But one evening it dawned on me - and how is Python bad? Setting looks nothing more complicated, even for someone unfamiliar with the language at all (except that the first two lines will seem magical to him):
')
Simply put, the configuration file can be a script that runs every time a page is accessed. For Django, this behavior is not typical, but we will make her do it.
I must say that I later thanked myself many times for this decision. Not only because it was easier to add new features, but also because in special cases it became possible to solve the problem with a simple Python hack right in the configuration file. For example, to perform different requests, depending on the conditions, or to generate several graphs of the same type. If the config were standardized as a static file, it is not known how such issues would be solved. But I suspect that it is very difficult. For each such case, I would have to finish the engine.
This is how the “interpreter” of the configuration files looks in the simplest form.
We execute with the help execfile setup file. All variables created in the script will be in the ctx_dict dictionary. We take the contents of the template variable and create a full-fledged template, into which we pass the standard RequestContext and the newly created context from the same script.
In urls.py add
Passing an arbitrary dictionary as a namespace for an executable script opens up interesting possibilities.
For example, we needed to access request get parameters in the configuration file. To do this, simply change ctx_dict before passing it to execfile
Now, in the configuration file, without any imports, the function get will be available, which gets the value of the desired parameter from the current request. Actually, imports would not help here, since request is new each time.
At the same time, it was necessary to post-process the data obtained from the configuration file. For example, it became necessary to assign each html-id to each graphic in accordance with its name. This is necessary in order for javascript to print the same name as in python (for interaction of graphs with each other). Of course, you can solve this with another parameter in Chart, but it's not very kosher to constantly write something in the style
It is better not to strain the users of the engine with its entrails, and assign the necessary id automatically, after the formation of ctx_dict in the execfile.
There is another interesting point with ctx_dict. Since all its values ​​fall into the context of the template, they rewrite those of the same name transmitted from RequestContext. For example, if some context processor calculates the value 'TITLE' to put it in the page header, then you can calculate your own in your configuration file and it will be displayed instead of the existing
It is clear that there is a danger of inadvertent rewriting. But this is decided by a naming convention (for example, in context processors only the upper case is used, and in the configuration files only the lower case).
In the end, it got to the point that it was necessary to post several sections with statistics on the Portal. Of course, they were slightly differently designed and required a little different logic, well, we ourselves were convenient to store the report groups separately.
So dynamic_page should be from a simple viewview generator. What was done.
This allowed us to create wrappers for different reporting sections. They were engaged in programmers. The principles of making reports did not change.
For example, in one case, the games mentioned above with html_id were needed.
In the other, fill out from the setup file not one block of the template, but two.
In this case, the blocks are specified in the file with the report.
In the third, to recognize the division in which the current user works, on its basis to determine how and what requests to perform, as well as in what form to show the submenu.
All these wrappers are added to the urls as regular views.
Here is such a mini framework over a large one. I hope he successfully demonstrates that if the framework of the system makes it difficult to solve the problem in a simple way, you can always slightly push them apart using the more powerful features of the language.
UPD: As suggested by magic4x , the __name__ property has been added to DynamicPage, which enhances the mimicry for the function.
Our task was to make an engine for statistical reporting on Django. We created selectors for getting data from Oracle and widgets for displaying this data in the form of tables or graphs (using HighChart). But these are all purely technological solutions, without much magic. If you are interested, I will tell in a separate post. And now I would like to draw attention to a more unusual part of the project. To provide report compilers with a convenient way to compile these reports.
Here are a few points:
- On the one hand, the compilers of the reports are with us in the same department, that is, in principle they can be shown the insides of the project. On the other hand, they are fluent in SQL, a little bit of HTML and not at all Python, and certainly not Django.
So, it is necessary, if possible, to relieve them of the load on the brain in the form of mastering the architecture of the framework. In addition, you need to put their work in the sandbox, so that no errors would affect the performance of the system as a whole. - Each page should contain several reports in a rather arbitrary form. There are a lot of pages and they are usually not related to each other (well, except that the sources in the database)
If you stuff the logic of a single report across different files, you will get huge files by which you need to search for a report in pieces.
But one ought to have the opportunity to open "something" and see in front of you the whole logic of building a report in and out.
- You need the ability to quickly edit the report without restarting Django.
- It is desirable to provide the ability to collaborate and track changes in reports.
There was an option to store the report settings in the database. But tracking changes is much easier in a version control system than in a database. In addition, it was clear in advance that the engine will develop, and changing the data scheme is perhaps the most painful for any system.
Means files. Which engine will read and do something based on them. The format was assumed to be different. And JSON, and ini, and invent some kind of your own. XML was flagged immediately as hard to read. But one evening it dawned on me - and how is Python bad? Setting looks nothing more complicated, even for someone unfamiliar with the language at all (except that the first two lines will seem magical to him):
# -*- coding: utf-8 -*- from statistics import OracleSelect, Chart, Table select_traf = OracleSelect('user/password@DB', """select DAY, NSS_TRAF, BSS_TRAF from DAY_TRAFFIC where DAY >= trunc(sysdate,'dd')-32""") chart_traf = Chart(selector=select_traf, x_column='DAY', y_columns=[('NSS_TRAF', u'NSS '), ('BSS_TRAF', u'BSS ')]) table_traf = Table(selector=select_traf, columns=['DAY', 'NSS_TRAF', 'BSS_TRAF']) template = """ {{ chart_traf }} {{ table_traf }} """ ')
Simply put, the configuration file can be a script that runs every time a page is accessed. For Django, this behavior is not typical, but we will make her do it.
I must say that I later thanked myself many times for this decision. Not only because it was easier to add new features, but also because in special cases it became possible to solve the problem with a simple Python hack right in the configuration file. For example, to perform different requests, depending on the conditions, or to generate several graphs of the same type. If the config were standardized as a static file, it is not known how such issues would be solved. But I suspect that it is very difficult. For each such case, I would have to finish the engine.
Reading (execution) of the setup file
This is how the “interpreter” of the configuration files looks in the simplest form.
import os from django.template import RequestContext, Template from django.http import HttpResponse, Http404 from settings import PROJECT_ROOT # , __file__ settings.py def dynamic_page(request, report_path): ctx_dict = {} execfile(os.path.join(PROJECT_ROOT, 'reports', report_path + '.py'), ctx_dict) templ_header = '{% extends "base.html" %}{% block content %}' templ_footer = '{% endblock %}' template = Template(templ_header + ctx_dict['template'] + templ_footer) context = RequestContext(request) context.autoescape = False context.update(ctx_dict) return HttpResponse(template.render(context)) We execute with the help execfile setup file. All variables created in the script will be in the ctx_dict dictionary. We take the contents of the template variable and create a full-fledged template, into which we pass the standard RequestContext and the newly created context from the same script.
In urls.py add
(r'^reports/(?P<report_path>.+)$', 'statistics.views.dynamic_page'), Passing context to and from report
Passing an arbitrary dictionary as a namespace for an executable script opens up interesting possibilities.
For example, we needed to access request get parameters in the configuration file. To do this, simply change ctx_dict before passing it to execfile
def dynamic_page(request, report_path): ctx_dict = {'get': request.GET.get} ... Now, in the configuration file, without any imports, the function get will be available, which gets the value of the desired parameter from the current request. Actually, imports would not help here, since request is new each time.
At the same time, it was necessary to post-process the data obtained from the configuration file. For example, it became necessary to assign each html-id to each graphic in accordance with its name. This is necessary in order for javascript to print the same name as in python (for interaction of graphs with each other). Of course, you can solve this with another parameter in Chart, but it's not very kosher to constantly write something in the style
chart_name = Chart(select, x_col, config, ..., html_id='chart_name') It is better not to strain the users of the engine with its entrails, and assign the necessary id automatically, after the formation of ctx_dict in the execfile.
... execfile(os.path.join(PROJECT_ROOT, 'reports', report_path + '.py'), ctx_dict) for (name, obj) in ctx_dict.items(): if isinstance(obj, (Chart, Table)): obj.html_id = name ... There is another interesting point with ctx_dict. Since all its values ​​fall into the context of the template, they rewrite those of the same name transmitted from RequestContext. For example, if some context processor calculates the value 'TITLE' to put it in the page header, then you can calculate your own in your configuration file and it will be displayed instead of the existing
bs = get('bs') if bs is not None: TITLE = u' %s' % bs It is clear that there is a danger of inadvertent rewriting. But this is decided by a naming convention (for example, in context processors only the upper case is used, and in the configuration files only the lower case).
Scaling to other url and base patterns
In the end, it got to the point that it was necessary to post several sections with statistics on the Portal. Of course, they were slightly differently designed and required a little different logic, well, we ourselves were convenient to store the report groups separately.
So dynamic_page should be from a simple viewview generator. What was done.
import os from django.template import RequestContext, Template from django.http import HttpResponse, Http404 from settings import PROJECT_ROOT from functools import partial def get_param(request, key=None, default=None, as_list=False): if key: if as_list: return request.GET.getlist(key) else: return request.GET.get(key, default) else: return request.GET.lists() class DynamicPage(object): " " # view def __init__(self, subpath, # , , parent_template = "base.html", load_tags = (), # block_name = 'content', pre_calc = lambda request, context: None, # execfile post_calc = lambda request, context: None): # execfile self.templ_header = ('{% extends "' + parent_template + '" %}' + DynamicPage.loading_tags(load_tags) + DynamicPage.block_top(block_name)) self.templ_footer = DynamicPage.block_foot(block_name) self.subpath = subpath self.pre_calc = pre_calc self.post_calc = post_calc @staticmethod def block_top(block_name): if block_name: return "{% block " + block_name + " %}" else: return '' @staticmethod def block_foot(block_name): if block_name: return "{% endblock %}" else: return '' @staticmethod def loading_tags(tags): return ''.join(['{% load ' + tag + ' %}' for tag in tags]) @property def __name__(self): return self.__class__.__name__ # view def __call__(self, request, pagepath): ctx_dict = self.get_context(request, pagepath) if 'response' in ctx_dict and isinstance(ctx_dict['response'], HttpResponse): return ctx_dict['response'] # response # , , html- else: template = Template(self.templ_header + ctx_dict['template'] + self.templ_footer) context = RequestContext(request) context.autoescape = False context.update(ctx_dict) return HttpResponse(template.render(context)) def get_context(self, request, pagepath): fullpath = os.path.join(PROJECT_ROOT, self.subpath, pagepath + '.py') if not os.path.exists(fullpath): raise Http404 ctx_dict = {'get': partial(get_param, request), 'request': request} self.pre_calc(request, ctx_dict) execfile(fullpath, ctx_dict) self.post_calc(request, ctx_dict) return ctx_dict This allowed us to create wrappers for different reporting sections. They were engaged in programmers. The principles of making reports did not change.
For example, in one case, the games mentioned above with html_id were needed.
def add_html_id(request, context): for (name, obj) in context.items(): if isinstance(obj, (Chart, Table)): obj.html_id = name show_report = DynamicPage('stat_tech/pages', parent_template='stat_tech/base.html', load_tags=['adminmedia', 'jquery', 'chapters'], post_calc=add_html_id) In the other, fill out from the setup file not one block of the template, but two.
show_weekly = DynamicPage('stat_weekly/pages', parent_template = 'stat_weekly/base.html', load_tags = ['chapters', ' employees'], block_name=None) In this case, the blocks are specified in the file with the report.
template = """ {% block chart %} {{ costs_monthly }} {{ costs_weekly }} {% endblock %} {% block responsible %} {% employee vasily_pupkin %}, {% employee ivan_ivanov %} {% endblock %} """ In the third, to recognize the division in which the current user works, on its basis to determine how and what requests to perform, as well as in what form to show the submenu.
def add_division(request, context): div = Division.get_by_user(request.user) context['DIVISION'] = div context['SUBMENU'] = calc_goal_submenu(request.path, div) show_goal = DynamicPage('stat_goals/pages', load_tags = ['chapters'], block_name='report', parent_template = 'stat_goals/base.html', pre_calc = add_division) All these wrappers are added to the urls as regular views.
(r'^stat/(?P<pagepath>.+)$', 'stat_tech.views.show_report'), (r'^weeklyreport/(?P<pagepath>.+)$', 'stat_weekly.views.show_weekly'), (r'^goals/(?P<pagepath>.+)$', 'stat_goals.views.show_goal'), Here is such a mini framework over a large one. I hope he successfully demonstrates that if the framework of the system makes it difficult to solve the problem in a simple way, you can always slightly push them apart using the more powerful features of the language.
UPD: As suggested by magic4x , the __name__ property has been added to DynamicPage, which enhances the mimicry for the function.
Source: https://habr.com/ru/post/185180/
All Articles