Project generator
This article will discuss a specific software product designed for the development of large software and information systems. This product is called “Project Generator”, and it is based on an instrumental approach to programming, when the process of developing software systems is considered as an object of automation. This topic was quite fashionable somewhere in the 70-80s of the last century, but in our opinion, this approach is not outdated today. At least, for many years our creative team managed to develop industrial automated systems for various purposes with the help of our own tools .
In the subsequent sections of the article, we will consider some technical aspects of our approach to the instrumental support for the development of large software and information systems, taking into account the historical development of our own views on this issue. That part of the audience, which is our “self-digging” seems too diligent, we suggest immediately proceed to the section “The current situation”, where we, summing up all the material, we will try to articulate as clearly as possible what we offer to evaluate and discuss to the wide developer community.
The project generator was laid more than a quarter of a century ago, when the development team was forced for obvious reasons to complete work on projects in aviation design bureaus and start looking for the use of acquired experience in other areas.
Working on several projects in the banking sector, it turned out that we had to write several systems consisting of many similar elements, differing only in the composition of parameters. The developed systems were built on the then new technology two-tier client-server: workplace - server - database. Each element of the system at the client workstation in general implemented a screen with a data entry form, several tables and several buttons or menu items to perform some actions and, possibly, switch to another screen. In the client-server protocol, it was necessary to program the transmission of the corresponding data and the reception of the server's response. The server had a procedure for receiving the request parameters, analyzing them, performing the required database queries, sending the results in the form of a response to a client request.
')
Thus, the work was carried out as if in two dimensions. One of them is the development of new such fragments, which we have called procedures. New procedures appeared as they were developed, as well as when the task was clarified. Old procedures have been modified.
Here the word “procedure” is used not in the sense, for example, of the Pascal language, i.e. something like subroutines, functions, etc. Rather, it is a kind of code pattern.
The second dimension is the process implementation technology itself. The protocol was changed, various security subsystems were connected, which led to the need to change the program code of all the procedures implemented previously, both in client and server programs.
In addition, with a large number of procedures, it became a problem to maintain the integrity of the project in terms of compliance of client and server programs with their interaction protocol. For example, if in some query a new parameter was added, then the corresponding edits should be performed consistently in two different programs - in the client and in the server.
In connection with the above, an idea arose to automate this process. By that time, we had experience in implementing programs for the parsing of complex languages (the Fakir conceptual programming system with a hierarchical description of algebraic models, analytical formula calculations, and other irrelevant tasks now).
As a result, a project description language was developed, consisting of a header with various project parameters and a set of procedure descriptions. Each procedure had a unique name, a list of input parameters, a list of server requests with a specification of input and output parameters and a response type — a table or one set, and a list of envisaged actions that can be performed in the context of this procedure.
The named data types used in the procedures were specified as part of the project in its description. The description was naturally monitored for compliance with the different components in composition.
Such a description was fed to the input of the project generator, which built the project model in memory, its analysis for integrity. Further, on the basis of the model, the texts of programs that implement client and server program modules were generated.
In the simplest case, client programs were generated in a complete form, but in special cases it was possible to order hand-made software inserts for performing special actions not provided for in the described simple project model.
Project programs themselves were implemented in C language. The server at that time was on the VAX VMS platform, client programs on an IBM PC running MS DOS (but also provided on VAX VMS). Server requests were written on the C preprocessor, which understood the SQL queries embedded in the C program text.
The project description also included a description of the database tables in terms of the types described. This made it possible in the project description to formulate simplified queries to the database on some subset of the SQL language with full syntactic analysis of them for compliance with the given base scheme. More complex queries were implemented in the server programs by means of the preprocessor.
The described technology took root and stood the test of time for a decade. With its help, a fair number of projects in various fields were made.
The model of the project was improved, complicated. Changed software and hardware platforms. Clients and servers began to function in MS Windows and Linux.
The development of the two dimensions mentioned above made it possible to separate the work of analysts from the work of system programmers. The analyst focused on applied tasks, and the system programmer on operating systems, protocols, languages, etc.
The main advantage of this technology is a significant reduction in the code that the system developer writes. The description of one procedure can take several dozen lines, and the generated C program code is usually an order of magnitude more. A change in system software affects the application developer code a little, such changes are taken into account in new versions of the generator.
The use of the project generator made it possible to implement complex volumetric projects with a minimum number of developers, often by one person, and this was done by people who were not well versed in the intricacies of programming, but had a mathematical background and experience in developing systems. Those. not a programmer, but an analyst could independently set up a complex project and bring it to working condition without the involvement of system programmers.
To ensure the convenience of development in the generator, various auxiliary functions of the project design have been constantly improved. In addition to the text of the programs, numerous command files were generated to perform the translation and assembly of all program components for different platforms. The utilities for creating and modifying the database were generated. Automated procedure for the formation of installation packages for transmission to the customer.
It should be noted a little terminological confusion. The project description language described above was often designated as a non-procedural language, in the sense that there were no program operators in it, as in C, Pascal, etc. And the procedures mentioned in the project language are completely different concepts.
As the appetite grew during the meal in the early 2000s, it became clear that it was necessary to improve the technology used in developing projects further.
First, the framework of the project model itself, as a set of types, a database and a set of procedures with queries, became close.
Secondly, the use of the C language in the server, and sometimes in client programs, has become boring. The analyst level is too low - to delve into the details and intricacies of passing parameters, pointers, zero bytes, etc.
On the other hand, there were projects in which it was necessary to write not simple requests to the database for the needs of banking accounting, but rather voluminous programs with complex data structures. In such projects, the effect of automatic generation of software text was reduced, since you had to write a lot of manual code.
With the transition to the window interface, it became possible, and, as a result, the need to depict not only tables with texts and numbers in client programs, but also all other content in the form of pictures and drawings.
In this connection, it was necessary to complicate the client-server interaction protocol. All this has become an incentive to develop a completely new project generator.
The basis of the new development was the concept of a structured document. The document in the context of the project generator occupies one of the central concepts. At the conversational level, a document is a database of a network structure in the program's memory.
The network structure of the document (database) assumes that tables are stored in it, between the rows of which one-to-many relationships are established. This is the old, forgotten concept of databases of the network structure, which we used in the old developments.
The main idea was that the client program receives information from the server not in the form of several tuples and tables, but in the form of an arbitrary document fixed for the given request structure. And, symmetrically, the client, fulfilling the request, also sends a document in general form.
Thus, on the server it is necessary to be able to read the input document from the protocol, having restored its structure in memory. Further, on the basis of the received document, execute a certain program, now called business logic, in which interaction with the database (real, out of memory) is assumed. As a result, the program must generate an output document for transmission to the client program via the protocol.
The client program should provide a program for generating the contents of the application window based on the data received.
The initial project assumed, as well as previous versions of the generator, the use of the C language for writing server request bodies, as well as programs for generating the contents of the application window. As an experimental tool, a relatively simple programming language was developed, designed in the simplest cases to replace the C language. This language was part of the project description language as a whole and received the conditional name of the basic generator language.
The following are the basic concepts of the base language.
The description of the named data type is a string of a given fixed length, numbers of different widths, an enumerated type, a mask type (a bitmask with named components), a structure, and, finally, the document mentioned above.
Procedure (analogue of a function in C) with parameters of the specified types, including documents.
The main standard operators are assignment, conditional, for loop, while loop, procedure call, expression. Everything is very similar to ordinary standard languages. Procedures for manipulating its components are automatically available for the document type.
The result was devastating - after some time in current projects the share of C programs has rapidly decreased to insignificant sizes. Somehow it turned out by itself that project developers began to write exclusively in this language not only the project model, but all the programs both on the client and on the server.
The apotheosis of this unexpected process was the writing of the generator itself in its basic language.
Initially, the structure of the generator assumed at the input a description of the project, a set of texts of manual programs in the C language and a database preprocessor for C. The output is a set of resulting C programs and scripts for translation, assembly, distribution kit formation and other useful actions.
The structure of the generator at this stage of development had the form in the figure below.
With the advent of the base language, the structure of the generator was as follows. Two components stood out in the generator - the C-code generator from the project model and the C-code generator from texts in the base language.
A project model contains high-level entities, such as a server, an application, a server port, a security subsystem, an application window type, a dialog, and many other useful things. All these model entities in the process of generating a project generate programs that implement them in the C language.
The developer also writes his pieces of C-programs, which according to certain rules are included in the resulting set of C-programs of the project.
Now part of the manual C-programs has become possible to replace with equivalent programs in the basic language, which are easier to integrate with the project model.
Thus, with the advent of the basic language, the structure of the generator acquired the form shown in the following figure.
In the process of developing a basic language, the volume of input texts on it grew, and in C it decreased. And as the model description language evolved, the amount of code in the base language decreased, being replaced by short and succinct descriptions of model entities.
Due to the complication of the model component of the project description, the idea arose to generate the model immediately after the C code, and the code in the base language. This simplified the generator itself, since there was no need to follow the subtleties of addressing in C to implement model entities. The translator of the base language in C follows all this.
As a result, the structure of the generator was as follows.
The base language is a part of the generator project description language that implements a universal programming language. The developer using the project generator has the ability to write ordinary programs in this language, but as part of some project.
Unlike C, there is no possibility to write and broadcast a separate program in the basic language. Even if your program consists of several dozen operators, you still need to describe a project in which your program will be the only component, as a utility-type package - a console application.
In a more complicated case, a program in the base language may consist of several packages.
A package is one of the basic concepts of a generator (and basic) language. The package has a unique name within the project, is declared as a package of some type and, as a rule, is presented as a separate file with the text of the program of the package.
A project description is a project description file with a header and an ordered list of packages.
The generator has a large set of so-called system packages of various types, which are written by the developers of the generator and can be used explicitly or implicitly through model entities in your projects. Developer packages do not have to match names with system packages.
Let us give an example of the simplest project of the “Hello world” type. We will not consider the project directory structure here. In our case, we have to create two files - a description of the project and a description of the utility as part of the project, inventing names for them.
Let the project name be hello, the utility name in the project be world. Then we need to create two text files hello.gen and world.utility.
File hello.gen:
World.utility file:
As a result of the generation and assembly of such a project, the executable world.exe (for MS Window) or world (for Unix) will be obtained. When launched, the program prints the string “Hello world!” In the console.
Consider a more complex example in which, in addition to the utility package, a simple package of the type package is used.
Pkgexample.gen file:
File mypkg.package:
File myutl.utility:
In this case, we have two packages as part of the project - the mypkg software package and the myutl utility.
The mypkg package in the external specifications section (before the implementation keyword) describes the type of t_myint derived from the base type int (a 32-bit integer), and the specification of the sum procedure with two input parameters a, b of type t_myint and one output parameter with the same type.
All that is written after the implementaion keyword is available only inside the package.
The procedure localsum has a similar header like the external procedure sum, but it starts with the keyword fproc, which means that it is a local object. After the header, instead of a semicolon, there is a block of operators in curly brackets, very similar to that in the C language. Inside the block there is one assignment operator. In contrast to the vulgar equality, the assignment here is given by the characters ": =".
Generator developers, as people with basic mathematical education, were in pain to look at such constructions in C-like languages, like a = a + 1. It was decided to use algo-pascal notation in such operators.
After the local procedure localsum, the body of the external procedure sum declared in the specifications section is located. The description of the previously specified procedure is similar to the description of the local procedure, but with the fprocdecl keyword and an exact copy of the parameters. Next comes the procedure body block, in which the local procedure call localsum is located.
In the myutl package, there is still a single main section with an operator block in curly braces. The main section must be in the utility exactly in one instance at the end of the package. This is an analogue of the main function in the C language. Operators of this block are executed when the utility is started.
In the block there are three operators. The first statement is a description of the three variables p, q, r, the first two being initialized by constant expressions. Variable types are specified as a package name, followed by a type name defined in this package after a period.
The second statement is a call to the sum procedure from the mypkg package, given, like the types, by a dotted expression.
The third operator is a debug print to the console.
Such an elaborate programming for calculating the sum of two integer constants was done solely in order to demonstrate as much as possible the capabilities of the basic language with the minimum size of the text presented to the public.
A more complex example demonstrates how to work with a document type.
File docexample.gen:
Doc.package file:
File doctest.utility:
In the docexample project, there are two packages - doc.package and doctest.utility.
The doc package describes the character types t_orgname, t_addr, t_phone, t_empname in UTF-8 encoding with the specified lengths.
The orgs type is a document (dqueue keyword). It describes two types of org and emp entries - organization and employee. In brackets the listed components of records of the specified types. In essence, this is similar to the description of a database with two tables and specified columns in them.
The network structure is defined by the relationship between one-to-many records. Such links are entered as sets. So, the set org_emp introduces a connection between an organization and an employee, many employees can work in an organization, but each employee works in no more than one organization. The keyword owner (set owner) sets the type of record owner of the set. The member keyword sets the type of the member of the set.
A set without an owner (in our case, orgs_org) is called singular, it exists in a single copy in the document (it is conditionally owned by the entire document / database).
The document type declaration automatically declares the record types of such a document with the same name as the record names. At the same time, one should remember about the uniqueness of type names within a package.
In addition, procedures for manipulating these objects are automatically generated and available for each record and for each set.
So, for the org record, the org_cre procedure is created with two parameters: a link to the document, an output variable of the record type to get a link to the created record instance.
For the org_emp set, a org_emp_ins procedure is created with three parameters: a reference to the org instance — the record owner of the set, a reference to the emp type instance — the member record of the set, an integer — the position in the set (negative numbers — numbering from the end of the set).
For the org_emp set, a org_emp_mem procedure is also created with three parameters: a reference to an instance of the org type — the record owner of the set; an integer number — the position in the set; an output parameter —a reference to an instance of the emp type — the member record of the set.
For the set org_emp, a procedure org_emp_next is created with one parameter of type emp (input and output) to move to the next element of the set.
The multifunctional expressions isnull (...) and isnotnull (...) apply to document type variables.
In this project, the rand and rand_test system packages are used to generate random test data.
To demonstrate the possibilities of parsing, consider another example of the project.
File calc.gen:
File calc.utility:
The development of a calculator utility is demonstrated, to which the expression is input, the output is the calculated value of the expression.
Using the system package tlex parse allows you to solve the problem of parsing the expression is very simple and the minimum amount of code, using the extensive built-in features of this package. In this example, you can see how easily and compactly it is possible to solve problems of syntactic analysis.
The project involved a system package tlex - lexical parsing in a line. The syntactic parsing is performed by the method of recursive descent. Each non-terminal grammar symbol corresponds to a procedure in the parsing program.
The grammar should be brought to such a form that, by the results of finding out which terminal symbol is located in the reading position, it was possible to decide what rule of grammar to apply for parsing. Bringing to this form for most existing languages is not difficult. To do this, it suffices to define a suply language, which allows incorrect texts from the point of view of the original grammar. Further, on the basis of the constructed parse tree (in one form or another), you can verify the correctness.
In fact, this is always done in real translators, since real programs are not described by context-free grammars. KS grammar can hardly describe a language in which the use of various objects (types, procedures, variables) is prohibited without their description. So these kinds of checks are done anyway.
In our case, parsing is performed in the calc utility for the input parameter string of a program call in the console. If exactly one parameter is not specified, an error message is displayed. If the parameter is set, then it is parsed with the calculation of the result. Upon successful analysis, the result of the calculation is printed in the console. In case of an error, a diagnostic message is displayed with information on which position of the text an error was detected.
Grammar in words. An additive expression is a sequence of multiplicative expressions separated by the signs of operations + or -. A multiplicative expression is a sequence of multipliers separated by signs of operations * or /. The multiplier is an additive expression in parentheses or a number.
The additive expression corresponds to the syn_add procedure, the multiplicative expression is syn_mul, and the multiplier is syn_fact.
Parameters of syntax procedures: buf - the analyzed text, xlex - the context of lexical analysis (including the reading position), dval - the output parameter - the calculated value.
Used lexical procedures of the tlex package.
Function tlex.sample - checks in the current position the presence of the sample presented in the parameter. If the pattern starts with a letter, then the recognized text should completely coincide with the pattern, and not contain the pattern as a substring. Those. if we check the presence of a sample proc, then the text procdecl in the reading position will not give an affirmative result. When recognizing any lexeme, all space characters, line breaks, tabs, etc. are skipped. C and C ++ style comments are also skipped.
Tlex.sample_err procedure - checks the tlex.sample procedure for the presence of a sample; if absent, it triggers a fatal error with the message text containing information about the reading position of the analyzed line.
Such function / procedure pairs are present in the tlex package for many kinds of tokens. The function allows you to check for the presence of a token, the procedure checks and fails to trigger an error.
Procedure tlex.double - checks in the current position the presence of a decimal number in the form 123.456. The NOSIGN parameter prohibits the + - sign at the vocabulary level; we recognize these signs at the grammar level. The NOFRAC parameter allows numbers with or without a decimal point.
The previous section provides examples of projects that demonstrate some of the capabilities of the base language. Note that this is not a reference book, not a language manual, or even a set of examples for learning. This is just an introductory text.
If the reader has reached this place, he has a quite logical question: what is in this basic language what is not in other languages, and why did it take time and effort to implement it?
First, the base language is part of a complete generating language for project description. In the generator language there are many different constructs for describing such model entities as a database, sql query, server, server port, application, application screen layout, dialog, configuration file, json conversion for arbitrary data types, abstract set of related tables for mappings in different environments (dom-model, gtk, qt), etc. This list is constantly updated, outdated entities die, new ones appear. And all these constructions are implemented mainly in the base language. In the early stages of the development of the generator, model structures were implemented directly in C. The use of a base language facilitated the integration of models and language.
Secondly, the generator is intended (and only then gives significant advantages) for the implementation of large projects, when the amount of program code is measured in hundreds of thousands of lines. The work of volume projects is fundamentally different from the development of small programs of up to several thousand lines. In a small program, the developer can keep in mind all the information about its structure. In a large program this is not possible. In the process of modifying such programs, redundant unused code inevitably increases. To counteract this process, numerous means and code checks are being undertaken in the generator.
In the procedure headers, formal parameters are specified as input, input and output, only output. During the translation of the procedure body, this specification is checked for correctness. For example, if a certain parameter is declared a holiday, and in the body of the procedure there is a path in the operator tree in which the variable is used for reading earlier than the value is assigned to it, then this situation leads to an error diagnosis. It uses information on the inputs and outputs of other procedures called in this procedure. If a variable has a structure type, then such an analysis is carried out for each of its components at any level.
For structured documents, you can order a complete set of all operations envisaged for all its records and sets. This will result in significant redundancy code. It is possible (and necessary) to explicitly specify a list of all the required operations for each record and set. In this case, operations that were not used in the project will be identified, and appropriate diagnostics will be issued. Moreover, if, for example, the operation of creating a record is not used, then this is also interpreted as an error. Or for a set there is an operation of inclusion in the set and no more are used, then an error is also generated.
At the time of this writing, there was just the process of deleting some obsolete entities in the project model (old servers and applications are being replaced with new ones). In the document representing the project model in the generator code at the time of the simultaneous existence of old and new objects there were about 250 record types and more than 800 set types. After deleting old entities, the number of records decreased to 190, sets to 580. It is clear that it is extremely difficult to track the usage of so many objects and operations with them manually.
As an illustration, we present data on one real, but already completed, project in the field of financial activity. The project on the generator contained 1294 files with a total of 203,829 lines. In the resulting C program code, there were 4,605 files with 260,8,812 lines. There are 23 different types of servers in the project, 29 ports through which they receive requests, 11 applications, 27 utilities (console programs), and this is only in one, but the largest project within the whole system. As part of this system, there were 7 projects combined by means of export and import of individual packages. The division of the system into 7 different generating projects was done to reduce their volume. Managing such an economy without paranoid integrity checks is very problematic.
The project generator during the whole period of work on it developed as an internal project in the development team. In the process of working on applied projects, our customers in some cases had the opportunity to use the generator for their own development, one way or another connected with our project (or maybe not only, it is difficult to control). As work on the projects clarified the formulation of the problem, new tool needs arose that were implemented in the next versions of the generator. As it was said at the beginning, it was a two-dimensional development, applied and instrumental. Those. the generator developed along with the next (s) project. Periodically, there was talk about making the generator itself a software product for the general public. This text is in a sense an attempt to present our system to this very public.
Currently, the generator has tools for working with database management systems Oracle, Postgres, MySql, MS Sql, Sqlite of different versions. Working with databases involves describing the database schema using generator tools, writing sql queries in a special language (using a representative enough subset of sql) with full syntactic and semantic control of the query text to the database schema. There are tools for specifying alternatives for different types of databases. And you can write the so-called dynamic queries that will be transferred to the database without control.
Servers are transferred to modern network tools - Epoll on Unix and I / O Completion on MS Windows.
Until recently, applications were developed using a single interface model (the so-called window layouts), which had two implementations - Microsoft WIN32 GUI and GTK-3 for Unix.Maintaining one model for these two systems has become increasingly difficult due to insufficiently developed tools in the WIN32 GUI. Therefore, two different approaches are being developed now - GTK-3 and Qt-5. Models are different, but very similar. It is not yet decided to separate them, or try to combine them with individual exceptions in different implementations.
The export of Java Script packages for use in WEB-development is actively developing. In the browser, you can receive data from the server in json-format and use the functions generated for Java Script with the types described in the project, for example, with a structured document.
For WEB-development, a fairly representative subset of the HTML language is implemented with inserts in the base language (similar to PHP). At the same time, HTML markup is controlled for correct nesting of tags. And, of course, inserts in the base language are subject to total control, as is customary throughout the generator.
Package export to Java language implemented mainly for developing components of systems on mobile platforms (Android).
In the subsequent sections of the article, we will consider some technical aspects of our approach to the instrumental support for the development of large software and information systems, taking into account the historical development of our own views on this issue. That part of the audience, which is our “self-digging” seems too diligent, we suggest immediately proceed to the section “The current situation”, where we, summing up all the material, we will try to articulate as clearly as possible what we offer to evaluate and discuss to the wide developer community.
Historical background for understanding the motivation of developers
The project generator was laid more than a quarter of a century ago, when the development team was forced for obvious reasons to complete work on projects in aviation design bureaus and start looking for the use of acquired experience in other areas.
Working on several projects in the banking sector, it turned out that we had to write several systems consisting of many similar elements, differing only in the composition of parameters. The developed systems were built on the then new technology two-tier client-server: workplace - server - database. Each element of the system at the client workstation in general implemented a screen with a data entry form, several tables and several buttons or menu items to perform some actions and, possibly, switch to another screen. In the client-server protocol, it was necessary to program the transmission of the corresponding data and the reception of the server's response. The server had a procedure for receiving the request parameters, analyzing them, performing the required database queries, sending the results in the form of a response to a client request.
')
Thus, the work was carried out as if in two dimensions. One of them is the development of new such fragments, which we have called procedures. New procedures appeared as they were developed, as well as when the task was clarified. Old procedures have been modified.
Here the word “procedure” is used not in the sense, for example, of the Pascal language, i.e. something like subroutines, functions, etc. Rather, it is a kind of code pattern.
The second dimension is the process implementation technology itself. The protocol was changed, various security subsystems were connected, which led to the need to change the program code of all the procedures implemented previously, both in client and server programs.
In addition, with a large number of procedures, it became a problem to maintain the integrity of the project in terms of compliance of client and server programs with their interaction protocol. For example, if in some query a new parameter was added, then the corresponding edits should be performed consistently in two different programs - in the client and in the server.
In connection with the above, an idea arose to automate this process. By that time, we had experience in implementing programs for the parsing of complex languages (the Fakir conceptual programming system with a hierarchical description of algebraic models, analytical formula calculations, and other irrelevant tasks now).
As a result, a project description language was developed, consisting of a header with various project parameters and a set of procedure descriptions. Each procedure had a unique name, a list of input parameters, a list of server requests with a specification of input and output parameters and a response type — a table or one set, and a list of envisaged actions that can be performed in the context of this procedure.
The named data types used in the procedures were specified as part of the project in its description. The description was naturally monitored for compliance with the different components in composition.
Such a description was fed to the input of the project generator, which built the project model in memory, its analysis for integrity. Further, on the basis of the model, the texts of programs that implement client and server program modules were generated.
In the simplest case, client programs were generated in a complete form, but in special cases it was possible to order hand-made software inserts for performing special actions not provided for in the described simple project model.
Project programs themselves were implemented in C language. The server at that time was on the VAX VMS platform, client programs on an IBM PC running MS DOS (but also provided on VAX VMS). Server requests were written on the C preprocessor, which understood the SQL queries embedded in the C program text.
The project description also included a description of the database tables in terms of the types described. This made it possible in the project description to formulate simplified queries to the database on some subset of the SQL language with full syntactic analysis of them for compliance with the given base scheme. More complex queries were implemented in the server programs by means of the preprocessor.
The described technology took root and stood the test of time for a decade. With its help, a fair number of projects in various fields were made.
The model of the project was improved, complicated. Changed software and hardware platforms. Clients and servers began to function in MS Windows and Linux.
The development of the two dimensions mentioned above made it possible to separate the work of analysts from the work of system programmers. The analyst focused on applied tasks, and the system programmer on operating systems, protocols, languages, etc.
The main advantage of this technology is a significant reduction in the code that the system developer writes. The description of one procedure can take several dozen lines, and the generated C program code is usually an order of magnitude more. A change in system software affects the application developer code a little, such changes are taken into account in new versions of the generator.
The use of the project generator made it possible to implement complex volumetric projects with a minimum number of developers, often by one person, and this was done by people who were not well versed in the intricacies of programming, but had a mathematical background and experience in developing systems. Those. not a programmer, but an analyst could independently set up a complex project and bring it to working condition without the involvement of system programmers.
To ensure the convenience of development in the generator, various auxiliary functions of the project design have been constantly improved. In addition to the text of the programs, numerous command files were generated to perform the translation and assembly of all program components for different platforms. The utilities for creating and modifying the database were generated. Automated procedure for the formation of installation packages for transmission to the customer.
It should be noted a little terminological confusion. The project description language described above was often designated as a non-procedural language, in the sense that there were no program operators in it, as in C, Pascal, etc. And the procedures mentioned in the project language are completely different concepts.
Transition from a non-procedural project model to a universal programming language
As the appetite grew during the meal in the early 2000s, it became clear that it was necessary to improve the technology used in developing projects further.
First, the framework of the project model itself, as a set of types, a database and a set of procedures with queries, became close.
Secondly, the use of the C language in the server, and sometimes in client programs, has become boring. The analyst level is too low - to delve into the details and intricacies of passing parameters, pointers, zero bytes, etc.
On the other hand, there were projects in which it was necessary to write not simple requests to the database for the needs of banking accounting, but rather voluminous programs with complex data structures. In such projects, the effect of automatic generation of software text was reduced, since you had to write a lot of manual code.
With the transition to the window interface, it became possible, and, as a result, the need to depict not only tables with texts and numbers in client programs, but also all other content in the form of pictures and drawings.
In this connection, it was necessary to complicate the client-server interaction protocol. All this has become an incentive to develop a completely new project generator.
The basis of the new development was the concept of a structured document. The document in the context of the project generator occupies one of the central concepts. At the conversational level, a document is a database of a network structure in the program's memory.
The network structure of the document (database) assumes that tables are stored in it, between the rows of which one-to-many relationships are established. This is the old, forgotten concept of databases of the network structure, which we used in the old developments.
The main idea was that the client program receives information from the server not in the form of several tuples and tables, but in the form of an arbitrary document fixed for the given request structure. And, symmetrically, the client, fulfilling the request, also sends a document in general form.
Thus, on the server it is necessary to be able to read the input document from the protocol, having restored its structure in memory. Further, on the basis of the received document, execute a certain program, now called business logic, in which interaction with the database (real, out of memory) is assumed. As a result, the program must generate an output document for transmission to the client program via the protocol.
The client program should provide a program for generating the contents of the application window based on the data received.
The initial project assumed, as well as previous versions of the generator, the use of the C language for writing server request bodies, as well as programs for generating the contents of the application window. As an experimental tool, a relatively simple programming language was developed, designed in the simplest cases to replace the C language. This language was part of the project description language as a whole and received the conditional name of the basic generator language.
The following are the basic concepts of the base language.
The description of the named data type is a string of a given fixed length, numbers of different widths, an enumerated type, a mask type (a bitmask with named components), a structure, and, finally, the document mentioned above.
Procedure (analogue of a function in C) with parameters of the specified types, including documents.
The main standard operators are assignment, conditional, for loop, while loop, procedure call, expression. Everything is very similar to ordinary standard languages. Procedures for manipulating its components are automatically available for the document type.
The result was devastating - after some time in current projects the share of C programs has rapidly decreased to insignificant sizes. Somehow it turned out by itself that project developers began to write exclusively in this language not only the project model, but all the programs both on the client and on the server.
The apotheosis of this unexpected process was the writing of the generator itself in its basic language.
Architecture project generator at different stages of its development
Initially, the structure of the generator assumed at the input a description of the project, a set of texts of manual programs in the C language and a database preprocessor for C. The output is a set of resulting C programs and scripts for translation, assembly, distribution kit formation and other useful actions.
The structure of the generator at this stage of development had the form in the figure below.
With the advent of the base language, the structure of the generator was as follows. Two components stood out in the generator - the C-code generator from the project model and the C-code generator from texts in the base language.
A project model contains high-level entities, such as a server, an application, a server port, a security subsystem, an application window type, a dialog, and many other useful things. All these model entities in the process of generating a project generate programs that implement them in the C language.
The developer also writes his pieces of C-programs, which according to certain rules are included in the resulting set of C-programs of the project.
Now part of the manual C-programs has become possible to replace with equivalent programs in the basic language, which are easier to integrate with the project model.
Thus, with the advent of the basic language, the structure of the generator acquired the form shown in the following figure.
In the process of developing a basic language, the volume of input texts on it grew, and in C it decreased. And as the model description language evolved, the amount of code in the base language decreased, being replaced by short and succinct descriptions of model entities.
Due to the complication of the model component of the project description, the idea arose to generate the model immediately after the C code, and the code in the base language. This simplified the generator itself, since there was no need to follow the subtleties of addressing in C to implement model entities. The translator of the base language in C follows all this.
As a result, the structure of the generator was as follows.
Basic Concepts of the Generator Base Language
The base language is a part of the generator project description language that implements a universal programming language. The developer using the project generator has the ability to write ordinary programs in this language, but as part of some project.
Unlike C, there is no possibility to write and broadcast a separate program in the basic language. Even if your program consists of several dozen operators, you still need to describe a project in which your program will be the only component, as a utility-type package - a console application.
In a more complicated case, a program in the base language may consist of several packages.
A package is one of the basic concepts of a generator (and basic) language. The package has a unique name within the project, is declared as a package of some type and, as a rule, is presented as a separate file with the text of the program of the package.
A project description is a project description file with a header and an ordered list of packages.
The generator has a large set of so-called system packages of various types, which are written by the developers of the generator and can be used explicitly or implicitly through model entities in your projects. Developer packages do not have to match names with system packages.
Let us give an example of the simplest project of the “Hello world” type. We will not consider the project directory structure here. In our case, we have to create two files - a description of the project and a description of the utility as part of the project, inventing names for them.
Let the project name be hello, the utility name in the project be world. Then we need to create two text files hello.gen and world.utility.
File hello.gen:
Hidden text
project hello /version="01.001" utility world World.utility file:
Hidden text
utility world:"world" main { dprint("Hello world!"); } As a result of the generation and assembly of such a project, the executable world.exe (for MS Window) or world (for Unix) will be obtained. When launched, the program prints the string “Hello world!” In the console.
Consider a more complex example in which, in addition to the utility package, a simple package of the type package is used.
Pkgexample.gen file:
Hidden text
project pkgexample /version="01.001" package mypkg utility myutl File mypkg.package:
Hidden text
package mypkg type t_myint : int; fprocdecl sum(t_myint a,t_myint b,out t_myint c); implementation fproc localsum(t_myint x,t_myint y,out t_myint z) { z := x+y; } fprocdecl sum(t_myint a,t_myint b,out t_myint c) { call localsum(a,b,c); } File myutl.utility:
Hidden text
utility myutl:"myutl" main { var mypkg.t_myint p := 3, mypkg.t_myint q := 5, mypkg.t_myint r; call mypkg.sum(p,q,r); dprint("p=",p," q=",q," r=",r,"\n"); } In this case, we have two packages as part of the project - the mypkg software package and the myutl utility.
The mypkg package in the external specifications section (before the implementation keyword) describes the type of t_myint derived from the base type int (a 32-bit integer), and the specification of the sum procedure with two input parameters a, b of type t_myint and one output parameter with the same type.
All that is written after the implementaion keyword is available only inside the package.
The procedure localsum has a similar header like the external procedure sum, but it starts with the keyword fproc, which means that it is a local object. After the header, instead of a semicolon, there is a block of operators in curly brackets, very similar to that in the C language. Inside the block there is one assignment operator. In contrast to the vulgar equality, the assignment here is given by the characters ": =".
Generator developers, as people with basic mathematical education, were in pain to look at such constructions in C-like languages, like a = a + 1. It was decided to use algo-pascal notation in such operators.
After the local procedure localsum, the body of the external procedure sum declared in the specifications section is located. The description of the previously specified procedure is similar to the description of the local procedure, but with the fprocdecl keyword and an exact copy of the parameters. Next comes the procedure body block, in which the local procedure call localsum is located.
In the myutl package, there is still a single main section with an operator block in curly braces. The main section must be in the utility exactly in one instance at the end of the package. This is an analogue of the main function in the C language. Operators of this block are executed when the utility is started.
In the block there are three operators. The first statement is a description of the three variables p, q, r, the first two being initialized by constant expressions. Variable types are specified as a package name, followed by a type name defined in this package after a period.
The second statement is a call to the sum procedure from the mypkg package, given, like the types, by a dotted expression.
The third operator is a debug print to the console.
Such an elaborate programming for calculating the sum of two integer constants was done solely in order to demonstrate as much as possible the capabilities of the basic language with the minimum size of the text presented to the public.
A more complex example demonstrates how to work with a document type.
File docexample.gen:
Hidden text
project docexample /version="01.001" package doc utility doctest Doc.package file:
Hidden text
package doc type t_orgname : char8[100]; type t_addr : char8[100]; type t_phone : char8[30]; type t_empname : char8[100]; type orgs : dqueue ( record org ( t_orgname orgname, t_addr addr, t_phone phone ); record emp ( t_empname empname1, t_empname empname2, t_empname empname3, t_addr addr, t_phone phone ); set orgs_org member org;/oper=(mem,next) set org_emp owner org member emp;/oper=(mem,next) ); procdecl fill(orgs porgs); implementation procdecl fill(orgs porgs) { var int iorg; rand.init(); for ( iorg := 0; iorg < 6; iorg += 1 ) { var org xorg, int iemp; org_cre(porgs,xorg); rand_test.firm8(xorg.orgname); rand_test.addr8(false,xorg.addr); rand_test.phone(xorg.phone); orgs_org_ins(porgs,xorg,-1); for ( iemp := 0; iemp < 8; iemp += 1 ) { var emp xemp; emp_cre(xorg.xdoc,xemp); rand_test.name8(xemp.empname1,xemp.empname2,xemp.empname3); rand_test.addr8(true,xemp.addr); rand_test.phone(xemp.phone); org_emp_ins(xorg,xemp,-1); } } } File doctest.utility:
Hidden text
utility doctest:"doctest" proc test(doc.orgs porgs) { call doc.fill(porgs); { var doc.org xorg; doc.orgs_org_mem(porgs,0,xorg); while ( isnotnull(xorg) ) { dprint("\n"); dprint(U": ",xorg.orgname,"\n"); dprint(U": ",xorg.addr,"\n"); dprint(U": ",xorg.phone,"\n"); var doc.emp xemp; doc.org_emp_mem(xorg,0,xemp); while ( isnotnull(xemp) ) { dprint("\n"); dprint(U" : ",xemp.empname1,"\n"); dprint(U" : ",xemp.empname2,"\n"); dprint(U" : ",xemp.empname3,"\n"); dprint(U" : ",xemp.addr,"\n"); dprint(U" : ",xemp.phone,"\n"); doc.org_emp_next(xemp); } doc.orgs_org_next(xorg); } } } main { varobj doc.orgs porgs; call test(porgs); } In the docexample project, there are two packages - doc.package and doctest.utility.
The doc package describes the character types t_orgname, t_addr, t_phone, t_empname in UTF-8 encoding with the specified lengths.
The orgs type is a document (dqueue keyword). It describes two types of org and emp entries - organization and employee. In brackets the listed components of records of the specified types. In essence, this is similar to the description of a database with two tables and specified columns in them.
The network structure is defined by the relationship between one-to-many records. Such links are entered as sets. So, the set org_emp introduces a connection between an organization and an employee, many employees can work in an organization, but each employee works in no more than one organization. The keyword owner (set owner) sets the type of record owner of the set. The member keyword sets the type of the member of the set.
A set without an owner (in our case, orgs_org) is called singular, it exists in a single copy in the document (it is conditionally owned by the entire document / database).
The document type declaration automatically declares the record types of such a document with the same name as the record names. At the same time, one should remember about the uniqueness of type names within a package.
In addition, procedures for manipulating these objects are automatically generated and available for each record and for each set.
So, for the org record, the org_cre procedure is created with two parameters: a link to the document, an output variable of the record type to get a link to the created record instance.
For the org_emp set, a org_emp_ins procedure is created with three parameters: a reference to the org instance — the record owner of the set, a reference to the emp type instance — the member record of the set, an integer — the position in the set (negative numbers — numbering from the end of the set).
For the org_emp set, a org_emp_mem procedure is also created with three parameters: a reference to an instance of the org type — the record owner of the set; an integer number — the position in the set; an output parameter —a reference to an instance of the emp type — the member record of the set.
For the set org_emp, a procedure org_emp_next is created with one parameter of type emp (input and output) to move to the next element of the set.
The multifunctional expressions isnull (...) and isnotnull (...) apply to document type variables.
In this project, the rand and rand_test system packages are used to generate random test data.
To demonstrate the possibilities of parsing, consider another example of the project.
File calc.gen:
Hidden text
project calc /version="01.001" utility calc File calc.utility:
Hidden text
utility calc:"Calculator" type t_double : double; /frac=6 procspec syn_add(string7 buf,tlex.t_lex xlex,out t_double dval); proc syn_fact(string7 buf,tlex.t_lex xlex,out t_double dval) { dval := 0.0; if ( tlex.sample(buf,xlex,"(") ) { call syn_add(buf,xlex,dval); tlex.sample_err(buf,xlex,")"); } else if ( tlex.double(buf,xlex,NOSIGN,NOFRAC,NOEXPON,dval) ) ; else tlex.message(buf,xlex,"syntax error"); } proc syn_mul(string7 buf,tlex.t_lex xlex,out t_double dval) { call syn_fact(buf,xlex,dval); for ( ; ; ) { var t_double dval1; if ( tlex.sample(buf,xlex,"*") ) { call syn_fact(buf,xlex,dval1); dval *= dval1; } else if ( tlex.sample(buf,xlex,"/") ) { call syn_fact(buf,xlex,dval1); dval /= dval1; } else break; } } proc syn_add(string7 buf,tlex.t_lex xlex,out t_double dval) { var bool minus; if ( tlex.sample(buf,xlex,"+") ) ; else if ( tlex.sample(buf,xlex,"-") ) minus := true; call syn_mul(buf,xlex,dval); if ( minus ) dval := -dval; for ( ; ; ) { var t_double dval1; if ( tlex.sample(buf,xlex,"+") ) { call syn_mul(buf,xlex,dval1); dval += dval1; } else if ( tlex.sample(buf,xlex,"-") ) { call syn_mul(buf,xlex,dval1); dval -= dval1; } else break; } } proc syn(string7 buf,tlex.t_lex xlex,out t_double dval) { call syn_add(buf,xlex,dval); tlex.eof_err(buf,xlex); } main { if ( utl.argc(xutl.yutl) <> 2 ) error U" "; var tlex.t_lex xlex, t_double dval; call syn(utl.argv7(xutl.yutl,1),xlex,dval); dprint("result=",dval,"\n"); } The development of a calculator utility is demonstrated, to which the expression is input, the output is the calculated value of the expression.
Using the system package tlex parse allows you to solve the problem of parsing the expression is very simple and the minimum amount of code, using the extensive built-in features of this package. In this example, you can see how easily and compactly it is possible to solve problems of syntactic analysis.
The project involved a system package tlex - lexical parsing in a line. The syntactic parsing is performed by the method of recursive descent. Each non-terminal grammar symbol corresponds to a procedure in the parsing program.
The grammar should be brought to such a form that, by the results of finding out which terminal symbol is located in the reading position, it was possible to decide what rule of grammar to apply for parsing. Bringing to this form for most existing languages is not difficult. To do this, it suffices to define a suply language, which allows incorrect texts from the point of view of the original grammar. Further, on the basis of the constructed parse tree (in one form or another), you can verify the correctness.
In fact, this is always done in real translators, since real programs are not described by context-free grammars. KS grammar can hardly describe a language in which the use of various objects (types, procedures, variables) is prohibited without their description. So these kinds of checks are done anyway.
In our case, parsing is performed in the calc utility for the input parameter string of a program call in the console. If exactly one parameter is not specified, an error message is displayed. If the parameter is set, then it is parsed with the calculation of the result. Upon successful analysis, the result of the calculation is printed in the console. In case of an error, a diagnostic message is displayed with information on which position of the text an error was detected.
Grammar in words. An additive expression is a sequence of multiplicative expressions separated by the signs of operations + or -. A multiplicative expression is a sequence of multipliers separated by signs of operations * or /. The multiplier is an additive expression in parentheses or a number.
The additive expression corresponds to the syn_add procedure, the multiplicative expression is syn_mul, and the multiplier is syn_fact.
Parameters of syntax procedures: buf - the analyzed text, xlex - the context of lexical analysis (including the reading position), dval - the output parameter - the calculated value.
Used lexical procedures of the tlex package.
Function tlex.sample - checks in the current position the presence of the sample presented in the parameter. If the pattern starts with a letter, then the recognized text should completely coincide with the pattern, and not contain the pattern as a substring. Those. if we check the presence of a sample proc, then the text procdecl in the reading position will not give an affirmative result. When recognizing any lexeme, all space characters, line breaks, tabs, etc. are skipped. C and C ++ style comments are also skipped.
Tlex.sample_err procedure - checks the tlex.sample procedure for the presence of a sample; if absent, it triggers a fatal error with the message text containing information about the reading position of the analyzed line.
Such function / procedure pairs are present in the tlex package for many kinds of tokens. The function allows you to check for the presence of a token, the procedure checks and fails to trigger an error.
Procedure tlex.double - checks in the current position the presence of a decimal number in the form 123.456. The NOSIGN parameter prohibits the + - sign at the vocabulary level; we recognize these signs at the grammar level. The NOFRAC parameter allows numbers with or without a decimal point.
Subtotals
The previous section provides examples of projects that demonstrate some of the capabilities of the base language. Note that this is not a reference book, not a language manual, or even a set of examples for learning. This is just an introductory text.
If the reader has reached this place, he has a quite logical question: what is in this basic language what is not in other languages, and why did it take time and effort to implement it?
First, the base language is part of a complete generating language for project description. In the generator language there are many different constructs for describing such model entities as a database, sql query, server, server port, application, application screen layout, dialog, configuration file, json conversion for arbitrary data types, abstract set of related tables for mappings in different environments (dom-model, gtk, qt), etc. This list is constantly updated, outdated entities die, new ones appear. And all these constructions are implemented mainly in the base language. In the early stages of the development of the generator, model structures were implemented directly in C. The use of a base language facilitated the integration of models and language.
Secondly, the generator is intended (and only then gives significant advantages) for the implementation of large projects, when the amount of program code is measured in hundreds of thousands of lines. The work of volume projects is fundamentally different from the development of small programs of up to several thousand lines. In a small program, the developer can keep in mind all the information about its structure. In a large program this is not possible. In the process of modifying such programs, redundant unused code inevitably increases. To counteract this process, numerous means and code checks are being undertaken in the generator.
In the procedure headers, formal parameters are specified as input, input and output, only output. During the translation of the procedure body, this specification is checked for correctness. For example, if a certain parameter is declared a holiday, and in the body of the procedure there is a path in the operator tree in which the variable is used for reading earlier than the value is assigned to it, then this situation leads to an error diagnosis. It uses information on the inputs and outputs of other procedures called in this procedure. If a variable has a structure type, then such an analysis is carried out for each of its components at any level.
For structured documents, you can order a complete set of all operations envisaged for all its records and sets. This will result in significant redundancy code. It is possible (and necessary) to explicitly specify a list of all the required operations for each record and set. In this case, operations that were not used in the project will be identified, and appropriate diagnostics will be issued. Moreover, if, for example, the operation of creating a record is not used, then this is also interpreted as an error. Or for a set there is an operation of inclusion in the set and no more are used, then an error is also generated.
At the time of this writing, there was just the process of deleting some obsolete entities in the project model (old servers and applications are being replaced with new ones). In the document representing the project model in the generator code at the time of the simultaneous existence of old and new objects there were about 250 record types and more than 800 set types. After deleting old entities, the number of records decreased to 190, sets to 580. It is clear that it is extremely difficult to track the usage of so many objects and operations with them manually.
As an illustration, we present data on one real, but already completed, project in the field of financial activity. The project on the generator contained 1294 files with a total of 203,829 lines. In the resulting C program code, there were 4,605 files with 260,8,812 lines. There are 23 different types of servers in the project, 29 ports through which they receive requests, 11 applications, 27 utilities (console programs), and this is only in one, but the largest project within the whole system. As part of this system, there were 7 projects combined by means of export and import of individual packages. The division of the system into 7 different generating projects was done to reduce their volume. Managing such an economy without paranoid integrity checks is very problematic.
Current situation
The project generator during the whole period of work on it developed as an internal project in the development team. In the process of working on applied projects, our customers in some cases had the opportunity to use the generator for their own development, one way or another connected with our project (or maybe not only, it is difficult to control). As work on the projects clarified the formulation of the problem, new tool needs arose that were implemented in the next versions of the generator. As it was said at the beginning, it was a two-dimensional development, applied and instrumental. Those. the generator developed along with the next (s) project. Periodically, there was talk about making the generator itself a software product for the general public. This text is in a sense an attempt to present our system to this very public.
Currently, the generator has tools for working with database management systems Oracle, Postgres, MySql, MS Sql, Sqlite of different versions. Working with databases involves describing the database schema using generator tools, writing sql queries in a special language (using a representative enough subset of sql) with full syntactic and semantic control of the query text to the database schema. There are tools for specifying alternatives for different types of databases. And you can write the so-called dynamic queries that will be transferred to the database without control.
Servers are transferred to modern network tools - Epoll on Unix and I / O Completion on MS Windows.
Until recently, applications were developed using a single interface model (the so-called window layouts), which had two implementations - Microsoft WIN32 GUI and GTK-3 for Unix.Maintaining one model for these two systems has become increasingly difficult due to insufficiently developed tools in the WIN32 GUI. Therefore, two different approaches are being developed now - GTK-3 and Qt-5. Models are different, but very similar. It is not yet decided to separate them, or try to combine them with individual exceptions in different implementations.
The export of Java Script packages for use in WEB-development is actively developing. In the browser, you can receive data from the server in json-format and use the functions generated for Java Script with the types described in the project, for example, with a structured document.
For WEB-development, a fairly representative subset of the HTML language is implemented with inserts in the base language (similar to PHP). At the same time, HTML markup is controlled for correct nesting of tags. And, of course, inserts in the base language are subject to total control, as is customary throughout the generator.
Package export to Java language implemented mainly for developing components of systems on mobile platforms (Android).
Source: https://habr.com/ru/post/326564/
All Articles