Experience creating frameworks
I work in a company that has been developing custom accounting systems for quite a long time. About the same time there is a technology department - a team of people who develop tools and frameworks for their comrades, and they, in turn, create systems for customers. Briefly we can say that we (technologists) unify the process of writing business logic using our own ORM, combine designers and programmers using a UML editor with code generators, provide various UI controls that allow users of end applications to work productively.
In this article I want to share my experience regarding the process of creating a technological layer.
There are several reasons why the company contains several people who do not bring direct profit; in my opinion, the most important thing is:
1. Reuse ideas and code. The technology department collects best practices as you complete various projects and distributes them to all teams. In this case, there is no need to arrange each connection with each in order to be aware of what is new and useful that a neighbor has. After all, a neighbor can immediately share with technologists and not to bother much about universalization, while technologists have the opportunity to inform everyone else (a list of improvements is regularly made that have fallen into the next version of technological products).
2. Unification of solutions. Programmers sometimes change jobs. To deal with the negative aspects of such situations, the technology provides some unified methods for solving typical problems. This reduces the number of bicycles and reduces the time of entry of a new person into the project.
A little bit about how a new technological direction is born. From time to time, the applied platforms are modified and you have to create technology for these new items.
')
Due to the fact that our product is not a complete application, to begin with, any subject area for the prototype project is selected (1-3 pieces). This application serves only to test solutions and approaches that we plan to implement. For fun, the last such system we used to account for domestic cats in the apartment. A special feature of the prototypes is that the code is quite mixed up and it is not clear where the technological layer is, and where the applied logic begins. On this "cat" project, all the necessary components, generators, user interface elements are worked out. Only technologists work on the project.
When it comes time to test the technology on something more serious, the “sacrificial” project is chosen from the commercial one. As a rule, it has a 1.5–2 times longer period that can be set aside for development than a typical project (everyone understands that there are risks, because the technology is used by us for the first time and the company itself pays for technological work, and not customer). A feature of this project is that the technology code is initially in a solution, and as it is developed it is divided into technology projects and application code (which depends on the subject area). The project work and technology and application programmers. It should be understood that the main approaches are still being debugged and not everything in the technology is 100% ready, so the project will be with innate “bikes” that complicate support. Sometimes these “bikes” are replaced with standard technological solutions, sometimes they remain until the end.
At this stage, we believe that the technology is ready for widespread use. Go to the normal "operating mode". Application developers no longer need direct assistance from technologists. Technological assemblies are delivered in compiled form. Refinement tasks and error messages are transferred to the TFS technology project.
From time to time, serious improvements of the technology are required, as a rule, this is adding functionality, for example, the general reporting subsystem, the authorization subsystem, the logging subsystem, etc ... for users, it appears already in its finished form. As a rule, such new products immediately become popular with all projects that are actively developing. It's funny that sometimes, at the end of the development of a regular feature, an application programmer comes in and asks for it to be implemented.
With the development of new directions, the old ones gradually disappear. Projects are closed, there are units of projects that require support. Together with them, the technology "retires", goes into Bug driven development. New features are no longer added, only fixes old bugs that have been accumulating over the years. Some bugs are fixed with crutches in the application project, since it is no longer advisable to make changes to the technology. Changes in technology should always be made with an eye to all projects that depend on it. One wrong move and problems are guaranteed both to the people on the applied project, and after them to the technologists.
If something needs to be rewritten, then for a technology with multiple entries in projects:
• We collect requirements that were satisfied by the old functionality (as a rule, after a series of improvements and minor fixes, there is no complete and up-to-date documentation for the programmer)
• We plan work (once again we look at the budget and evaluate the feasibility)
• We write tests to fix the state. According to them we will determine that what has worked before will not break.
• We write functionality
• Check the quality and compliance with the stated requirements
Now a little about techniques and techniques. It's about the Microsoft .NET Framework, because our solutions are built on this platform.
The technology should not limit programmers, but give advantages over a solution from scratch, and sometimes completely ready-made solutions. In order for a programmer to show his individuality where the technology is “too standard” we use the following approaches:
• Inheritance. There is a base class that implements the standard behavior and is located in the technological assembly. The code generated for an application project is inherited from the base class and in special methods it assumes a change in the standard behavior.
• Developments. Technology classes contain a series of events whose processing allows you to change behavior. It should be remembered that an event can be subscribed to many times.
• Delegates. Similar to events, but there can be only one implementation. This prevents unwanted triggering several times.
• App.config. Various trifles fall into the configuration file. As a rule, these are some flags and paths (do you need to write debugging information, connection strings to services, etc.).
• DI. This option implies redefinition of the whole module.
From the technology layer, there is rarely normal access to the user interface. As a rule, if something went wrong, it indicates that there is an error either in technology (yes, they are and we understand it), or in the application code (ideally, you can indicate to the application programmer clearly what he is doing wrong and in which place). In any case, you need to somehow signal this. We use Log4net. To highlight technological messages, you can use either a separate logger, or fix the level of errors, for example, Warn, to the technology.
XML comments in the technology code are extremely important. It is necessary to prescribe them for public classes, methods and properties. This will allow the generation of help (http://sandcastle.codeplex.com/) and provide hints in application projects thanks to IntelliSense. Do not forget to specify in the project settings whether to generate an xml file with comments.
It is extremely rare that such a change in technology occurs, which requires the intervention of an application programmer to update the technology layer. If such an update can be predicted, then you need to warn in advance, for example, using the Obsolete attribute:
[Obsolete (“This function will be removed in the next version, use the function”)]
Sometimes you have to go on about and apply the rule: “it’s better to add 1 time than to update 2 times”, i.e. if the method signature is used in 1000 places in each application project, changing it in technology is not a good idea. The most important rule: always a detailed description of possible problems with the update - forewarned, then armed.
For process assemblies, documentation is essential. To store information, we use the Wiki engine with the right to write to the technologists. To keep information up to date, 2 rules apply:
1. Made a deal - write it in the Wiki!
2. Answered the question - write it in the Wiki!
Description Levels:
1. For yourself (as arranged)
2. For programmers of applied projects (how to use methods and classes)
3. For system users (how to use components)
Sometimes you want to minimize the number of assemblies issued for projects. This allows you to deal with incorrect updates when you need to update several assemblies at once, otherwise something will not work and minimizes the risk that something will be forgotten to be installed by the customer. However, different projects need slightly different functions. What can be applied?
1. Links to cs-files between projects (work with projects becomes a bit more complicated)
2. ILMerge (this can be assigned to the Build server, but debugging is difficult)
3. Assemblies in resources (with a separate assembly loading mechanism)
Another iron rule: all process assemblies must be signed. This gives us the following benefits:
1. Signed assemblies are uniquely identified in the GAC.
2. You can only sign if all dependent assemblies are signed (application assemblies will not be able to sign, if technological assemblies are not signed)
3. Some degree of protection (to change the code, you will need to rebuild all application builds)
If it so happens that the code from the assembly is lost, then signing can be done by rebuilding the assembly. Rebuilding is done this way (to specify a file with a key, look for a special parameter):
1. ildasm.exe sampleA.exe / source /out:sampleA.il
2. ilasm.exe sampleA.il / exe /out:sampleA.exe
The more often a product is released, the less it seems to users that programmers do nothing. We tried 2 versions of the release of technological product versions:
1. The latest builds are the best.
Everything that is hosted in SourceControl is considered verified and application programmers can take any server build of technological assemblies. At the same time, technologists before each Check-in must carefully check their changes.
Pros:
• Instant posting of changes
Minuses:
• Each Check-In requires an extended check with lots of different functionality that may suffer.
• Errors can easily penetrate the version at the customer
2. Periodic release of stable version
Release of the version of technological assemblies on a periodic basis.
Pros:
• More reliable integration verification
Minuses:
• There is a risk of losing control of the code in the middle of the iteration.
• It takes time to check everything again, no instant fixes
During the release of the version for 1-2 days, all technologists are distracted by checking and correcting the errors found. This is the option we now apply.
There are 2 strategies for naming build versions:
1. Variable build versions
• Updating of applied projects only with recompilation of the project (inability to send a patch to the technology directly to the customer)
• Part of the problems will be visible when compiling (relevant for complex dependencies between assemblies)
• Problem with type serialization (serialized data stored by the customer may cause problems when updating build versions)
2. Fixed versions of builds
• Update simple substitution of assemblies
• Problems will be visible at startup because no recompilation required during upgrade
• You can only send the corrected assembly to the customer as a last resort
• The release date of the assembly as a conditional version for identification
Our team stopped at the second version because Often it is necessary to check new technological assemblies on applied projects, and there is no desire or time to recompile them. Server builds are configured in Team Foundation Server and only the server build can be transferred for deployment.
We considered 2 approaches to source code storage:
Branches for adding features, branches for implementations
A plus:
• You can make a small change directly to the desired version.
Minus:
• Need to do a lot of code merges
Single version without branches
A plus:
• No merger
Minus:
• The disassembled version cannot be fixed quickly.
We stopped at a simpler second version. We try not to make long-term changes without Check-In, but Check-in should not destroy the performance of our solutions.
A few words about OpenSource in a commercial product: if you wish, you can find OpenSource solutions with a good license to solve any necessary problem. However, when making such a decision, you should think hard about how changes will be made there:
1. Edit everything yourself locally
A plus:
• You can do anything
Minus:
• When updating the official version, you will have to merge your changes yourself.
2. Send updates to the project supporting team.
A plus:
• Additional check for quality patches
Minuses:
• Risk that support will cease.
• M. b. long term review and release fixes
• Corrections may be rejected for one reason or another.
Sometimes you have to make a choice in favor of a paid decision only because the policy of making changes (at least error correction) is fixed in the license for which money is paid.
How to protect against disassembling code that is available is available from both application and technological assemblies?
You can obfustsirovat before assembling the technological assembly, but then we get:
A plus:
• Application programmers will not pry into the process code and ask stupid questions about it
Minuses:
• Difficulty debugging for application developers
• Possible instability during development
• Obfuscation and verification will be performed twice.
The second option is obfuscation of the complete application with all technological assemblies.
Pros:
• Obfuscation at the application assembly stage
• Ability to debug an application by entering the framework code. Verification of the application operation is performed once
Minus:
• Protection of technological assemblies involved in application programmers
We apply the last option.
I will not talk about unit tests - this is necessary for any layer, not just the technology. However, always technological code must be written taking into account the performance requirements and work flawlessly in multi-threaded mode.
Testing of technological projects, as a rule, is impossible without an end application in which the subject area is described. In some cases, in order to understand exactly where the error is hidden, you have to create your own separate application and try to repeat the necessary conditions. Only this option helps to separate the "pickups" from the application code.
I can note that working on technology is a very exciting and, in some places, fun activity. The main thing - do not get carried away too much and sometimes communicate with end users, trying to understand their needs. Thanks to everyone who read this long monologue to the end.
In this article I want to share my experience regarding the process of creating a technological layer.
Why do we need frameworks?
There are several reasons why the company contains several people who do not bring direct profit; in my opinion, the most important thing is:
1. Reuse ideas and code. The technology department collects best practices as you complete various projects and distributes them to all teams. In this case, there is no need to arrange each connection with each in order to be aware of what is new and useful that a neighbor has. After all, a neighbor can immediately share with technologists and not to bother much about universalization, while technologists have the opportunity to inform everyone else (a list of improvements is regularly made that have fallen into the next version of technological products).
2. Unification of solutions. Programmers sometimes change jobs. To deal with the negative aspects of such situations, the technology provides some unified methods for solving typical problems. This reduces the number of bicycles and reduces the time of entry of a new person into the project.
Growth points or maturing technology
A little bit about how a new technological direction is born. From time to time, the applied platforms are modified and you have to create technology for these new items.
')
Kindergarten
Due to the fact that our product is not a complete application, to begin with, any subject area for the prototype project is selected (1-3 pieces). This application serves only to test solutions and approaches that we plan to implement. For fun, the last such system we used to account for domestic cats in the apartment. A special feature of the prototypes is that the code is quite mixed up and it is not clear where the technological layer is, and where the applied logic begins. On this "cat" project, all the necessary components, generators, user interface elements are worked out. Only technologists work on the project.
School time
When it comes time to test the technology on something more serious, the “sacrificial” project is chosen from the commercial one. As a rule, it has a 1.5–2 times longer period that can be set aside for development than a typical project (everyone understands that there are risks, because the technology is used by us for the first time and the company itself pays for technological work, and not customer). A feature of this project is that the technology code is initially in a solution, and as it is developed it is divided into technology projects and application code (which depends on the subject area). The project work and technology and application programmers. It should be understood that the main approaches are still being debugged and not everything in the technology is 100% ready, so the project will be with innate “bikes” that complicate support. Sometimes these “bikes” are replaced with standard technological solutions, sometimes they remain until the end.
Higher education
At this stage, we believe that the technology is ready for widespread use. Go to the normal "operating mode". Application developers no longer need direct assistance from technologists. Technological assemblies are delivered in compiled form. Refinement tasks and error messages are transferred to the TFS technology project.
Professional Development
From time to time, serious improvements of the technology are required, as a rule, this is adding functionality, for example, the general reporting subsystem, the authorization subsystem, the logging subsystem, etc ... for users, it appears already in its finished form. As a rule, such new products immediately become popular with all projects that are actively developing. It's funny that sometimes, at the end of the development of a regular feature, an application programmer comes in and asks for it to be implemented.
Retirement
With the development of new directions, the old ones gradually disappear. Projects are closed, there are units of projects that require support. Together with them, the technology "retires", goes into Bug driven development. New features are no longer added, only fixes old bugs that have been accumulating over the years. Some bugs are fixed with crutches in the application project, since it is no longer advisable to make changes to the technology. Changes in technology should always be made with an eye to all projects that depend on it. One wrong move and problems are guaranteed both to the people on the applied project, and after them to the technologists.
If something needs to be rewritten, then for a technology with multiple entries in projects:
• We collect requirements that were satisfied by the old functionality (as a rule, after a series of improvements and minor fixes, there is no complete and up-to-date documentation for the programmer)
• We plan work (once again we look at the budget and evaluate the feasibility)
• We write tests to fix the state. According to them we will determine that what has worked before will not break.
• We write functionality
• Check the quality and compliance with the stated requirements
The technical side of the issue
Now a little about techniques and techniques. It's about the Microsoft .NET Framework, because our solutions are built on this platform.
Expansion points
The technology should not limit programmers, but give advantages over a solution from scratch, and sometimes completely ready-made solutions. In order for a programmer to show his individuality where the technology is “too standard” we use the following approaches:
• Inheritance. There is a base class that implements the standard behavior and is located in the technological assembly. The code generated for an application project is inherited from the base class and in special methods it assumes a change in the standard behavior.
• Developments. Technology classes contain a series of events whose processing allows you to change behavior. It should be remembered that an event can be subscribed to many times.
• Delegates. Similar to events, but there can be only one implementation. This prevents unwanted triggering several times.
• App.config. Various trifles fall into the configuration file. As a rule, these are some flags and paths (do you need to write debugging information, connection strings to services, etc.).
• DI. This option implies redefinition of the whole module.
Event logging
From the technology layer, there is rarely normal access to the user interface. As a rule, if something went wrong, it indicates that there is an error either in technology (yes, they are and we understand it), or in the application code (ideally, you can indicate to the application programmer clearly what he is doing wrong and in which place). In any case, you need to somehow signal this. We use Log4net. To highlight technological messages, you can use either a separate logger, or fix the level of errors, for example, Warn, to the technology.
XML comments
XML comments in the technology code are extremely important. It is necessary to prescribe them for public classes, methods and properties. This will allow the generation of help (http://sandcastle.codeplex.com/) and provide hints in application projects thanks to IntelliSense. Do not forget to specify in the project settings whether to generate an xml file with comments.
Breaking changes
It is extremely rare that such a change in technology occurs, which requires the intervention of an application programmer to update the technology layer. If such an update can be predicted, then you need to warn in advance, for example, using the Obsolete attribute:
[Obsolete (“This function will be removed in the next version, use the function”)]
Sometimes you have to go on about and apply the rule: “it’s better to add 1 time than to update 2 times”, i.e. if the method signature is used in 1000 places in each application project, changing it in technology is not a good idea. The most important rule: always a detailed description of possible problems with the update - forewarned, then armed.
Documenting
For process assemblies, documentation is essential. To store information, we use the Wiki engine with the right to write to the technologists. To keep information up to date, 2 rules apply:
1. Made a deal - write it in the Wiki!
2. Answered the question - write it in the Wiki!
Description Levels:
1. For yourself (as arranged)
2. For programmers of applied projects (how to use methods and classes)
3. For system users (how to use components)
Framework structure
Sometimes you want to minimize the number of assemblies issued for projects. This allows you to deal with incorrect updates when you need to update several assemblies at once, otherwise something will not work and minimizes the risk that something will be forgotten to be installed by the customer. However, different projects need slightly different functions. What can be applied?
1. Links to cs-files between projects (work with projects becomes a bit more complicated)
2. ILMerge (this can be assigned to the Build server, but debugging is difficult)
3. Assemblies in resources (with a separate assembly loading mechanism)
Signing assemblies or Strong name
Another iron rule: all process assemblies must be signed. This gives us the following benefits:
1. Signed assemblies are uniquely identified in the GAC.
2. You can only sign if all dependent assemblies are signed (application assemblies will not be able to sign, if technological assemblies are not signed)
3. Some degree of protection (to change the code, you will need to rebuild all application builds)
If it so happens that the code from the assembly is lost, then signing can be done by rebuilding the assembly. Rebuilding is done this way (to specify a file with a key, look for a special parameter):
1. ildasm.exe sampleA.exe / source /out:sampleA.il
2. ilasm.exe sampleA.il / exe /out:sampleA.exe
Release Strategy
The more often a product is released, the less it seems to users that programmers do nothing. We tried 2 versions of the release of technological product versions:
1. The latest builds are the best.
Everything that is hosted in SourceControl is considered verified and application programmers can take any server build of technological assemblies. At the same time, technologists before each Check-in must carefully check their changes.
Pros:
• Instant posting of changes
Minuses:
• Each Check-In requires an extended check with lots of different functionality that may suffer.
• Errors can easily penetrate the version at the customer
2. Periodic release of stable version
Release of the version of technological assemblies on a periodic basis.
Pros:
• More reliable integration verification
Minuses:
• There is a risk of losing control of the code in the middle of the iteration.
• It takes time to check everything again, no instant fixes
During the release of the version for 1-2 days, all technologists are distracted by checking and correcting the errors found. This is the option we now apply.
Assembly Versions
There are 2 strategies for naming build versions:
1. Variable build versions
• Updating of applied projects only with recompilation of the project (inability to send a patch to the technology directly to the customer)
• Part of the problems will be visible when compiling (relevant for complex dependencies between assemblies)
• Problem with type serialization (serialized data stored by the customer may cause problems when updating build versions)
2. Fixed versions of builds
• Update simple substitution of assemblies
• Problems will be visible at startup because no recompilation required during upgrade
• You can only send the corrected assembly to the customer as a last resort
• The release date of the assembly as a conditional version for identification
Our team stopped at the second version because Often it is necessary to check new technological assemblies on applied projects, and there is no desire or time to recompile them. Server builds are configured in Team Foundation Server and only the server build can be transferred for deployment.
Branching: Pros and Cons
We considered 2 approaches to source code storage:
Branches for adding features, branches for implementations
A plus:
• You can make a small change directly to the desired version.
Minus:
• Need to do a lot of code merges
Single version without branches
A plus:
• No merger
Minus:
• The disassembled version cannot be fixed quickly.
We stopped at a simpler second version. We try not to make long-term changes without Check-In, but Check-in should not destroy the performance of our solutions.
External assembly
A few words about OpenSource in a commercial product: if you wish, you can find OpenSource solutions with a good license to solve any necessary problem. However, when making such a decision, you should think hard about how changes will be made there:
1. Edit everything yourself locally
A plus:
• You can do anything
Minus:
• When updating the official version, you will have to merge your changes yourself.
2. Send updates to the project supporting team.
A plus:
• Additional check for quality patches
Minuses:
• Risk that support will cease.
• M. b. long term review and release fixes
• Corrections may be rejected for one reason or another.
Sometimes you have to make a choice in favor of a paid decision only because the policy of making changes (at least error correction) is fixed in the license for which money is paid.
Obfuscation
How to protect against disassembling code that is available is available from both application and technological assemblies?
You can obfustsirovat before assembling the technological assembly, but then we get:
A plus:
• Application programmers will not pry into the process code and ask stupid questions about it
Minuses:
• Difficulty debugging for application developers
• Possible instability during development
• Obfuscation and verification will be performed twice.
The second option is obfuscation of the complete application with all technological assemblies.
Pros:
• Obfuscation at the application assembly stage
• Ability to debug an application by entering the framework code. Verification of the application operation is performed once
Minus:
• Protection of technological assemblies involved in application programmers
We apply the last option.
Testing
I will not talk about unit tests - this is necessary for any layer, not just the technology. However, always technological code must be written taking into account the performance requirements and work flawlessly in multi-threaded mode.
Testing of technological projects, as a rule, is impossible without an end application in which the subject area is described. In some cases, in order to understand exactly where the error is hidden, you have to create your own separate application and try to repeat the necessary conditions. Only this option helps to separate the "pickups" from the application code.
Conclusion
I can note that working on technology is a very exciting and, in some places, fun activity. The main thing - do not get carried away too much and sometimes communicate with end users, trying to understand their needs. Thanks to everyone who read this long monologue to the end.
Source: https://habr.com/ru/post/150766/
All Articles