Overview of the software development process

Introduction

Before offering an overview of the development process that has developed as a result of the accumulation of experience in recent years, I would like to make a few general explanations that seem to me to be significant.

I have been working in IT for the last 15 years, although I started programming much earlier. The main direction of my activity as a system architect was the organization of program development, development of concepts and high-level architecture, and control over the implementation of the concept throughout the project. In addition to managing software development and building architecture, I occasionally tackle complex technical problems and write some critical sections of code where you need not only knowledge of the language itself and the development environment, but also their internal organization, sometimes presenting unpleasant surprises.

The projects I work on are most often associated with custom or investment software development. I also had to work with embedded software and programs focused on the release of "hits" (which, with the light hand of Joel Spolsky, I call the game software, although in fact some game projects are closer to investment ones).
')
Custom software can be designed for internal or external customer. Exclusive rights to the developed system are received by the customer, and work on the development of the system can later be transferred to another performer.

Unlike custom software, work on investment software is carried out by the contractor himself with the money of an internal or external investor. As a rule, the rights to the system code remain with the contractor, which stimulates continuous work to improve their product and the consistent release of versions with more advanced functionality.

The embedded software comes with the hardware and, roughly speaking, is not subject to maintenance, since recalling a batch of devices by the manufacturer is very expensive and therefore exceptional.

The development of game hits also contains almost no maintenance phase. In addition, users of gaming programs, even when faced with an error in the game, very rarely download an updated version. Therefore, the development of games, as a rule, has its own economy and its development process.

Our customers are authorities, large state and commercial organizations and, of course, ourselves. Therefore, in terms of custom software, there is often some difference in our process between product development processes for internal and external customers. Some nuances I will point out in this article. The level of formalization of relations with the customer varies widely from project to project. In general, the larger the project budget, the higher the formality. A government customer or large commercial enterprises (especially with state participation) usually have legal restrictions on the formation, placement of an order and acceptance of work results. Another limitation of large organizations is the fact that their personnel, who are the source of requirements and the main user of our systems, have very limited accessibility for executors, if only because of their employment. However, for small organizations, the level of formalization falls and sometimes goes to the opposite extreme, where there is an insufficient level of responsibility of the customer within the project.

The other side of our custom projects is high functionality requirements. This is a high load on all systems, and a large geographical distribution, and high demands on the accuracy of calculations with a very limited time frame. Often in our projects there are elements of research and creative search aimed at solving non-trivial design problems. Sometimes we have to combine different methodologies in one development process, for example, inserting one or several stages of an almost pure scrum into a general process close to RUP, generating something like a project in a project. This allows us to maintain a low level of user involvement associated with the nature of the project, with the flexibility of development in conditions of high uncertainty of requirements. In this regard, it is for me the preparatory stage that is important, during which you can choose the necessary methodology and build the optimal development process. I described one of the examples of applying a flexible methodology in the article “Using agile in developing a project for a state customer” .

As an example of work on an investment project, I can cite the development of an integrated security system that we created as a “boxed” product. Under my leadership, four versions of this system were successively released, the users of which were various commercial and state organizations, including the Moscow mayor's office, Sistema, banks, business centers and, of course, our own office. The first version was not very successful, but we had a development strategy that allowed us to successfully capture the market and survive the difficult times of crisis. The experience of working on this and several other investment projects was also taken into account when forming the development process I used.

Our process is a sequence of specific steps. The software classification I quoted is done only to show a possible difference in the organization of the development of various software tools. Making an overview of the development process, I will focus only on the differences of the process itself with regards to different types of software. However, we must remember that the differences between the processes of developing different types of software are much deeper, therefore, when planning each stage, these nuances should be taken into account.

It is important to understand that the transition of the process from one stage to another does not have a clear boundary. As a rule, the work of the next stage begins as 80-90% of the work at the previous stage is completed. This especially concerns the development of requirements, when in some cases the removal of uncertainty occurs only towards the end of the project. Of course, the presence of such uncertainty in the project is a significant risk and should be under constant control.

Custom Software Development Process

An overview of the development process will begin with the most common case - the development of custom software. The process diagram is shown in Figure 1.


Figure 1. Custom software development process.

Work on the project begins with a preparatory stage. The purpose of the stage is to create some concept of the future system on the basis of the customer's proposals and, starting from this concept, to assess the relevance and feasibility of the project. If the decision on engaging a performer is made by the customer on a competitive basis, then the preliminary stage is in fact the stage of preparing the potential performer for the competition, including the formation of the necessary documentation.

No need to spend time and resources on a project whose concept is deemed unclaimed or unrealizable. Such a project must be completed. In some cases, some iterative work is required with the customer to correct the concept of the project, until either an acceptable balance of customer requirements and costs of the contractor is reached, or a decision is made to minimize the work.

The project, the concept of which looks acceptable for implementation, is entering the stage of developing requirements. At this stage, the contractor must create a list of all the obvious and hidden needs of the customer. It often turns out that the customer has either not decided on his needs, or his needs conflict with each other, with the capabilities of the customer or with the capabilities of the performer. The objectives of the stage are the identification of all hidden needs, the resolution of conflicts of requirements, the formation of a holistic technical solution and an analysis of the feasibility of the prepared solution.

Sometimes clarification of requirements leads to a revision of the project concept. If, after clarifying all the requirements, it is not possible to find an acceptable technical solution, the project has to be curtailed or put off for some time, waiting for more acceptable circumstances.

If a technical solution is found, the contractor starts developing the architecture of the future system. The goal of the stage is to define a high-level logical and physical architecture that fully covers all customer requirements. When developing the architecture, the concept and requirements and preliminary technical solutions are reviewed and clarified, which makes it possible to prevent the most dangerous risks.

After completing the design of the architecture, it is necessary to revise the main parameters of the project and decide whether the executor is able to complete the project. It is useful to abandon unnecessary and too cumbersome functions at the architecture development stage. Optimizing an architectural solution often helps fit into acceptable project parameters. In other cases, a more radical reduction in the functionality of the system being developed is required. However, even stopping the project at this stage, if it occurs for good reasons, should be perceived as a victory: the continuation of work in this case can only lead to even greater losses.

If a balance has been found, and it has been possible to create an acceptable system architecture, the contractor can proceed to the implementation and delivery of the system. Implementation can take place in one or several stages. For small projects, one-stage delivery of the entire system functionality may be quite acceptable. However, the larger the project, the higher the dependencies of the subsystems within the system being created. Under these conditions, the implementation should be divided into several stages so that at the end of each stage the development team has a product ready for delivery. In this case, the most important, fundamental functionality should be developed in the early stages, and add-ins that work on top of these main components should be implemented later. In this case, the most dangerous errors for the system will be corrected in the first stages, and the risk that the applied functionality of the system will be based on an unstable basis will be significantly reduced.
After the delivery of a fully completed system, the custom design project usually proceeds to the trial operation phase. The purpose of this stage is to check the quality of the developed system in real conditions of operation. As a rule, at this stage, the contractor, together with the customer, takes measurements of quantitative metrics to determine the quality of the system created. First of all, functional characteristics of quality are checked, then non-functional. If there are inconsistencies, the performer adjusts the system code.

Fully debugged and customized system is introduced into industrial operation. As a rule, the contractor must accompany the system, at least during the warranty period. Detected inconsistencies should be corrected. Users and customer service staff should receive operational advice.

Finally, the moment comes when the system ceases to arrange the customer for any reason. The stage of system decommissioning begins. However, for custom software, this stage is not always relevant, because the customer can use his exclusive rights to the system and remove the contractor from further work on the maintenance and development of the system even before it loses relevance.

Any project ultimately comes to an end. The project termination phase aims at analyzing the results, making changes to the development process based on the experience gained and adding new effective solutions and precautions to the developers knowledge base, as well as new off-the-shelf components that can be used in future projects.

It remains to note two more stages of the development process. It happens that circumstances do not allow the project to continue, but the results of the work done show that the project may have a future. To close such a project is premature. Therefore, instead of completely stopping the work, the contractor may temporarily suspend the project activity, recording the results achieved. As soon as circumstances allow, the project can be resumed, having the infrastructure re-established, having returned the developers to the project and restored the state of the project. It is important, however, to resume work from the stage at which the project was interrupted, re-conducting an audit of the results achieved.

Investment software development process

The process of developing investment software is different in that the work on several versions of the product can go in parallel: while the first version is being followed, the second is already being implemented, and for the third the requirements are formulated. The process is shown in Figure 2.


Figure 2. The process of developing investment software.

As it is easy to see, in the development of investment software, the same steps take place that were discussed above for the process of developing custom software. But the difference is that the stages do not apply to the whole product, but to a separate version of the product. An exception is the stage of project termination: the project cannot be completed while work is in progress on at least one version of the product.

Pay attention to the beginning of work on the next version of the product. This moment comes as soon as the stage of creating the architecture of the current development version is completed. Prior to this, at the stages of forming requirements and creating architecture, as a rule, there is a discussion about which functions should be implemented in the current version and which ones should be transferred to the future. And only when the requirements for the current version are formulated, reviewed and confirmed by the system architecture, does it make sense to think about the next version.

In addition, after the development of the architecture, as a rule, analysts and architects of the project have some leeway, because during the delivery stages, the main burden falls on programmers. This freedom can be used to work out the concept and requirements for the next version.

In principle, it is possible to postpone the start of work on the next version to a later date. For example, it is quite possible to first enter the current version into experimental or even commercial operation, and only after that begin work on the next version. But it must be remembered that such a decision is not applicable in case of high competition: you will simply be outstripped and squeezed out of the market. The decision must be made on the basis of the whole complex of circumstances affecting your business.

Speaking about the process of developing investment software, you need to understand that the work on several versions has a number of explicit and hidden interdependencies between parallel branches of the process.

First, corrections of inconsistencies found in the earlier version should be made in the version where they were found, and in all later versions, including those developed. This applies not only to the program code, but also to all other project artifacts: technical and user documentation, help system, assessments and work plans, etc. Moreover, corrections must be made immediately, because you will not be able to reduce the cost of corrections, but if you do not make corrections immediately, their cost at later stages may increase by tens and even hundreds of times.

Secondly, for parallel work on several versions, a special project infrastructure is needed, including organizing control of code versions and documentation, control of tasks and inconsistencies, automatic build and testing utilities, etc. We can not allow the work on one version of the product to block the execution of tasks on other versions only because the project infrastructure does not allow starting two assembly processes simultaneously for different versions of the product.

Special attention should be paid to the stands on which testing is being carried out: all versions of the product that were previously released (at least those versions that are accompanied) and all versions that are currently being developed should be deployed.

Thirdly, the same participants can be simultaneously involved in the work on several versions. There is a big risk that a key employee may become bogged down with work on one version of the program and allow significant time overruns in tasks related to another version.

Fourthly, there is a reverse situation, when the personnel working on one version do not know anything about what decisions are being made as part of work on another version. Part of the problem is resolved if the fixes of all the documentation and code are immediately extended to the later versions I mentioned above. But the matter should not be limited to corrections. It is necessary for the team working on one version to understand why certain decisions were taken when working on another version. This requires a knowledge base for developers - a special information system in which all the problems that developers have encountered while working on a particular version of a product, and ways to solve these problems should be described. The knowledge base should send notifications to all project participants about new entries. You can not let the interaction of two teams working on different versions of the same product take their toll.

Firmware development process

As noted above, firmware is different from custom software in that it is extremely difficult to maintain.

Let's say you are releasing programs for refrigerators. Once the software is delivered to the manufacturer, tens of thousands of devices begin to diverge around the world, and you have no idea where they will be. And if one of the refrigerators fails due to the fault of your software, it is easier to pay a penalty than to return the refrigerator to the factory and carry out diagnostics. Of course, you can prepare engineers for dealerships who can conduct on-site diagnostics and replace the firmware of your system, but this is still very expensive.

Thus, when developing firmware, several important limitations arise at once.

First, the delivery is carried out in only one stage: no one will embed a half-working program into the devices.

Secondly, when delivering, you should pay particular attention to the quality of the program, since it will be very difficult to change it from the moment it is introduced into the iron box. Particular attention should be paid to the trial operation phase, when the program is introduced into a limited batch of devices, and these devices pass complex tests in various operating modes. You must collect as much information as possible about the dynamics of your system behavior, analyze this information and refine the software.

Thirdly, when the device with your software went into a series, you have very little opportunity to correct errors. In fact, such corrections are possible only in the case of software malfunction, which leads to the inoperability of the entire batch of devices, which is why the manufacturer will have to withdraw this batch, and you will get a big black spot on your reputation.

Finally, fourthly, the firmware does not have a decommissioning phase. The program is simply thrown away with the device. Therefore, as soon as the batch of devices in which your software runs, the warranty period expires, you can proceed to closing the project.

The firmware development process is shown in Figure 3.


Figure 3. Embedded software development process.

Game development process

The game software was allocated by me because of the specifics of their production and operation. The gaming software business is based on the release of hits. One successful hit pays for the creation of several games that remain unnoticed by users. Therefore, the development process of one game is interconnected with the development processes of other games.

Another factor highlighting the production of games is the fact that the game is interesting to the user either until he passed the last level, or until he had a fatal error. This means that he will not buy the second version of the game or even download it for free just to fix a few errors.

These factors affect the development of gaming software. The process is shown in Figure 4.


Figure 4. Gaming software development process.

It should be noted the following features of the gaming software development process.

First of all, the quality of the concept is crucial in the production of games. If the concept of the game does not allow you to create a hit, then further work is meaningless. The situation when most projects end in the preparatory stage is typical for developing game software.

When developing requirements and architectures for gaming software, the lessons learned from previous projects are often reused. In this regard, the project termination phase also receives additional weight, when all useful developments should be recorded in the developers knowledge base.

The supply of gaming software occurs within one single stage. Even if you first create a kind of core, "engine" of the gaming system, its work can not be verified without implementing all the functionality of the system.

For gaming software there are no stages of trial operation and decommissioning. Games immediately go on sale, and after use are simply removed by the user as they lose interest in them.

Conclusion

Within the framework of this article, I tried to make an overview of the “top level” of the application software development process. Each stage of the process, of course, needs a separate discussion with the obligatory consideration of the features of the software being developed.

I note that the process scheme considered here is the result of the generalization of my personal experience in the development of various software tools. Like any generalization, my scheme is an abstraction. And, like any abstraction, it has its limits of applicability. You can not mindlessly apply this scheme to a specific project. It is important to understand that each project has its own nuances that affect the organization of the development process. And therefore, for each project, the scheme presented here needs to be adapted, and in some cases it will be necessary to develop a fundamentally different approach.

Continued: Preparatory stage of software development