📜 ⬆️ ⬇️

Dependency management in a complex Agile environment

Translation of the article “Dependency Management in a Large Agile Environment” .

Short review


The Salesforce.com development department includes more than 30 Scrum teams working together on common code in the same version control system branch. The article describes the methods used by salesforce.com to scale the Scrum approach and to manage inter-team relationships.

1. Introduction


In October 2006, the grandiose transition of the salesforce (R & D) division of salesforce.com from the waterfall model to flexible methodologies based on Scrum began. At that time, 10 months had passed from the previous major release, and the release date of the new one was postponed five times already. Many were upset that the product is rarely produced and with serious delays. We did not wait for the release to finish, reorganized the existing teams into Scrum teams and released the release in February 2007 using Scrum processes. Since then, using our new flexible approach, we have already released five major releases (lasting 3-4 months) of our set of SaaS applications and the Force.com platform. Each of them took place exactly on the scheduled day.
')
During the reorganization, we followed the Scrum recommendations for individual teams, but did not pay much attention to the interaction between the teams. Forming teams, we sought to minimize the dependencies between them, but the code did not change overnight, so many interconnections remained. Pretty soon we introduced Scrum-of-Scrum meetings. These meetings helped to discuss problems and the state of affairs, but meetings alone were not enough. While working on the last five releases, we tried and polished additional approaches that improve team interaction. Later in the article we will discuss some of the difficulties with dependency management and how we have overcome these problems.

2 Team structure


Product development at salesforce.com consists of three divisions: Applications, Platform and Core Infrastructure. These three units contain a total of more than 30 Scrum teams. In each Scrum team there are developers, testers and Product Owner, as well as Scrum Master (usually this is the head of developers or testers, or Program Manager). For complex features, Scrum teams also involve representatives of other functional teams, such as System Testing, Documentation, UI Design, Usability, Technology Operations, Release Engineering.
Usually, all Scrum teams work on the same release in the same code branch. Our products are interconnected, so the features of different teams can intertwine very closely - both functionally and technically. In terms of technical implementation, many teams use common code. In terms of functionality, teams working on features for the same end user should work closely together to create a harmonious and coherent user experience.

3 Why is managing dependencies difficult?


When so many teams are working on one release, introducing common features and changing the common code, it is impossible to foresee all the problems, surprises, changes, failures and successes that will occur in the development process. Such unpredictability is one of the reasons why managing dependencies is so difficult. Further in this section we will note a few more reasons. Any decisions should be directed to these fundamental conflicts and challenges.

3.1 System complexity

Identifying dependencies is the first step to managing them. Like many mature products, salesforce.com is too complex for one person or team to know and remember all the relationships. In order to identify and realize addictions, we increasingly have to rely on the collective knowledge of many people. It is extremely important to combine such knowledge and connect people who have information with people who need it. Experts with a broad understanding of the system play a key role in determining the dependencies and consequences of the proposed changes. However, as the system grows and becomes more complex, it is difficult to avoid specialization and fragmentation of knowledge. Moreover, with the current number of changes, it is not advisable to ask the experts to check the details of each new feature. We need a way to identify changes that have a high probability of affecting (or depending on) the work of other teams in order to pay special attention to such changes. Of course, this is not so easy, because sometimes relatively small changes seriously affect the entire system.

3.2 Conflict Priorities

Dependencies can lead to conflicts between commands. If one team wants the other team to do something, then the Product Owners of these teams discuss and determine the priority of this work. If, following the results of the discussion, it is possible to agree on the deadlines, then in the future, managing this dependency is quite simple. If a clear agreement is not reached, or if the priority of work is still low, then the uncertainty and the level of risk around this dependence increase.

Also, conflicts arise when one team makes a change that adds work to another team. For example, when one team changes the architecture, so the rest need to adapt to these changes. Change does not always bring obvious benefits to other teams, so sometimes people get offended and resent the appearance of extra work, especially when it appears suddenly. We try to identify such dependencies in the early stages, but some of them inevitably appear later - as a result of changes already made or after a revision of priorities.

3.3 Changes in plans

One advantage of Scrum is the ability to change team priorities for each sprint. However, this complicates the work with dependencies. You can not just identify and think about all the dependencies at the very beginning of the release cycle. Relationship analysis is more like a continuous ongoing process. As work progresses, dependencies appear and disappear, while teams clarify the details of what needs to be done. Therefore, the process of managing such dependencies should be dynamic and active.

3.4 Short and intersecting release cycles

In short release cycles, there is less time for coordination between teams, so relationships and their effects need to be identified early. In salesforce.com, release cycles overlap, in the sense that planning for a new release begins before the previous one is completed. This was done intentionally, because some teams are already free and can begin work on a new release. However, other teams may be late with planning and skip the discussion of dependencies for the next release. This is a serious problem, given the importance of collective knowledge for clarifying relationships.

4 Special approaches


In this section, we will discuss the specific approaches that we use to manage dependencies and cope with the problems listed above. Common strategies used in these approaches:



4.1 Release Start

About a month before the release of the current major release, we have a starting meeting for the next release. At the meeting there are all employees of the company. Vice-presidents of business units tell in general what features each team plans to work on in a new release. This collection noticeably helps to manage relationships. First of all, it encourages teams to prepare their initial plans for release by the scheduled date. If a team has an unfinished work from the current release, then this team starts to delay the planning of the next release; in this case, the meeting helps to complete the planning by the deadline. It is difficult to identify dependencies and discuss obligations with a team that has no idea what it will do. The starting meeting serves as a point of command synchronization and helps provide a productive discussion of team-team interactions.

In addition, after the meeting, plans for release teams become well known. We strive to achieve high awareness of what all the teams are doing, because we believe that it makes people think about and discuss relationships. All R & D executives are participating in the launch meeting, which helps to attract people and focus attention on release planning. A high level of participation helps align expectations and inform the board about release plans.

Of course, plans can change and change in the process. Not so long ago, we began to review the updated presentation of the launch meeting during the monthly sprint reviews. The updated presentation shows which features have been removed or added, and it helps a lot to maintain the clarity and publicity of the plans during the release.

4.2 Session to identify relationships

Having prepared the initial plans for release, the teams can more substantively discuss interconnections, and many such discussions arise spontaneously. In addition to this, we found it very valuable to hold a more formal session of identifying interconnections, when representatives of all teams work together. It is most useful to hold such a session shortly before the launch meeting of the release, in order to improve the quality of the plans announced at this meeting.

The session of identifying relationships is pretty simple. Representatives (most often Scrum Master or Product Owner) of all Scrum teams gather in a room with a large board. In the first part of the session, all participants simultaneously draw on the board two diagrams of dependencies between the teams. One diagram represents the teams that need the other team to do something for them. The second diagram depicts commands that do something that will affect the work of other teams. We use a few simple rules for these diagrams:


At first, confusion reigns, while each draws its own relationships, and observers ask questions and point out additional dependencies. Twenty minutes later the chart stabilizes, and the comments dry out. After that, we ask the representative of each team to briefly describe their dependencies.

In a very short time, we create a complete picture of the relationships in the release. In fig. Figure 1 shows a typical session result after being more readable. Hot spots (i.e., commands with a large number of dependencies) are immediately detected. Also see commands that have inconsistent dependencies. Such teams are most at risk of not reaching their plans until their interconnections are agreed.


Fig.1. Sample command-line dependency diagram

4.3 Open discussions

Recently, during the week after the launch meeting, we have open meetings, the main purpose of which is to create a platform for discussing issues and problems related to the team’s plans for release. We ask that at these meetings there should be at least one representative from each Scrum-team, otherwise they are not necessary to attend. Usually at the beginning of an open meeting, participants suggest topics for discussion. Then, within 45 minutes, discussions are held on topics in groups, and then the groups tell the others about the results of the discussion. Most often, participants want to know the details of the new and useful functionality, as well as the changes that will affect other teams. Participants in open meetings noted the instructiveness of such discussions; in addition, people found it interesting to communicate with colleagues from other teams with whom they do not interact in their usual work. Open meetings meanwhile a new format for us, so that we try different options to understand what works better, for example:



4.4 Design Review

The main purpose of the functional design review meetings is to increase the overall level of quality and convenience of our products by:


These team meetings focus on the design of functionality and user interaction for features that we consider key, the most complex, or particularly prone to risk. At such meetings, we strive to analyze and identify previously unnoticed inconsistencies and dependencies.

There are two main types of such meetings. In the early stages of the release cycle, participants focus on design concepts and strive to achieve consistency and effective interaction between existing and new features. At this stage, product owners and user interface (UI) designers usually participate in meetings, and occasionally representatives from other functional teams, such as development and QA.

Around the middle of the release cycle, the composition of participants changes: now they are mainly representatives of QA and, to some extent, development, and discussions focus on implementation details. These meetings provide understanding and clarity regarding:



4.5 Virtual architecture team

The virtual architecture team (Virtual Architecture Team, VAT) is “virtual” because it includes developers from all Scrum teams. Team members work in VAT without interrupting their main responsibilities within the Scrum team from the Applications, Platform or Core Infrastructure business units.

VAT is responsible for the support and development of our software architecture. The team determines the road map for the development of the architecture, reviews major changes in terms of architecture, defines coding standards to ensure compatibility and maintainability of the code.

VAT manages the backlog of important architectural projects and the necessary refactoring. As the product develops, we have to redesign the features and get rid of the old non-optimal code. Sometimes developers do not want to change the programs that are already running, even when they understand that the program code leaves much to be desired. Product Owners prefer to see how new features are developed, and not how something that already works is redone. To counter such sentiments, each of our business units must plan at least 20% of the time in each release for changes recommended by VAT.

VAT is collected twice a week for two hours to review the technical implementation of the products and features created by Scrum teams. Teams working on the most difficult features of the release are required to submit their VAT. VAT provides the Scrum team with feedback on how their technical solutions will affect other teams, and what other teams' development may affect the feature. VAT focuses primarily on technical implementation, especially on scalability and performance. If the design of the feature requires major changes, then the Scrum team is obliged to re-submit the feature in the same release cycle and show VAT how they changed their design.

4.6 Continuous Integration

Our web infrastructure of automated assembly, testing and sorting (triage) provides continuous integration of the build system and allows you to track the status of each line of code. This infrastructure is a key aspect that allows all developers and QA engineers to work with a common code base.

The main test suite built on the JUnit extension allows you to create functional tests using the Force.com API. In addition, we have a user interface testing framework that uses Selenium to automate test cases that need to be done via the UI.

Here are some of the fundamental principles that determine our approach to automation:

  1. Give developers quick feedback so they can see the results of their changes. Each commit starts the build build and execution of a test suite. In the event of an error, responsible engineers immediately receive a failure message. The basic set of tests is performed in half an hour, and the developer receives the results of checking the changes he has made. Periodically, an extended test suite is run throughout the day;
  2. Quickly repair and test builds . To do this, we have the role of "assembly wizard", changing owners every week. Usually this role is performed by two closely interacting people - a development manager and a senior developer. The build wizard is responsible for checking build results, tracking test history and failures on various lines of code, and assigning specific bug fixes to specific developers. We have a special panel that displays various metrics based on the test results collected in the database (share of successful tests, build time, number of failures, etc.). The assembly wizard uses this data when preparing an assembly status report;
  3. Maintain a high level of coverage by automated tests . Usually, Scrum teams tend to cover 70-80% of the code with automated tests. One of our most important assets is a large set of autotests, which play a key role in delivering quality releases on time every 3-4 months.


4.7 Scrum-of-Scrums

Each business unit has a weekly Scrum-of-Scrums meeting, there is also a "Scrum-of-Scrum-of-Scrums". Sometimes we organize an additional Scrum-of-Scrums for a group of teams that closely interact when working on a common goal. We tried out several different Scrum-of-Scrums formats. We started with the standard “4 questions” format, when the teams reported that they had done what they were going to do, that they were blocked, and whether they were doing something that affected the work of other teams. At first it worked, but it soon became tedious and boring due to the large number of teams. Then we moved to a more open format of self-organization, when participants suggested topics for discussion, writing them out on the blackboard at the beginning of the meeting. This made the participants take responsibility for the content of the rally and led to more productive discussions. The issue of command-line dependencies is often raised during Scrum-of-Scrums, especially in the early stages of development. The relationship recognition session is held during Scrum-of-Scrums, and for the next two to three weeks, changes in command-line dependencies are often discussed at this meeting.

4.8 Status Reports

When a team moves to Scrum, written reports often become unnecessary, as state clarity is now provided at the expense of other things (sprint reviews, burndown chart, daily stand-up meetings). When we switched to Scrum, we decided to keep the lightweight weekly status reports that each Scrum master prepares. It looked redundant while we used the “4 questions” format for Scrum-of-Scrums. However, with the current open Scrum-of-Scrums format, weekly reports have become an important addition to this meeting. We do not discuss the status of each team during the Scrum-of-Scrums, unless it is rendered as a separate issue, but this status is always available in the weekly report. The report identifies all dependencies, blockers and risks. Of course, such reports require a bit of extra effort from Scrum Masters, but the clarity they provide justifies the cost of creating them.

5 Conclusion


Salesforce.com has proven that Scrum is scalable, and we are confident that we can continue to scale as the company grows. As the number of teams grows, managing relationships and inter-team coordination becomes a daunting task. A critical infrastructure for building and automated testing is critical. Awareness raising and synchronization with the help of the launch meeting, the session of identifying relationships, Scrum-of-Scrums and lightweight status reports helps a lot. Ultimately, this leads to effective communication, interaction and the exchange of knowledge between teams. Approaches such as a virtual architecture team, open discussion and Scrum-of-Scrums are designed to help communication and support interaction.

Source: https://habr.com/ru/post/240275/


All Articles