Corporate GitHub: how Azure increased the number of employees on GitHub to two thousand
We at Latera are creating billing for telecom operators . In a blog on Habré, we not only talk about the features of our system and the details of its development (for example, ensuring fault tolerance ), but also publish materials about how other companies work with IT infrastructure. Today we bring to your attention an adapted translation of the note by Azure’s main developer, Jeff Wilcox, about how more than two thousand project team members moved to GitHub.
Over the past four years, Azure has learned a lot from the opensource community, so we had to reconsider approaches to work and minimize the number of manual operations needed to maintain a large GitHub organization. We automated most of the management functions by creating an open source web application that uses some of the Azure services (these are App Service, Active Directory and Azure Storage), the GitHub API, and other free solutions.
In this post, I will introduce you to our portal for working with GitHub, the source code of which is now available there. I will talk about the functions, features and main software blocks of the application, as well as share with you the story of how GitHub contributed to our development. At the end I will add a few links where you can get more information. I also prepared a separate post , revealing the details of the project: how it all began, and what we came to.
')
Developers and organizations of any size will immediately feel how useful our portal can be: Azure Active Directory gives the engineering team access to the tools to work with GitHub.
A web portal created using Azure, the GitHub API and Node.js allows you to automate GitHub and speed up the adaptation of new employees.
To invite someone to a GitHub organization, an administrator must send an invitation to a person, stating their username on GitHub, which is often not related to the company name or email address.
GitHub is a great service for teams of small and medium sizes: simple solutions that it offers work in most cases. However, we have outgrown it - we needed not simplicity, but automation, in order not to waste time on routine tasks and concentrate on important tasks.
Although GitHub Enterprise has some integration capabilities, in the case of GitHub.com, where we interact with the world of open source software, every employee needs an official account.
The Azure portal for GitHub began as a project for the hackathon, when we needed to offer a new solution to the problem of the complexity of manually managing organizations and teams. By giving users the ability to authenticate using both personal and corporate accounts and automating the process of joining teams, we can significantly simplify working with GitHub, take a course on open source software and make our lawyers happier.
Over time, we adjusted to the changes made by GitHub and set up integration with other systems in the company. Microsoft also went through this.
New tools and simple organization of the workflow allowed delegating part of the tasks to the maintainers (specialists who are directly related to the project who are not administrators), which increased the productivity of engineers, giving them the opportunity to work at a comfortable pace.
Despite the fact that we were interested in developing our project, we hope that with the development of GitHub we will be able to reduce part of the portal’s functionality. GitHub is our reliable partner, and we are happy to grow together.
Let's take a look at some simple work scenarios and features of the portal.
With the growing number of organizations there is a need to increase the number of administrators, but this is not the best solution. Administrators have all the rights to read / write, access to critical areas of the repository, manage teams and organization, because they probably have more important tasks than manually inviting new members to the group. When employees become too many, you have to ask important questions: “Is this user an employee of Microsoft? Do we know who he is? Is this girl still working with us, or has she quit long ago? ”
The portal uses the GitHub API to allow some operations to be performed as an administrator. Using the open OAuth authorization protocol, we can send invitations to users and accept their requests without delay, and also check whether they have two-factor authentication enabled (this is necessary for full compliance with standards). After that we offer them the opportunity to join the teams in which they will work.
Another portal provides an opportunity to share experiences, distribute links to available training materials and guidelines. Some of our engineers start working with GitHub for the first time, so their training is a very important aspect.
The portal supports many organizations. Users can join several at once. When personnel changes occur, we can be sure that we have excluded all former employees from all organizations. An administrator can be assigned to only one organization, since each of them has its own unique credentials.
There are no administrators in the Azure Github family of organizations that are regular users - all administrator accounts are specifically protected and assigned to roles. This reduces the likelihood of errors and eliminates random "push" commits to the repository.
Each organization has a special command that issues individual sudo-rights to the users - they will help in cases when the administrator is unavailable for some reason, as well as in emergency situations.
The list of employees and their GitHub data are displayed side by side during any of the operations on the portal: adding team members and maintainers, viewing requests to join the team, viewing requests to create new repositories. This simplifies the process of joining workers to teams, since they no longer need to additionally recognize their username on GitHub.
The application also provides several hooks and REST APIs that allow other open source tools used by Microsoft (including the Azure CLA bot) to interact with the data store.
We trust our engineers, so part of the important decisions are made not by the administrators, but by team maintainers . They have their own access rights. Each organization has a special team whose task is to approve the creation of repositories.
Accompanying people have several big rights and can make the repositories public, add new team members, manage access rights, web-hooks and connect freelancers to the work.
Until October 2015, our portal had such a “position” as a team owner. These people could attract new members to their work and manage repositories. After GitHub released an update of organizations, we switched to a new model.
Instead of a database for managing admissions and creating repositories, we use a lightweight NoSQL database combined with tasks in GitHub. When a request is made to join a team in a private repository, a task is opened, to which all team members have access, and the team maintainer receives a notification with a link by which this request can be processed. With this approach, we have achieved a new degree of transparency and simplicity.
The portal uses open source software, in particular, it uses the following Azure services:
This is what the Azure portal looks like. The main page contains basic data and information about the portal. Here you can find analytical data of the application, response time graphs and time graphs of the number of get / set Redis requests. Also from here you can quickly go to other Azure services and follow the current sweep process on the App Service
Among the front-end components used in the project are Bootstrap, jQuery and a few other small plug-ins of the latter. The portal itself is a Node.js application that uses the following NPN modules: applicationinsights, async, azure-storage, express, jade, moment, node-uuid, octonode, passport-passport-azure-ad, passport-github, redis. The application is available on Mac, Windows and Linux platforms.
In the development process, we participated in the creation of octonode Pavan Kumar Sunkara - pksunkara , implementing a large number of additional API calls for the GitHub organization management service.
I hope that many of you will be able to benefit from our work, as well as share your thoughts with us, because the code is publicly available, and everyone can contribute. Azure coordinates its work with the Microsoft office responsible for distributing open source software, delivering the portal as part of the corporation’s Open Source tools.
When we created our first organization at the end of 2011, we already had a development plan, set out before June 2012 , and GitHub was part of it: a new space outside of the area familiar to Microsoft. It all started with the fact that we pulled out a corporate credit card to pay for GitHub, and immediately hired 20 people as administrators of organizations.
Instead of developing projects “in secret from everyone”, we got the opportunity to work in a team and created many of our open projects right on GitHub. Early feedback gathering and code verification by the programmer community had a positive effect on product quality. It turns out that a huge number of Azure clients use GitHub - working with them was very exciting.
Today, our GitHub organization has more than 2,000 people from all corners of Microsoft. At the same time, we have a small amount of reporting documentation, a contract with the GitHub sales department, a high level of involvement in the work of the organization, and many engineers enjoying the newfound opportunities.
Part of the decision-making tasks were given to the maintainers, who understand the subtleties of the development process. Our engineering teams consult the Open Source community on a wide range of issues: from resource templates ( deploy a highly reliable MongoDB cluster in the cloud directly from GitHub) to better documentation on the Azure website, SDK and trial versions of the service. I'd like to see what happens next.
Over the past four years, Azure has learned a lot from the opensource community, so we had to reconsider approaches to work and minimize the number of manual operations needed to maintain a large GitHub organization. We automated most of the management functions by creating an open source web application that uses some of the Azure services (these are App Service, Active Directory and Azure Storage), the GitHub API, and other free solutions.
In this post, I will introduce you to our portal for working with GitHub, the source code of which is now available there. I will talk about the functions, features and main software blocks of the application, as well as share with you the story of how GitHub contributed to our development. At the end I will add a few links where you can get more information. I also prepared a separate post , revealing the details of the project: how it all began, and what we came to.
')
Developers and organizations of any size will immediately feel how useful our portal can be: Azure Active Directory gives the engineering team access to the tools to work with GitHub.
A web portal created using Azure, the GitHub API and Node.js allows you to automate GitHub and speed up the adaptation of new employees.
Azure open source portal for GitHub
To invite someone to a GitHub organization, an administrator must send an invitation to a person, stating their username on GitHub, which is often not related to the company name or email address.
GitHub is a great service for teams of small and medium sizes: simple solutions that it offers work in most cases. However, we have outgrown it - we needed not simplicity, but automation, in order not to waste time on routine tasks and concentrate on important tasks.
Although GitHub Enterprise has some integration capabilities, in the case of GitHub.com, where we interact with the world of open source software, every employee needs an official account.
The Azure portal for GitHub began as a project for the hackathon, when we needed to offer a new solution to the problem of the complexity of manually managing organizations and teams. By giving users the ability to authenticate using both personal and corporate accounts and automating the process of joining teams, we can significantly simplify working with GitHub, take a course on open source software and make our lawyers happier.
Over time, we adjusted to the changes made by GitHub and set up integration with other systems in the company. Microsoft also went through this.
New tools and simple organization of the workflow allowed delegating part of the tasks to the maintainers (specialists who are directly related to the project who are not administrators), which increased the productivity of engineers, giving them the opportunity to work at a comfortable pace.
Despite the fact that we were interested in developing our project, we hope that with the development of GitHub we will be able to reduce part of the portal’s functionality. GitHub is our reliable partner, and we are happy to grow together.
Let's take a look at some simple work scenarios and features of the portal.
Instant introduction to the work of new employees
With the growing number of organizations there is a need to increase the number of administrators, but this is not the best solution. Administrators have all the rights to read / write, access to critical areas of the repository, manage teams and organization, because they probably have more important tasks than manually inviting new members to the group. When employees become too many, you have to ask important questions: “Is this user an employee of Microsoft? Do we know who he is? Is this girl still working with us, or has she quit long ago? ”
The portal uses the GitHub API to allow some operations to be performed as an administrator. Using the open OAuth authorization protocol, we can send invitations to users and accept their requests without delay, and also check whether they have two-factor authentication enabled (this is necessary for full compliance with standards). After that we offer them the opportunity to join the teams in which they will work.
Another portal provides an opportunity to share experiences, distribute links to available training materials and guidelines. Some of our engineers start working with GitHub for the first time, so their training is a very important aspect.
Support for multiple organizations
The portal supports many organizations. Users can join several at once. When personnel changes occur, we can be sure that we have excluded all former employees from all organizations. An administrator can be assigned to only one organization, since each of them has its own unique credentials.
Distribution of roles between administrators
There are no administrators in the Azure Github family of organizations that are regular users - all administrator accounts are specifically protected and assigned to roles. This reduces the likelihood of errors and eliminates random "push" commits to the repository.
Each organization has a special command that issues individual sudo-rights to the users - they will help in cases when the administrator is unavailable for some reason, as well as in emergency situations.
The list of employees is mapped to usernames on github
The list of employees and their GitHub data are displayed side by side during any of the operations on the portal: adding team members and maintainers, viewing requests to join the team, viewing requests to create new repositories. This simplifies the process of joining workers to teams, since they no longer need to additionally recognize their username on GitHub.
The application also provides several hooks and REST APIs that allow other open source tools used by Microsoft (including the Azure CLA bot) to interact with the data store.
Transferring some tasks to the maintainer
We trust our engineers, so part of the important decisions are made not by the administrators, but by team maintainers . They have their own access rights. Each organization has a special team whose task is to approve the creation of repositories.
Accompanying people have several big rights and can make the repositories public, add new team members, manage access rights, web-hooks and connect freelancers to the work.
Until October 2015, our portal had such a “position” as a team owner. These people could attract new members to their work and manage repositories. After GitHub released an update of organizations, we switched to a new model.
Approving Requests Using Tasks in GitHub
Instead of a database for managing admissions and creating repositories, we use a lightweight NoSQL database combined with tasks in GitHub. When a request is made to join a team in a private repository, a task is opened, to which all team members have access, and the team maintainer receives a notification with a link by which this request can be processed. With this approach, we have achieved a new degree of transparency and simplicity.
Using Azure Cloud Infrastructure
The portal uses open source software, in particular, it uses the following Azure services:
- Application hosting, high availability, continuous deployment and configuration are all possible thanks to the Azure App Service . CI / CD are customized in order to simplify the process of scanning and testing, to speed them up. Deployment slots allow you to release updates on the fly using the preparatory Git branch. In extreme cases, the Diagnostic Log Stream service, which operates in real time, can quickly and painlessly diagnose problems, and Application Settings will notify you of an existing API or portal problem.
- Using Azure Active Directory and a list of Microsoft employees, our application can authenticate and authorize employees.
- Virtual links to user profiles are stored in the Azure Table Storage , a storage service element. This is a fast and inexpensive highly reliable, distributed storage (geo-redundant storage).
- Azure Redis Cache is a managed cache whose task is to reduce the load on the GitHub API, storage, and other important work data. This allows you to deploy the required number of applications with shared cache in any number of regions or instances.
- Visual Studio Application Insights and the Node.js NPM module are used to monitor the performance of the web client, monitor the response time from the server, and conduct availability tests from anywhere in the world.
This is what the Azure portal looks like. The main page contains basic data and information about the portal. Here you can find analytical data of the application, response time graphs and time graphs of the number of get / set Redis requests. Also from here you can quickly go to other Azure services and follow the current sweep process on the App Service
Excellent open source tools
Among the front-end components used in the project are Bootstrap, jQuery and a few other small plug-ins of the latter. The portal itself is a Node.js application that uses the following NPN modules: applicationinsights, async, azure-storage, express, jade, moment, node-uuid, octonode, passport-passport-azure-ad, passport-github, redis. The application is available on Mac, Windows and Linux platforms.
In the development process, we participated in the creation of octonode Pavan Kumar Sunkara - pksunkara , implementing a large number of additional API calls for the GitHub organization management service.
I hope that many of you will be able to benefit from our work, as well as share your thoughts with us, because the code is publicly available, and everyone can contribute. Azure coordinates its work with the Microsoft office responsible for distributing open source software, delivering the portal as part of the corporation’s Open Source tools.
Azure on GitHub: four years later
When we created our first organization at the end of 2011, we already had a development plan, set out before June 2012 , and GitHub was part of it: a new space outside of the area familiar to Microsoft. It all started with the fact that we pulled out a corporate credit card to pay for GitHub, and immediately hired 20 people as administrators of organizations.
Instead of developing projects “in secret from everyone”, we got the opportunity to work in a team and created many of our open projects right on GitHub. Early feedback gathering and code verification by the programmer community had a positive effect on product quality. It turns out that a huge number of Azure clients use GitHub - working with them was very exciting.
Today, our GitHub organization has more than 2,000 people from all corners of Microsoft. At the same time, we have a small amount of reporting documentation, a contract with the GitHub sales department, a high level of involvement in the work of the organization, and many engineers enjoying the newfound opportunities.
Part of the decision-making tasks were given to the maintainers, who understand the subtleties of the development process. Our engineering teams consult the Open Source community on a wide range of issues: from resource templates ( deploy a highly reliable MongoDB cluster in the cloud directly from GitHub) to better documentation on the Azure website, SDK and trial versions of the service. I'd like to see what happens next.
Additional materials
An open source portal for github
- A blog in which Jeff Wilcox tells in detail the story of creating an open source portal and describes its architecture;
- Azure portal for GitHub on GitHub - you can use clone (), you can fork (), or you can just see;
- Check out how the Azure engineering team started using Git version control at Microsoft.
Node.js on Azure
- Tutorial : Create a Node.js web application in the Azure App Service;
- Learn about the capabilities of Azure Table Storage and Node.js;
- Azure SDK for Node.js and Azure Storage SDK for Node.js on GitHub.
Azure services and cloud infrastructure
- Azure App Service : Build and deploy smart mobile and web apps;
- Azure Active Directory : Access control for the cloud;
- Azure Redis Cache : Dedicated Redis running Microsoft;
- Azure Storage : Storage for blobs, tables, queues, and other important service data;
- Visual Studio Application Insights : Find problems, diagnose problems, and use statistics for your web applications.
Source: https://habr.com/ru/post/273627/
All Articles