Developers in the struggle for the effectiveness of the programmer, team, teams
Hello.
Today we would like to discuss one very important aspect of effective work - reuse.
Routine, which does not want to write twice, and even more than three times. Infrastructure, which has to be written from project to project. Ingenious, which simply expresses a great idea and which sin is not to allow others to use it.
')
Yes, and let's admit - we all love to write “frameworks”, which we use ourselves later and which others use. This adds a little courage to our daily activities.
And here the author finds himself doing what is wrong, starting to describe the situation - everyone already understands what this is about. And you need to describe the technology and process. So let's just dwell on an abstract thought, that suddenly there was a lot of code that could be useful to colleagues and it would be good to make it easily accessible. We also take into account the second, much less pleasant, thought that in “mature” projects you can find a large number of copy-paste code, which significantly decreases maintainability. Yes, and work on such projects is difficult to call pleasant. And with this we need to do something.
The path we have taken was not so difficult, we can advise to repeat it to others, because the time spent pays for itself. And since at some point this path became interesting and even funny, we decided to share our experience.
So, today we will tell in detail:
So nuget is great and fun. This is not just a new tool that increases productivity, it is a new approach, a new way of thinking. Almost as useful as a healthy diet and morning exercises. So you need to use.
If we are talking about the deployment of a corporate server, then there are two installation options:
Actually, on this and everything - the process is extremely simple. However, there are a few things that are not described in the documentation that have appeared in us and that may be useful to others:
1. IIS express . The option of working on IIS express using localdb is difficult to consider as production ready for a number of reasons. Therefore, we moved our newborn to a full-featured IIS server. Fortunately, there are no difficulties with this. Just go to the properties tab of the “Website” project and remove the “Use IIS Express” checkbox in the “Web” section. Visual Studio will instantly react to such changes - will offer to create a Virtual Directory and change the Connection Strings in the web.config from localdb to SQL express.
2. Database Migrations . Nuget uses the Entity Framework to access data and actively uses the Database Migrations mechanism. In the deployment guide there is a step number 4 - creating a database with the help of dancing with sabers around the order of opening tabs in the studio. In fact, everything is easier. Simply enter the name of the project in which migrations are stored when updating via the console. And no dancing:
3. Security settings . The manual does not describe the moments concerning the security settings (although they were in the outdated manual ). Apparently, it is assumed that everything is put unconditional admin and everything will be spinning under his own account. If this is not the case, then we need:
4. GalleryOwner . It makes sense to change to the actual settings of the item “Gallery.GalleryOwner” in the web.config. From him will be sent letters at registration. And you can completely disable this newsletter. And be sure to list the current SiteRoot.
5. In the stats.js file there is a getStats function . At the time of this writing, invoking it resulted in an error on the main page.
The reason is that in the local version of nuget, statistics are disabled . At the moment it is intended only for the cloud version.
You can create your own implementation of IStatisticsService or simply disable the call. We chose the second path.
On this, in fact, everything. Now we have a nuget server with a gallery and packages .
But what if in our package (dll-ku from it) the bug crept in? Or does the developer just want to understand what our “DoStuff” function is doing? Obviously, the library needs debugging. But it's not so easy when we only have the dll at our disposal. That is why nuget can build not only library packages, but also symbol packages .
As a solution for debugging support, we decided to use SymbolSource , because it is so good friends with nuget that it supports it as a default symbol-server.
Here we make a small digression and tell you how it all works.
When creating a nuget package using the nuget pack command, we can specify the “–Symbols” flag. As a result, nuget creates, in addition to the standard library package, the <package_name> .symbols.nupkg package, which will contain the pdb file and the sources. Then this package can be placed on the SymbolSource server, which will index the pdb file and place the project sources in its internal directory. Next, Visual Studio configures access to the symbol server and, if necessary, it downloads the pdb file and sources from the server. This is how debugging becomes possible.
But, if the library we decided not to add to the public feed, then there’s nothing to say about the source code. In principle, SymbolSource makes it possible to start a private tape. But it also supports the Community version, which allows you to deploy your internal server. For a variety of reasons, we decided to go the last way.
As planned by the authors, deploying your symbol server is almost as simple as deploying your nuget feed — creating an MVC application, downloading the nuget-package , giving rights to the “App_Data” and “Data” folders — everything works.
Oh yes, there is one additional requirement - you need to put the Debugging Tools for Windows and set the path to them in the web.config:
Actually, that's what we did - we downloaded version 1.3.2 of SymbolServer, set it up, set it up, saw the magic screen.
Diagnostics launched - all is well. We set up Visual Studio exactly as instructed .
The problem was found quickly enough - the service did not save the source code in the “Data” directory when loading the symbol-package. Fiddling with the IIS settings for a while, we realized that the problem was different, and started debugging.
The cause of the problem was also found quickly enough, and at first it seemed even ridiculous. In the ManagedSourceExtractor class, there is an IsTemporaryCompilerFile method that returns, as the name implies, a sign if a certain file is a temporary compiler file. And in this method there was an extra logical negation operator. That is, this method reported that tempo files were just those that were not. So one extra character broke the entire server.
Well, what can you do ... They laughed, forked the source code , corrected it, got ready to do a pull request. And here we find that a similar request already exists , and it has been created for a long time. Yes, and the package version 1.3.2 has been released for two months already, but the version of the package with fix has not been released yet. This was not so funny. More precisely, it is not funny at all.
For a while, we even thought about whether we should use this server at all or better search for another one. But the positive previous experience of using ... And the developers of nuget made it the default server, it means they trust. And the package version 1.3.1 works fine. As a result, we decided to run everything on the corrected source code. For this, we additionally copied the settings from “Web.config.transform” into “Web.config”, since the local assembly does not perform transform:
Now it is ready and also works as it should. And then it is time to think about
But it is not enough for a very simple reason: nuget, as it is, is an accessible server for the entire company.
Therefore, it is impossible to put packages into it, which may be unstable. But when the development of final projects required changes to the assemblies hosted in nuget, then to make changes so that it worked immediately, the task from the category “write code without bugs”.
You can find a lot of different techniques and squats in order to somehow work with one server. For example, replace the dll-ki pens in the “packages” folder of the final project, and publish to nuget only when everything is ready. But we do it all to make life easier, and not vice versa. Therefore, we decided to approach the issue more thoroughly. A little thought, we came up with something.
Written below is a beta version of the process that we are still experiencing and comprehending. Therefore, it would be very nice to hear the views of readers in the comments.
So, we decided that we would have three nuget tapes.
It turns out, on the one hand, that build machines that compile release versions of final products will never build a project if it refers to a test version of the package.
On the other hand, the order in which the servers are specified in the Package Sources settings in Visual Studio (Tools-> Library Package Manager-> Package Manager Settings) sets the priority for downloading packages. Therefore, for example, developers of the base code have access to all three tapes and the order in the settings is set as release feed, test feed, local feed.
In Visual Studio, it looks like this:
By the way, it is here that the argument in favor of local server deployment becomes quite weighty - independence from access to the Internet and greater speed when working in a local network.
“What if one of the servers fell?” You ask. Well, it does not matter. After all, the build process “framework” is one for all tapes. That is, if the package version 1.2.3.4 is declared release, then it lies in all three tapes. And access to it, for example, the developer will always be.
On the one hand, this process is not very simple, but on the other hand, it turned out to be very natural, so it’s not often necessary to think about its complexity - it just falls on the standard work flow.
Well, thought out. Now you can go back to technology and think about Continuous Integration, which automates the whole process.
Therefore, we will say goodbye and leave some material for the second part. And here we would like to appeal to our readers. If the topic seemed interesting to you - let us understand it, leave comments. They will help us to make the second part more useful and informative, as well as motivate for its early release.
Thank you for your attention.
UPD: as it usually happens, exactly the next day after the article was added, the SymbolSource.Server.Basic update (version 1.3.3) was updated, fixing the bug mentioned in the section on deploying the symbol-server.
Today we would like to discuss one very important aspect of effective work - reuse.
It will, of course, about the code.
Routine, which does not want to write twice, and even more than three times. Infrastructure, which has to be written from project to project. Ingenious, which simply expresses a great idea and which sin is not to allow others to use it.
')
Yes, and let's admit - we all love to write “frameworks”, which we use ourselves later and which others use. This adds a little courage to our daily activities.
And here the author finds himself doing what is wrong, starting to describe the situation - everyone already understands what this is about. And you need to describe the technology and process. So let's just dwell on an abstract thought, that suddenly there was a lot of code that could be useful to colleagues and it would be good to make it easily accessible. We also take into account the second, much less pleasant, thought that in “mature” projects you can find a large number of copy-paste code, which significantly decreases maintainability. Yes, and work on such projects is difficult to call pleasant. And with this we need to do something.
And we decided to do something about it.
The path we have taken was not so difficult, we can advise to repeat it to others, because the time spent pays for itself. And since at some point this path became interesting and even funny, we decided to share our experience.
Well, enough of the preface.
So, today we will tell in detail:
- How to deploy a corporate nuget-server (we omit the basics of nuget , assuming that most readers know what it is).
- How to deploy a corporate symbol-server integrated with nuget.
- Why a single nuget server is not enough to build an efficient process.
- We will also designate topics for the second part in advance, which we will be happy to produce if this topic finds its readers:
- What is SandCastle and what it is eaten with.
- How to configure continuous integration to automate everything we have done.
- A few thoughts about the (self) organization.
Deploy corporate nuget server.
So nuget is great and fun. This is not just a new tool that increases productivity, it is a new approach, a new way of thinking. Almost as useful as a healthy diet and morning exercises. So you need to use.
If we are talking about the deployment of a corporate server, then there are two installation options:
- Simple deployment of tape without UI and buns. For this option, it is enough to create the MVC application and install the nuget server by downloading the nuget package (such is the recursion).
- The second option is created for a little less lazy, but it’s with UI and buns. For it, you need to download the source and collect them according to the instructions on the same page. After that, by pressing Ctrl + F5, we should see a (almost) complete analogue of the official tape.
Actually, on this and everything - the process is extremely simple. However, there are a few things that are not described in the documentation that have appeared in us and that may be useful to others:
1. IIS express . The option of working on IIS express using localdb is difficult to consider as production ready for a number of reasons. Therefore, we moved our newborn to a full-featured IIS server. Fortunately, there are no difficulties with this. Just go to the properties tab of the “Website” project and remove the “Use IIS Express” checkbox in the “Web” section. Visual Studio will instantly react to such changes - will offer to create a Virtual Directory and change the Connection Strings in the web.config from localdb to SQL express.
2. Database Migrations . Nuget uses the Entity Framework to access data and actively uses the Database Migrations mechanism. In the deployment guide there is a step number 4 - creating a database with the help of dancing with sabers around the order of opening tabs in the studio. In fact, everything is easier. Simply enter the name of the project in which migrations are stored when updating via the console. And no dancing:
3. Security settings . The manual does not describe the moments concerning the security settings (although they were in the outdated manual ). Apparently, it is assumed that everything is put unconditional admin and everything will be spinning under his own account. If this is not the case, then we need:
- a. Write permissions to the application’s “AppData” folder. There will be nuget-packages. We give rights to the account under which the AppPool application is running. In case if in the IIS settings there is an ApplicationPoolIdentity, and not a specific account, we give rights to the IIS_IUSRS group.
- b. By the same principles, we grant rights to access the database in SQL.
- c. After registration you can give yourself the same admin role. To do this, after the standard registration through the UI, we insert the entry into the dbo.UserRoles table, having looked at the Id roles in the dbo.Roles table. After that, at the address “<path to server> / admin”, you will have access to the admin panel with all sorts of buns and bookmarklet glimpse .
4. GalleryOwner . It makes sense to change to the actual settings of the item “Gallery.GalleryOwner” in the web.config. From him will be sent letters at registration. And you can completely disable this newsletter. And be sure to list the current SiteRoot.
<appSettings> <add key="Gallery.LuceneIndexLocation" value="AppData" /> <add key="Gallery.Environment" value="Development" /> <add key="Gallery.SiteRoot" value="http://your-server-name/" /> <add key="Gallery.GalleryOwner" value="%username% <%usermail%>" /> <add key="Gallery.ConfirmEmailAddresses" value="true" /> <add key="Gallery.HasWorker" value="false" /> </appSettings> 5. In the stats.js file there is a getStats function . At the time of this writing, invoking it resulted in an error on the main page.
The reason is that in the local version of nuget, statistics are disabled . At the moment it is intended only for the cloud version.
You can create your own implementation of IStatisticsService or simply disable the call. We chose the second path.
On this, in fact, everything. Now we have a nuget server with a gallery and packages .
But what if in our package (dll-ku from it) the bug crept in? Or does the developer just want to understand what our “DoStuff” function is doing? Obviously, the library needs debugging. But it's not so easy when we only have the dll at our disposal. That is why nuget can build not only library packages, but also symbol packages .
And we are going to learn to collect our own symbol server.
As a solution for debugging support, we decided to use SymbolSource , because it is so good friends with nuget that it supports it as a default symbol-server.
Here we make a small digression and tell you how it all works.
When creating a nuget package using the nuget pack command, we can specify the “–Symbols” flag. As a result, nuget creates, in addition to the standard library package, the <package_name> .symbols.nupkg package, which will contain the pdb file and the sources. Then this package can be placed on the SymbolSource server, which will index the pdb file and place the project sources in its internal directory. Next, Visual Studio configures access to the symbol server and, if necessary, it downloads the pdb file and sources from the server. This is how debugging becomes possible.
But, if the library we decided not to add to the public feed, then there’s nothing to say about the source code. In principle, SymbolSource makes it possible to start a private tape. But it also supports the Community version, which allows you to deploy your internal server. For a variety of reasons, we decided to go the last way.
And then the adventure began ...
As planned by the authors, deploying your symbol server is almost as simple as deploying your nuget feed — creating an MVC application, downloading the nuget-package , giving rights to the “App_Data” and “Data” folders — everything works.
Oh yes, there is one additional requirement - you need to put the Debugging Tools for Windows and set the path to them in the web.config:
<appSettings> <add key="SrcSrvPath" value="C:\Program Files\Debugging Tools for Windows (x64)\srcsrv" /> </appSettings> Actually, that's what we did - we downloaded version 1.3.2 of SymbolServer, set it up, set it up, saw the magic screen.
Diagnostics launched - all is well. We set up Visual Studio exactly as instructed .
But Debug doesn't work ...
The problem was found quickly enough - the service did not save the source code in the “Data” directory when loading the symbol-package. Fiddling with the IIS settings for a while, we realized that the problem was different, and started debugging.
Exactly. Debugging the SymbolSource server using the SymbolSource server.
The cause of the problem was also found quickly enough, and at first it seemed even ridiculous. In the ManagedSourceExtractor class, there is an IsTemporaryCompilerFile method that returns, as the name implies, a sign if a certain file is a temporary compiler file. And in this method there was an extra logical negation operator. That is, this method reported that tempo files were just those that were not. So one extra character broke the entire server.
Well, what can you do ... They laughed, forked the source code , corrected it, got ready to do a pull request. And here we find that a similar request already exists , and it has been created for a long time. Yes, and the package version 1.3.2 has been released for two months already, but the version of the package with fix has not been released yet. This was not so funny. More precisely, it is not funny at all.
For a while, we even thought about whether we should use this server at all or better search for another one. But the positive previous experience of using ... And the developers of nuget made it the default server, it means they trust. And the package version 1.3.1 works fine. As a result, we decided to run everything on the corrected source code. For this, we additionally copied the settings from “Web.config.transform” into “Web.config”, since the local assembly does not perform transform:
<location path="WinDbg/pdbsrc"> <system.webServer> <handlers> <clear /> <add name="Deny" verb="*" path="*.config" type="System.Web.HttpForbiddenHandler" /> <add name="Allow" verb="GET,HEAD" path="*" type="System.Web.StaticFileHandler" /> </handlers> <security> <requestFiltering> <fileExtensions allowUnlisted="true"> <clear /> </fileExtensions> </requestFiltering> </security> </system.webServer> </location> Now it is ready and also works as it should. And then it is time to think about
Why one nuget-server is not enough to build an effective process (beta).
But it is not enough for a very simple reason: nuget, as it is, is an accessible server for the entire company.
Therefore, it is impossible to put packages into it, which may be unstable. But when the development of final projects required changes to the assemblies hosted in nuget, then to make changes so that it worked immediately, the task from the category “write code without bugs”.
You can find a lot of different techniques and squats in order to somehow work with one server. For example, replace the dll-ki pens in the “packages” folder of the final project, and publish to nuget only when everything is ready. But we do it all to make life easier, and not vice versa. Therefore, we decided to approach the issue more thoroughly. A little thought, we came up with something.
Written below is a beta version of the process that we are still experiencing and comprehending. Therefore, it would be very nice to hear the views of readers in the comments.
So, we decided that we would have three nuget tapes.
- Local feed . In fact, this is just a network folder (in Visual Studio, when setting up sources, you can specify not only the server address, but also the UNC path to the folder). Packs are assembled on this tape for every commit in VCS. No one except the developers reusable code source for this tape does not configure.
- Test feed . This role quite logically falls on the SymbolSource server, which has its own tape (just a tape, not a gallery) for packages, where the usual nuget-package is automatically placed when the symbol-package is published. The publication in this tape is done when the changes are made to the base code and you can start using it. The source for this tape is for all the developers of the company and for the build-machines that assemble test versions of the final products.
- Server release packages . This server was the one that we deployed at the beginning of the article - beautiful, with UI and other buns. The most important bun is the ApiKey requirement. The local tape does not require it, just as the SymbolSource tape (more precisely, SymbolSource requires, but does not verify). And there they are not needed. Here everything is more serious, so it makes sense to make it so that not everyone can publish. And if someone could, it would be clear who it is. The publication on this server goes only when the basic code is tested both by itself and in work with end products. Source for this server is configured in all developers, build-machines, collecting test versions of the final products, build-machines, collecting release versions of the final products.
It turns out, on the one hand, that build machines that compile release versions of final products will never build a project if it refers to a test version of the package.
On the other hand, the order in which the servers are specified in the Package Sources settings in Visual Studio (Tools-> Library Package Manager-> Package Manager Settings) sets the priority for downloading packages. Therefore, for example, developers of the base code have access to all three tapes and the order in the settings is set as release feed, test feed, local feed.
In Visual Studio, it looks like this:
By the way, it is here that the argument in favor of local server deployment becomes quite weighty - independence from access to the Internet and greater speed when working in a local network.
“What if one of the servers fell?” You ask. Well, it does not matter. After all, the build process “framework” is one for all tapes. That is, if the package version 1.2.3.4 is declared release, then it lies in all three tapes. And access to it, for example, the developer will always be.
On the one hand, this process is not very simple, but on the other hand, it turned out to be very natural, so it’s not often necessary to think about its complexity - it just falls on the standard work flow.
Well, thought out. Now you can go back to technology and think about Continuous Integration, which automates the whole process.
But the article has already dragged on.
Therefore, we will say goodbye and leave some material for the second part. And here we would like to appeal to our readers. If the topic seemed interesting to you - let us understand it, leave comments. They will help us to make the second part more useful and informative, as well as motivate for its early release.
Thank you for your attention.
UPD: as it usually happens, exactly the next day after the article was added, the SymbolSource.Server.Basic update (version 1.3.3) was updated, fixing the bug mentioned in the section on deploying the symbol-server.
Source: https://habr.com/ru/post/191406/
All Articles