Another HighLoadCup solution on Go
I think already many users of Habr know that the HighLoadCup from Mail.ru ended last week (due to the abundance of articles from the participants). I would also like to share my decision with the community.
There are three kinds of entities: User, Location, Visit. You must write a REST-API to access them, i.e. it turns out you need to process 6 requests.
As with any service, requests may be invalid and this also needs to be processed, but the task is simplified by the fact that, in general, the HTTP packet is always valid.
Initially, I started writing in C ++, but it never came to a release. After seeing a large number of solutions in the top on Go, I decided to try it too. This language seemed to me much more suitable for developing server applications, it has all the necessary functionality out of the box, and in a very high-quality version. However, after the first tests, it became clear that neither net / http, nor encoding / json is suitable for this competition due to the large amount of garbage that is generated inside them.
')
Initially, the data was only 200 MB in the unpacked form, so I thought that it was possible to even store ready-made JSON strings for each entity. As the http-server on the advice of the contest goshniki (thanks to them very much for the help at the start of acquaintance with the language), I chose fasthttp , for parsing JSON buger / jsonparser (allows parsing without allocation and working only with the necessary information, ignoring the rest), and generating I did it with my hands, since no processing of Russian-language lines was required.
Such entities I came out at the start, in a visit to speed up, I decided to store immediately the pointers to the corresponding user and location. The result was very satisfactory, all the data fit perfectly and in the end I was even able to get into the top 10 (not for long though).
I want to pay special attention to the issue of calculating the age of the visitor. This question was acute for many participants in spite of the presence of an example of the FAQ, this fate also did not pass me and in the first days a lot of mistakes were precisely due to an incorrect calculation. Also several times in the generated data there were users whose birthday was on the day of the tests, which also brought some difficulties.
The result was the following code:
A few days before the final, after the democratic and open voting, the contest creators increased the amount of data 10 times and the maximum RPS by 2 times. After that, my decision stopped getting into my memory and required changes. I had to remove the pre-generated JSON from the structures and create it on the fly when prompted, which increased, of course, realism. At the same time, I thought, why not add links to visits that are associated with them to the visitor’s structures and locations. This significantly increased the speed of the program, since it did not have to go through the entire array of visits each time (which now contained 10 million entries).
The decision after that earned, but in speed it became inferior to other decisions and I risked not going through to the final. Without hesitation, I threw out fasthttp and switched to tcp-sockets and epoll. The size of the window in our system was around 65kb and the packages exactly came and went completely and this gave a large field for crutches that will not work in production exactly.
I came to the finals at the 39th place, which I was undoubtedly very pleased about. This is my first participation in such a contest, the first acquaintance with Go and highload (although I would not call it highload). The finale began badly, the server crashed due to an error of simultaneous access to read and write (there were no such problems before the final and the lockers were cut out), however, one of the waves managed to show the best result for all starts, which made it possible to take 28th place.
In general, it was a very interesting and informative (for me, at least) competition. I would like to express my gratitude to the organizers and look forward to the next one taking into account all the errors and features of the current one (busy-polling, for example). True, in the future, it was promised more emphasis on logic, and not on the network stack, which would be more interesting.
PS Waiting for a t-shirt :)
The code (suddenly it will be interesting to someone) can be viewed in my repository .
Task Description
There are three kinds of entities: User, Location, Visit. You must write a REST-API to access them, i.e. it turns out you need to process 6 requests.
- {GET, POST} / user /: id - get or change user
- {GET, POST} / location /: id - get or change location
- {GET, POST} / visit /: id - receiving or changing the site visit by the user
- POST / user / new - add new user
- POST / location / new - add a new location
- POST / visit / new - add a new visit to the location by the user
As with any service, requests may be invalid and this also needs to be processed, but the task is simplified by the fact that, in general, the HTTP packet is always valid.
Start
Initially, I started writing in C ++, but it never came to a release. After seeing a large number of solutions in the top on Go, I decided to try it too. This language seemed to me much more suitable for developing server applications, it has all the necessary functionality out of the box, and in a very high-quality version. However, after the first tests, it became clear that neither net / http, nor encoding / json is suitable for this competition due to the large amount of garbage that is generated inside them.
')
Round one
Initially, the data was only 200 MB in the unpacked form, so I thought that it was possible to even store ready-made JSON strings for each entity. As the http-server on the advice of the contest goshniki (thanks to them very much for the help at the start of acquaintance with the language), I chose fasthttp , for parsing JSON buger / jsonparser (allows parsing without allocation and working only with the necessary information, ignoring the rest), and generating I did it with my hands, since no processing of Russian-language lines was required.
type User struct { id uint email string first_name string last_name string gender bool birth_date int64 age int visits Visits json []byte } type Location struct { id uint place string country string city string distance int64 visits Visits json []byte } type Visit struct { id uint location *Location user *User visited_at int64 mark int64 json []byte } Such entities I came out at the start, in a visit to speed up, I decided to store immediately the pointers to the corresponding user and location. The result was very satisfactory, all the data fit perfectly and in the end I was even able to get into the top 10 (not for long though).
Age
I want to pay special attention to the issue of calculating the age of the visitor. This question was acute for many participants in spite of the presence of an example of the FAQ, this fate also did not pass me and in the first days a lot of mistakes were precisely due to an incorrect calculation. Also several times in the generated data there were users whose birthday was on the day of the tests, which also brought some difficulties.
The result was the following code:
func countAge(timestamp *int64) int { now := time.Now() t := time.Unix(*timestamp, 0) years := now.Year() - t.Year() if now.Month() > t.Month() || now.Month() == t.Month() && now.Day() >= t.Day() { years += 1 } return years } Load increase
A few days before the final, after the democratic and open voting, the contest creators increased the amount of data 10 times and the maximum RPS by 2 times. After that, my decision stopped getting into my memory and required changes. I had to remove the pre-generated JSON from the structures and create it on the fly when prompted, which increased, of course, realism. At the same time, I thought, why not add links to visits that are associated with them to the visitor’s structures and locations. This significantly increased the speed of the program, since it did not have to go through the entire array of visits each time (which now contained 10 million entries).
The decision after that earned, but in speed it became inferior to other decisions and I risked not going through to the final. Without hesitation, I threw out fasthttp and switched to tcp-sockets and epoll. The size of the window in our system was around 65kb and the packages exactly came and went completely and this gave a large field for crutches that will not work in production exactly.
The final
I came to the finals at the 39th place, which I was undoubtedly very pleased about. This is my first participation in such a contest, the first acquaintance with Go and highload (although I would not call it highload). The finale began badly, the server crashed due to an error of simultaneous access to read and write (there were no such problems before the final and the lockers were cut out), however, one of the waves managed to show the best result for all starts, which made it possible to take 28th place.
In general, it was a very interesting and informative (for me, at least) competition. I would like to express my gratitude to the organizers and look forward to the next one taking into account all the errors and features of the current one (busy-polling, for example). True, in the future, it was promised more emphasis on logic, and not on the network stack, which would be more interesting.
PS Waiting for a t-shirt :)
The code (suddenly it will be interesting to someone) can be viewed in my repository .
Source: https://habr.com/ru/post/338156/
All Articles