Load Test with Go

Good afternoon, Habrahabr.
You are probably familiar with JMeter . If in short - a very convenient tool for carrying out load testing, it has a huge functionality and many, many useful chips. But the article is not about him.

How did it start

There is a pretty heavy node in our project, JMeter has been helping for a long time. Profiling and optimization gave their profit, but everything came up against a small problem. JMeter could not create a very large traffic, and if more precisely, after 10 seconds of the mode we need, OutOfMemory occurred and testing stopped, in some cases there was no problem, but the speed of sending requests noticeably decreased, while the CPU load was 400%, solved restarting the program. It was extremely uncomfortable to use.
So, we have a problem, and it needs to be solved, the first thing that came to mind was to make a mini-test that meets the minimum requirements. It has long been interesting to try Go to taste. This is how the go-meter application was born. When writing, there were a lot of questions, answers to which either did not exist or they did not explain the problem, so I decided to share my experience and an example of working code, if you are interested, I’ll ask for a tackle.

Foreword

I think writing that this language does not make sense, you can always see a tour of the language , which reveals the main elements. How to install and configure the environment is also not worth it, everything is written in a completely understandable language in the documentation .
Why chose Go? There are several criteria that are very important to me: it works quickly, cross-platform, there are streams that are easy to manage, unusual. Of course, you say that you can write it in any other language. I agree with you, but the task was not only to write, but also to learn something new.

Let's get started

Without hesitation, it was decided to store the test profile in JSON format, after running the application, the profile is read and testing is started. During testing, a pivot table is displayed in the console (response time, number of requests per second, and the percentage of errors, warnings, and successful requests). With JSON, everything is simple; to do this, you need to make structures for each element, open and read the file:

func (this *Settings) Load(fileName string) error { file, e := ioutil.ReadFile(fileName); if e != nil { return e } e = json.Unmarshal(file, this); if e != nil { return e } return nil } 

')
Let's go further. After starting, we need to start the N-streams, and after working on each of them, aggregate the data, then output beautifully to the console. For this in this interesting language there are Channels. A kind of "pipe" between different streams. No need for synchronization, locking, everything is done for us. The idea is this: the thread sends the request, determines the result and reports it to the main thread, which in turn waits until all the threads have completed and output all the received data. The streams will communicate by means of the structure transfer:

 type Status struct { IsError bool IsWarning bool IsSuccess bool Duration *time.Duration Size int64 IsFinished bool Error *error FinishedAt *time.Time StartedAt *time.Time } 

Each thread we will have to perform an M-time HTTP request to the specified resource. If we have a POST request, then still sending certain data that the user wants:

 func StartThread(setts *settings.Settings, source *Source, c chan *Status){ iteration := setts.Threads.Iteration //  key, value    header := map[string]string{} for _, s := range setts.Request.Headers { keyValue := regexp.MustCompile("=").Split(s, -1) header[keyValue[0]] = keyValue[1] } sourceLen := len(*source) // URL url := setts.Remote.Protocol + "://" + setts.Remote.Host + ":" + strconv.Itoa(setts.Remote.Port) + setts.Request.Uri if iteration < 0 { iteration = sourceLen } index := -1 for ;iteration > 0; iteration-- { status := &Status{false, false, false, nil, 0, false, nil, nil, nil} index++ if index >= sourceLen { if setts.Request.Source.RestartOnEOF { index = 0 } else { index-- } } //     var s *bytes.Buffer if strings.ToLower(setts.Request.Method) != "get" { s = bytes.NewBuffer((*source)[index]) } // HTTP  req, err := http.NewRequest(setts.Request.Method, url, s); if err != nil { status.Error = &err status.IsError = true c <- status break } //  for k,v := range header { req.Header.Set(k,v) } //  startTime := time.Now() //  res, err := http.DefaultClient.Do(req); if err != nil { status.Error = &err status.IsError = true c <- status break } endTime := time.Now() //   status.FinishedAt = &endTime status.StartedAt = &startTime diff := endTime.Sub(startTime) //        3  (Error, Warning, Success) checkStatus(setts.Levels, res, diff, status) //  ioutil.ReadAll(res.Body) res.Body.Close() //   c <- status //    ,   if setts.Threads.Delay > 0 { sleep := time.Duration(setts.Threads.Delay) time.Sleep(time.Millisecond * sleep) } } //      status := &Status{false, false, false, nil, 0, true, nil, nil, nil} c <- status } 

It remains only to start our threads at program start and listen to the data from them.

 c := make(chan *Status, iteration * setts.Threads.Count) for i := 0; i < setts.Threads.Count; i++{ go StartThread(&setts, source, c) } for i := iteration * setts.Threads.Count; i>0 ; i-- { counter(<-c) } fmt.Println("Completed") 

Instead of conclusion

These are the most interesting moments, in my opinion. All source codes are available on GitHub , where you can see the whole cycle of work with an example of use. In fact, this miracle has coped with this task with a vengeance. When generating traffic with a volume of 3 times more than was the case with JMeter, the processor load rarely exceeds 15%.
If it will be interesting, I will tell about the process of writing HTTP Restfull Web service with storage in MongoDB and Redis.

Thanks for attention!