We write the simulator of slow connections on Go

In this article I want to show how simple you can do quite complex things in Go, and how powerful the interfaces are in yourself. We will talk about simulating a slow connection - but, unlike popular solutions in the form of rules for iptables, we implement this on the side of the code - so that you can easily use it, for example, in tests.

There will be nothing complicated here, and for the sake of greater clarity I recorded ascii animations (using the asciinema service), but I hope it will be informative.

Interfaces

Interfaces are a special type in the Go type system that allows you to describe the behavior of an object. Any static type for which methods (behavior) are defined implicitly implements an interface that describes these methods. The most famous example is the interface from the standard io.Reader library:

// Reader is the interface that wraps the basic Read method. // ... type Reader interface { Read(p []byte) (n int, err error) } 

Any structure for which you define the method Read ([] byte) (int, error) - can be used as io.Reader.
')
A simple idea, not seemingly too valuable and powerful at first, takes on a completely different look when interfaces are used by other libraries. To demonstrate this, the standard library and io.Reader are ideal candidates.

Console output

So, let's start with the simplest use of Reader, we will display a line in stdout. Of course, for this task it is better to use functions from the fmt package, but we want to demonstrate how Reader works. Therefore, we will create a variable of the strings.Reader type (which implements io.Reader) and, using the io.Copy () function, which also works with io.Reader, copy it into os.Stdout (which, in turn, implements io.Writer).

 package main import ( "io" "os" "strings" ) func main() { r := strings.NewReader("Not very long line...") io.Copy(os.Stdout, r) } 

And now, using composition, we will create our own SlowReader type, which will read one character from the original Reader with a delay of, say, 100 milliseconds - thus providing a speed of 10 bytes per second.

 // SlowReader reads 10 chars per second type SlowReader struct { r io.Reader } func (sr SlowReader) Read(p []byte) (int, error) { time.Sleep(100 * time.Millisecond) return sr.r.Read(p[:1]) } 

I hope that there is no need to explain what p [: 1] is - just a new slice consisting of 1 first character from the original slice.

All we have to do is use our strings.Reader as the original io.Reader, and pass a slow SlowReader to io.Copy ()! See how simple and cool at the same time.
(ascii-caste opens in a new window, js-scripts are not allowed to be embedded in the habr)


You should already begin to suspect that this simple SlowReader can be used not only for displaying on the screen. You can also add a parameter like delay. Better yet, put the SlowReader into a separate package so that it is easy to use in the following examples. A little comb the code.

Combing code

Create the test / habr / slow directory and transfer the code there:

 package slow import ( "io" "time" ) type SlowReader struct { delay time.Duration r io.Reader } func (sr SlowReader) Read(p []byte) (int, error) { time.Sleep(sr.delay) return sr.r.Read(p[:1]) } func NewReader(r io.Reader, bps int) io.Reader { delay := time.Second / time.Duration(bps) return SlowReader{ r: r, delay: delay, } } 

Or, who is interested in watching ascii-castes, like this - we put it in a separate package:


And add the delay parameter of the type time.Duration:


(It would be more correct, after removing the code in a separate package, to name the Reader type - so that it was slow.Reader, and not slow.SlowReader, but the screencast is already written like this).

Read from file

And now, almost without effort, we will check our SlowReader for slow reading from files. Having received a variable of type * os.File, which stores the open file descriptor, but at the same time implements the io.Reader interface - we can work with the file in the same way as before with strings.Reader.

 package main import ( "io" "os" "test/habr/slow" ) func main() { file, err := os.Open("test.txt") if err != nil { panic(err) } defer file.Close() // close file on exit r := slow.NewReader(file, 5) // 5 bps io.Copy(os.Stdout, r) } 

Or so:

JSON decoding

But reading from a file is too easy. Let's look at an example a little more interesting - the JSON decoder from the standard library. Although for convenience the encoding / json package provides the json.Unmarshal () function, it also allows you to work with io.Reader using json.Decoder — you can deserialize streaming data in json-format with it.

We will take a simple json-encoded string and read it “slowly” using our SlowReader, and json.Decoder will return the finished object only after all the bytes have reached. To make this obvious, we will add to the slow.SlowReader.Read () function a display of each character read:

 package main import ( "encoding/json" "fmt" "strings" "test/habr/slow" ) func main() { sr := strings.NewReader(`{"value": "some text", "id": 42}`) // encoded json r := slow.NewReader(sr, 5) dec := json.NewDecoder(r) type Sample struct { Value string `json:"value"` ID int64 `json:"id"` } var sample Sample err := dec.Decode(&sample) if err != nil { panic(err) } fmt.Println("Decoded JSON value:", sample) } 

This is also in the ascii caste:


If the awareness of the possibilities that such a simple interface concept gives us has not yet fallen upon you, then we go further - in fact, we come to the topic of the post - use our SlowReader to slowly download the page from the Internet.

Slow HTTP Client

You shouldn’t be surprised that io.Reader is used everywhere in the standard library - for everything that can read something from somewhere. Reading from the network is not an exception - io.Reader is used on several levels, and is hidden under the hood of such a seemingly simple http.Get single-line call (url string).

First, we write the standard code for the HTTP GET request and output the answer to the console:

 package main import ( "io" "net/http" "os" ) func main() { resp, err := http.Get("http://golang.org") if err != nil { panic(err) } defer resp.Body.Close() io.Copy(os.Stdout, resp.Body) } 

For those who have not had time to get acquainted with the net / http-library - a few explanations. http.Get () is a wrapper for the Get () method implemented for the http.Client type - but this wrapper uses an “initialized for most cases” already initialized variable called DefaultClient. Actually, the Client further performs all the dusty work, including reading from the network using an object of the Transport type, which in turn uses a lower-level object of the net.Conn type. At first, this may seem confusing, but in fact it is quite easy to learn by simply reading the source code of the library — that’s what, and the standard library in Go, unlike most other languages, is exemplary code that you can (and should) learn to go and take an example from him.

A little earlier, I mentioned “io.Reader is used on several levels” and this is true - for example, resp.Body is io.Reader too, but we are not interested in it, because we are interested in simulating not slowed down browser, but slow connection then you need to find io.Reader, which reads from the network. And this, running ahead, is a variable of the net.Conn type - which means we need to redefine it for our custom http-client. We can do this by embedding:

 type SlowConn struct { net.Conn // embedding r slow.SlowReader // in ascii-cast I use io.Reader here, but this one a bit better } // SlowConn is also io.Reader! func (sc SlowConn) Read(p []byte) (int, error) { return sc.r.Read(p) } 

The most difficult thing here is to sort out a little deeper into the net and net / http packages from the standard library, and correctly create our http.Client using a slow io.Reader. But as a result, nothing complicated - I hope the logic is visible on the screencast, as I look into the code of the standard library.

The result is the following client (for a real code, it is better to bring it into a separate function and comb it a bit, but for an example of proof-of-concept it will do):

  client := http.Client{ Transport: &http.Transport{ Dial: func(network, address string) (net.Conn, error) { conn, err := net.Dial(network, address) if err != nil { return nil, err } return SlowConn{conn, slow.NewReader(conn, 100)}, nil }, }, } 

Well, now we glue it all together and see the result:


In the end, it is clear that HTTP headers are output to the console normally, and the text, in fact, the page is displayed with doubling each character - this is normal, since we output resp.Body using io.Copy () and at the same time our slightly modified implementation SlowReader.Read () displays each character too.

Conclusion

As mentioned at the beginning of the article, interfaces are an extremely powerful toolkit, and the very idea of ​​separating types for properties and behavior is very correct. But truly this power manifests itself when the interfaces are actually used for their intended purpose in different libraries. This allows you to combine very different functionality, and use someone else's code for things that the original author could not even suspect. And it's not just about standard interfaces - inside large projects, interfaces provide tremendous flexibility and modularity.

Links

Since the idea of ​​this post was brazenly pulled from the Francesc Campoy twitter, there is only one link :)
twitter.com/francesc/status/563310996845244416