Writing a web service on Go (part one)
In this article, I would like to tell you how you can quickly and easily write a small web application in the Go language, which, despite its young age, managed to win favor with many developers. Usually, for such articles they write artificial applications, such as the TODO sheet. We will try to write something useful that already exists and is used.
Often, when developing services, you need to understand what data is sent to another service, and the ability to capture traffic is not always there. And just to catch such requests, there is a project requestb.in , which allows you to collect requests on a specific URL and display them in the web interface. Writing a similar application, we will do. To simplify a task a little, let's take some framework, for example Martini , as a basis.
In the end, we should have a service like this:
')
This article will be divided into steps, each of which will contain code stored in a separate repository branch on GitHub. You can always run and see the results, as well as play around with the code.
To run the application, you need to have Go compiler on your machine. I proceed from the assumption that you already have it and are configured as you like. If not, then you can find out how to do this on the project page .
As a development environment, you can use what is more convenient for you, thank you. Go plugins are available for almost every editor. GoSublime is most popular. But I would advise IntelijIdea + go-lang-ide-plugin , which has recently been very actively developing, for example, from the last added application debug.
You can try the ready-made service at work via the link skimmer.tulu.la .
To get started, you need to clone the repository to your machine in any directory, like this:
You can add a project to your work environment (you can read more about this on the project website ), or arrange the code as you like. For simplicity, I use goenv , which allows you to specify the go compiler versions and create a clean working environment in the project directory.
Now we need to go to the skimmer's cloned directory and install the necessary dependencies with the command:
After the dependency installation is complete, you can run the project:
You should start the web service on port 3000 (the port and the host can be specified via the PORT and HOST environment variables, respectively). Now you can open it in the browser at 127.0.0.1.1000 and try the ready-made service at work.
Ahead of us are the following stages:
Special thanks to kavu for correcting the first and second parts of the article.
Let's start the development.
Load the first step code:
To get started, just try to print the request that comes to us. The entry point to any Go application is the main function of the main package. Create the main.go file in the src directory. Martini already has an application stub that adds logs, error handling, recovery, and a router; and in order not to repeat, we will use it.
Martini itself is pretty simple:
It implements the http.Handler interface by implementing the ServeHTTP method. Further, all incoming requests are passed through various handlers stored in handlers and at the end executes Handler action.
Classic Martini:
In this constructor, an object of type Martini and Router is created, handler handlers are added via the martini.Use method to request logging, intercept panic ( more about this mechanism), return static, and the last action sets the handler of the router.
We will intercept any HTTP requests to our application using the
If you really want to - you can implement your implementation of the address handler, but we will use the one that goes to Martini by default.
The first parameter is the location. Locations in Martini support parameters via
Using the ready-made DumpRequest function from the httputil package , we preserve the structure of the http.Request request, and write it in the http.ResponseWriter response. Just do not forget to handle possible errors. The api.Run function simply starts the built-in go server from the standard library, specifying the port and host it takes from the PORT (3000 by default) and HOST environment parameters.
Run our first application:
Let's try to send a request to the server:
It was just a test of strength, now let's start writing this application.
Do not forget to download the code:
Placing the code inside the main package is not very correct, because, for example, Google Application Engine creates its main package, in which yours are already connected. Therefore, we move the creation of the API into a separate module, let's call it, for example, skimmer / api.go.
Now we need to create an entity in which we can store captured requests, let's call it Bin, by analogy with requestbin. With the model, we will have just the usual Go data structure.
So, our Bin model will contain fields with the name, the number of requests caught, and the dates of creation and modification. Each field is also described by a tag.
We also describe the auxiliary function NewBin, in which the values of the Bin object are initialized (a kind of constructor):
Now in more detail about generation of lines through object rs. It is initialized as follows:
The code itself is in the utils.go file. In the function, we pass an array of characters from which we need to generate a string and create a RandomString object:
Here we use the math / rand package, which gives us access to random number generation. Most importantly, sow the generator before starting work with it, so that we do not get the same sequence of random numbers with each launch.
In the Generate method, we create an array of bytes, and each of the bytes is filled with a random character from the string pool. The resulting string is returned.
Let us proceed, in fact, to the description of Api. To begin, we need three methods for working with objects of type Bin, displaying a list of objects, creating and obtaining a specific object.
Earlier, I wrote that martini accepts a handler function with the HandlerFunc interface, in fact, the received function in Martini is described as interface {} - that is, it can be absolutely any function. How are arguments inserted into this function? This is done with the help of a well-known pattern - Dependency injection (hereinafter DI) with the help of a small inject package from the author martini. I will not go into details regarding how this is done, you can look into the code yourself, since it’s not great and everything is pretty simple. But if in two words, with the help of the already mentioned package reflect, the types of the function arguments are obtained and after that the necessary objects of this type are substituted. For example, when inject sees the type * http.Request, it substitutes the object req * http.Request in this parameter.
We can add the necessary objects ourselves for reflection through the methods of the Map and MapTo object globally, or through the martini.Context request context object for each request separately.
We will declare temporary variables history and bins, the first will contain the history of the Bin objects we created, and the second will be a certain short version of the Bin object storage.
Now consider the created methods.
The method allows you to get the Bin object by its name, in it we use the martini.Params object (in fact, simply map [string] string), through which we can access the parsed address parameters.
Let's experiment with our application. First, run it:
Add a new Bin object:
Get a list of available Bin objects:
Request a specific Bin object by taking the name value from the previous query:
Great, now we have learned how to create models and respond to inquiries, it seems now nothing will keep us from completing everything else.
Now we need to learn how to save requests that come to us in the desired Bin object.
Download the code for the third step.
To begin, create a model that will store an HTTP request.
I think it makes no sense to explain what kind of field it is for, but there are a couple of notes: for the files we will store only their names, and for the form data, we will store the ready dictionary of meanings.
By analogy with the creation of the Bin object, we write the function that creates the Request object from the HTTP request:
The function turned out to be quite large, but in general, understandable, I will explain only some of the points. In the http.Request object, the request body - Body is a kind of buffer that implements the io.ReadCloser interface; for this reason, after parsing the form (a call to the ParseMultipartForm method), we can’t get the raw request data. Therefore, for starters, we copy the Body into a separate variable and then replace the original buffer with our own. Next, we invoke the parsing of incoming data and collect information about the values of the forms and files.
In addition to the Bin objects, now we also need to store the requests, so it's time to add the ability to store data in our project. We describe its interface in the file storage.go:
In addition, we will create a basic storage object, which will have auxiliary fields that we will need in each implementation:
Now it's time to implement the behavior of our storage interface. To begin with, we will try to store everything in memory, delimiting parallel access to data by mutexes .
Create a memory.go file. Our storage will be based on a simple data structure:
It consists of nested, anonymous BaseStorage and sync.RWMutex fields.
We need RWMutex to block simultaneous work with the binRecords dictionary, since Go does not guarantee the correct behavior when changing data in dictionaries in parallel.
The data itself will be stored in the binRecords field, which is a dictionary with the keys from the name Bin field of objects and data of the BinRecord type.
This structure contains all the necessary data. References to queries are stored in two fields, in the list, where they go in the order of addition and in the dictionary, for faster search by identifier.
Also, for the BinRecord object, a method is implemented for trimming unnecessary requests, which simply removes unnecessary elements from requests and requestMap.
All MemoryStorage methods implement the behavior of the Storage interface, and we also have a getBinRecord helper method in which we can read the record we need. At the moment when we read the record, we put the lock on the read and immediately indicate the deferred call of the unlock in the defer. The defer expression allows us to specify a function that will always be executed when the function completes, even if the function was interrupted by panic. You can read more about defer in the documentation.
It makes no sense to examine each MemoryStorage method in more detail, everything is not so difficult there, you can look into the code yourself.
, , api. .
.
Storage . , Bin Storage.
, Request.
, .
Bin HTTP
, :
, , .
, , - AngularJS Bootstrap, Redis .
Often, when developing services, you need to understand what data is sent to another service, and the ability to capture traffic is not always there. And just to catch such requests, there is a project requestb.in , which allows you to collect requests on a specific URL and display them in the web interface. Writing a similar application, we will do. To simplify a task a little, let's take some framework, for example Martini , as a basis.
In the end, we should have a service like this:
')
Training
This article will be divided into steps, each of which will contain code stored in a separate repository branch on GitHub. You can always run and see the results, as well as play around with the code.
To run the application, you need to have Go compiler on your machine. I proceed from the assumption that you already have it and are configured as you like. If not, then you can find out how to do this on the project page .
As a development environment, you can use what is more convenient for you, thank you. Go plugins are available for almost every editor. GoSublime is most popular. But I would advise IntelijIdea + go-lang-ide-plugin , which has recently been very actively developing, for example, from the last added application debug.
You can try the ready-made service at work via the link skimmer.tulu.la .
To get started, you need to clone the repository to your machine in any directory, like this:
git clone https://github.com/m0sth8/skimmer ./skimmer You can add a project to your work environment (you can read more about this on the project website ), or arrange the code as you like. For simplicity, I use goenv , which allows you to specify the go compiler versions and create a clean working environment in the project directory.
Now we need to go to the skimmer's cloned directory and install the necessary dependencies with the command:
go get -d ./src/ After the dependency installation is complete, you can run the project:
go run ./src/main.go You should start the web service on port 3000 (the port and the host can be specified via the PORT and HOST environment variables, respectively). Now you can open it in the browser at 127.0.0.1.1000 and try the ready-made service at work.
Ahead of us are the following stages:
- Step one. Meet Martini;
- Step two. Create a Bin model and respond to requests;
- Step three. We accept requests and save them in storage;
- Step Four. But what about the tests?
- Step five - decorations and a web interface;
- Step Six. Add a bit of privacy;
- Step Seven. We clear the unnecessary;
- Step Eight. Use Redis for storage.
Special thanks to kavu for correcting the first and second parts of the article.
Let's start the development.
Step one. Meet the Martini.
Load the first step code:
git checkout step-1 To get started, just try to print the request that comes to us. The entry point to any Go application is the main function of the main package. Create the main.go file in the src directory. Martini already has an application stub that adds logs, error handling, recovery, and a router; and in order not to repeat, we will use it.
Martini itself is pretty simple:
// Martini represents the top level web application. inject.Injector methods can be invoked to map services on a global level. type Martini struct { inject.Injector handlers []Handler action Handler logger *log.Logger } It implements the http.Handler interface by implementing the ServeHTTP method. Further, all incoming requests are passed through various handlers stored in handlers and at the end executes Handler action.
Classic Martini:
// Classic creates a classic Martini with some basic default middleware - martini.Logger, martini.Recovery, and martini.Static. func Classic() *ClassicMartini { r := NewRouter() m := New() m.Use(Logger()) m.Use(Recovery()) m.Use(Static("public")) m.Action(r.Handle) return &ClassicMartini{m, r} } In this constructor, an object of type Martini and Router is created, handler handlers are added via the martini.Use method to request logging, intercept panic ( more about this mechanism), return static, and the last action sets the handler of the router.
We will intercept any HTTP requests to our application using the
Any method from the router, intercepting any URLs and methods. The router interface is described in Martini like this: type Router interface { // Get adds a route for a HTTP GET request to the specified matching pattern. Get(string, ...Handler) Route // Patch adds a route for a HTTP PATCH request to the specified matching pattern. Patch(string, ...Handler) Route // Post adds a route for a HTTP POST request to the specified matching pattern. Post(string, ...Handler) Route // Put adds a route for a HTTP PUT request to the specified matching pattern. Put(string, ...Handler) Route // Delete adds a route for a HTTP DELETE request to the specified matching pattern. Delete(string, ...Handler) Route // Options adds a route for a HTTP OPTIONS request to the specified matching pattern. Options(string, ...Handler) Route // Any adds a route for any HTTP method request to the specified matching pattern. Any(string, ...Handler) Route // NotFound sets the handlers that are called when a no route matches a request. Throws a basic 404 by default. NotFound(...Handler) // Handle is the entry point for routing. This is used as a martini.Handler Handle(http.ResponseWriter, *http.Request, Context) } If you really want to - you can implement your implementation of the address handler, but we will use the one that goes to Martini by default.
The first parameter is the location. Locations in Martini support parameters via
":param" , regular expressions, as well as glob . The second parameter and the following ones take the function that will handle the request. Since Martini supports a chain of handlers, you can add various auxiliary handlers here, such as checking access rights. We still have nothing to do with it, so we will add only one handler with an interface that is processed by the usual Go web handler (an example of development on it can be found in the documentation ). Here is the code of our handler: func main() { api := martini.Classic() api.Any("/", func(res http.ResponseWriter, req *http.Request,) { if dumped, err := httputil.DumpRequest(req, true); err == nil { res.WriteHeader(200) res.Write(dumped) } else { res.WriteHeader(500) fmt.Fprintf(res, "Error: %v", err) } }) api.Run() } Using the ready-made DumpRequest function from the httputil package , we preserve the structure of the http.Request request, and write it in the http.ResponseWriter response. Just do not forget to handle possible errors. The api.Run function simply starts the built-in go server from the standard library, specifying the port and host it takes from the PORT (3000 by default) and HOST environment parameters.
Run our first application:
go run ./src/main.go Let's try to send a request to the server:
> curl -X POST -d "fizz=buzz" http://127.0.0.1:3000 POST / HTTP/1.1 Host: 127.0.0.1:3000 Accept: */* Content-Type: application/x-www-form-urlencoded User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 fizz=buzz It was just a test of strength, now let's start writing this application.
Step two. Create a Bin model and respond to requests.
Do not forget to download the code:
git checkout step-2 Placing the code inside the main package is not very correct, because, for example, Google Application Engine creates its main package, in which yours are already connected. Therefore, we move the creation of the API into a separate module, let's call it, for example, skimmer / api.go.
Now we need to create an entity in which we can store captured requests, let's call it Bin, by analogy with requestbin. With the model, we will have just the usual Go data structure.
The order of the fields in the structure is quite important, but we will not think about it, but those who want to know how order affects the size of the structure in memory can read these articles here - www.goinggo.net/2013/07/understanding-type-in -go.html and www.geeksforgeeks.org/structure-member-alignment-padding-and-data-packing .
So, our Bin model will contain fields with the name, the number of requests caught, and the dates of creation and modification. Each field is also described by a tag.
Tags are normal lines that do not affect the program as a whole, but you can read them using the package of reflection while the program is running (so-called introspection), and based on this, change your behavior (about how to work with tags through reflection ). In our example, the json package takes the tag value into account when encoding / decoding, something like this:package main import ( "reflect" "fmt" ) type Bin struct { Name string `json:"name"` } func main() { bin := Bin{} bt := reflect.TypeOf(bin) field := bt.Field(0) fmt.Printf("Field's '%s' json name is '%s'", field.Name, field.Tag.Get("json")) }
Will outputField's 'Name' json name is 'name'
The encoding / json package supports various options when generating tags:// Field int `json:"-"` // json myName Field int `json:"myName"`
The second parameter may be, for example, the omitempty option - if the value in json is omitted, the field is not filled. So for example, if the field is a link, we can find out if it is in the json object by comparing it with nil. You can read more about json serialization in the documentation.
We also describe the auxiliary function NewBin, in which the values of the Bin object are initialized (a kind of constructor):
type Bin struct { Name string `json:"name"` Created int64 `json:"created"` Updated int64 `json:"updated"` RequestCount int `json:"requestCount"` } func NewBin() *Bin { now := time.Now().Unix() bin := Bin{ Created: now, Updated: now, Name: rs.Generate(6), } return &bin } Structures in Go can be initialized in two ways:
1) Mandatory listing of all fields in order:Bin{rs.Generate(6), now, now, 0}
2) Indicating the fields for which values are assigned:Bin{ Created: now, Updated: now, Name: rs.Generate(6), }
Fields that are not specified accept default values. For example, for integers it will be 0, for strings - the empty string "", for links, channels, arrays, slices and dictionaries - this will be nil. Read more in the documentation . The main thing to remember is that you cannot mix these two types of initialization.
Now in more detail about generation of lines through object rs. It is initialized as follows:
var rs = NewRandomString("0123456789abcdefghijklmnopqrstuvwxyz") The code itself is in the utils.go file. In the function, we pass an array of characters from which we need to generate a string and create a RandomString object:
type RandomString struct { pool string rg *rand.Rand } func NewRandomString(pool string) *RandomString { return &RandomString{ pool, rand.New(rand.NewSource(time.Now().Unix())), } } func (rs *RandomString) Generate(length int) (r string) { if length < 1 { return } b := make([]byte, length) for i, _ := range b { b[i] = rs.pool[rs.rg.Intn(len(rs.pool))] } r = string(b) return } Here we use the math / rand package, which gives us access to random number generation. Most importantly, sow the generator before starting work with it, so that we do not get the same sequence of random numbers with each launch.
In the Generate method, we create an array of bytes, and each of the bytes is filled with a random character from the string pool. The resulting string is returned.
Let us proceed, in fact, to the description of Api. To begin, we need three methods for working with objects of type Bin, displaying a list of objects, creating and obtaining a specific object.
Earlier, I wrote that martini accepts a handler function with the HandlerFunc interface, in fact, the received function in Martini is described as interface {} - that is, it can be absolutely any function. How are arguments inserted into this function? This is done with the help of a well-known pattern - Dependency injection (hereinafter DI) with the help of a small inject package from the author martini. I will not go into details regarding how this is done, you can look into the code yourself, since it’s not great and everything is pretty simple. But if in two words, with the help of the already mentioned package reflect, the types of the function arguments are obtained and after that the necessary objects of this type are substituted. For example, when inject sees the type * http.Request, it substitutes the object req * http.Request in this parameter.
We can add the necessary objects ourselves for reflection through the methods of the Map and MapTo object globally, or through the martini.Context request context object for each request separately.
We will declare temporary variables history and bins, the first will contain the history of the Bin objects we created, and the second will be a certain short version of the Bin object storage.
Now consider the created methods.
Creating a Bin Object
api.Post("/api/v1/bins/", func(r render.Render){ bin := NewBin() bins[bin.Name] = bin history = append(history, bin.Name) r.JSON(http.StatusCreated, bin) }) Getting a list of Bin objects
api.Get("/api/v1/bins/", func(r render.Render){ filteredBins := []*Bin{} for _, name := range(history) { if bin, ok := bins[name]; ok { filteredBins = append(filteredBins, bin) } } r.JSON(http.StatusOK, filteredBins) }) Getting a specific instance
api.Get("/api/v1/bins/:bin", func(r render.Render, params martini.Params){ if bin, ok := bins[params["bin"]]; ok{ r.JSON(http.StatusOK, bin) } else { r.Error(http.StatusNotFound) } }) The method allows you to get the Bin object by its name, in it we use the martini.Params object (in fact, simply map [string] string), through which we can access the parsed address parameters.
In Go, we can access the dictionary element in two ways:
- By requesting the value of the key
a := m[key], in this case either the key value in the dictionary, if any, or the default value of the initialization of the value type is returned. Thus, for example for numbers, it is difficult to understand whether the key contains 0 or just the value of this key does not exist. Therefore, the second option is provided for in go.- In this way, by requesting by key and get its value by the first parameter and the indicator of the existence of this key by the second parameter -
a, ok := m[key]
Let's experiment with our application. First, run it:
go run ./src/main.go Add a new Bin object:
> curl -i -X POST "127.0.0.1:3000/api/v1/bins/" HTTP/1.1 201 Created Content-Type: application/json; charset=UTF-8 Date: Mon, 03 Mar 2014 04:10:38 GMT Content-Length: 76 {"name":"7xpogf","created":1393819838,"updated":1393819838,"requestCount":0} Get a list of available Bin objects:
> curl -i "127.0.0.1:3000/api/v1/bins/" HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 03 Mar 2014 04:11:18 GMT Content-Length: 78 [{"name":"7xpogf","created":1393819838,"updated":1393819838,"requestCount":0}] Request a specific Bin object by taking the name value from the previous query:
curl -i "127.0.0.1:3000/api/v1/bins/7xpogf" HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 03 Mar 2014 04:12:13 GMT Content-Length: 76 {"name":"7xpogf","created":1393819838,"updated":1393819838,"requestCount":0} Great, now we have learned how to create models and respond to inquiries, it seems now nothing will keep us from completing everything else.
Step three. We accept requests and save them in storage.
Now we need to learn how to save requests that come to us in the desired Bin object.
Download the code for the third step.
git checkout step-3 Model Request
To begin, create a model that will store an HTTP request.
type Request struct { Id string `json:"id"` Created int64 `json:"created"` Method string `json:"method"` // GET, POST, PUT, etc. Proto string `json:"proto"` // "HTTP/1.0" Header http.Header `json:"header"` ContentLength int64 `json:"contentLength"` RemoteAddr string `json:"remoteAddr"` Host string `json:"host"` RequestURI string `json:"requestURI"` Body string `json:"body"` FormValue map[string][]string `json:"formValue"` FormFile []string `json:"formFile"` } I think it makes no sense to explain what kind of field it is for, but there are a couple of notes: for the files we will store only their names, and for the form data, we will store the ready dictionary of meanings.
By analogy with the creation of the Bin object, we write the function that creates the Request object from the HTTP request:
func NewRequest(httpRequest *http.Request, maxBodySize int) *Request { var ( bodyValue string formValue map[string][]string formFile []string ) // if body, err := ioutil.ReadAll(httpRequest.Body); err == nil { if len(body) > 0 && maxBodySize != 0 { if maxBodySize == -1 || httpRequest.ContentLength < int64(maxBodySize) { bodyValue = string(body) } else { bodyValue = fmt.Sprintf("%s\n<<<TRUNCATED , %d of %d", string(body[0:maxBodySize]), maxBodySize, httpRequest.ContentLength) } } httpRequest.Body = ioutil.NopCloser(bytes.NewBuffer(body)) defer httpRequest.Body.Close() } httpRequest.ParseMultipartForm(0) if httpRequest.MultipartForm != nil { formValue = httpRequest.MultipartForm.Value for key := range httpRequest.MultipartForm.File { formFile = append(formFile, key) } } else { formValue = httpRequest.PostForm } request := Request{ Id: rs.Generate(12), Created: time.Now().Unix(), Method: httpRequest.Method, Proto: httpRequest.Proto, Host: httpRequest.Host, Header: httpRequest.Header, ContentLength: httpRequest.ContentLength, RemoteAddr: httpRequest.RemoteAddr, RequestURI: httpRequest.RequestURI, FormValue: formValue, FormFile: formFile, Body: bodyValue, } return &request } The function turned out to be quite large, but in general, understandable, I will explain only some of the points. In the http.Request object, the request body - Body is a kind of buffer that implements the io.ReadCloser interface; for this reason, after parsing the form (a call to the ParseMultipartForm method), we can’t get the raw request data. Therefore, for starters, we copy the Body into a separate variable and then replace the original buffer with our own. Next, we invoke the parsing of incoming data and collect information about the values of the forms and files.
In addition to the Bin objects, now we also need to store the requests, so it's time to add the ability to store data in our project. We describe its interface in the file storage.go:
type Storage interface { LookupBin(name string) (*Bin, error) // get one bin element by name LookupBins(names []string) ([]*Bin, error) // get slice of bin elements LookupRequest(binName, id string) (*Request, error) // get request from bin by id LookupRequests(binName string, from, to int) ([]*Request, error) // get slice of requests from bin by position CreateBin(bin *Bin) error // create bin in memory storage UpdateBin(bin *Bin) error // save CreateRequest(bin *Bin, req *Request) error } The interfaces in Go are a contract linking the expected functionality and the actual implementation. In our case, we described the storage interface, which we will use later in the program, but depending on the settings, the implementation may be completely different (for example, it may be Redis or Mongo). Learn more about interfaces .
In addition, we will create a basic storage object, which will have auxiliary fields that we will need in each implementation:
type BaseStorage struct { maxRequests int } Now it's time to implement the behavior of our storage interface. To begin with, we will try to store everything in memory, delimiting parallel access to data by mutexes .
Create a memory.go file. Our storage will be based on a simple data structure:
type MemoryStorage struct { BaseStorage sync.RWMutex binRecords map[string]*BinRecord } It consists of nested, anonymous BaseStorage and sync.RWMutex fields.
Anonymous fields enable us to call methods and fields of anonymous structures directly. For example, if we have a variable obj of type MemoryStorage, we can access the maxRequests field directly obj.BaseStorage.maxRequests, or as if they are members of the MemoryStorage obj.maxRequests itself. More information about anonymous fields in data structures can be found in the documentation .
We need RWMutex to block simultaneous work with the binRecords dictionary, since Go does not guarantee the correct behavior when changing data in dictionaries in parallel.
The data itself will be stored in the binRecords field, which is a dictionary with the keys from the name Bin field of objects and data of the BinRecord type.
type BinRecord struct { bin *Bin requests []*Request requestMap map[string]*Request } This structure contains all the necessary data. References to queries are stored in two fields, in the list, where they go in the order of addition and in the dictionary, for faster search by identifier.
Dictionaries in Go in the current implementation are a hash of the table, so searching for an element in the dictionary has a constant value. More information about the internal device can be found in this excellent article .
Also, for the BinRecord object, a method is implemented for trimming unnecessary requests, which simply removes unnecessary elements from requests and requestMap.
func (binRecord *BinRecord) ShrinkRequests(size int) { if size > 0 && len(binRecord.requests) > size { requests := binRecord.requests lenDiff := len(requests) - size removed := requests[:lenDiff] for _, removedReq := range removed { delete(binRecord.requestMap, removedReq.Id) } requests = requests[lenDiff:] binRecord.requests = requests } } All MemoryStorage methods implement the behavior of the Storage interface, and we also have a getBinRecord helper method in which we can read the record we need. At the moment when we read the record, we put the lock on the read and immediately indicate the deferred call of the unlock in the defer. The defer expression allows us to specify a function that will always be executed when the function completes, even if the function was interrupted by panic. You can read more about defer in the documentation.
It makes no sense to examine each MemoryStorage method in more detail, everything is not so difficult there, you can look into the code yourself.
MemoryStorage code
package skimmer import ( "errors" "sync" ) type MemoryStorage struct { BaseStorage sync.RWMutex binRecords map[string]*BinRecord } type BinRecord struct { bin *Bin requests []*Request requestMap map[string]*Request } func (binRecord *BinRecord) ShrinkRequests(size int) { if size > 0 && len(binRecord.requests) > size { requests := binRecord.requests lenDiff := len(requests) - size removed := requests[:lenDiff] for _, removedReq := range removed { delete(binRecord.requestMap, removedReq.Id) } requests = requests[lenDiff:] binRecord.requests = requests } } func NewMemoryStorage(maxRequests int) *MemoryStorage { return &MemoryStorage{ BaseStorage{ maxRequests: maxRequests, }, sync.RWMutex{}, map[string]*BinRecord{}, } } func (storage *MemoryStorage) getBinRecord(name string) (*BinRecord, error) { storage.RLock() defer storage.RUnlock() if binRecord, ok := storage.binRecords[name]; ok { return binRecord, nil } return nil, errors.New("Bin not found") } func (storage *MemoryStorage) LookupBin(name string) (*Bin, error) { if binRecord, err := storage.getBinRecord(name); err == nil { return binRecord.bin, nil } else { return nil, err } } func (storage *MemoryStorage) LookupBins(names []string) ([]*Bin, error) { bins := []*Bin{} for _, name := range names { if binRecord, err := storage.getBinRecord(name); err == nil { bins = append(bins, binRecord.bin) } } return bins, nil } func (storage *MemoryStorage) CreateBin(bin *Bin) error { storage.Lock() defer storage.Unlock() binRec := BinRecord{bin, []*Request{}, map[string]*Request{}} storage.binRecords[bin.Name] = &binRec return nil } func (storage *MemoryStorage) UpdateBin(_ *Bin) error { return nil } func (storage *MemoryStorage) LookupRequest(binName, id string) (*Request, error) { if binRecord, err := storage.getBinRecord(binName); err == nil { if request, ok := binRecord.requestMap[id]; ok { return request, nil } else { return nil, errors.New("Request not found") } } else { return nil, err } } func (storage *MemoryStorage) LookupRequests(binName string, from int, to int) ([]*Request, error) { if binRecord, err := storage.getBinRecord(binName); err == nil { requestLen := len(binRecord.requests) if to >= requestLen { to = requestLen } if to < 0 { to = 0 } if from < 0 { from = 0 } if from > to { from = to } reversedLen := to - from reversed := make([]*Request, reversedLen) for i, request := range binRecord.requests[from:to] { reversed[reversedLen-i-1] = request } return reversed, nil } else { return nil, err } } func (storage *MemoryStorage) CreateRequest(bin *Bin, req *Request) error { if binRecord, err := storage.getBinRecord(bin.Name); err == nil { storage.Lock() defer storage.Unlock() binRecord.requests = append(binRecord.requests, req) binRecord.requestMap[req.Id] = req binRecord.ShrinkRequests(storage.maxRequests) binRecord.bin.RequestCount = len(binRecord.requests) return nil } else { return err } } , , api. .
.
memoryStorage := NewMemoryStorage(MAX_REQUEST_COUNT) api.MapTo(memoryStorage, (*Storage)(nil)) Storage . , Bin Storage.
api.Post("/api/v1/bins/", func(r render.Render, storage Storage){ bin := NewBin() if err := storage.CreateBin(bin); err == nil { history = append(history, bin.Name) r.JSON(http.StatusCreated, bin) } else { r.JSON(http.StatusInternalServerError, ErrorMsg{err.Error()}) } }) api.Get("/api/v1/bins/", func(r render.Render, storage Storage){ if bins, err := storage.LookupBins(history); err == nil { r.JSON(http.StatusOK, bins) } else { r.JSON(http.StatusInternalServerError, ErrorMsg{err.Error()}) } }) api.Get("/api/v1/bins/:bin", func(r render.Render, params martini.Params, storage Storage){ if bin, err := storage.LookupBin(params["bin"]); err == nil{ r.JSON(http.StatusOK, bin) } else { r.JSON(http.StatusNotFound, ErrorMsg{err.Error()}) } }) , Request.
// api.Get("/api/v1/bins/:bin/requests/", func(r render.Render, storage Storage, params martini.Params, req *http.Request){ if bin, error := storage.LookupBin(params["bin"]); error == nil { from := 0 to := 20 if fromVal, err := strconv.Atoi(req.FormValue("from")); err == nil { from = fromVal } if toVal, err := strconv.Atoi(req.FormValue("to")); err == nil { to = toVal } if requests, err := storage.LookupRequests(bin.Name, from, to); err == nil { r.JSON(http.StatusOK, requests) } else { r.JSON(http.StatusInternalServerError, ErrorMsg{err.Error()}) } } else { r.Error(http.StatusNotFound) } }) // Request api.Get("/api/v1/bins/:bin/requests/:request", func(r render.Render, storage Storage, params martini.Params){ if request, err := storage.LookupRequest(params["bin"], params["request"]); err == nil { r.JSON(http.StatusOK, request) } else { r.JSON(http.StatusNotFound, ErrorMsg{err.Error()}) } }) // http Request Bin(name) api.Any("/bins/:name", func(r render.Render, storage Storage, params martini.Params, req *http.Request){ if bin, error := storage.LookupBin(params["name"]); error == nil { request := NewRequest(req, REQUEST_BODY_SIZE) if err := storage.CreateRequest(bin, request); err == nil { r.JSON(http.StatusOK, request) } else { r.JSON(http.StatusInternalServerError, ErrorMsg{err.Error()}) } } else { r.Error(http.StatusNotFound) } }) , .
Bin HTTP
> curl -i -X POST "127.0.0.1:3000/api/v1/bins/" HTTP/1.1 201 Created Content-Type: application/json; charset=UTF-8 Date: Mon, 03 Mar 2014 12:19:28 GMT Content-Length: 76 {"name":"ws87ui","created":1393849168,"updated":1393849168,"requestCount":0} > curl -X POST -d "fizz=buzz" http://127.0.0.1:3000/bins/ws87ui {"id":"i0aigrrc1b40","created":1393849284,...} , :
> curl http://127.0.0.1:3000/api/v1/bins/ws87ui/requests/ [{"id":"i0aigrrc1b40","created":1393849284,...}] , , .
, , - AngularJS Bootstrap, Redis .
Source: https://habr.com/ru/post/208680/
All Articles