Go and Protocol Buffers, acceleration
Some continuation of the article Go and Protocol Buffers is a bit of practice (or quick start, for those who are not familiar yet) . The coding / decoding processes in certain formats in Go are closely related to reflection. And as we, dear reader, we know that reflection is a long time. About what methods of struggle there is this article. I think that the experienced ones will hardly find anything new in it.
Actually, in the mentioned article it was told about packages
Reference: Marshaler , Unmarshaler .
If the encoder encounters a type embodying one of these interfaces, then in this case the whole work is assigned to their methods.
')
The solution is always simple. Use another package:
These packages simply add convenience and acceleration.
About package (links):
As you can see acceleration is from 1.10x and higher. It is possible to simply use a set of extensions - without acceleration. It is possible to just speed up. I settled on this command:
As you can see, some extensions have the status of beta , and this remark is also about proto3. Do not doubt. This package is successfully used by many (see home page). Still, it does not exempt from writing tests. If extensions and others are not interested, then (as noted in the project's README) this command will suffice:
a spoon of tar
If you have not looked at the previous spoiler, then I would like to emphasize one of its fragments, here it is
The fact is that
Approach
Call
Approach
Well. You can try using the
A new slice is created - the old one flies into the firebox of the GC. Actually, if you have small structures ( fields of structures - and all together too ) - then the easiest way is not to steam. For large ones, look for workarounds (read rewrite generated code). With the current implementation to use the pool does not make sense.
Library convenient for streaming. Writing messages to
Since I started talking about
Since we are talking about speed and about the pool: github.com/valyala/fasthttp . "Fast" replacement of
Root
Actually, in the mentioned article it was told about packages
github.com/golang/protobuf/{proto,protoc-gen-go} . What is wrong with them? That is what is used reflection. Suppose you have a project that works with a specific set of structures. And these structures are now and again encoded in Protocol Buffers and back. If it were always different, unpredictable types, then there are no problems. But if the set is known in advance, there’s absolutely no need to use reflection. As you know, it is customary to use some interface that is responsible for encoding. Here's an example of a piece of encoding/json : type Marshaler interface { MarshalJSON() ([]byte, error) } type Unmarshaler interface { UnmarshalJSON([]byte) error } Reference: Marshaler , Unmarshaler .
If the encoder encounters a type embodying one of these interfaces, then in this case the whole work is assigned to their methods.
Simple json example
Not always
type X struct { Name string, Value int, } func (x *X) MarshalJSON() ([]byte, error) { return []byte(fmt.Sprintf(`{"name": %q, "value": %d}`, x.Name, x.Value)) } Not always
(Un)Marshaler look so rosy. For example, here you can read about yaml ( English ) and in general on this topic.')
Key
The solution is always simple. Use another package:
go get github.com/gogo/protobuf/{proto,protoc-gen-gogo,gogoproto,protoc-gen-gofast} These packages simply add convenience and acceleration.
About package (links):
As you can see acceleration is from 1.10x and higher. It is possible to simply use a set of extensions - without acceleration. It is possible to just speed up. I settled on this command:
protoc \ --proto_path=$GOPATH/src:$GOPATH/src/github.com/gogo/protobuf/protobuf:. \ --gogofast_out=. *.proto and you will get both extensions (if any) and acceleration.example
Not encouraging to use extensions, but for review.
will turn out
syntax="proto3"; package some; //protoc \ // --proto_path=$GOPATH/src:$GOPATH/src/github.com/gogo/protobuf/protobuf:. \ // --gogofast_out=. *.proto import "github.com/gogo/protobuf/gogoproto/gogo.proto"; // , Equal, option (gogoproto.equal_all) = true; option (gogoproto.goproto_stringer_all) = false; // Stringer ( ) option (gogoproto.stringer_all) = true; // - option (gogoproto.populate_all) = true; // option (gogoproto.testgen_all) = true; // option (gogoproto.benchgen_all) = true; // option (gogoproto.marshaler_all) = true; // option (gogoproto.sizer_all) = true; // option (gogoproto.unmarshaler_all) = true; // enums, - option (gogoproto.goproto_enum_prefix_all) = false; enum Bool { Yes = 0; No = 1; DontCare = 2; } message Some { option (gogoproto.goproto_unrecognized ) = false; option (gogoproto.goproto_getters) = false; Bool Waht = 1; int64 Count = 2; bytes Hash = 3; } will turn out
/* (Size, String ..) */ type Bool int32 const ( Yes Bool = 0 No Bool = 1 DontCare Bool = 2 ) // ... type Some struct { Waht Bool `protobuf:"varint,1,opt,name=Waht,proto3,enum=some.Bool" json:"Waht,omitempty"` Count int64 `protobuf:"varint,2,opt,name=Count,proto3" json:"Count,omitempty"` Hash []byte `protobuf:"bytes,3,opt,name=Hash,proto3" json:"Hash,omitempty"` } // proto.Message (github.com/golang/protobuf/proto) func (m *Some) Reset() { *m = Some{} } func (*Some) ProtoMessage() {} // func (m *Some) Marshal() (data []byte, err error) { // ... } // func (m *Some) Unmarshal(data []byte) error { // ... } As you can see, some extensions have the status of beta , and this remark is also about proto3. Do not doubt. This package is successfully used by many (see home page). Still, it does not exempt from writing tests. If extensions and others are not interested, then (as noted in the project's README) this command will suffice:
protoc --gofast_out=. myproto.proto Stones
a spoon of tar
If you have not looked at the previous spoiler, then I would like to emphasize one of its fragments, here it is
func (m *Some) Reset() { *m = Some{} } // The fact is that
gogo allows you to generate "fast" structures. In this case, you can use them with the "old" github.com/golang/protobuf/proto . This will use the Marshal and Unmarshal - this is not a problem. But what if you use the same instance of a structure many times. If the structure is large (no, huge), then by and large it would not hurt to use the pool and save the “waste” structures, and then retrieve them back - reuse them.Approach
github.com/golang/protobuf/proto . Reference . func Unmarshal(buf []byte, pb Message) error { pb.Reset() // return UnmarshalMerge(buf, pb) } Call
Reset . And consequently from *m = Some{} - the old structure is thrown out, a new one is created. This structure is small - do not care - but I would like to save Hash []byte ( I mean the allocated memory ), in case you use a big-hash.Approach
github.com/gogo/protobuf/proto similar - “copy-paste”. No glimpse.Well. You can try using the
Unmarshal method directly or UnmarshalMerge - just add your MyReset method, trim the length of the slice - leave the capacity. Not! Here is a line from the generated Unmarshal : m.Hash = append([]byte{}, data[iNdEx:postIndex]...) A new slice is created - the old one flies into the firebox of the GC. Actually, if you have small structures ( fields of structures - and all together too ) - then the easiest way is not to steam. For large ones, look for workarounds (read rewrite generated code). With the current implementation to use the pool does not make sense.
Bonus
Library convenient for streaming. Writing messages to
io.Writer , reading from io.Reader - this bike already exists.Since I started talking about
json : github.com/pquerna/ffjson . Similarly for json. Not just a generator - but a Swiss knife for json + Go .Since we are talking about speed and about the pool: github.com/valyala/fasthttp . "Fast" replacement of
net/http . Acceleration due to reuse of memory. And the same with additional features.Source: https://habr.com/ru/post/276905/
All Articles