We do applications with search on Go

Once in the Golang Weekly mailing list, I came across the Bleve project. This is a full-text search that is written in Go. The project is interesting, and there was a fierce desire to get experience with it.

Bleve can store data in different embedded databases:

• BoltDB (uses by default)
• Leveldb
• RocksDB
• Goleveldb
• forestdb
• Gtreap

Working with Bleve is simple:

import "github.com/blevesearch/bleve" func main() { //    mapping := bleve.NewIndexMapping() index, err := bleve.New("example.bleve", mapping) //     err = index.Index(identifier, your_data) //  - query := bleve.NewMatchQuery("text") search := bleve.NewSearchRequest(query) searchResults, err := index.Search(search) } 

Everything is simple and clear. But it looks not from the real world. To be closer to the real world, let's make a bot for Slack, which will keep history.

Bot architecture

Service for working with slack;
Service index. To store and search messages.

Plan

• We take https://api.slack.com/methods/channels.history for working with slack;
• We take Bleve to search and store history;
• If a message came to us not by the bot's order, we put it in the index;
• If a message with a minor has arrived, we clean and search by the current channel;

Slack

With a sleek, everything is simple and the example will be a little more complicated than the example from the repo

The only thing we need is two methods to check if the bot is addressed to the message and clear it on behalf of the bot

 import "strings" func isToMe(message string) bool { return strings.Contains(message, fmt.Sprintf("<@%s>", ss.me)) } func cleanMessage(message string) string { return strings.Replace(message, fmt.Sprintf("<@%s> ", ss.me), "", -1) } 

Bleve

Considering that I like to use goleveldb as an embedded database for my projects. In this project I decided to use it.
Store in Bleve data will be more difficult in the form of:

 type IndexData struct { ID string `json:"id"` Username string `json:"username"` Message string `json:"message"` Channel string `json:"channel"` Timestamp string `json:"timestamp"` } 

Create an index with Goleveldb as a database:

 import ( "github.com/blevesearch/bleve" "github.com/blevesearch/bleve/index/store/goleveldb" ) func createIndex() (bleve.Index, error) { indexName := "history.bleve" index, err := bleve.Open(indexName) if err == bleve.ErrorIndexPathDoesNotExist { mapping := buildMapping() kvStore := goleveldb.Name kvConfig := map[string]interface{}{ "create_if_missing": true, } index, err = bleve.NewUsing(indexName, mapping, "upside_down", kvStore, kvConfig) } if err != nil { return err } } 

and the buildMapping method, which will create us a mapping for storage:

 func (ss *SearchService) buildMapping() *bleve.IndexMapping { ruFieldMapping := bleve.NewTextFieldMapping() ruFieldMapping.Analyzer = ru.AnalyzerName eventMapping := bleve.NewDocumentMapping() eventMapping.AddFieldMappingsAt("message", ruFieldMapping) mapping := bleve.NewIndexMapping() mapping.DefaultMapping = eventMapping mapping.DefaultAnalyzer = ru.AnalyzerName return mapping } 

The search is a bit more complicated:

 func (ss *SearchService) Search(query, channel string) (*bleve.SearchResult, error) { stringQuery := fmt.Sprintf("/.*%s.*/", query) // NewTermQuery  Query     ,      ch := bleve.NewTermQuery(channel) //  Query     .    .    .         .       . mq := bleve.NewMatchPhraseQuery(query) //  Query         rq := bleve.NewRegexpQuery(query) //  Query   ,     . qsq := bleve.NewQueryStringQuery(stringQuery) //   Query       Query. q := bleve.NewDisjunctionQuery([]bleve.Query{ch, mq, rq, qsq}) search := bleve.NewSearchRequest(q) search.Fields = []string{"username", "message", "channel", "timestamp"} return ss.index.Search(search) } 

Putting it all together, we get a bot that saves history and can search for it without heavy weight using the examples of ElasticSearch, Solr.

The full project code is available on Github.