Flexible logging system on Go

This article is the hellish invention of the new bicycle. So on the production use only at your own peril and risk. I have been looking for a system for logging on Go for a long time that would satisfy my requests (flexible, the ability to notify by email, very fast and keeping logs in the muscle).

Frankly, I was looking for three days and did not find anything. Then I started writing my bike (the first version was very crooked and barely worked). Then I deleted all that code and started ~~thinking about~~ writing again.

I immediately understood what to write in the database every time is very tiring. By this I did this:

The library for each type of log makes a key in radish where it writes data in this format:

')

Data in radish

('Debug','2015-11-05 20:12:37.700052989 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700506704 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700663127 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700803651 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700987999 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701128513 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701293643 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701433496 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701602372 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701745287 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701925988 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702093499 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702276867 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702431455 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702581625 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702738953 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702899007 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703055622 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703210768 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.70340691 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703566623 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.7037252 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703954549 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704119435 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704281902 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704536707 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704721061 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704901908 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705106033 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705284342 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705465074 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705633484 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705802108 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705962381 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706129288 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706314702 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706463092 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706674268 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706848586 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707050005 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707221136 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707379335 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707583978 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707742422 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707967253 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708164671 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708410554 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708578324 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708775197 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708955609 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709184168 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709349784 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709510939 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709726286 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709940253 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710141611 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.71034329 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710537637 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710763157 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710969449 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711167704 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711355522 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711550562 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711756 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712048767 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712273974 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712517739 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712828333 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.71306392 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.713335398 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.713570618 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.71389819 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.714182802 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.714448273 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.714754937 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715018147 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715291228 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715596998 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715910118 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.7162719 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.716552975 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.716807074 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717153412 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717434854 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717704591 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717991896 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.718283451 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.718590239 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.718849058 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.719152303 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.719424972 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.719734567 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.720070491 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.720386241 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.720651655 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.72094698 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.721207595 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.721514296 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.721776408 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.722090163 +0200 EET','Testing'),

I immediately wanted to embed them in the sql query but came across an error in the sql syntax. For a long time I ~~fought about the keyboard,~~ looking for a bug in the code, but it turned out that there would always be a coma because of the way to add data to the radish. When I noticed her, I asked ~~Google for a long time~~ on the toaster how to remove this coma. It turned out to be easier nowhere:

 strings.TrimRight(data, ",")

Then I started trying to start the function from the library in a separate thread. Over time, I realized ~~that I’m a donkey~~ that I can’t do this, and so I brought it into a demon.

At the exit, I have:

-> .

-> mysql

On a weak VPS (frequency 1.6; 1 core, 2 gigabytes of RAM), 1,000,000 records were added to Muskul for 25 million seconds (DB without tuning), although these data were written in radishes for about 6 minutes. The whole load on the radish.

That's all.

Config

{
    

      
      
      
    

  
    

      
      
      
    

  "MailConf" : [
    

      
      
      
    

  " smtp ",
    

      
      
      
    

  "",
    

      
      
      
    

  "smtp ",
    

      
      
      
    

  " smtp "
    

      
      
      
    

  ],
    

      
      
      
    

  
    

      
      
      
    

  "MailTo" : [
    

      
      
      
    

  "___@gmail.com",
    

      
      
      
    

  "v.grabko99@yandex.ru"
    

      
      
      
    

  ],
    

      
      
      
    

  
    

      
      
      
    

  "Types" : [
    

      
      
      
    

  "Debug",
    

      
      
      
    

  "Info",
    

      
      
      
    

  "Warn",
    

      
      
      
    

  "Error",
    

      
      
      
    

  "Fatal"
    

      
      
      
    

  ],
    

      
      
      
    

  
    

      
      
      
    

  "EmailSend" : [
    

      
      
      
    

  "Error",
    

      
      
      
    

  "Fatal"
    

      
      
      
    

  ],
    

      
      
      
    

  
    

      
      
      
    

  "Redis" : [
    

      
      
      
    

  "localhost:6379",
    

      
      
      
    

  "parsh888",
    

      
      
      
    

  "log_"
    

      
      
      
    

  ],
    

      
      
      
    

  
    

      
      
      
    

  "MysqlConnect" : [
    

      
      
      
    

  "",
    

      
      
      
    

  "",
    

      
      
      
    

  " "
    

      
      
      
    

  ],
    

      
      
      
    

  
    

      
      
      
    

  "MysqlTable" : "log",
    

      
      
      
    

  "ReplicationTimeSecond" : 320
    

      
      
      
    

  
    

      
      
      
    

  }

Library

 package GeneralsLog import ( "encoding/json" "gopkg.in/redis.v3" "io/ioutil" "log" "microService/libs/mail" "time" ) type Config struct { MailTo, MailConf, Types, EmailSend, Redis, MysqlConnect []string MysqlTable string } var ( R *redis.Client MailConf map[string]string MailTo []string Types []string EmailSend []string RedisConfig []string //     config_file string = "/home/v-smerti/localhost/api/src/microService/config/log.json" ) func init() { //  bs, err := ioutil.ReadFile(config_file) if err != nil { log.Panicln(err) } b := []byte(bs) var conf Config err = json.Unmarshal(b, &conf) if err != nil { log.Panicln(err) } //        MailConf = map[string]string{ "username": conf.MailConf[0], "password": conf.MailConf[1], "host": conf.MailConf[2], "port": conf.MailConf[3], } MailTo = conf.MailTo Types = conf.Types EmailSend = conf.EmailSend RedisConfig = conf.Redis //   R = redis.NewClient(&redis.Options{ Addr: RedisConfig[0], Password: RedisConfig[1], DB: 0, }) // .         .     for _, typ := range Types { _, err := R.Get(RedisConfig[2] + typ).Result() if err == redis.Nil { if err := R.Set(RedisConfig[2]+typ, " ", 0).Err(); err != nil { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init____R.Set (      "+typ+")") } } else if err != nil { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init____R.Set (  "+typ+"  )") } } } func New(types string, messages string) { for _, typ := range Types { if typ == types { if data, err := R.Get(RedisConfig[2] + typ).Result(); err == nil { //           e-mail for _, b := range EmailSend { //  if b == typ { mail.Send(MailConf, MailTo, typ, messages) } } data = data + "('" + types + "','" + time.Now().String() + "','" + messages + "')," if err := R.Set(RedisConfig[2]+typ, data, 0).Err(); err != nil { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func "+typ+"____R.Set (   )") } } else { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func "+typ+"____R.Get (   )") } } else { } } }

Demon

package main

import (

"Database / sql"

_ "Github.com/go-sql-driver/mysql" // you can connect any sql database

"Gopkg.in/redis.v3"

"Log"

"MicroService / libs / mail"

"Strings"

"Time"

"Encoding / json"

"Io / ioutil"

)

type Config struct {

MailTo, MailConf, Types, EmailSend, Redis, MysqlConnect [] string

MysqlTable string

ReplicationTimeSecond time.Duration

}

var (

DB * sql.DB

R * redis.Client

MailConf map [string] string

MailTo [] string

Types [] string

EmailSend [] string

RedisConfig [] string

MysqlTable string

ReplicationSecond time.Duration

// Path to the file with configs

config_file string = "/home/v-smerti/localhost/api/src/microService/config/log.json"

)

func init () {

print ("Starting ...")

// Sparsim config

bs, err: = ioutil.ReadFile (config_file)

if err! = nil {

log.Panicln (err)

}

b: = [] byte (bs)

var conf Config

err = json.Unmarshal (b, & conf)

if err! = nil {

log.Panicln (err)

}

// Transfer data from file config to global variables

MailConf = map [string] string {

"Username": conf.MailConf [0],

"Password": conf.MailConf [1],

"Host": conf.MailConf [2],

"Port": conf.MailConf [3],

}

MailTo = conf.MailTo

Types = conf.Types

EmailSend = conf.EmailSend

RedisConfig = conf.Redis

MysqlTable = conf.MysqlTable

ReplicationSecond = conf.ReplicationTimeSecond

// Kontek with radish

R = redis.NewClient (& redis.Options {

Addr: RedisConfig [0],

Password: RedisConfig [1],

DB: 0,

})

// connection with database

db, err: = sql.Open ("mysql", conf.MysqlConnect [0] + ":" + conf.MysqlConnect [1] + "@ /" + conf.MysqlConnect [2])

if err! = nil {

log.Fatal (err)

}

DB = db

// Initialize the package. Here, check whether there are such types of logs in radish. If not, then create

for _, typ: = range Types {

_, err: = R.Get (RedisConfig [2] + typ) .Result ()

if err == redis.Nil {

if err: = R.Set (RedisConfig [2] + typ, "", 0) .Err (); err! = nil {

mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init ____ R.Set (Creating an empty radish entry key" + typ + ")")

}

} else if err! = nil {

mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init ____ R.Set (Check entry" + typ + "in radish)")

}

}

print ("Ok!")

}

func main () {

for {

replication_db ()

time.Sleep (time.Second * ReplicationSecond)

}

}

func replication_db () {

for _, typ: = range Types {

data, err: = R.Get (RedisConfig [2] + typ) .Result ()

if err == redis.Nil {

mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.Set (Not in radish" + typ + ")")

if err: = R.Set (RedisConfig [2] + typ, "", 0) .Err (); err! = nil {

mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.Set (Creating an empty radish entry key" + typ + ")")

}

} else if err! = nil {

log.Fatal (err)

mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.GET (Fatal radish error)")

} else {

if data! = "" {

_, err: = DB.Exec ("INSERT INTO" + MysqlTable + "(type, time, messages) VALUES" + strings.TrimRight (data, ","))

if err! = nil {

log.Fatal (err)

} else {

log.Println ("replication")

}

if err: = R.Set (RedisConfig [2] + typ, "", 0) .Err (); err! = nil {

mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.Set (Clearing the entry" + typ + ")")

}

}

}

}

}

The code still has the import of the mail package.

Package mail

 package mail import ( "fmt" "net/smtp" ) func Send(conf map[string]string, to []string, subject string, msg string) error { auth := smtp.PlainAuth( "", conf["username"], conf["password"], conf["host"], ) address := fmt.Sprintf("%v:%v", conf["host"], conf["port"]) body := []byte("Subject: " + subject + "\r\n\r\n" + msg) err := smtp.SendMail( address, auth, conf["username"], to, body, ) if err != nil { return err } return nil }

 -- --   `log` -- CREATE TABLE IF NOT EXISTS `log` ( `id` int(11) NOT NULL AUTO_INCREMENT, `type` text NOT NULL, `time` datetime NOT NULL, `messages` text CHARACTER SET utf32 NOT NULL, PRIMARY KEY (`id`), KEY `type` (`type`(191)) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;

Source: https://habr.com/ru/post/270263/

All Articles

Flexible logging system on Go

More articles: