📜 ⬆️ ⬇️

Flexible logging system on Go

This article is the hellish invention of the new bicycle. So on the production use only at your own peril and risk. I have been looking for a system for logging on Go for a long time that would satisfy my requests (flexible, the ability to notify by email, very fast and keeping logs in the muscle).

Frankly, I was looking for three days and did not find anything. Then I started writing my bike (the first version was very crooked and barely worked). Then I deleted all that code and started thinking about writing again.

I immediately understood what to write in the database every time is very tiring. By this I did this:

The library for each type of log makes a key in radish where it writes data in this format:
')
Data in radish
('Debug','2015-11-05 20:12:37.700052989 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700506704 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700663127 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700803651 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.700987999 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701128513 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701293643 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701433496 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701602372 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701745287 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.701925988 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702093499 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702276867 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702431455 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702581625 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702738953 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.702899007 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703055622 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703210768 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.70340691 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703566623 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.7037252 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.703954549 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704119435 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704281902 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704536707 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704721061 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.704901908 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705106033 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705284342 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705465074 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705633484 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705802108 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.705962381 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706129288 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706314702 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706463092 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706674268 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.706848586 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707050005 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707221136 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707379335 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707583978 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707742422 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.707967253 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708164671 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708410554 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708578324 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708775197 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.708955609 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709184168 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709349784 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709510939 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709726286 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.709940253 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710141611 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.71034329 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710537637 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710763157 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.710969449 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711167704 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711355522 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711550562 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.711756 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712048767 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712273974 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712517739 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.712828333 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.71306392 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.713335398 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.713570618 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.71389819 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.714182802 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.714448273 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.714754937 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715018147 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715291228 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715596998 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.715910118 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.7162719 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.716552975 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.716807074 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717153412 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717434854 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717704591 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.717991896 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.718283451 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.718590239 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.718849058 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.719152303 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.719424972 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.719734567 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.720070491 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.720386241 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.720651655 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.72094698 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.721207595 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.721514296 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.721776408 +0200 EET','Testing'),('Debug','2015-11-05 20:12:37.722090163 +0200 EET','Testing'),


I immediately wanted to embed them in the sql query but came across an error in the sql syntax. For a long time I fought about the keyboard, looking for a bug in the code, but it turned out that there would always be a coma because of the way to add data to the radish. When I noticed her, I asked Google for a long time on the toaster how to remove this coma. It turned out to be easier nowhere:

 strings.TrimRight(data, ",") 

Then I started trying to start the function from the library in a separate thread. Over time, I realized that I’m a donkey that I can’t do this, and so I brought it into a demon.

At the exit, I have:

-> .
-> mysql

On a weak VPS (frequency 1.6; 1 core, 2 gigabytes of RAM), 1,000,000 records were added to Muskul for 25 million seconds (DB without tuning), although these data were written in radishes for about 6 minutes. The whole load on the radish.

That's all.

Config
{

"MailConf" : [
" smtp ",
"",
"smtp ",
" smtp "
],

"MailTo" : [
"___@gmail.com",
"v.grabko99@yandex.ru"
],

"Types" : [
"Debug",
"Info",
"Warn",
"Error",
"Fatal"
],

"EmailSend" : [
"Error",
"Fatal"
],

"Redis" : [
"localhost:6379",
"parsh888",
"log_"
],

"MysqlConnect" : [
"",
"",
" "
],

"MysqlTable" : "log",
"ReplicationTimeSecond" : 320

}


Library
 package GeneralsLog import ( "encoding/json" "gopkg.in/redis.v3" "io/ioutil" "log" "microService/libs/mail" "time" ) type Config struct { MailTo, MailConf, Types, EmailSend, Redis, MysqlConnect []string MysqlTable string } var ( R *redis.Client MailConf map[string]string MailTo []string Types []string EmailSend []string RedisConfig []string //     config_file string = "/home/v-smerti/localhost/api/src/microService/config/log.json" ) func init() { //  bs, err := ioutil.ReadFile(config_file) if err != nil { log.Panicln(err) } b := []byte(bs) var conf Config err = json.Unmarshal(b, &conf) if err != nil { log.Panicln(err) } //        MailConf = map[string]string{ "username": conf.MailConf[0], "password": conf.MailConf[1], "host": conf.MailConf[2], "port": conf.MailConf[3], } MailTo = conf.MailTo Types = conf.Types EmailSend = conf.EmailSend RedisConfig = conf.Redis //   R = redis.NewClient(&redis.Options{ Addr: RedisConfig[0], Password: RedisConfig[1], DB: 0, }) // .         .     for _, typ := range Types { _, err := R.Get(RedisConfig[2] + typ).Result() if err == redis.Nil { if err := R.Set(RedisConfig[2]+typ, " ", 0).Err(); err != nil { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init____R.Set (      "+typ+")") } } else if err != nil { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init____R.Set (  "+typ+"  )") } } } func New(types string, messages string) { for _, typ := range Types { if typ == types { if data, err := R.Get(RedisConfig[2] + typ).Result(); err == nil { //           e-mail for _, b := range EmailSend { //  if b == typ { mail.Send(MailConf, MailTo, typ, messages) } } data = data + "('" + types + "','" + time.Now().String() + "','" + messages + "')," if err := R.Set(RedisConfig[2]+typ, data, 0).Err(); err != nil { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func "+typ+"____R.Set (   )") } } else { mail.Send(MailConf, MailTo, "Fatal error game", "package GeneralsLog func "+typ+"____R.Get (   )") } } else { } } } 


Demon
package main

import (
"Database / sql"
_ "Github.com/go-sql-driver/mysql" // you can connect any sql database
"Gopkg.in/redis.v3"
"Log"
"MicroService / libs / mail"
"Strings"
"Time"

"Encoding / json"
"Io / ioutil"
)

type Config struct {
MailTo, MailConf, Types, EmailSend, Redis, MysqlConnect [] string
MysqlTable string
ReplicationTimeSecond time.Duration
}

var (
DB * sql.DB
R * redis.Client
MailConf map [string] string
MailTo [] string
Types [] string
EmailSend [] string
RedisConfig [] string
MysqlTable string
ReplicationSecond time.Duration
// Path to the file with configs
config_file string = "/home/v-smerti/localhost/api/src/microService/config/log.json"
)

func init () {
print ("Starting ...")
// Sparsim config
bs, err: = ioutil.ReadFile (config_file)
if err! = nil {
log.Panicln (err)
}

b: = [] byte (bs)
var conf Config
err = json.Unmarshal (b, & conf)
if err! = nil {
log.Panicln (err)
}
// Transfer data from file config to global variables
MailConf = map [string] string {
"Username": conf.MailConf [0],
"Password": conf.MailConf [1],
"Host": conf.MailConf [2],
"Port": conf.MailConf [3],
}

MailTo = conf.MailTo
Types = conf.Types
EmailSend = conf.EmailSend
RedisConfig = conf.Redis
MysqlTable = conf.MysqlTable
ReplicationSecond = conf.ReplicationTimeSecond

// Kontek with radish
R = redis.NewClient (& redis.Options {
Addr: RedisConfig [0],
Password: RedisConfig [1],
DB: 0,
})

// connection with database
db, err: = sql.Open ("mysql", conf.MysqlConnect [0] + ":" + conf.MysqlConnect [1] + "@ /" + conf.MysqlConnect [2])
if err! = nil {
log.Fatal (err)
}
DB = db

// Initialize the package. Here, check whether there are such types of logs in radish. If not, then create
for _, typ: = range Types {
_, err: = R.Get (RedisConfig [2] + typ) .Result ()
if err == redis.Nil {
if err: = R.Set (RedisConfig [2] + typ, "", 0) .Err (); err! = nil {
mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init ____ R.Set (Creating an empty radish entry key" + typ + ")")
}
} else if err! = nil {
mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func Init ____ R.Set (Check entry" + typ + "in radish)")
}
}
print ("Ok!")
}

func main () {
for {
replication_db ()
time.Sleep (time.Second * ReplicationSecond)
}
}

func replication_db () {

for _, typ: = range Types {
data, err: = R.Get (RedisConfig [2] + typ) .Result ()
if err == redis.Nil {

mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.Set (Not in radish" + typ + ")")
if err: = R.Set (RedisConfig [2] + typ, "", 0) .Err (); err! = nil {
mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.Set (Creating an empty radish entry key" + typ + ")")
}
} else if err! = nil {
log.Fatal (err)
mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.GET (Fatal radish error)")

} else {
if data! = "" {
_, err: = DB.Exec ("INSERT INTO" + MysqlTable + "(type, time, messages) VALUES" + strings.TrimRight (data, ","))
if err! = nil {
log.Fatal (err)
} else {
log.Println ("replication")
}
if err: = R.Set (RedisConfig [2] + typ, "", 0) .Err (); err! = nil {
mail.Send (MailConf, MailTo, "Fatal error game", "package GeneralsLog func replication_db ____ R.Set (Clearing the entry" + typ + ")")
}
}

}
}
}


The code still has the import of the mail package.

Package mail
 package mail import ( "fmt" "net/smtp" ) func Send(conf map[string]string, to []string, subject string, msg string) error { auth := smtp.PlainAuth( "", conf["username"], conf["password"], conf["host"], ) address := fmt.Sprintf("%v:%v", conf["host"], conf["port"]) body := []byte("Subject: " + subject + "\r\n\r\n" + msg) err := smtp.SendMail( address, auth, conf["username"], to, body, ) if err != nil { return err } return nil } 


 -- --   `log` -- CREATE TABLE IF NOT EXISTS `log` ( `id` int(11) NOT NULL AUTO_INCREMENT, `type` text NOT NULL, `time` datetime NOT NULL, `messages` text CHARACTER SET utf32 NOT NULL, PRIMARY KEY (`id`), KEY `type` (`type`(191)) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ; 

Source: https://habr.com/ru/post/270263/


All Articles