Elegant Go Web Server (Graceful Restart)

In this article I am going to describe Graceful Restart on Go. Graceful Restart is important for Go web applications. Go has one drawback. In Go, there is no way to reload the code during execution. Therefore, developers on Go encounter a problem that is not found in servers written in Java, .NET, or PHP. If you need to update the server code written on Go, then the server process must be stopped and a new process started. This reduces the availability of the server when the code is updated.

In the previous article I described the Balancer on Go in 200 lines . On the basis of the balancer, you can ensure high availability during the application update, but how then to update the balancer itself. Using a balancer can often just be superfluous. If your server is running on Mac OS X or Linux, then there is another way to update the server code and process all requests received at the time of restarting the server. This method is Graceful Restart.

The essence of Graceful Restart is that in unix / linux systems, open files and sockets are accessible to spawned processes. It is enough for them to know the value of the file descriptor (the file descriptor is an integer) in order to get access to the file or socket opened by the ancestor.

Here is a list of problems that need to be addressed to implement Graceful Restart on Go.
')

1. In Go, all open files are automatically closed at the end of the process (close-on-exec)
2. It is necessary to do something with the old keep-alive connections opened in the ancestor.

The first problem is solved in two ways. Using fnctl, you can remove the syscall.FD_CLOEXEC flag, or syscall.Dup creates a copy of the file descriptor, without the syscall.FD_CLOEXEC flag. These calls are not available in the Windows Go implementation, so this technique works on Mac OS X and Linux. In this example, I use syscall.Dup. This is easier than the first approach.

I solve the second problem by setting Timeout for connections in 10 seconds and turning off the server 11 seconds after Graceful Restart. The second problem can also be solved in two other ways: a net.Listner wrapper to calculate the number of open connections and the predetermination of func (c * conn) serve () , which is quite difficult in Go. Other behavior may be desirable. For example, what would the old process after Graceful Restart report an error and close the connections.

It is important to understand that after Graceful Restart, part of the web browsers will be connected to the old server thanks to keep-alive. New connections will be established with the new server. For clarity, which server processed which request I was in response from the server to indicate the PID of the process.

grace1.go

package main import ( "flag" "fmt" "net" "net/http" "os" "os/exec" "syscall" "time" "log" ) var FD *int = flag.Int("fd", 0, "Server socket FD") var PID int = syscall.Getpid() var listener1 net.Listener var file1 *os.File = nil var exit1 chan int = make(chan int) var stop1 = false func main() { fo1, err := os.Create(fmt.Sprintf("pid-%d.log", PID)) if err != nil { panic(err) } log.SetOutput(fo1) log.Println("Grace1 ", PID) flag.Parse() s := &http.Server{Addr: ":8080", ReadTimeout: 10 * time.Second, WriteTimeout: 10 * time.Second, } http.HandleFunc("/", DefHandler) http.HandleFunc("/stop", StopHandler) http.HandleFunc("/restart", RestartHandler) http.HandleFunc("/grace", GraceHandler) http.HandleFunc("/think", ThinkHandler) if *FD != 0 { log.Println("Starting with FD ", *FD) file1 = os.NewFile(uintptr(*FD), "parent socket") listener1, err = net.FileListener(file1) if err != nil { log.Fatalln("fd listener failed: ", err) } } else { log.Println("Virgin Start") listener1, err = net.Listen("tcp", s.Addr) if err != nil { log.Fatalln("listener failed: ", err) } } err = s.Serve(listener1) log.Println("EXITING", PID) <-exit1 log.Println("EXIT", PID) } func DefHandler(w http.ResponseWriter, req *http.Request) { fmt.Fprintf(w, "def handler %d %s", PID, time.Now().String()) } func ThinkHandler(w http.ResponseWriter, req *http.Request) { time.Sleep(5 * time.Second) fmt.Fprintf(w, "think handler %d %s", PID, time.Now().String()) } func StopHandler(w http.ResponseWriter, req *http.Request) { log.Println("StopHandler", req.Method) if(stop1){ fmt.Fprintf(w, "stopped %d %s", PID, time.Now().String()) } stop1 = true fmt.Fprintf(w, "stop %d %s", PID, time.Now().String()) go func() { listener1.Close() if file1 != nil { file1.Close() } exit1<-1 }() } func RestartHandler(w http.ResponseWriter, req *http.Request) { log.Println("RestartHandler", req.Method) if(stop1){ fmt.Fprintf(w, "stopped %d %s", PID, time.Now().String()) } stop1 = true fmt.Fprintf(w, "restart %d %s", PID, time.Now().String()) go func() { listener1.Close() if file1 != nil { file1.Close() } cmd := exec.Command("./grace1") err := cmd.Start() if err != nil { log.Fatalln("starting error:", err) } exit1<-1 }() } func GraceHandler(w http.ResponseWriter, req *http.Request) { log.Println("GraceHandler", req.Method) if(stop1){ fmt.Fprintf(w, "stopped %d %s", PID, time.Now().String()) } stop1 = true fmt.Fprintf(w, "grace %d %s", PID, time.Now().String()) go func() { defer func() { log.Println("GoodBye") }() listener2 := listener1.(*net.TCPListener) file2, err := listener2.File() if err != nil { log.Fatalln(err) } fd1 := int(file2.Fd()) fd2, err := syscall.Dup(fd1) if err != nil { log.Fatalln("Dup error:", err) } listener1.Close() if file1 != nil { file1.Close() } cmd := exec.Command("./grace1", fmt.Sprint("-fd=", fd2)) err = cmd.Start() if err != nil { log.Fatalln("grace starting error:", err) } log.Println("sleep11", PID) time.Sleep(10 * time.Second) log.Println("exit after sleep", PID) exit1<-1 }() } 

Run this program without go run .

 go build grace1.go ./grace1 

Now that the server is running, we have the following handlers

http://127.0.0.1:8080/ - default handler
http://127.0.0.1:8080/restart - normal server restart
http://127.0.0.1:8080/grace - Graceful server restart
http://127.0.0.1:8080/think - handler with delay

In order to check how it all works, I wrote another program on Go. She makes successive requests to the server, if there is no error, then the letter g is displayed on the screen, if the error is E. After each request, the program falls asleep for 10ms.

bench1.go

 package main import ( "net/http" "time" ) func main() { nerr := 0 ngood := 0 for i := 0; i < 10000; i++ { resp, err := http.Get("http://127.0.0.1:8080/") if err != nil { // error print("E") nerr++ }else{ print("g") ngood++ resp.Body.Close() } time.Sleep(10 * time.Millisecond) } println() println("Good:", ngood, "Error", nerr) } 

If you restart the server under load, then bench1.go gives the following picture.

 gggggggggggggggggggggggggggggggggggggggggggggggggggggggEEEEEgggggggggggggggggggg gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg gggggggggggggggggggggggggggggggggEEggggggggggggggggggggggggggggggggggggggggggggg ggggggggggggggggggggggggggggggggggggggggggggggggEEgggggggggggggggggggggggggggggg ggggggggggggggggggggggggEggggggggggggggggggggggggggggggggggggggggggggggggggggggg gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg gggEEggggggggggggggggggEgggggggggggggggggggEggggggggggggggggEEgggggggggggggggggE gggggggggggggggggggggEEgggggggggggggggggEggggggggggggggggggggEggggggggggggggggEE gggggggggggggggggEEgggggggggggggggggEEggggggggggggggggggEgggggggggggggggEEgggggg 

One or several letters, E, symbolizes the error and inaccessibility of the server during the restart. (I have repeatedly overloaded the server, so the letters E are often found)

If to use Graceful Restart I did not observe errors in general.