Running SSH commands on hundreds of servers using Go

What is the article about

In this article, we will write a simple Go program (100 lines long) that can execute commands via the SSH protocol on hundreds of servers, making it quite effective. The program will be implemented using go.crypto / ssh , an implementation of the SSH protocol by the authors of Go.

A more “advanced” version of the program written in this article is available on the githab called GoSSHa (Go SSH agent).

Introduction

In the company in which I work, a little more than 1 server, and to work effectively with our number of servers using the SSH protocol, the library libpssh was written based on libssh2. This library was written in C using libevent many years ago, and still copes well with its responsibilities, but is very difficult to maintain. Also, Google’s Go language began to gain popularity, including within our company, so I decided to try writing a replacement for libpssh with Go, and fix some of its flaws, while at the same time greatly simplifying the code and the complexity of support.

To get started, we will need the Go compiler (available at golang.org ) and the working hg command to download go.crypto / ssh using “go get”.
')

Beginning of work

Create a file "main.go" in some directory, preferably empty. Let's now write the “framework” of our program, and then implement the missing functions in the course of the article:

package main import ( "code.google.com/p/go.crypto/ssh" // ... ) // ... func main() { cmd := os.Args[1] //   - ,       hosts := os.Args[2:] //   (  ) -   results := make(chan string, 10) //        timeout := time.After(5 * time.Second) //  5    timeout   //       ssh.  makeKeyring()   config := &ssh.ClientConfig{ User: os.Getenv("LOGNAME"), Auth: []ssh.ClientAuth{makeKeyring()}, } //    goroutine (  OS thread)  ,  executeCmd()   for _, hostname := range hosts { go func(hostname string) { results <- executeCmd(cmd, hostname, config) }(hostname) } //     ,   "Timed out",      for i := 0; i < len(hosts); i++ { select { case res := <-results: fmt.Print(res) case <-timeout: fmt.Println("Timed out!") return } } } 

Apart from the fact that we need to write the functions makeKeyring () and executeCmd (), our program is ready! Thanks to the “magic of Go” we will establish a connection to all servers in parallel and execute the specified command on them, and in any case we will finish in 5 seconds by printing the results from all the servers that have managed to execute. Such a simple way of implementing a common timeout for all concurrently executing operations is possible thanks to the concept of channels and the presence of a select construct that allows you to communicate simultaneously between several channels: as soon as at least one of the structures in the case can be executed, the corresponding block of code will be executed.

Initializing data structures for go.crypto / ssh

We have not yet written makeKeyring () and executeCmd (), but most likely you will not see anything much interesting here. We will only log in using SSH keys and assume that the keys are located in .ssh / id_rsa or .ssh / id_dsa:

 type SignerContainer struct { signers []ssh.Signer } func (t *SignerContainer) Key(i int) (key ssh.PublicKey, err error) { if i >= len(t.signers) { return } key = t.signers[i].PublicKey() return } func (t *SignerContainer) Sign(i int, rand io.Reader, data []byte) (sig []byte, err error) { if i >= len(t.signers) { return } sig, err = t.signers[i].Sign(rand, data) return } func makeSigner(keyname string) (signer ssh.Signer, err error) { fp, err := os.Open(keyname) if err != nil { return } defer fp.Close() buf, _ := ioutil.ReadAll(fp) signer, _ = ssh.ParsePrivateKey(buf) return } func makeKeyring() ssh.ClientAuth { signers := []ssh.Signer{} keys := []string{os.Getenv("HOME") + "/.ssh/id_rsa", os.Getenv("HOME") + "/.ssh/id_dsa"} for _, keyname := range keys { signer, err := makeSigner(keyname) if err == nil { signers = append(signers, signer) } } return ssh.ClientAuthKeyring(&SignerContainer{signers}) } 

As you can see, we are returning the ssh.ClientAuth interface, which has the necessary methods for authorization on the server. For brevity, error handling is almost completely absent; in production-mode, the amount of code will be 1.5 times larger.

To execute the command on the server, the code is also quite trivial (error handling is dropped for brevity):

 func executeCmd(cmd, hostname string, config *ssh.ClientConfig) string { conn, _ := ssh.Dial("tcp", hostname+":22", config) session, _ := conn.NewSession() defer session.Close() var stdoutBuf bytes.Buffer session.Stdout = &stdoutBuf session.Run(cmd) return hostname + ": " + stdoutBuf.String() } 

For simplicity and brevity, we always use the current username for authorization on servers, as well as port 22 by default.

Our program is ready! The full source code of the program is under the spoiler:

Run our application:

 $ vim main.go #   :) $ go get #    $ time go run main.go 'hostname -f; sleep 4.7' localhost srv1 srv2 localhost: localhost srv1: srv1 Timed out! real 0m5.543s 

Works! The localhost, srv1 and srv2 servers had only 0.3 seconds to execute all the commands, and the slow srv2 did not have time. Together with the compilation of the program on the fly from source, the execution of the program took 5.5 seconds, of which 5 seconds is our default timeout for executing the command.

Conclusion

The article turned out to be short, but at the same time we wrote a very useful application that you can safely use in production. We tested the more advanced version of this application in the production environment and it showed excellent results.

References:

1. Go language: golang.org
2. go.crypto library: code.google.com/p/go/source/checkout?repo=crypto
3. GoSSHa (SSH-proxy with communication with the outside world via JSON): github.com/YuriyNasretdinov/GoSSHa