Under my supervision about 1000 iron servers, I don’t even start counting VPS. A couple of dozen of them have very critical data. And a
banal ssh with keys in a standard situation is not safe enough . Not all “leather people” take care of their keys, let's talk how to protect ourselves against the possibility of losing a key by the user.
Whom we protect
"Standard" hosting company. The largest number of personnel in customer service and support. Honest SSH servers have access only to support 3 lines, about 12 people. The technical avant-garde of the company completes the set - “service department” by 8 people
Mass user
For the support department and part of the service department, an SSH authorization server is configured. This is the server that has the key for authorization almost anywhere. Almighty Key is one of the most valuable information not to be missed. 4 people in the office can read it directly. Backup on paper lies in the safe. Also on this server get temporary access developers who need to personally see where and how it broke.
Employee can get ssh access through the authorization server using the all-powerful key. Technically, these employees can call only one command - ssh, ssh, in turn, uses a private key to connect to a remote server. As a rule, employees use a local script to quickly use an authorization server. Historically, it is called go. Here are its contents:
')

Another plus of the common authorization server is the ability to log and read statistics for all employees. We know which employee spent much time on which server. In case of emergency situations, we know that the employee entered there and what answered him, since all input and output of the ssh session is copied.
Demigods
The second serious vulnerability is the omnipotent keys of several employees. Such keys are allowed to register on the server only with the parameter from = ””. At authorized_keys, it looks like this:

The mentioned omnipotent 4 people realize their importance and encrypt the disk of the laptop without forgetting about the password to the private part of the key.
Work outside the office is only possible using VPN to the office. If there is no electricity in the office, we have a backup VPN server that can also announce our office network.
Almighty server
The last serious point is a large service server. The server monitors almost everything and knows about all the pieces of iron and why they are needed, besides that, all the ansible tasks are launched on it. On the server is a private key with a password. After logging in to the server, the magic with ssh-agent in bashrc prompts you to enter a password for the key. Then you can work in full force. There is no direct ssh on this server, “two subheams, three whims” and you're on the server.
These rules do not override the normal configuration of the firewall on the servers. But the firewall is usually configured a little wider and lets ssh from the office network, then through sshd_config we press the list of ip which can become root:

And if all this does not work
The last bastion of verification on each server is the .bashrc file, when the shell initializes via ssh, bash starts, and it checks the source of the connection:

This number of restriction options is used to group the necessary level of protection without blocking the employee’s work. But at the same time to maintain confidence that the server is covered. As the company grows, I am less confident in programs running in the office. I trust all employees, but I sleep with difficulty.
Plans for finishing touches
Extend the functionality of the authorization server so that you can specify a list or mask of servers for each user. It is possible to add a bashrc so that at the time of connection, he dumped information about the user and ip. Next, raise the checkboxes if the user has a new ip or something like that. I'm not sure that this is a working scheme, but you can try it at your leisure.
Ps: Message from
bfuvx how to get around bashrc
ssh -vvv -i everebody root@149.154.64.101
We catch something like “debug2: shell request accepted on channel 0” and send at this moment “Ctrl + c”:
Last login failed: Tue Mar 20 02:41:12 EDT 2018 from 58.242.83.24 on ssh: notty
There were 23 failed login attempts since the last successful login.
Last login: Tue Mar 20 02:40:14 2018 from 95.154.75.23
^ C-bash-4.2 #