Robbie Cannings

Consolidated SSH remote admin for multiple devices

Robbie Cannings -

This is perhaps one of the most significant manageability accomplishments in my current role, I'd highly recommend this for anyone running one or more devices which can be managed via SSH (unless of course you're able to better leverage APIs relevant to each managed platform).

Essentially a centralised Linux host is established, with SSH public keys exported to each managed device (see my other posts on importing keys to various devices). The outputs of commands remotely executed on each managed device can then be parsed through standard UNIX utilities (sed/awk/grep/etc), reported & emailed, scheduled, etc.

For this example, I will have four NetApp filers and four Brocade FC switches. This approach (with a single management host) is much more scalable, although after about 50 managed devices, I'd suggest scaling out to multiple management hosts.

Security

It probably goes without saying but any system which is permitted to remotely execute commands must be adequately secured. It goes without saying - if this management system is compromised, an attacker could easily infiltrate any system it has access to. I'd consider all of these points crucial to operating a central management system

  • Network segregation (separate VLAN etc to PROD, also separate to the infrastructure you're managing)
  • firewalled access only (all inbound and outbound traffic to pass through a firewall, no direct access
  • Individual user login into this system (accountability) - root login only via sudo, not directly permitted except for console
  • All shell activity recorded using utilities such as script
  • Patching patching patching - keep this system up to date

So now onto the more interesting functionality

SSH Key Exchange

Some of my other posts outline specific steps for key exchange with NetApp, Brocade, etc. Most enterprise Linux/UNIX shell based CLI permits key authentication, although sometimes they make you really work for it.

choose a local user on the Linux management system which will act as the "remote admin" user. In this example, I've chosen the user "remoteadmin". When exchanging your keys, be sure to use this designated shared remote account.

If you would prefer, you can add individual users' keys to each managed device, however this introduces some major scalability issues, you need to balance security with useability. Imagine if you have 150 devices, you have 2 new admins join and one admin departing. That is a lot of messing around with keys every time staff change. If you can find a way of orchestrating key updates, this is fantastic, but often not easily achievable

sudoers

Add all administrator accounts to the sudoers list, permitting them only to execute "sudo su - remoteadmin". You can either do this with an alias in the sudoers file, or create a small shell script (only root can modify, everyone can read) which does the same thing then permit sudo execution of that script.

Aliases

Each hostname can be set up as a shell alias. For instance, if I'm managing filer01 I would configure the following

linuxmgthost:remoteadmin> alias filer01='ssh -l root filer01'
linuxmgthost:remoteadmin> alias filer02='ssh -l root filer02'
linuxmgthost:remoteadmin>  alias filer03='ssh -l root filer03'
linuxmgthost:remoteadmin> alias filer04='ssh -l root filer04'
linuxmgthost:remoteadmin> alias switch01='ssh -l admin switch01'    
linuxmgthost:remoteadmin> alias switch02='ssh -l admin switch02'
linuxmgthost:remoteadmin> alias switch03='ssh -l admin switch03'
linuxmgthost:remoteadmin> alias switch04='ssh -l admin switch04'

Device_Lists

For scripting purposes, I generate device lists which are loaded as variables whenever the administrative account login script runs. For instance

linuxmgthost:remoteadmin> cat /etc/filer_list
filer01 cdot dubbo shared
filer02 cdot dubbo shared
filer03 7mode sydney government
filer04 7mode sydney government

filer_list=`awk '{print $1}' /etc/filer_list`

In this example, I've created a simple config file defining all my filers, including useful information such as ONTAP family, location and customer. In my bashrc file, I've generated a variable for all filers. This way, if I want to run a command on every system, I can do something like the following (assuming the admin account is the same on each device)

linuxmgthost:remoteadmin> for filer in ${filer_list} ; do echo ${filer} `ssh -l admin_acct ${filer} version` ; done

filer01 NetApp Release 8.2.1P3 Cluster-Mode: Fri Aug 22 03:58:40 PDT 2014
filer02 NetApp Release 8.2.1P3 Cluster-Mode: Fri Aug 22 03:58:40 PDT 2014
filer03 NetApp Release 8.1.4P1 7-Mode: Tue Feb 11 23:23:31 PST 2014
filer04 NetApp Release 8.1.4P1 7-Mode: Tue Feb 11 23:23:31 PST 2014

Other useful stuff

There are countless things you can easily orchestrate using a centralised Linux management system, such as

  • Periodic reports (capacity, performance snapshots, etc)
  • configuration backups (supportshow on a brocade switch, etc)
  • Deployments (scripts for FC zoning, volume provisioning, etc)
  • Chargeback data gathering

Ordinarily, vendor supplied reporting/orchestration is preferred (supportability, feature compatibility between versions, etc), however in many cases the vendor supplied utilities are simply not sufficient for custom requirements. For instance, if you use a naming scheme which contains a chargeback item (e.g. department or customer code), most vendor tools won't allow you to create a lookup table to translate this into a human-readable string (e.g. CG005 = "fred's Automotive"). In a Linux environment however, you can extract the code, translate it using a custom lookup table (e.g. exported daily from your ERP to a csv on the Linux management host) and dynamically produce reports against it.

To date, I have not found a repeatable task yet which cannot be orchestrated in Linux. Sometimes you need to work really hard to get the outcome you're after, but it can be done.