Let’s say one of your techs is configuring a firewall and makes a mistake that results in the firewall (and the rest of the network) no longer being accessible.
What are your options for fixing the issue?
Plugging in with a serial cable
Network elements (such as firewalls, routers and switches) can often be configured through serial connections to the device. A serial connection provides command line management of the device even when the device’s network interfaces aren’t accessible.
You can plug a laptop into the serial port of the firewall with a USB to serial cable and restore a previous configuration to bring the firewall back up.
Doing this—connecting to a device through an alternate connection (like a serial interface) without using the production network—is called out-of-band management. You can use out-of-band management to access network devices even when the production network isn’t accessible.
When bare metal can’t be reached
Another problem is when a bare metal server becomes completely inaccessible. If the server’s operating system has become corrupt to the point of no longer booting, you can’t connect to the machine remotely using RDP or SSH, so you can’t reinstall the OS or restore a backup remotely.
In these cases, you can manage the server out-of-band using intelligent platform management interfaces (IPMI). Different vendors have different names for IPMI, such as iLO, iDRAC, and IMM, but they all work essentially the same way.
The interfaces act as an IP switch, providing keyboard, video, and mouse access to the server remotely over the network. Even if the operating system becomes corrupt and the server is no longer accessible, it can be rebooted over IPMI and the OS repaired or even reinstalled.
The only issue with these out-of-band management options is they both require a truck roll—you need to send a tech onsite to physically plug into the devices.
But what if your site or client is three hours away? Now a simple fix has quickly become an expensive fix.
There is another way though…
Console servers to the rescue
Console servers have multiple serial ports that can be connected to the serial console ports of multiple network elements. In the past, console servers were accessed remotely using dial-up modems. These days most console servers support 3G or LTE connections, or an alternate internet connection using a secondary ISP.
You can then configure the network device remotely using the CLI over the serial console ports of multiple devices, as if you were actually sitting on site with a laptop plugged in.
Even if the internet connection is broken during a storm, you can still remotely manage network devices through the console server’s 3G or LTE connections. The site won’t have internet access until the line is repaired by the ISP, but you’ll still be able to ensure local infrastructure is accessible. And you’ll look like a superstar when you’re able to fix local connectivity issues remotely even with the ISP down.
Since IPMI uses NICs for access, IPMI devices aren’t accessible if the network isn’t accessible. Fortunately, console servers come with multiple NICs allowing server IPMIs to connect directly to the console server, separate from the rest of the local LAN.
In this way, you can also remotely access an IPMI-equipped server through the console server even if the internet connection is down.
Console servers can be costly—but for top-tier clients or sites that are essential to keep up, especially ones located far from you, they can be totally worth the investment to ensure you can remotely administer the network even when the network isn’t accessible.
Great article! Do you have any recommendations for console servers? As an MSP, while it may not make financial sense to keep a console server at each client’s remote location, it may make sense to keep one on hand that we can overnight ship to a client to temporarily diagnose and configure.
Thanks for leaving a comment, Sam! Unfortunately, for a console server to work in such a situation, it would be necessary to connect the console server to the serial and / or IPMI interfaces of the devices, so a truck roll would still be necessary. It’s better to set up the console server beforehand. For the customer who has a remote location hundreds of miles away that you haven’t yet had the chance to plan for, there are a number of console servers on the market with low port density. Models with cellular modems (for 3G or LTE) will be more expensive than those without that feature.
For such remote access situations, I’ve previously worked with a company that provides dispatch techs to put hands on far-flung equipment. I’m specifically thinking of Cook Technologies based in Jax, FL but I’m sure there are similar contractors in other areas. Sometimes it is cheaper or more time effective to pay someone else to put hands on the equipment. They may not know the ins and outs of the data center, but all they really need to know in this case is which cable to plug into the iLo port and power it up. I briefly was employed by them and drove hundreds of miles sometimes to flip a switch or reset and provision a VOIP phone or rack a server.