Network Time Synchronization: Why and How It Works

When something goes wrong, you need to look through your log messages and figure out important things like which device saw the problem first. This automatically tells you where to start looking for the root cause. If your clocks aren’t synchronized, it becomes much more difficult to correlate log messages between devices.

More generally, you want to know if the similar log messages you’re seeing are related to the same incident or if maybe some of them happened much earlier or later. And, related to this, are you seeing similar problems at around the same time every day?

The other important reason to synchronize the clocks on your network devices is so they can in turn provide time services to other devices at the same site.

The importance of time synchronization

Time, more importantly, accurate time shared by all devices is vital for network communications to be successful. When events happen, the order in which they happen, and how that data is then collected and logged depends almost entirely on devices being in synchronization.

According to time synchronization device manufacturer Symmetricom, key areas where time synchronization directly affects network operations are:

• Log file accuracy, auditing & monitoring
• Network fault diagnosis and recovery
• File time stamps
• Directory services
• Access security and authentication
• Distributed computing
• Scheduled operations
• Real-world time values

Accurate time may also be a legal requirement. Depending on your industry, such as healthcare, some legal regulations require accurate timestamps on all logs.

Time synchronization is also something that needs to be constantly maintained. No time-keeping device is truly accurate, so without proper synchronization, the natural drift of between devices’ internal time-keeping will ultimately lead to issues.

How to synchronize clocks on network devices

There are really two ways to synchronize clocks for network optimization.

The first is to do it manually by logging into every network device and manually setting the clocks against some common source, like your watch.

This is a terrible solution.

The clock in every device will slip a little bit, probably in random directions, so there could be a difference of several seconds between the fastest and the slowest clocks by the end of the day. Then you’ll need to do it again tomorrow. And it’s incredibly time-consuming to log into all of those devices and manually set the clocks.

The second way is to use an automatic time synchronization protocol. There are two in common use.

The one we normally use is Network Time Protocol (NTP). It’s a mature protocol that’s been around for decades and is currently in its fourth version. NTP is accurate to about 1 ms, although you can get sub-millisecond accuracy with NTPv4 if you design the network carefully and have access to your own Stratum 1 clocks. (I’ll explain what that means in a moment.)

The other common network clocking protocol is called Precision Time Protocol (PTP), or IEEE 1588. This protocol is able to synchronize clocks to sub-microsecond accuracy. It is, however, more complicated to deploy and not universally available on network equipment.

For most applications, NTP is sufficient.

People tend to use PTP for applications where a microsecond is a long time. This includes things like industrial and financial (stock trading) applications.

Power grids, in particular, need extraordinary accuracy for ensuring the phases of the 60Hz (or 50Hz in many locations) AC currents are correctly matched. If the phase isn’t matched, power is lost. Lost power is heat. So losing a lot of power means possibly creating a lot of heat. In this environment, unsynchronized clocks could result in fires or damaged equipment.

At 60Hz, one full cycle is about 17ms. If you want to match the phase to within 1%, you need the clocks synchronized to 170 microseconds. This is just within the reach of NTPv4, but not previous versions. And there are other industrial applications where even more accuracy is required.

Why syncing clocks is hard

Accurate clock synchronization presents several interesting problems. The first is latency.

If I send you a packet saying to set your clock to 12:00, you need to know very accurately how long it took for that packet to reach you. You can make a pretty good guess at how long it takes for that packet to get to you by sending the server a packet and making careful notes of how long it takes to get the response.

This assumes the path to the server is the same as the path from the server, which is usually a fairly reasonable assumption in most modern networks. It also assumes the server is able to turn around the response instantaneously, which is clearly not all that accurate. So the NTP protocol has been refined by including additional timestamps for when the packets are sent and received at each end.

You also need some statistical information on that network latency so you know how much it varies over time when a packet occasionally needs to be buffered because a link is busy.

Then you need to keep track of something called clock skew. This is a quantitative measurement of how accurate your clock is. Is it a little bit fast or a little bit slow? And does it gradually speed up and slow down over time?

All of these parameters can be estimated by keeping careful statistics on how much the clock needs to be adjusted each time there’s new time information from the server. Cloud-based network monitoring can typically help.

The more network hops the packets need to cross on their way from the server to the clock being synchronized, the more chances there are for small random queuing delays. Since we’re collecting statistical information on the delays, the protocol is still able to get a pretty good result. But clearly, the more sources of random delays, the less accurate those statistics will be.

And on top of all these issues, we ideally want to keep the total number of time synchronization packets fairly low to avoid causing congestion either on the network or on the NTP server. So, once it has established a reliable synchronization, NTP very cleverly adjusts the timers to back off the time between requests it sends to the server.

Designing NTP into a network

NTP is a hierarchical protocol. The source clock is called Stratum 1. The clocks synchronized from the original source are Stratum 2. Clocks synchronized to the Stratum 2 clocks are called Stratum 3, and so forth.

A typical NTP network design involves using a pair of Stratum 1-time sources. Using multiple sources improves overall accuracy, as well as provides some redundancy in case one of the clocks is temporarily unreachable.

There are several Stratum 1 clocks available on the public internet, free of charge. Refer to https://ntp.org/ for useful lists of load-balanced public NTP servers.

Generally, you’ll want to use the servers closest to you to minimize network hops. To help with this, the NTP.org site includes lists of geographically sorted, publicly available NTP servers.

You’ll generally want to have a small number of servers inside your network that act as the Stratum 2 servers for your organization. I like to use either the Active Directory domain controllers or some network device located close to the Internet edge, like a firewall or a router.

Again, for reliability reasons, you should have a few Stratum 2 servers. Then you can make all other internal devices synchronize their clocks to these Stratum 2 servers so that nothing is more than Stratum 3.

In organizations with multiple data centers or multiple geographically diverse internet connections, I like to put at least one of my main Stratum 2 servers in each location.

Note that you should not expose your Stratum 2 devices directly to the internet unless they’re firewalls. There could be bugs in the NTP implementation that might compromise your network security.

If you need particularly accurate clock synchronization, it’s worthwhile investigating a standalone Stratum 1 server. These can be purchased as appliances you install in your data center. Your own Stratum 1 server means you’re not dependent on external resources, and you can control the latency between the devices.

There are generally two types of Stratum 1 devices. The less expensive variety has an antenna and takes its clock signal from the GPS satellite network. The more expensive variety has its own internal atomic clocks (usually using a Rubidium oscillator) or some other high-accuracy clock. These clocks need to take their initial time signal from some external source.

Cisco NTP configuration

Configuring NTP on Cisco devices is quite simple. You just need to point to the server that you’ll be synchronizing to.

!
ntp server 192.168.1.1
!

On ASA firewalls, this configuration is slightly different because you need to specify which interface you’ll be using to reach the NTP server.

!
ntp server 192.168.1.1 source management
!

In addition to these commands, on several router and switch models, you want to let NTP update the hardware clock, rather than just trying to keep track of the time in software. This provides greater accuracy.

!
ntp update-calendar
!

For edge devices, you’ll probably want to use a DNS name. This requires that your device be configured to use DNS as well. Then you can specify one of the load-balanced groups mentioned on the NTP.org lists.

!
ntp server 0.north-america.pool.ntp.org
ntp server 1.north-america.pool.ntp.org
!

Please carefully review the restrictions and guidelines on the use of these public pools before you use them.

Other time considerations

At the beginning of this article, I mentioned that one of the important reasons to synchronize clocks on network devices is part of network monitoring and troubleshooting: to allow correlation of log messages. So you’ll also need to make sure your devices include a timestamp in each log message.

!
logging enable
logging timestamp
!

I also like to include the date in my timestamps, so I can easily look at a log message and know whether it came today or weeks ago.

!
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
!

And, of course, you’ll need to set the appropriate time zone. The first of these commands set the timezone on this device to EST, which is 5 hours west of UTC (London, UK). The second command indicates we’re following normal daylight savings time rules.

!
clock timezone EST -5
clock summer-time EDT recurring
!

There are several different philosophies on what time zone to set your device clocks to. Some people like to set the time zone on the device to the actual location of that device.

Some like to set the time zone on all devices to a common central zone, usually where the network management staff are located.

A third approach, which makes sense in extremely large networks that span the globe, particularly when the network management staff are located in many different places, is to simply leave the clocks at UTC (-0).