NTP - Time and tide wait for no man. —Geoffrey Chaucer

mail

NTP best practices

The Internetz are full of advice on everything. Some are part of the DO's and some clearly belong to the DONT's. Here's my summary when it comes to NTP.

Do not run a time server in a virtual machine :

  • NTP server was not designed to run inside of a virtual machine. It requires a high resolution system clock, with response times to clock interrupts that are serviced with a high level of accuracy. (source)
  • Clock accuracy within a VM is still really bad. This comes from a few spots, but the killer thing is that the time drift is not constant; the drift factor changes from moment to moment. NTP is a protocol that has clock compensation built within it, but it was designed with a static drift factor built in. For example, if a physical machine loses 12 seconds every 30 days, NTP can compensate for that and does so very well. But if that machine can lose anywhere from 4 to 70 seconds every 30 days, NTP isn't so good at tracking that level of change. (source)
NTP client is ok to run in some virtualization solutions. (source)

Use at least 4 time sources (source) :

Having the best algorithms and lots of CPU to spare is useless if your inputs are poor :
  • 1 source :
    • you have no clue if it's right or wrong, how far from right it is, and no way to get a hint about these
    • if it dies : game over
  • 2 sources : worst possible situation because there will ALWAYS be a difference, but you can't tell which is right / closer to be right : A man with a watch knows what time it is. A man with two watches is never sure. (Segal's law)
  • 3 sources :
    • minimum number of sources to allow ntpd to detect if one time source is a "falseticker"
    • in the case there is a falseticker, you fall in the "2 sources" scenario, providing no real redundancy
  • 4 sources : one (or more) can be a "falseticker", or just unreachable, and ntpd will have a sufficient number of sources to choose from

When using public NTP servers (source) :

  • Use a local NTP service, which itself updates from an external one. This :
    • survives internet connectivity outage
    • avoids DDoSing public servers with requests coming from hosts of a large network
    • reduces network latency and round-trip delays and provides a more robust time service
  • Avoid excessive use of public NTP servers : only query servers at reasonable intervals (from once a day to a few times an hour)
  • As for NTP clients :
    • they should never be configured to request time from a NTP server more frequently than once every four seconds. Clients that exceed this rate may be flagged as attempting a DoS attack and may be refused service.
    • configure your own internal NTP service for your network to limit the load on public servers

Do NOT run ntpdate with CRON (sources : 1, 2, 3)

This is a BAD1000 practice for many reasons :
  • clock jolt, inconsistent time : what ntpdate does :
    1. get the current time from a time server
    2. set the local clock to the received value
    The local clock may jump abruptly several seconds/minutes in the past or in the future. This can have dramatic consequences on time-sensitive applications by un-ordering logged events and altering/reversing the delay between them.
  • poor "time quality" : ntpdate just "resets" the clock at regular intervals, which cancels the clock drift but never fixes / compensates it. So the longer since the previous reset, the higher the drift, which will have a sawtooth shape.
  • thunderclap effect : if numerous lazy admins setup a crontab rule such as : 
    0 * * * * /usr/local/bin/ntpdate
    every hour, at the exact hour, all NTP clients will contact the NTP server simultaneously, which ends up in a kind of DDoS attack .
mail

NTP

For details on what NTP is and what it is for, see Wikipedia.

Software "doing" NTP come in several flavors :

... as well as some best practices.

"Full" NTP implementation

  • implement the full protocol and algorithms
  • have the best precision (provided they have an adequate number of time sources)
  • takes precautions by filtering out time discrepancies
  • has numerous security features, like authentication, which clarifies the origin of received time-stamps, assuring that data received is not malicious
  • can act as a time server
  • examples :

"Light" NTP implementation (NTP vs SNTP)

Here comes SNTP (more at Wikipedia, RFC) :
  • it doesn't implement all the NTP protocols and algorithms, offering less functionality than full implementation with the benefit of a smaller system footprint
  • this suits embedded systems (DVR, routers, ...) and applications where full NTP capacity is not required such as workstations
  • does not assess the stability or quality of time references
  • may accept any time-stamp that is provided, without taking any precautions to filter out time discrepancies. As a result, SNTP applications :
    • are vulnerable to malicious users as they have no security in place
    • deliver a lower-quality time sync solution than its NTP counterpart, lacking certain algorithms that ensure total time accuracy
  • can not act as a time server
  • examples :

sidekicks

These are mostly utilities coming along with a daemon :
mail

ntpd vs ntpdate service

Situation

I've been pretty puzzled by the fact that RHEL7 has a ntpdate service :
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)
systemctl list-unit-files | grep ntp
ntpd.service				enabled
ntpdate.service				enabled

Details

The common practice is to run :
  1. ntpdate at boot to set the clock to the right time
  2. then ntpd to stay synchronized and fix drift (How far is "too far" ?)

Solution

ntpdate is NOT a service, but may still appear when listing services. This is because ntpdate has been "wrapped" as a service by RHEL7 (details), so that it fits the boot process and daemons start order. This way, ntpdate is executed :