Month: July 2026

Trust Stores need updates too

In contemporary IP network connected devices, encryption is a requirement. Not just on inbound connections for administration, but outbound integrations as well.

Those updates obviously includes adding new protocols when they are new, as well as removing older protocols when they are dangerous to keep using (though there is some wiggle room there if users insist on using insecure crypto if they have dependencies they cannot update). An example of this is adding in TLS 1.3 support (released 2018, some 8 years ago), and removing TLS 1.0 (deprecated by IEFT in 2021). That wiggle room is demonstrated by Microsoft only retiring TLS 1.0 much later (see this).

And while protocols are one thing, we have two other areas:

  • Encryption Ciphers
  • TLS Certificate Trust Store

The Trust Store is the set of pre-defined TLS Certificate Authorities Root Certificates that your device will trust by default. Depending on the fatures of your device, you may be able to extend this with additional Customer trusted certificate authorities, and potentially disable some of the default trusted CAs.

The Certificate Authority (CA) Root Trust store is often included in the base firmware or software of your computer or device, issued by the manufacturer. They get to chose what Certificate Authorities they will put in here. This gets added into the firmware you download (or is pre-installed).

The challenge here is, like with all TLS Certificates, they have an expiry date, and new ones are being added.

My recent experience with this was a Yealink VOIP/DECT phone system trying to make an outbound HTTPS connection to fetch a centralised address book (“Phone Book”). Yealink defines an XML format that can be placed (or generated) on a webserver, and then fetched by the base station periodically.

However, even with the latest firmware 146.87.0.30 (as at this date), it doesn’t support the ISRG Root X3 certificate, as used by Let’s Encrypt, with a NotBefore date of 2020. and expiring in 2040. Luckily, Yealink supports users adding additional Custom CA certificates to the local trust store. That’s a good work around, but really the internal trust store from the manufacturer should include the current and upcoming CA Root Certificates for all the major worldwide public CAs of strong reputation (those that confirm to the Baseline Requirements of the Browser/CA Forum).

At the same time, its also good to remove the expired Root certificates as a janitorial service. If an older CA is compromised, and local time is badly skewed, then this could prevent an compromise vector.

Vendors should monitor global CAs and watch for their new Root certificates being generated, and add them early to avoid connectivity issued. In “this case with the YeaLink W70B, the only error message was “Connect Error”.

Dear Yealink, see ticket: ID 519067.

The importance of time in networks

Last week saw a major incident for an Australian telco that literally stopped trains, business, and other services across Australia. After a week it was revealed that the company’s internal time servers had not been well maintained, and flipped over 19.x years of uptime, causing them to warp back in time20 years.

Apparently it was a known issue; engineers had warned about it, the vendor had many outstanding updates to be applied. But these devices had just been kept up and running.

There were multiple of them, but when operations teams are stripped down, clearly what should be essential maintenance just doesn’t happen.

The original Network Time Protocol (NTP, and the newer Precision Time Protocol (PTP) is one that often originates from atomic clocks. One readily available source is from GPS satellites, happily blasting the time across your location routinely. These days that’s also joined by Europe’s Galileio, the Russian GLONASS, and CHina’s BeiDou.

And while you think the correct time is universal these days (check your mobile/cell phone – you probably find it has the right time), these signals are often jammed and interferredwith by various national entities during conflicts and other activities to confuse positioning systems. Yes, it’s like the plot of James Bond’s Tomorrow Never Dies.

In every network I have ever had, pre cloud or post cloud, having a reliable source of time was always critical. Logs must line up to the millisecond, if not then more precise than that.

Having scalable time services is even more important, because at scale you are relying on correct time even more. Most network operators normally have multiple time servers, listening to upstream time providers, locate din different buildings, on different UPS, or in different data centers, etc..

In order to protect their primary time servers, organisations then have a second level of server, available to clients – these are the only ones that can talk tot he primary server. If you’re using NTP, then the Stratum number will help show this, as eveerly level down (away) from the atomic clock is a higher stratum:

  • Stratum 0: the atomic clock
  • Stratum 1: the server physically wired to the atomic clock
  • Stratum 2: downstream from the Stratum 1 servers
  • etc….

On my small Debian system, I have the SystemD NTP service (timesyncd) that is listening, and I can see from the comment timedatectl show-timesync the current status:

# timedatectl status
Local time: Mon 2026-07-13 22:51:00 AWST
Universal time: Mon 2026-07-13 14:51:00 UTC
RTC time: Mon 2026-07-13 14:51:00
Time zone: Australia/Perth (AWST, +0800)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
# timedatectl show-timesync
FallbackNTPServers=169.254.169.123 fd00:ec2::123
ServerName=fd00:ec2::123
ServerAddress=fd00:ec2::123
RootDistanceMaxUSec=5s
PollIntervalMinUSec=32s
PollIntervalMaxUSec=34min 8s
PollIntervalUSec=34min 8s
NTPMessage={ Leap=0, Version=4, Mode=4, Stratum=3, Precision=-18, RootDelay=198us, RootDispersion=335us, Reference=A9FEA97A, OriginateTimestamp=Mon 2026-07-13 22:39:05 AWST, ReceiveTimestamp=Mon 2026-07-13 22:39:05 AWST, TransmitTimestamp=Mon 2026-07-13 22:39:05 AWST, DestinationTimestamp=Mon 2026-07-13 22:39:05 AWST, Ignored=yes, PacketCount=1166, Jitter=489us }

Here my internal NTP daemon is a Stratum 3, which means there is a Stratum 2 and 1 above me. IN this case, you can also see the address being used: 196.254.169.123 – which is the AWS Time Sync Service, a scalable time source across the entire EC2 fleet. In pre-cloud days, I would have two or more NTP hosts exchanging NTP traffic direct to upstream peers, and then have hundreds of servers query those three.

I would also have monitoring on those to ensure that the the three had a reasonably consistent view of the current time, and were not drifting off into the past (or future).

And lastly, the NTP software would be some of the core packages to get routine package updates to address vulnerabilities and bugs over time. You would do these one at a time, to ensure the other NTP servers remained available, and the one being updated had time to reconnect, sync up, and start having (even network-internal) clients use it.

Not maintaining hardware (firmwares) is a clear piece of not taking the responsibility for basic operational defence of these systems. Once upon a time (30+ years ago) a long “system uptime” was an admired feat of endurance. For the last decade, I looking at the age of your software (and firmware) to determine the oldest pieces and prioritising everything having a low median age is a better measure. Something that has been unpatched, but still running, doesn’t mean it is secure and reliable. If it hasn’t been restarted in the last year, do you know if it can restart after a power outage? Does it have patches that are not yet available? We know that cryptographic support changes over time (see TLS 1.3), but so has basic networking addressing protocols (see the IPv4 and IPv6 changes).