Ubiquiti’s Unifi Site Magic

I’ve switched to using Unifi a few years back, and made the investment in a Unifi Dream Machine when my home Internet connection exceeded 25 MB/sec — basically when NBN Fibre to the Premises (FTTP) became available in my area, replacing the ADSL circuit previously used.

I’d also shifted ISPs, as I wanted one that gave me native IPv6 so that I can test customer deployments (and be ahead of the curve). I selected Aussie Broadband, and they have been very good.

Here’s what my home network topology looks like, according to Unifi:

But actually, this is what it really is like:

The difference is the top cluster, which has the two point-to-point devices, separated by around 105 meters (300ft), which the Unifi device does not see.

Meanwhile, my family had a separate property 300 kms away in the southwest of Western Australia, which until two months ago, had been a totally disconnected site: no Internet and no telephone. With aging family members, it was becoming more pressing to have a telephone service available at the property, and resolve the issue of using a mobile hot spot when on site.

At the same time, we had a desire to get some CCTV set up, and my Unifi Protect had been working particularly well for several years now at my location(s) – including over a 100m point-to-point WiFi link.

Once again, we selected Aussie Broadband as an ISP, but a slower Fibre to the Node (FTTN) was delivered, which we weren’t expecting. This required an additional VDSL bridge to convert from the analogue phone line, to ethernet presentation to the UDM SE gateway/router.

Here’s the topology of the new site:

Easy. The Unifi device has some great remove administration capabilities, which means ensuring everything is working is easy to do when 300 kms away.

Unifi Site Magic

And then this week, I see this:

So I wander over to unifi.ui.com, and try to link my two sites – one subnet at each of the two sites, and it starts to try to connect:

But after a while, I give up. It won’t connect.

I see it only supports IPv4 (at this time). Everything else looks fine…

It’s only then (after a post to the Ubiquiti forums) that I’m pointed at the face that both sites are on 100.xxx, which are reserved addresses for Carrier Grade NAT.

A quick look up on the Aussie Broadband site, and I see I can opt out of CGNAT, and today I made that call. Explaining the situation, I requested one site be moved out (I’m not greedy, and IPv4 space is scarce).

Am hour later, and I have a better outcome:

And now, from Perth, I can ping the VoIP phone on-site 300 kms away behind the router:

C:\Users\james>ping 192.168.100.122

Pinging 192.168.100.122 with 32 bytes of data:
Reply from 192.168.100.122: bytes=32 time=19ms TTL=62
Reply from 192.168.100.122: bytes=32 time=20ms TTL=62
Reply from 192.168.100.122: bytes=32 time=19ms TTL=62
Reply from 192.168.100.122: bytes=32 time=19ms TTL=62

Ping statistics for 192.168.100.122:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 19ms, Maximum = 20ms, Average = 19ms

What’s interesting is that only ONE of my sites had to be popped out from behind the ISP’s CGNAT, and the Site Magic worked.

Of course in future, having IPv6 should be sufficient without having to deal with CGNAT.

Software License Depreciation in a Cloud World

Much effort is spent on preserving and optimising software licenses when organisations shift their workloads to a cloud provider. It’s seen as a “sunk cost”, something that needs to be taken whole into the new world, without question.

However, some vendors don’t like their customers using certain cloud providers, and are making things progressively more difficult for those organisations that value (or are required) to keep their software stack well maintained.

Case in point, one software vendor who has their own cloud provider made significant changes to their licensing, removing rights progressively for customers to have the choice to run their acquired licences in a competitors cloud.

I say progressively, customers can continue to run (now) older versions of the software before that point in time the licensing was modified.

The Security Focus

Security in IT is a moving target. Three’s always better ways of doing something, and previous ways which, once were the best way, but are now deemed obsolete.

Let me give you a clear example: network encryption in flight. The dominant protocol used to negotiate this is called Transport Layer Security (TLS), and its something I’ve written about many times. There’s different versions (and if you dig back far enough, it even had a different name – SSL or Secure Sockets Layer).

Older TLS versions have been found to be weaker, and newer versions implemented.

But certain industry regulators have mandated only the latest versions be used.

Support for this TLS is embedded in both your computer operating system, and certain applications that you run. This permits the application to make outbound connections using TLS, as well as listen and receive connections protected with TLS.

Take a database server: its listening for connections. Unless you’ve been living under a rock, the standard approach these days is to insist on using encryption in flight in each segment of your application. Application servers may access your database, but only if the connection is encrypted – despite them sitting in the same data centre, possibly in the same rack or same physical host! It’s an added layer of security, and the optimisations done mean its rarely a significant overhead compared to the eavesdropping protection it grants you.

Your operating system from say 2019 or before may not support the latest TLS 1.3 – some vendors were pretty slow with implementing support for it, and only did so when you installed a new version of the entire operating system. And then some application providers didn’t integrate the increased capability (or a control to permit or limit the version of TLS) in their software in those older versions from 2019 or earlier.

But in newer versions they have fixed this.

Right now, most compliance programs require only TLS 1.2 or newer, but it is foreseeable that in future, organisations will be required to “raise the bar” (or drawbridge) to use only TLS 1.3 (or newer), at which time, all that older software becomes unusable.

Those licences become worthless.

Of course, the vendor would love you to take a new licence, but only if you don’t use other cloud providers.

Vendor Stickiness

At this time, you may be thinking that this is not a great customer relationship. You have an asset that, over time, will become useless, and you are being restricted from using your licence under newer terms.

The question then turns to “why do we use this vendor”. And often it is because of historical reasons. “We’ve always used XYZ database”, “we already have a site licence for their products, so we use it for everything”. Turns out, that’s a trap. Trying to smear cost savings by forcing technology decisions because of what you already have may preclude you from having flexibility in your favour.

For some in the industry, the short term goal is the only objective; they signa purchase order to reach an immediate objective, without taking the longer term view of where that is leading the organisation – even if that’s backing hem into a corner. They celebrate the short term win, get a few games of golf out of it, and then go hunting for their next role elsewhere, using the impressive short term saving as their report card.

A former colleague of mine once wrote that senior executive bonuses shouldn’t be paid out in the same calendar year, but delayed (perhaps 3 years) to ensure that the longer term success was the right outcome.

Those with more fortitude with change have, over the last decade, been embracing Open Source solutions for more of their software stack. The lack of licence restriction – and licence cost – makes it palatable.

The challenge is having the team who can not only implement potential software changes, but also support a new component in your technology stack. For incumbent operations and support teams, this can be an upskilling challenge; some wont want to learn something new, and will churn up large amounts of Fear, Uncertainty and Doubt (FUD). Ultimately, they argue it is better to just keep doing what we’ve always done, and pay the financial cost, instead of the effort to do something better.

Because better is change, and change is hard.

An Example

Several years ago, my colleagues helped rewrite a Java based application and change the database from Oracle, to PostgreSQL. It was a few months from start to finish, with significant testing. Both the Oracle and PostgreSQL were running happily on AWS Relational Database Service (RDS). The database was simple table storage, but the original application developers already had a site license for Oracle, and since that’s what they had, that’s what they’ll use.

At the end of the project, the cost savings were significant. The return on investment for the project services to implement the change was around 3 months, and now, years later, the client is so much better off financially. It changed the trajectory of the TCO spend.

The coming software apocalypse

So all these licences that are starting to hold back innovation are becoming progressively problematic. The time that security requirements tighten again, you’re going to hear a lot of very large, legacy software license agreements disintegrate.

Meanwhile, some clod providers can bundle the software licence into the hourly compute usage fee. If you use it, you pay for it; when you don’t use it, you don’t pay for it. if you want a newer version, then you have flexibility to do so. Or perhaps event to stop using it.

Time to minimise public IPv4 usage in the AWS Cloud

It was always going to happen. We’ve been watching the exhaustion of the 32 bit address space of IPv4 for more than 20 years, and we’ve had the solution available for even longer: IPv6.

I’ve written many times about IPv6 adoption and migration on this blog. I’ve spoken many times with colleagues about it. I’ve presented at AWS User Groups about using IPv6 in AWS. And when I worked at AWS 10 years ago, I championed that s a competitive advantage to IPv6 all the things where IPv4 was in use.

The adoption has been slow. Outside of the Cloud, ISP support has been mixed, depending if they have the engineering capability to uplift legacy networks, or not. Let’s be clear – those ISPs who removed their engineers, and minimise the innovation, are about to have a lot of work to do, or face tough conversations with customers.

For those that have already done the work, then this weeks AWS annoucement about the start of charging for public IPv4 address space from 2024 is a non-issue. For others, its going to start to mean some action.


Lets start with the basics; go have a read of the AWS Announcement: New – AWS Public IPv4 Address Charge + Public IP, posted 28 July 2023.

You’re back, ok, so at time of blogging, charges start in 2024. Currently, your first IPv4 assigned to an instance is not charged for, but soon it will be half a US cent per hour, or on a 744 hour month, US$3.72 a month. Not much, unless you have hundreds of them.

Selling an IPv4 netblock

In the last few years I helped a government agency “sell” an unused /16 IPv4 netblock for several million dollars. They had two of them, and had only ever used a few /24 ranges from their first block; the second block was not even announced anywhere. There was no sound plan for keeping them.

The market price to sell a large contiguous block of addresses keeps going up – 4 years ago it was around $22 per IPv4 address (and a /16 is 65,536 of them, so just over US$1.4M). Over time, large contiguous address blocks were becoming more valuable. Only one event would stop this from happening: when no one needed them any more. And that event was when the tipping point into the large spread (default) usage of IPv6, at which time, they drop towards worthless.

The tipping point just got closer.

Bringing it back to now

So with this announcement, what do we see. Well, this kind of sums it up:

Congratulations, your IPv6 migration plan just got a business case, AWS is now charging for v4 addresses. v6 is free, and the sky has finally fallen:

Nick Matthews @nickpowpow

There have been many IPv6 improvements over the years, but few deployments are ready to ditch IPv4 all together. Anything with an external deployment that only supports IPv4 is going to be a bit of a pain.

Luckily, AWS has made NAT64 and DNS64 available, which lets IPv6 only hosts contact IPv4 hosts.

The time has come to look at your business partners you work with – those you have API interfaces to, and have the IPv6 conversation. It’s going to be a journey, but at this stage, its one that some in the industry have been on since the last millennium (I used to use Hurricane Electric’s TunnelBroker IPv6 tunnelling service in the late 1990s from UWA for IPv6).

Looking at your personal ISP and Mobile/Cell provider

It’s also time to start to reconsider your home ISP and cell phone provider if they aren’t already providing you with real IPv6 addresses. I personally swapped home Internet provider in Australia several years ago, tired of the hollow promises of native IPv6 implementation from one of Australia’s largest and oldest ISPs, started by an industry friend of mine in Perth many years ago (who has not been associated with it for several years). When the ISP was bought out, many of the talented engineers left (one way or another), and it was clear they weren’t going to implement new and modern transport protocols any time soon.

Looking at your corporate IT Dept

Your office network is going to need to step up, eventually. This is likely to be difficult, as often corporate IT departments are understaffed when it comes to these kinds of changes. They often outsource to managed service providers, many of whom don’t look to the future to see what they need to anticipate for their customers, but minimise the current present cost to “keep the lights on”. This is because customers often buy on cost, not on quality or value, in which case, the smart engineers are elsewhere.

Your best hope is to find the few technically minded people in your organisation who have already done this, or are talking about this, and getting them involved.

Looking at your internet-facing services

There’s only one thing to do, ASAP: dual-stack everything that is [public] Internet facing. Monitor your integration partners for traffic that uses IPv4, and talk to them about your IPv6 migration plans.

Its worth watching for when organisations make this switch. There are many ways to do this.

For web sites and HTTP/HTTPS APIs, consider using a CDN that can sit in front of your origin server, and as the front-door to your service, can be dual stack for you. Amazon CloudFront has been a very flexible way to do this for years, but you must remember both steps in doing this:

  1. Tick the Enable IPv6 on the CloudFront distribution
  2. Add a record to your DNS for the desired hostname as an AAAA record, alongside the existing A record.

The Long Term Future

IPv4 will go away, one day.

It may be another 20 years, or it may now be sooner given economic pressures starting to appear. Eventually the world will move on past Vint Cerf’s experimental range that, from the 1970s, has outlasted all expectations. IPv4 was never supposed to scale to all of humanity. But its replacement, IPv6, is likely to outlast all of us alive today.


EDIT: Cross link to Greg Cockburn’s recent AWS IPv6 post, and Corey Quinn’s post on the topic.

More TLS 1.3 on AWS

Earlier this week, AWS posted about their expanded support for TLS 1.3, clearly jumping on the reduced handshake as a speed improvement in their blog post entitled: Faster AWS cloud connections with TLS 1.3.

Back in 2017, (yes, 6 years ago) we started raising Product Feature Requests for AWS products to enable this support, and at the same time, customer control to be able to limit the acceptable TLS versions. This makes perfect sense in customer applications (the data plane). Not only do we not want our applications supporting every possible historic version of cryptography, various compliance programs require us to disable them.

Most notable in this was PCI DSS 3.1, the Payment Card (credit card) Industry Association’s Data Security Standard, which drove the nail in to the coffin of TLS 1.1 and everything before it.

Over time, TLS versions (and SSL before it) have fallen from grace. Indeed, SSL 1.0 was so bad it never saw the light of day outside of Netscape.

And it stands to reason that, in future, newer versions of TLS will come to life, and older versions will, eventually, have to be retired; and between those two, is another transition. However, this transition requires deep upgrades from cryptography libraries, and sometimes to client code to support the lower level library’s new capability..

On the server side, we often see a more proactive implementation of what currently supported TLS versions are permitted. Great services like SSLLabs.com, Hardenize.com, and testssl.sh have guided many people to what today’s current state of “acceptable” and “good” would generally look like. And the key item of those services, is their continual uplift as the state of “acceptable” and “good” changes over time.

On the client side, its not always been as useful. I may have a process that establishes outbound connections to a server, but as a client, I amy wan tto specify some minimum version for my compliance, and not just rely upon the remote party to do this for me. Not many software packages do this – the closest control you get is an integration possibly using HTTPS (or TLS), and not the next level down of “yeah, so which versions are OK to use when I connect outbound”. Of course, having specified HTTPS (or TLS) and doing server certificate validation against our local trust store, we then have a degree of confidence hat its probably the right provider, given that one of my 500 trusted CAs signed that certificate. we got given back during the handshake

This sunrise/sunset is even more important to understand in the case of managed services from hyperscaler cloud providers. AWS speaks of the deprecation of TLS 1.1 and prior in this article (June 2022).

If you have solutions that use AWS APIs, such as applications talking to DynamoDB, then this is part of your technical debt you should be actively, regularly addressing. If you haven’t been including updated AWS SDKs in your application, and updating your installed SSL libraries, updating your OS, then you may not be prepared for this. Sure, it may be “working” fine right now.

One option you have is to look at your application connection logs, and see if the TLS version for connections is being logged. If not, you probably want to get that level of visibility. Sure, you could Wireshark (packet dump) a few sample connections, but it would probably be better not to have to resort to that. Having the right data logged is all part of Observability.

June 28 is the (current) deadline for AWS to raise the minimum supported TLS version. That’s a month away from today. Let’s see who hasn’t been listening…

IoT Trackers and the AWS Cloud

I continued my IoT project over the recent end-of-year/Christmas break period, picking up from where I was 6 months ago.

Since then, a new firmware version had become available for the RAK Wireless RAK10700 GNSS (Global Navigation Satellite System) solar powered device. These devices shipped without battery (due to postal limitations), and came with firmware 1.0.4.

I failed completely last time to get these to associate with my local IoT gateway (rak7246, basically a raspberry PI in a box that bridges LoRaWAN and WiFi).

Since then, a new firmware 1.0.6 has been released.

Documentation for the RAK10700 was OK, until you get to the page that says the latest firmware is version 1.0.1; given this device already shipped with 1.0.4, I dived in deeper; the link to the firmware is https://downloads.rakwireless.com/LoRa/WisBlock/Solutions/LPWAN-Tracker-Latest.zip, and the contents of this zip file, at the time of writing, are three files:

  • Manifest
  • WisBlock_SENS_V1.0.6_2022.04.07.13.36.27.bin
  • WisBlock_SENS_V1.0.6_2022.04.07.13.36.27.dat

Caveat Elit (developer beware): this appears to be firmware version 1.0.6.

Flashing this was interesting: the device, connected via its USB cable to my laptop, had to be reset into DFU mode, which required double-tapping the reset pin in quick succession (its located next to the USB port). Once done, the device presented as USB storage, and the adafruit-nrfutil tool could update it (check the COM port from Device Manager).

adafruit-nrfutil dfu serial --package LPWAN-Tracker-Latest.zip -p COM19 -b 115200

When in DFU mode, the device turned up as a different COM port compared to when it was in its normal mode. It took me two attempts for this to be successful, and then pressing the rest button to have the device return to normal mode.

Next came the interface to AWS IoT Core for LoRaWAN. I’d previously been using the LGT-92 (now not available), but had to abandon these as no amount of protection in waterproof bags had made them durable enough to last the distance of my use case; tracking a small sail boat.

The configuration that eventually worked for me was to define a profile with MAC version 1.0.3, Regional Parameters RP002-1.01, Max EIRP 15, for AU915 frequencies (I am in Australia):

AWS IoT Core for LPWAN: Profile Configuration for RAK10700

Now with the profile defined, I can add the two Devices in, using the AppKey, DevKey, etc:

With data coming through it was now time to decide the Payload. These devices use a format called CayenneLPP to stuff as much data into as small a payload as possible. One of the first things you’ll want to do is decide the data to check it looks legitimate. Using a small Python script, I can unpack it – after doing a pip install cayennelpp:

#!/usr/bin/python3
import base64
import sys
from cayennelpp import LppFrame
d=base64.standard_b64decode(sys.argv[1])
f=LppFrame().from_bytes(d)

for i in f.data:
  if len(i.value) == 1:
    print("Ch {}, {}: {}".format(i.channel, i.type, i.value[0]))
  else:
    print("Ch {}, {}: {}".format(i.channel, i.type, i.value))

By routing the incoming IoT messages to a Lambda function, I can now pick out the PayloadData from the event and see the string being send. Here’s what I am seeing in CloudWatch logs when I just print(event):

{'WirelessDeviceId': 'b15xxxx-xxxx-47df-8a5d-f57800c170b5', 'PayloadData': ' AXQBqwZoRwdnARQIcydWCQIG6Q==', 'WirelessMetadata': {'LoRaWAN': {'ADR': False, 'Bandwidth': 125, 'ClassB': False, 'CodeRate': '4/5', 'DataRate': '3', 'DevAddr': 'xxxxx', 'DevEui': 'xxxxx', 'FCnt': 73, 'FOptLen': 0, 'FPort': 2, 'Frequency': '917200000', 'Gateways': [{'GatewayEui': 'xxxxx930e93', 'Rssi': -31, 'Snr': 9.5}], 'MIC': 'xxxxx95b', 'MType': 'UnconfirmedDataUp', 'Major': 'LoRaWANR1', 'Modulation': 'LORA', 'PolarizationInversion': False, 'SpreadingFactor': 9, 'Timestamp': '2023-01-04T13:46:42Z'}}}

While inside, with no satellite lock, that PayloadData translates out to:

Ch 1, Voltage: 4.27
Ch 6, Humidity: 35.5
Ch 7, Temperature: 27.6
Ch 8, Barometer: 1007.0
Ch 9, Analog Input: 17.69

Now I have the two sensors, its time to get them outside, with a bit of soldering of the LiPo battery on to the right connector…