Category: Rants

CloudPets security fail is not a Cloud failure

I spent several years at Amazon Web Services as the Solution Architect with a depth in Security in A/NZ. I created and presented the Security keynotes at the AWS Summits in Australia and New Zealand. I teach Advanced Security and Operations on AWS. I have run online share-trading systems for many of the banks in Australia. I help create the official Debian EC2 AMIs. I am the National Cloud Lead for AWS Partner Ajilon, and via Ajilon, I also secure the State Government Land Registry in Ec2 with Advara.

So I am reasonably familiar with configuring AWS resources to secure workloads.

Last week saw a poor security failure; the compromise of a company that makes Internet-connected plush toys for children that lets users record and playback audio via the toys: CloudPets. Coverage from Troy Hunt,  The Register, ArsTechnica.

As details emerged, a few things became obvious. But here are the highlights (low-lights, really) to me that apparently occurred:

  • A production database (MongoDB) was exposed directly to the Internet with no authentication required to query it
  • Audio files in S3 were publicly, anonymously retrievable. However, they were not listable directly (no worries, the object URLs were in that open Mondo database)
  • Non-production and production systems were co-tenanted

There’s a number of steps that should have been taken technically to secure this:

  1. Each device should have had a unique certificate or credential on each of them
  2. This certificate/credential should have been used to authenticate to an API Endpoint
  3. Each certificate/credential could then be uniquely invalidated if someone stole the the keys from it
  4. Each certificate/credential should only have been permitted access to fetch/retrieve its own recordings, not any recording from any customer
  5. The Endpoint that authenticates the certificate should have generated Presigned URLs for the referenced recordings. PreSigned URLs contain a timestamp set in the future, after which the Presigned URL is no longer valid. Each time the device (pet) would want a file, it could ask the Endpoint to generate the Presigned URL, and then fetch it from S3
  6. The Endpoint could rate limit the number of requests per certificate pre minute/hour/day. Eg, 60 per minute (for burst fetches), 200 per hour, 400 per day?

If the Endpoint for the API was an Ec2 instance (or better yet, an AutoScale Group of them), then it could itself be running in the context of an IAM Role, with permission to create these Presigned URLs. Similarly an API Gateway running a Lambda in a Role.

Indeed, that Endpoint would have been what would have used the MongoDB (privately), removing the publicly facing database.

I’ve often quoted Voltaire (or Uncle Ben from Spider Man, take your pick): “with great power comes great responsibility“. There’s no excuse from the series of failures that were conducted here; the team apparently didn’t understand security in their architecture.

Yet security is in all the publicly facing AWS customer documents (joint responsibility). It’s impossible to miss this. AWS even offers a free security fundamentals course, which I recommend as a precursor to my own teachings.

Worse is the response and lack of action from the company when they were alerted last year.

PII and PHI is stored in the cloud. Information that the economy, indeed modern civilisation depends upon. The techniques used to secure workloads are not overly costly, they mostly require knowledge and implementation.

You don’t need to be using Hardware Security Modules (HSMs) to have a good security architecture, but you do need current protocols, ciphers, authentication and authorisation. The protocols and ciphers will change over time, so IoT devices like this need to also update over time to support Protocols and Ciphers that may not exist today. It’s this constant stepping-stone approach, to continually be moving to the next implementation of transport and at-rest ciphers that is becoming a pattern.

Security architecture is not an after-thought that can be left on the shelf of unfulfilled requirements, but a core enabler of business models.

Arlec Wireless LED Sensor Kit review

I was wandering around my esteemed local hardware store (Bunnings) and obtained an Arlec Wireless LED Sensor Kit. I’d been looking for something to give “under cabinet” lighting, particularly at night in my bathroom. Not wanting to get mains power outed to near floor height in a wet room, this looked like a great solution.

The kit consists of three “bars” of LEDs. One of these bars is a control unit, and also has an IR sensor in it. The other two are slave units to the master containing just an array of LEDs.

For the most part, this does as you want, but with a bit of thinking the product could do so much more. So Arlec, here’s some product research that frankly, you could have done in 20 minutes of thinking about your product:

  1. The light stays on for one minute, and then goes off. Regardless if the IR has been triggered again, during the last 60 seconds, its on for 60 and then off. Followed by madly trying to re-trigger this in the darkness. Surely if the IR triggers again you should reset the timer.
  2. Who choose 60 seconds? This should be configurable by the user. Minimum 5 seconds, maximum an hour?
  3. When the lights come on, they come on at 100%. Making an ease-in, ease-out to bring them up to “full brightness” would be much nicer.
  4. Why have the LEDs always go to 100% brightness. Perhaps that should be configurable.
  5. The LEDs are quite a cold white color. For me, Warm white would have been nicer. Others may want specific colour.
  6. When triggering, the slave units take some time to come on, and they trigger on in a random order. I have mine all in a tight two, and I’d be happy to run a small 2 or 3 wire cable between them and have them trigger simultaneously. Furthermore, with only 3 channel available, I’m limited to deploying this in larger settings. If I wire slaves together, then they should ignore their wireless receivers.
  7. The random order triggering of the slave units should be configurable. I may have a set of 10 of them going up some stairs, and want to put a 50ms delay as the light appears going up the stairs. Coupled with ease-in and brightness control this could look quite good.
  8. The master unit has the IR and a bank of LEDs, but I may not want LEDs where my IR trigger is: separate the IR and control unit into its own module.
  9. Give me the option of having multiple IR sensors (perhaps either end of the array of lights) to trigger the LEDs.
  10. Sell additional slave units individual, and in 5 packs.
  11. 3 channels is not enough if I have multiple sets in close proximity, and subject to interference. So give me the option of disabling the wireless signalling completely.

A smaller form factor would also be neat – perhaps a hard-wired version that could sit flatter under surfaces and be less obtrusive. But that’s my first few things that I think a bit or R&D would uncover.

Woodlands Primary School Song

In the mid 80s, my Dad wrote a lyrics for a song for my primary school – Woodlands -  with one of our neighbours creating the music. It became the official school song for nearly 30 odd years, and was only recently supplanted. I was trying to remember the lyrics, and found only one document with it left on line, so I thought I ‘d paste it here to preserve it a little longer.

At the bottom of the hill
Nestling by the trees
Warmed by the sun
Cooled by the breeze
There’s a place for learning
There’s a place for fun
It’s the school at Woodlands
We welcome everyone
Banksia gum and wattle
They are just a few
Of the many trees around us
That make our little school
A good place to learn in
A good place for fun
It’s the school at Woodlands
We welcome everyone
The Banksia is our emblem
We wear it with pride
Endeavour is our motto
It means we always try
A good place to learn in
A good place for fun
The BEST school in W.A.
Woodlands number one.
– John A N Bromberger

SysOps Mantras

One saying I’ve used with my teams over the years has been:

Have Backups

Have Monitoring

Monitor your Backups

Backup your monitoring

Its usually lead to things being under control. That extends to things like RAID sets – its nice to have, but its useless unless you’re monitoring for failed disks, otherwise you only notice an issue when the last disk fails.

Hurricane Electric IPv6 tunnel MTU

I’ve been running an IPv6 tunnel for a long time, but occasionally I’ve been seeing traffic hang on it. It looks like it was the MTU, defaulting at 1500 bytes, causing issues when large amounts of data were being shuffled OUT from my Linux box, back to the ‘net’.

The fix is easy: /etc/network/interfaces should have an “up” line for the interface definition saying: up ip link set mtu 1280 dev henet, where henet is the name of your tunnel interface.

Easy enough to skip this line if your tunnel appears to be working OK, but interesting to track down.