Staring at tomorrow: Technology Impacts on society

This started as musings on where broadband connections enter buildings. It went further.


There’s a few technical specifications that we’ve become very used to in our homes and offices for the last few decades, but they’ve not always been there. Their introduction to our lives has spurred changes in the construction of our dwellings, and our expectations of what we do in our lives. And another one is unfolding now.

Plumbing (Scheme Water, Sewerage), Electricity, and Broadcast TV have all had their impacts. Water in & out has remained largely unchanged in the last 50 years, having moved from outhouses to servicing dedicated ‘wet’ rooms in our buildings (kitchens, bathrooms, WCs). Electricity has crept from being provided for lighting purposes only, to additionally providing a single power socket per home, to multiple sockets per room at convenient places for us to leave devices semi-permanent plugged in. We’ve become accustomed to the 110v/60Hz and 230v/50Hz, and the sockets and plugs are generally down to several well known arrangements of pins and locking mechanisms.

With the dawn of broadcast TV (UK: 1936, US: 1948, Australia: 1965), and excluding set-top “rabbit ears”, we’ve had antennas on roofs with coaxial cables leading to specific rooms: family rooms, bedrooms, etc. Using the superior reception of the roof-top antenna has meant that the placement of TVs was pretty much determined by the location of Co-Axial sockets in walls. Indeed, not just which rooms, but which corners and walls of specific rooms, close to these antenna sockets. While we’ve happily added multiple power sockets to rooms for future proofing our desire to rearrange our lives, co-axial cables suffered more loss with more joins to other sockets, so it was generally avoided.

Thus our layout of our homes has been throttled by the tether to the roof top antenna.

We are well into the next revolution delivery of our video feeds over a new carrier: IP and the Internet. The age of radio frequency broadcast is declining, and with it some of our cultural norms.

One impact is building design: a time will come when we’ll no longer request TV broadcast antennas on buildings, and coaxial sockets on walls. This is replaced with either short range high speed WiFi such as 802.11ac, or wired Ethernet ports at gigabit speed (no less). Wireless has continued to increase in speed, but as with all broadcast spectrum, it is subject to interference from other wireless signals, which can impact throughput. A continual battle between ever faster wireless speeds, improved signal processing and interference handling, and cost considerations will play out to determine if our TVs will have a wired or wireless IP connection within the home.

Wireless has been through several generations, some of which are now well and truly dead. WEP encryption, once the critical protection for wireless networks, is known to be compromisable, and thus is abandoned today. Hardware devices that only implemented WEP are effectively junk. Devices with physical Ethernet socket have not suffered so – the trusty RJ45 100 base TX CSMACD works as well today as it did 20 years ago.

We’ll see people opt to use wireless for video delivery within homes for short term savings, the aggressive rate of change on encryption and signalling standards for wireless networks will find many wireless-only devices become redundant before their time. We expect to get 10 – 20 years from a TV (its often an expensive item), but should WPA2 get compromised, then those hosts are vulnerable. We’ve already seen this with the KRACK vulnerability, and those vendors that did not patch their firmwares are probably vulnerable (both the Access Point and clients need patches to address KRACK for WPA2).

Of course, the interactions that your TV makes over whatever connection should indeed be over an encrypted transport, but even the encryption that uses – the key lengths, signature algorithms, all will be refreshed and strengthened over time.  In the last 5 years we’ve seen web site certificates change in key length (1024 -> 2048 bit), signatures for asserting a chain-of-trust from a certificate authority move from what was MD5 to SHA1 to SHA256, the bulk encryption algorithms move from RC4 to AES128 to AES256 to Elliptical Curve algorithms, and message digest checksums from MD5 to SHA1, to SHA256, and not SHA384. Each of these changes needs to be applied to the software in your appliances for them to maintain the security you expect to be there.

Many manufacturers won’t bother updating this on their already-shipped units; the disposable consumer sales cycle continues unabated and effectively is helped by these changes, and customers barely make any mention of the trouble of having to replace these items.

But as we move to IP delivery of video content, we also move to more of a video-on-demand world, and away from a time-of-broadcast (playout) world where all consumers – the audience in a broadcast area – would all get the same content at the same time.

30 years ago, when I was at school, the talk in the playground was of the latest episode of some show: MacGuyver, Airwolf, The A-Team, comedies like ‘Allo ‘Allo, The D-Generation, The Comedy Company, etc. With only a few select channels to choose from (3 where I was), your peers likely saw the same content at the same time, making for a shared experience that you wove into the daily fabric of society.

Shows with catch phrases became common place, and everyone knew them. Mark Mitchell gave Australia the “Couple o’ days, Beautiful”, and everyone knew the setting and connotation.

As we move to a VOD, subscription based scenarios, this disappears. Shows on one subscription service are not on others, or appear at different times. Subscription services cross local boundaries in a way that broadcast spectrum could never do with the limitation of transmitter power. People watch content at different times, in different months, from different countries (despite the continued attempt for geo-blocking). The social background noise of modern society is changed, disjointed.

I suspect that in the next decade, broadcast TV will disintegrate. Advertising spend will continue to shift to product placement in original content, and customised pre- and mid-roll ad insertion, customised to the view, tracked and targeted.

As broadcast TV declines, it heavily modifies the stalwart of most TV stations: TV news. While news has been available form multiple media sources, local broadcast TV news was always curated to somewhat balance local, national and global current affairs. If all journalism production costs the same, then stories filed that can be replayed in multiple territories have more value per second, and the decline of local stories.

I’ve watched local news for decades, paying attention to their mix of slow news days stories (car through a fence, cat up a tree) and more interesting ones (state general or by election results, perhaps investigative journalism – but that’s on the decline). Its even interesting to sometimes see the format of content being padded out with in-content extended adverts for news clips that are scheduled later, the tweaks to the lower thirds straps, even the background animations that engage the reader in subtle ways. These things often annoy me at how much they con the audience’s emotional engagement with a story. One local broadcast TV channel has a habit of applying a cinematic dust/sparkle video filter to most human-interest stories (cat up tree). They’ve played games with putting weather forecasts showing MIN (overnight) and MAX (daytime) temperatures on top of each other, and then sometimes MAX (daytime) on top of MIN.

(While I’m ranting, the “cross to the local hospital to a reporting standing out front” seems a waste of time, as is a “cross to the next room” scenario which an anchor could have just read out and moved on from.)

One media that has withstood much change until now is broadcast radio. I suspect that because radio has had one place where its continued its existence without much challenge: vehicles. Vehicle manufacturers have been pretty slow at putting DAB radios into vehicles by default. AM and FM radios are still universal. I wrote in 1995 in the UWA student magazine Pelican about the dawn of MP3 as an audio format, and the start of Digital Audio Broadcasting (DAB). The MP3 player was born – Apple seized on it with the iPod, and the Sony Walkman receded to the annals of history. And while DAB has been around for some time, the licensing and hardware has been at an expense that did not generally warrant the improvement in delivery, and clearly not supported by vehicle manufacturers.

But while broadcast radio has been compatible with what the listener is doing in vehicles: a threat is coming to the radio’s last bastion. It’s the same threat that is coming to organisations that live off driving license fees: self-driving cars. Drivers can then do other tasks: they can then look at screens. The radio will stop becoming an in-vehicle companion to millions of single-occupant cars, as those occupants will start viewing content instead of just listening to it. Eyes will come off the road.

(And if eyes are off the road, do we have the demise of billboards and placards as an advertising medium – as no one is looking out the window any more?)

By extension to this, the music industry relies on radio for socialising its content, encouraging people to either purchase content, or become customers to performers when they tour. With broadcast music no longer curated to seed the introduction of new content over time, artists will find it difficult to get established.

Coupled with the self-driving car is the move to electric vehicles, and the eventual drop in the petroleum fuel use, and the tax (excise) that is collected from this. Governments often use this as a source of funding road building. This model will have to change, probably to a tax-per-kilometer travelled on the owner of these vehicles.

And with self-driving cars, we can finally have some backward parts of the world switch to the metric system for units of distance and speed, without the risk of the human population getting it wrong and going too fast/slow/far.

Eventually, as my friend Paul Fenwick (PJF) has spoken of, the population will move to not owning vehicles, but calling them completely on demand – a la Uber/taxi, but without the human driver. These vehicles will get corporate ownership, and will all have live mobile broadband data links to each vehicle. They’ll all have built-in dash cams, and logging of all activities in and out of the vehicles at all times. New advertising opportunities will rise up – screens in these vehicles will know the route you’re about to take to advertise products and services on or near your route, so you can chose to stop off. They’ll know your preferences, the environmental factors (warm, cold, sunny, rainy) and advertisers will bid to produce targeted advertising at you while you travel.

Next knock on effect from this is the car insurance industry. If the self-driving vehicle has less accidents, and the risk of death or major damage is less, then disruption will arrive here too. With only a few major self-driving-taxi companies that require insurance, the number of insurance companies will consolidate.

The self driving vehicle probably needs less street signs. It may require less street lighting. It may require less lane markings, cats eyes, number of lanes.

This probably sounds like a diatribe version of a Swardley map (hat tip to Simon). All these things are connected, generally by revenue streams or shared interests, and always by data and technology. As always, its all change, and resistance is futile.

Google Pixel & Pixel XL: impressions

It’s a little late in the release cycle, with the Pixel 2 and Pixel XL 2 having been released, but there’s a number of points I’ve been contemplating on this premium-priced phone for some time that I’ve wanted to Blog about.  Here goes…

Phone Retailer: Avoiding the Bloat-ware

I purchased my Pixel (in Australia) direct from Google about 12 months ago (as at December 2017). One of my primary reasons for purchasing a phone direct from the vendor is to explicitly avoid 3rd party (Tel co) pre-installed, forced additional ‘value’ software.

Telephone companies (collectively, Wireless Providers, Tel Co, Phone company, Mobile Company, Cell Provider or whatever your term is) seem to take vanilla smart phone firmware, and force-install their own additional software that they see as adding essential functionality. They also mark such software on Android as being not uninstallable, leaving the consumer with space consumed on their device for software they potentially don’t want, or may want to free up later.

Telcos have a history of producing some fairly horrible 3rd Party software. Somehow they get the combination of inefficient software that drains battery life, causes system reboots, consumes inordinate amount of phone storage capacity for no obvious reason, and often has horrible security throughout, none of which is in the consumer’s interest.

Given this software is not uninstallable, the consumer has two options over the life of the phone: either put up with the issues, or apply security updates for this bloat ware — if they are made available — which inevitable consumes more device storage space (apparently never less), and spin the wheel on changes around battery life, stability and security.

You’ll note I say ‘consumer‘ in the above, because if the Telcos treated the people paying them as customers, then perhaps they’d pay a bit more attention to customer experience and customer satisfaction, rather than forcing their own poorly implemented branded bloat ware on these devices. Even a boot logo — I’d rather have the default boot logo rather than have to fetch the animated loop for a Tel co to be displayed to me when I turn my phone on.

I had this with the original Google Nexus phone perhaps a decade ago, but phones I have used since then have suffered this bloat infestation. My wife has a Samsung Galaxy S4, with a combination of additional Optus and Samsung software crowding out the fixed-storage-space in the S4.

The Pixel

While Nexus returned with the 5P, it was the time that the Pixel launched that my S3 was on its last legs; and with an option of going direct to Google, I ordered one; a reasonably easy ordering process, good tracking for delivery.

The install looked great: just hook up a USB cable to the older Android phone, and everything should transfer — except it didn’t work at all. The S3 (from Optus) was capped at Android 4.4, the Pixel shipped with Android 7, an the delta was too long a divide for a promised smooth transfer.

Oh well, looks likes this may be useful in future for easing the upgrade/transition/replacement path I thought.

Pixel Sound Issues

Then the speaker started to play up.

Over the course of three months, the sound quality from the speakers (ie, when playing music, YouTube content, and phone Speakerphone mode) degraded (and eventually stopped, later). When the phone ‘rang’, it would be highly distorted audio. Speakerphone was not possible – you couldn’t make out the words the caller was saying.

This is when I first contacted Google support.

Google Support

Conveniently, Google support was contacted through a menu on the phone; either text chat, or a ring-back system that must have registered me into a queue, and called me when an agent was available. Neat.

After performing a few checks (ie, volume turned up), I was asked to firmware reset the phone. With MFA enabled on my phone for a few accounts (>30), I didn’t want to loose those seeds; but would like to transfer them to a new handset, especially if the promised transfer experience was going to avoid me having to recover MFA set up. After explaining this, the call ended, but no replacement was forthcoming.

Fast forward until October, and the audio on speaker phone was completely dead, and I’d even tried the new Oreo release and any other software solution. So now another call with Google support; this time they confirmed on the phone they would send a replacement.

Replacement procedure

What they didn’t say was that they would send a time sensitive email to my Gmail address (not my primary address) that Gmail would automatically filter into a folder called “updates” (ie, not my default view of my Gmail inbox) that required me to a lick a link to order a replacement model within 5 days.

So a week after the call, wondering where the process was up to, I discovered email (having not been told to click a link in an email); but the link had expired, so another call to get a fresh time based link generated.

Confusingly, while I was trying to replace a Pixel, support sent me a link that would only order a Pixel XL. I wasn’t looking to change form factor (the Pixel fits nicely in my pocket). Another call – to sort this out, turned up that there were no Pixel replacements for RMA, and I would have to move to a larger handset.

The RMA procedure also required me to order a new one, a daunting process of having a UAD$1400 hold on my credit card, especially late in the pay cycle when there wasn’t $1400 clear on my card to hold. A few days later, another support call, a fresh link to click and start the “order” (RMA) again.

Transferring Pixels

Finally, it arrived. I connected the magic USB cable to initiate the transfer… hoping to keep copy media on the device, and the precious MFA seeds.

But it failed to start. Pixel 8.1 → Pixel XL 8.0 wouldn’t connect over the USB cable, but after trying various options, and proceeding to join a common WiFi network, it did promise to copy over WiFi.

Sadly, account logins only for Gmail. No media, no seeds. Not even the set of applications installed on the old phone.

So the promise of a seamless upgrade over a back-to-back connection between handsets seems unfulfilled.

Symantec Touchdown

For my various work email addresses, I purchased and have been using Symantec Touchdown for about 6 years. Its a reasonable exchange client, and consisted of the Touchdown application, and the Touchdown License application (ie, two installations).

Now as stars align, Symatec have End-of-Lifed Touchdown. They did this by pulling the license installation from the Play store. So I am transferring my applications, and can no longer install the license I purchased from Symantec.

Pixel & Pixel XL USB-C PD (Power Delivery) charging

One of the nice points about the Pixel was that it charges quick., using a new USB connector. This rectangular connector is effectively symmetrical; it can be plugged in either way, and starts a very rapid charging process (like a percent per 30 seconds or so).

However, it appears to wear loose pretty quickly. Even on the Pixel XL (now two months old) the USB-C PD connector actually needs to be held in place to acquire a rapid charge. Numerous times I have connected it, seen the rapid charge begin, but returned to find that it had dislodged and not been charging at all!

So now I have to regularly check on a charging phone to ensure I don’t need to grab-and-go and find its flat.

Pixel & Pixel XL Performance

So some positives: the snapdragon processor seems pretty speedy; applications respond well.

The Chrome browser is regularly updated, and Security updates come through each month without delay (didn’t get that with a Telco branded firmware).

The camera takes nice photos and videos, including some reasonable slow motion (either 120 or 240 fps) and nice panorama and photo-sphere pics (stitched on camera). The integration of photos.google.com into the phone to backup (and offload) media from the device works well.

The placement of the fingerprint sensor works well on the rear; with the same hand I am holing the phone I can unlock it. And unlike FaceID, it doesn’t stink: I can register multiple fingers (ie, one from left hand, and one from right – H/A for my hands).

Wish List

Google:

  1. make transferring phones also install the applications from the older phone into the new, and set them up with the same settings
  2. transfer media from old to new over the back-to-back USB-C link
  3. improve the support experience for RMA; perhaps extend the link validity a little longer (2 weeks?), tell customers to look for the email that customers have to order the device
  4. have the USB-C click and lock into place, or something else to help it not spring back and loose connection

Symantec:

  1. can I get my licence key or a refund?

Tel cos in general:

  1. stop forcing your software onto customers phones; make your ‘essential’ services available as web apps without requiring client side bloat, make them uninstallable, and ensure that Androind updates flow to customers as soon as possible (have you pushed WPA Krack updates yet?).

Web Security 2017

Stronger encryption requirements for PCI compliance is having a good effect on purging the scourge of the web: legacy browsers, and as they disappear comes even more capability client side for security.

I started web development around late 1994. Some of my earliest paid web work is still online (dated June 1995). Clearly, that was a simpler time for content! I went on to be ‘Webmaster’ (yes, for those joining us in the last decade, that was a job title once) for UWA, and then for Hartley Poynton/JDV.com at time when security became important as commerce boomed online.

At the dawn of the web era, the consideration of backwards compatibility with older web clients (browsers) was deemed to be important; content had to degrade nicely, even without any CSS being applied. As the years stretched out, the legacy became longer and longer. Until now.

In mid-2018, the Payment Card Industry (PCI) Data Security Standard (DSS) 3.2 comes into effect, requiring card holder environments to use (at minimum) TLS 1.2 for the encrypted transfer of data. Of course, that’s also the maximum version typically available today (TLS 1.3 is in draft 21 at this point in time of writing). This effort by the PCI is forcing people to adopt new browsers that can do the TLS 1.2 protocol (and the encryption ciphers that permits), typically by running modern/recent Chrome, Firefox, Safari or Edge browsers. And for the majority of people, Chrome is their choice, and the majority of those are all auto-updating on every release.

Many are pushing to be compliant with the 2018 PCI DSS 3.2 as early as possible; your logging of negotiated protocols and ciphers will show if your client base is ready as well. I’ve already worked with one government agency to demonstrate they were ready, and have already helped disable TLS 1.0 and 1.1 on their public facing web sites (and previously SSL v3). We’ve removed RC4 ciphers, 3DES ciphers, and enabled ephemeral key ciphers to provide forward secrecy.

Web developers (writing Javascript and using various frameworks) can rejoice — the age of having to support legacy MS IE 6/7/8/9/10 is pretty much over. None of those browsers support TLS 1.2 out of the box (IE 10 can turn this on, but for some reason, it is off by default). This makes Javascript code smaller as it doesn’t have to have conditional code to work with the quirks of those older clients.

But as we find ourselves with modern clients, we can now ask those clients to be complicit in our attempts to secure the content we serve. They understand modern security constructs such as Content Security Policies and other HTTP security-related headers.

There’s two tools I am currently using to help in this battle to improve web security. One is SSLLabs.com, the work of Ivan Ristić (and now owned/sponsored by Qualys). This tool gives a good view of the encryption in flight (protocols, ciphers), chain of trust (certificate), and a new addition of checking DNS records for CAA records (which I and others piled on a feature request for AWS Route53 to support). The second tool is Scott Helm’s SecurityHeaders.io, which looks at the HTTP headers that web content uses to ask browsers to enforce security on the client side.

There’s a really important reason why these tools are good; they are maintained. As new recommendations on ciphers, protocols, signature algorithms or other actions become recommended, they’re updated on these tools. And these tools are produced by very small, but agile teams — like one person teams, without the bureaucracy (and lag) associated with large enterprise tools. But these shouldn’t be used blindly. These services make suggestions, and you should research them yourselves. For some, not all the recommendations may meet your personal risk profile. Personally, I’m uncomfortable with Public-Key-Pins, so that can wait for a while — indeed, Chrome has now signalled they will drop this.

So while PCI is hitting merchants with their DSS-compliance stick (and making it plainly obvious what they have to do), we’re getting a side-effect of having a concrete reason for drawing a line under where our backward compatibility must stretch back to, and the ability to have the web client assist in ensure security of content.

Inspecting the AWS RDS CA Certificates

Trying to fetch all the RDS CA certificates as a bundle, and inspect them:

#!/usr/bin/python3
# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4
import urllib.request
import re
from OpenSSL import crypto
from datetime import datetime


def get_certs():
    url = ("https://s3.amazonaws.com/rds-downloads/"
           "rds-combined-ca-bundle.pem")
    with urllib.request.urlopen(url=url) as f:
        pem_certs = []
        current_cert = ''
        for line in f.read().decode('utf-8').splitlines():
            current_cert = current_cert + line + "\n"
            if re.match("^-----END CERTIFICATE-----", line):
                pem_certs.append(current_cert)
                current_cert = ""
        return pem_certs


def validate_certs(certs):
    ca = None
    for cert_pem in certs:
        cert = crypto.load_certificate(crypto.FILETYPE_PEM, cert_pem)
        if cert.get_issuer().CN == cert.get_subject().CN:
            ca = cert
    for cert_pem in certs:
        cert = crypto.load_certificate(crypto.FILETYPE_PEM, cert_pem)
        start_time = datetime.strptime(
            cert.get_notBefore().decode('utf-8')[0:14], "%Y%m%d%H%M%S")
        end_time = datetime.strptime(
            cert.get_notAfter().decode('utf-8')[0:14], "%Y%m%d%H%M%S")
        print("%s: %s (#%s) exp %s" %
              (cert.get_issuer().CN, cert.get_subject().CN,
               cert.get_serial_number(), end_time))
        if end_time < datetime.now():
            print("EXPIRED: %s on %s" % (cert.get_subject().CN,
                                         cert.get_notAfter()))
        if start_time > datetime.now():
            print("NOT YET ACTIVE: %s on %s" % (cert.get_subject().CN,
                                                cert.get_notBefore()))
    return

pem_certs = get_certs()
validate_certs(pem_certs)

Output

Today this gives me::

Amazon RDS Root CA: Amazon RDS Root CA (#66) exp 2020-03-05 09:11:31
Amazon RDS Root CA: Amazon RDS ap-northeast-1 CA (#68) exp 2020-03-05 22:03:06
Amazon RDS Root CA: Amazon RDS ap-southeast-1 CA (#69) exp 2020-03-05 22:03:19
Amazon RDS Root CA: Amazon RDS ap-southeast-2 CA (#70) exp 2020-03-05 22:03:24
Amazon RDS Root CA: Amazon RDS eu-central-1 CA (#71) exp 2020-03-05 22:03:31
Amazon RDS Root CA: Amazon RDS eu-west-1 CA (#72) exp 2020-03-05 22:03:35
Amazon RDS Root CA: Amazon RDS sa-east-1 CA (#73) exp 2020-03-05 22:03:40
Amazon RDS Root CA: Amazon RDS us-east-1 CA (#67) exp 2020-03-05 21:54:04
Amazon RDS Root CA: Amazon RDS us-west-1 CA (#74) exp 2020-03-05 22:03:45
Amazon RDS Root CA: Amazon RDS us-west-2 CA (#75) exp 2020-03-05 22:03:50
Amazon RDS Root CA: Amazon RDS ap-northeast-2 CA (#76) exp 2020-03-05 00:05:46
Amazon RDS Root CA: Amazon RDS ap-south-1 CA (#77) exp 2020-03-05 21:29:22
Amazon RDS Root CA: Amazon RDS us-east-2 CA (#78) exp 2020-03-05 19:58:45
Amazon RDS Root CA: Amazon RDS ca-central-1 CA (#79) exp 2020-03-05 00:10:11
Amazon RDS Root CA: Amazon RDS eu-west-2 CA (#80) exp 2020-03-05 17:44:42

S3 MFA Delete

The Simple Storage Service (S3, or S3) has made long term durable storage simple for the masses. The democratisation of object storage with well documented, stable APIs has been incorporated into many products. The API is part of the product.

But despite the word Simple, there are more and more advanced features: storage tiers, security policies, life-cycle policies, logging, versioning, requestor-pays, and more recently, Inventory generation and more.

S3 features prominently in long-term retention of important data due to its high durability. But today I’m diving into the another benefit: MFA Delete.

Simple CRUD

Create, Read, Update, Delete: the basics of a REST interface for sending and manipulated a data store. In AWS, IAM policy (or Bucket Policy) can permit or limit the actions that a user can perform.  If you delete an Object, then it’s gone. If you overwrite an object (using the same Prefix or name), then the original is lost, as you would expect.

We can limit a calls to s3:DeleteObject, either with a explicit DENY, or carefully only permitting the fine grained controls we intend (s3:PutObject, s3:GetObject) for the role., groups or users we confer privileges to. However, we still run the change of an unintended overwrite.

Furthermore, there may be privileged users or roles that have elevated access, so while your general work-flow is protected by policy from accidental deletion, you’re not protected from accidents from other source (eg, humans with admin privs).

S3 Versioning

To help with this, S3 Versioning permits you to retain multiple revisions of the same object. When listing the bucket naturally, you see the current revision in the list. But a few API calls and you can drill into the previous revisions of the same object, helping you recover from object overwrites.

When a file is deleted from a Versioned S3 Bucket, its really just updated with a new version as a designated Delete Marker. This Marker prevents the object being included in a natural bucket listing. Without further action, the previous versions are still present, and you’re still paying for their storage.

Lifecycle Policies

I always recommend agreeing a life cycle retention policy for S3 buckets – possibly by agreed prefix – upon creation of the Bucket. It makes the creator of the data set really consider how permanent their data must be.

Lifecycle policies can change data storage tiers, but my favourite is the expiry of “previous revisions” after a customer-defined number of days. This gives me a kind of “S3 Undelete” window, and its saved my bacon on several occasions; the accidental Admin delete can easily be undone within the number of days you have specified.

But I want to go further, I want to have some buckets that I know are my “keep forever” bucket, and I want to make any kind of delete of even previous revisions difficult: enter MFA Delete.

Enabling MFA Delete

MFA delete works on Versioned S3 Buckets, and protects all revisions (including delete markers) from being deleted with a corresponding special delete command that includes a valid MFA token from an authorised user.

In my experimentation, I had an existing bucket that I had Versioning enabled. To enable this feature I had to turn to the API – this isn’t available in the AWS Console at this time. I also had to us an IAM User with MFA or the master Root identity – federated users or Ec2 Instances in IAM Roles cannot do this, as they have no MFA associated with them directly.

In this example, I created a profile for the AWS CLI called MasterUser, and had root IAM keys created (which I immediately rescinded). I had a bucket called MyVersionBucket, that I had set up just as I liked it. I also grabbed the ARN of my Virtual MFA I had for the Root user in this account (the ARN is listed as a SERIAL number in the console).

To enable MFA Delete:

aws s3api put-bucket-versioning –profile MasterUser –bucket MyVersionBucket –versioning-configuration MFADelete=Enabled,Status=Enabled –mfa ‘arn:…. 012345

Note: the MFA is referenced with quotes around it, as the single argument contains a space between the serial (ARN in this case) and the current value on the MFA).

To then see the configuration:

aws s3api get-bucket-versioning –profile MasterUser –bucket MyVersionBucket

With this in place it was time to test it out.

(Not really) Deleting from an MFA-Delete protected Bucket

The first thing I did was upload a file (same as normal), and then delete it. Using the “current view” of the bucket, the file vanished. In the new AWS console I could see the deleted item listed, and drilling into it, I could see the revisions there as with a regular Versioning bucket.

The next thing I tried was to “undelete” an object, an option that has just appeared in the revised S3 console, however this silently failed.

I then looked at the revisions of my sample file, and could see the delete marker sitting there. I attempted to delete the Delete Marker, but without an MFA I was blocked. This seemed to make sense: previously “undeleting” an object from S3 meant removing the delete marker, and clearly that’s just a version that I cannot really delete.

I looked at the other revisions of my sample file, and I was likewise blocked from deleting them.

Next looked at adding a Lifecycle policy to the bucket, and discovered that no Lifecycle policies can be added to an MFA protected bucket. So three’s no opportunity to move to the Infrequent Access tier of storage after a period automatically.

To truly empty the bucket, I deleted the a version of the file:

aws s3api delete-object –bucket MyVersionBucket –key sample.png –version-id Foo1234 –mfa ‘arn:… 123456

The VersionID was displayed to me in the list versions’ output.

Of course, I could potentially have suspended MFA delete, tidied up, and then re-enabled it.

At the end of my experiment, with MFA Delete Enabled, I could dimply delete the empty bucket as normal – there were no further challenges.

When to use MFA Delete

As MFA-Delete is a bucket-wide policy, you need to ensure that all objects that will be in this bucket are right to be considered permanent. You’ll want to limit who has access to put-version policy (perhaps your PowerUsers should have an explicit deny on this API call). If you have temporary or staging data in the bucket, or data that you want a lifecycle policy to automaticlaly clean up, then MFA Delete is not for you.