AWS VPCs: Calculating Subnets in CloudFormation

Virtual Private Cloud is a construct in AWS that gives the customer their own, er, virtual network for the deployment of network based resources such as virtual machines and more. Its been around for nearly a decade, and is a basic construct that helps provide security of those resources within an AWS Region.

CloudFormation is the (text, either YAML or JSON) templating language (service) that can take a definition of resources you would like configured, and does the execution of creating these resources for you, saving you the hassle of having to either navigate the web console for hours, or scripting up many API calls (which could be thousands of API create calls).

VPCs can be quite complex; they can specify subnets for resources, across multiple Availability Zones within a Region, define routing tables, Endpoints to create, and much more. So it probably comes as no surprise that managing a VPC via CloudFormation is a natural desire. The configuration of the virtual network for a workload needs to be as management in a CI/CD fashion as the workload that will live in there.

But there’s often been a limitation in making this simple; mathematics.
Continue reading “AWS VPCs: Calculating Subnets in CloudFormation”

AWS CloudFront launches in Perth

I moved back to Perth in 2010, having grown up here, gone to school, University and started my career here. It’s a lovely city, with the metropolitan area sprawling north and south along the blue Indian Ocean for some 50+kms. They says it’s a bit of a Mediterranean climate, normally never going below 0°C, and the heat of summer hitting mid 40°C, but with a fresh westerly coastal breeze appearing most afternoons to cool the place down.

But it is rather remote from other major population centers. The next nearest capital city, Adelaide, is 2,600 kms (1,600 miles) by road. Melbourne is 3,400 kms (2,100 miles) on the road, and Sydney is 3,900 kms (2,400 miles).  It’s a large state, some 2.5 million square kilometers of land, the size of the US Alaska and Texas states combined.

So one thing those in technology are well aware of is latency. Even with fibre to the premises (NBN in Australia), the Round Trip Time to Sydney is around 55ms – which is a similar time to Singapore. Melbourne comes in around 45ms.Latency from Perth to Singapore, Sydney, Melbourne, and New Zealand to Sydney

In 2013 I met with the AWS CloudFront team in Seattle, and was indicating the distances and population size (circa 2 million) in Perth. There’s a lot of metrics that goes in to selecting roll-out locations (Points of Presence) for caching services, with latency, population size, economic prosperity, cost of doing business, customer demand from a direct customer model, and customer demand from an end-consumer model being weighed up.

This week (1st week of January 2018) AWS CloudFront launched in Perth.

This impact on this is that all web sites that people of Perth that use CloudFront will now appear to be faster for cachable content. The latency has dropped from the 45ms (to Melbourne) to around 3ms to 5ms (from a residential NBN FTTP @ 50 Mbit/sec).

Test at 9:30pm from Perth (iiNet NBN).

In addition, the ability to upload/send data to applications (Transfer Acceleration) on-Region via the Edge (Edge Upload) may now also make a difference; with 45 ms to Melbourne, its been a largely unused feature as the acceleration hadn’t made much of a difference. There is a Transfer Acceleration test tool that shows what effect this will give you; and right now, while it shows an advantage to Singapore, just a 7% increase in performance to the AWS Sydney Region. Its not clear if TA via the Perth PoP is enabled at this point, so prehaps this will change the result over time.

And so, after several years, and with other improvements like the ability to restrict HTTPS traffic to TLS 1.2, it now makes sense to me to use CloudFront for my personal blog. In an hour, I had applied a new (additional) hostname against my origin server (a Linux box running WordPress) by editing the Apache config, symlinking the wordpress config file, and adding a Route53 CNAME for the host. I had certbot on Linux then add the new name to the Let’s Encrypt certificate on the origin. Next I applied for an Amazon Certificate Manager SSL certificate, with the hostname blog, and (if you inspect it) I then created a Cloudfront Distribution, with one origin, but two behaviours – one for the WordPress admin path, and one for the default paths, so that I could apply additional rules to protect the administration interface.

With this in place I could then update the DNS CNAME to move traffic to CloudFront, without any downtime. Not that downtime matters on my personal blog, but doing exercises like this you need to practice.

Welcome to Perth, CloudFront.

PS: It’s worth noting that IPv4 DNS resolution for my CloudFront distribiution is giving me 4ms RTT from Perth, but IPv6 RTT is 52ms, which indicates that IPv6 CloudFront has not yet arrived here.

AWS Certifications in Perth (II)

I wrote last year about sitting AWS Certifications in Perth. I’ve done another two AWS Certifications in the last month (Networking Specialty, and Cloud Practitioner), and a few things have changed. Gone is Kryterion as the assessment provider, and in has come PSI; this means new venues- and there’s now only one in Perth at 100 Havelock St, West Perth.

It’s a new-ish building I know well; an old friend was working on the top floor for a while, and I spoke to his teams about AWS several times (they became and AWS reference customer). Small Italian-inspired coffee shop on the ground floor (more on this later).

The booking process for exams is much the same, but now via (funky new DNS TLD). The certifications with PSI happen via their customer rigged Kiosk systems: a PC with two webcams, one mounted on the monitor facing the candidate, and one positioned on mast protruding above the screen facing the desk (down). With these two cameras, a remote monitor can view the candidate and the desk at all times to ensure there is no compromise of reference materials; and one person remotely monitoring can theoretically be proctoring multiple students in many locations simultaneously (I suspect they are listening too).

With this customer rig, there are only limited seats — in Perth, there are two. And the booking process is scheduling candidates to one of these Kiosks — literally called Kiosk 1 and Kiosk 2 — are located in a small room on the 1st floor of 100 Havelock St, looked after by the friendly Regus staff.

The exam start time is often 8:30am, and advise on the booking emails recommends turning up 15 minutes before this. By contrast, some non-AWS exams scheduled with PSI on the same Kiosks recommend arriving 30 minutes before hand. But there’s a catch; the doors on the ground floor do not unlock for access until around 8:25am, and Regus doesn’t often get staffed until 8:30am (Regus checks you in and sets you up at the Kiosk).

Unlike the Kryterion centers, this doesn’t seem to be a big problem — previously being just a few minutes late was an issue; so, if you do get there with plenty of time, the aforementioned cafe on the ground floor is open much earlier (there were open at 8:00am they day I got there early).

Photo ID is critical to have with you; a scanner mounted on the Kiosk rig is used to get an image of documents like Passports and Drivers’ Licences. You should have two forms of photo ID, but if you have bank cards or others they can supplement (just cover some of your card numbers for security’s sake). The moderator looking at the camera compares the Photo ID with the image of you sitting there in real time.

The assessment interface itself is then very similar, with the addition of a chat window to communicate with the moderator at any time. Feedback comments can be left on questions. I found one question had assumed that multi-choice answers that did not include the answer that had changed in mid-December (just a few weeks ago) so I left a commend for the AWS certification team on this and followed up by my contacts directly.

I’ve had no problem scheduling certifications with a week’s notice, but I envisage that as demand grows, the lead time to book a slot may become an issue until more Kiosks are added (or additional venues). But that’s not an issue right now.

AWS GuardDuty: taking on the undifferentiated heavy lifting of network security analytics

Guard Duty is a machine learning security analytics service for AWS

Several years ago saw the introduction of AWS CloudTrail, the ‘almost’ audit log of API calls performed by a customer against an AWS Account. This was a huge security milestone; the ability for the customer to play back what they had asked for.

I say ‘almost’, as a critical design decision was for CloudTrail in no way to inhibit the already authenticated API call that had been made by the customer. If the internal logging mechanism of CloudTrail were to ever fail, it should not stop the API call that was issued. Other logging mechanisms in computing may place logging in the critical path of call execution, and if logging fails, then the API call fails.

With CloudTrail (and the ability to go directly cross-account to from AWS direct to a trusted independent account, came the second task – looking at the data. Its all JSON text, and it has a corresponding chain of check-summed and signed digest files meaning the set of log files cannot be tampered with, and cannot be removed without breaking the chain.

Numerous solutions were put in place, but they were mostly basic individual pattern matches against single lines of logs. If you see X, then alert with a message Y: If there is a Console Login event, and it doesn’t come from XX.YY.ZZ.AA/32, then alert.

Similarly, VPC introduced VPC flow logs, tracking the authorisation or rejection of connections through the VPC (no payload content, just payload size, start time, ports, addresses).

In December, AWS introduced a managed service that would use a private copy of the VPC Flow Logs, a private copy of the CloudTrail log, as well as a Route53 query log, and supplement this with some centrally managed, maintained and updated threat lists, mix in some customer defined threat lists and white lists, mix with a bit of machine learning, and produce much richer alerting.

Guard duty currently has not finished yet. At re:Invent, Tom Stickle indicated in a graph that there is a slew of additional capability coming shortly to GuardDuty, and now that it’s GA, more customers will have feedback and input into the future direction of the service.

However, this doesn’t replace the need to have your own, secured and trusted copy of your CloudTrail logs, and your own alerting for events that you think are particularly significant, such as a SAML Identity Provider being updated with a new Metadata document!

But between this, and Amazon Macie (for analysing and helping you review and secure your S3 documents), your visibility of security compliance and issues continues to get even higher.

Staring at tomorrow: Technology Impacts on society

This started as musings on where broadband connections enter buildings. It went further.

There’s a few technical specifications that we’ve become very used to in our homes and offices for the last few decades, but they’ve not always been there. Their introduction to our lives has spurred changes in the construction of our dwellings, and our expectations of what we do in our lives. And another one is unfolding now.

Plumbing (Scheme Water, Sewerage), Electricity, and Broadcast TV have all had their impacts. Water in & out has remained largely unchanged in the last 50 years, having moved from outhouses to servicing dedicated ‘wet’ rooms in our buildings (kitchens, bathrooms, WCs). Electricity has crept from being provided for lighting purposes only, to additionally providing a single power socket per home, to multiple sockets per room at convenient places for us to leave devices semi-permanent plugged in. We’ve become accustomed to the 110v/60Hz and 230v/50Hz, and the sockets and plugs are generally down to several well known arrangements of pins and locking mechanisms.

With the dawn of broadcast TV (UK: 1936, US: 1948, Australia: 1965), and excluding set-top “rabbit ears”, we’ve had antennas on roofs with coaxial cables leading to specific rooms: family rooms, bedrooms, etc. Using the superior reception of the roof-top antenna has meant that the placement of TVs was pretty much determined by the location of Co-Axial sockets in walls. Indeed, not just which rooms, but which corners and walls of specific rooms, close to these antenna sockets. While we’ve happily added multiple power sockets to rooms for future proofing our desire to rearrange our lives, co-axial cables suffered more loss with more joins to other sockets, so it was generally avoided.

Thus our layout of our homes has been throttled by the tether to the roof top antenna.

We are well into the next revolution delivery of our video feeds over a new carrier: IP and the Internet. The age of radio frequency broadcast is declining, and with it some of our cultural norms.

One impact is building design: a time will come when we’ll no longer request TV broadcast antennas on buildings, and coaxial sockets on walls. This is replaced with either short range high speed WiFi such as 802.11ac, or wired Ethernet ports at gigabit speed (no less). Wireless has continued to increase in speed, but as with all broadcast spectrum, it is subject to interference from other wireless signals, which can impact throughput. A continual battle between ever faster wireless speeds, improved signal processing and interference handling, and cost considerations will play out to determine if our TVs will have a wired or wireless IP connection within the home.

Wireless has been through several generations, some of which are now well and truly dead. WEP encryption, once the critical protection for wireless networks, is known to be compromisable, and thus is abandoned today. Hardware devices that only implemented WEP are effectively junk. Devices with physical Ethernet socket have not suffered so – the trusty RJ45 100 base TX CSMACD works as well today as it did 20 years ago.

We’ll see people opt to use wireless for video delivery within homes for short term savings, the aggressive rate of change on encryption and signalling standards for wireless networks will find many wireless-only devices become redundant before their time. We expect to get 10 – 20 years from a TV (its often an expensive item), but should WPA2 get compromised, then those hosts are vulnerable. We’ve already seen this with the KRACK vulnerability, and those vendors that did not patch their firmwares are probably vulnerable (both the Access Point and clients need patches to address KRACK for WPA2).

Of course, the interactions that your TV makes over whatever connection should indeed be over an encrypted transport, but even the encryption that uses – the key lengths, signature algorithms, all will be refreshed and strengthened over time.  In the last 5 years we’ve seen web site certificates change in key length (1024 -> 2048 bit), signatures for asserting a chain-of-trust from a certificate authority move from what was MD5 to SHA1 to SHA256, the bulk encryption algorithms move from RC4 to AES128 to AES256 to Elliptical Curve algorithms, and message digest checksums from MD5 to SHA1, to SHA256, and not SHA384. Each of these changes needs to be applied to the software in your appliances for them to maintain the security you expect to be there.

Many manufacturers won’t bother updating this on their already-shipped units; the disposable consumer sales cycle continues unabated and effectively is helped by these changes, and customers barely make any mention of the trouble of having to replace these items.

But as we move to IP delivery of video content, we also move to more of a video-on-demand world, and away from a time-of-broadcast (playout) world where all consumers – the audience in a broadcast area – would all get the same content at the same time.

30 years ago, when I was at school, the talk in the playground was of the latest episode of some show: MacGuyver, Airwolf, The A-Team, comedies like ‘Allo ‘Allo, The D-Generation, The Comedy Company, etc. With only a few select channels to choose from (3 where I was), your peers likely saw the same content at the same time, making for a shared experience that you wove into the daily fabric of society.

Shows with catch phrases became common place, and everyone knew them. Mark Mitchell gave Australia the “Couple o’ days, Beautiful”, and everyone knew the setting and connotation.

As we move to a VOD, subscription based scenarios, this disappears. Shows on one subscription service are not on others, or appear at different times. Subscription services cross local boundaries in a way that broadcast spectrum could never do with the limitation of transmitter power. People watch content at different times, in different months, from different countries (despite the continued attempt for geo-blocking). The social background noise of modern society is changed, disjointed.

I suspect that in the next decade, broadcast TV will disintegrate. Advertising spend will continue to shift to product placement in original content, and customised pre- and mid-roll ad insertion, customised to the view, tracked and targeted.

As broadcast TV declines, it heavily modifies the stalwart of most TV stations: TV news. While news has been available form multiple media sources, local broadcast TV news was always curated to somewhat balance local, national and global current affairs. If all journalism production costs the same, then stories filed that can be replayed in multiple territories have more value per second, and the decline of local stories.

I’ve watched local news for decades, paying attention to their mix of slow news days stories (car through a fence, cat up a tree) and more interesting ones (state general or by election results, perhaps investigative journalism – but that’s on the decline). Its even interesting to sometimes see the format of content being padded out with in-content extended adverts for news clips that are scheduled later, the tweaks to the lower thirds straps, even the background animations that engage the reader in subtle ways. These things often annoy me at how much they con the audience’s emotional engagement with a story. One local broadcast TV channel has a habit of applying a cinematic dust/sparkle video filter to most human-interest stories (cat up tree). They’ve played games with putting weather forecasts showing MIN (overnight) and MAX (daytime) temperatures on top of each other, and then sometimes MAX (daytime) on top of MIN.

(While I’m ranting, the “cross to the local hospital to a reporting standing out front” seems a waste of time, as is a “cross to the next room” scenario which an anchor could have just read out and moved on from.)

One media that has withstood much change until now is broadcast radio. I suspect that because radio has had one place where its continued its existence without much challenge: vehicles. Vehicle manufacturers have been pretty slow at putting DAB radios into vehicles by default. AM and FM radios are still universal. I wrote in 1995 in the UWA student magazine Pelican about the dawn of MP3 as an audio format, and the start of Digital Audio Broadcasting (DAB). The MP3 player was born – Apple seized on it with the iPod, and the Sony Walkman receded to the annals of history. And while DAB has been around for some time, the licensing and hardware has been at an expense that did not generally warrant the improvement in delivery, and clearly not supported by vehicle manufacturers.

But while broadcast radio has been compatible with what the listener is doing in vehicles: a threat is coming to the radio’s last bastion. It’s the same threat that is coming to organisations that live off driving license fees: self-driving cars. Drivers can then do other tasks: they can then look at screens. The radio will stop becoming an in-vehicle companion to millions of single-occupant cars, as those occupants will start viewing content instead of just listening to it. Eyes will come off the road.

(And if eyes are off the road, do we have the demise of billboards and placards as an advertising medium – as no one is looking out the window any more?)

By extension to this, the music industry relies on radio for socialising its content, encouraging people to either purchase content, or become customers to performers when they tour. With broadcast music no longer curated to seed the introduction of new content over time, artists will find it difficult to get established.

Coupled with the self-driving car is the move to electric vehicles, and the eventual drop in the petroleum fuel use, and the tax (excise) that is collected from this. Governments often use this as a source of funding road building. This model will have to change, probably to a tax-per-kilometer travelled on the owner of these vehicles.

And with self-driving cars, we can finally have some backward parts of the world switch to the metric system for units of distance and speed, without the risk of the human population getting it wrong and going too fast/slow/far.

Eventually, as my friend Paul Fenwick (PJF) has spoken of, the population will move to not owning vehicles, but calling them completely on demand – a la Uber/taxi, but without the human driver. These vehicles will get corporate ownership, and will all have live mobile broadband data links to each vehicle. They’ll all have built-in dash cams, and logging of all activities in and out of the vehicles at all times. New advertising opportunities will rise up – screens in these vehicles will know the route you’re about to take to advertise products and services on or near your route, so you can chose to stop off. They’ll know your preferences, the environmental factors (warm, cold, sunny, rainy) and advertisers will bid to produce targeted advertising at you while you travel.

Next knock on effect from this is the car insurance industry. If the self-driving vehicle has less accidents, and the risk of death or major damage is less, then disruption will arrive here too. With only a few major self-driving-taxi companies that require insurance, the number of insurance companies will consolidate.

The self driving vehicle probably needs less street signs. It may require less street lighting. It may require less lane markings, cats eyes, number of lanes.

This probably sounds like a diatribe version of a Swardley map (hat tip to Simon). All these things are connected, generally by revenue streams or shared interests, and always by data and technology. As always, its all change, and resistance is futile.