Category: Rants

AWS Certification trends (on LinkedIn)

I am always trying to find great talent; it’s part of being a Practice Lead in a large consulting organisation to find and develop talent. I work with a team recruiters who are constantly finding and screening people for the many roles we have.

I’ve been a big proponent of the AWS Certifications for a number of reasons; amongst which are value and confidence to the holder, value to the partner, value to the customer. I helped contribute questions to the AWS Solution Architect Professional certification in 2014 whilst passing through Herndon, Washington DC as an AWS employee, and again in February 2020 in San Francisco as an industry Subject Matter Expert, just before COVID-19 started closing down travel.

Today I took to LinkedIn, and did a search for the various AWS Certifications, and found a tally that looked interesting. These numbers are by no means authoritative, and could just be a reflection of the network of connections that I have.

AWS CertificationTallyLaunch Year#/Year (to 2020)
Solution Architect Associate*311,000201344,428
Developer Associate*189,000201431,500
Cloud Practitioner*103,000201734,333
Solution Architect Professional*94,000201415,667
DevOps Engineer Professional*57,00020149,500
SysOps Associate*29,00020179,667
Security Specialty*12,00020186,000
Networking Specialty*7,80020183,900
Database Specialty*7,20020197,200
Data Analytics Specialty6,30020196,300
Big Data Specialty (retired/renamed to Data Analytics)81,0002014 – 201916,200 #
Machine Learning Specialty5,30020195,300
Alexa Skill Builder Specialty5462019549
AWS Certifications as found on Linked In, 18/9/2020. * Denotes certifications I hold. # only calculated over the five years this was active.

With such a low number for the Alexa certification, I expect the source numbers is not be complete. Many people in certain industries (eg, intelligence services) will not put their profile online.

But regardless, lets review what we see…

The clear winner is the venerable Solution Architect Associate with the largest number per annum and largest number in total. Its seen as the initial certification in the technical certs, and is regularly reported as one of the most valuable in the industry with respect to salary expectations. Its also the longest cert I have held – being part of the very first cohort to pass this in January 2013.

While the Developer Associate certification is in second place by total number, it is just eclipsed by the number of people who have taken the Cloud Practitioner Foundational certification, on a yearly basis. The Cloud Prac is billed as an entry level, non-technical certification, so its appeal is to an even wider audience – the technical team can obtain it relatively easily, and the non-technical roles involved in total service delivery can achieve this as well.

At the Professional level, it seems the demand for certified Architects outweighs the DevOps Engineers almost 2:1; I suspect this is as a natural progression from that initial SA Associate.

The Data Analytics certification replaced the original Big Data cert last year; this gives us an insight into the change in demand. Over its active lifetime, Big Data drove 16,200 per year – its replacement sites at almost a third the prior demand. Perhaps the data analytics hype is stablising?

The total number of certifications reported above is 903,146; just shy of a million certifications in 7 years (and probably more given the validity of the data) excluding re-certifications (after 3 years, now).

Lets see what this looks like in a year from now. New AWS certifications will likely launch, continuing to help validate and differentiate experienced Cloud engineers.

S3 Public Access: Preventable SNAFUs

It’s happened again.

This time it is Facebook who left an Amazon S3 Bucket with publicly (anonymously) accessible data. 540 million breached records.

Previously, Verizon, PicketiNet, GoDaddy, Booz Allen Hamilton, Dow Jones, WWE, Time Warner, Pentagon, Accenture, and more. Large, presumably trusted names.

Let’s start with the truth: objects (files, data) uploaded to S3, with no options set on the bucket or object, are private by default.
Someone has to either set a Bucket Policy to make objects anonymously accessible, or set each object as Public ACL for objects to be shared.

Lets be clear.

These breaches are the result of someone uploading data and setting the acl:public-read, or editing a Bucket’s overriding resource policy to facilittate anonymous public access.

Having S3 accessible via authenticated http(s) is great. Having it available directly via anonymous http(s) is not, but historically that was a valid use case.

This week I have updated a client’s account, that serves a static web site hosted in S3, to have the master “Block Public Access” enabled on their entire AWS account. And I sleep easier. Their service experienced no downtime in the swap, no significant increase in cost, and the CloudFront caching CDN cannot be randomly side-stepped with requests to the S3 bucket.

Serving from S3 is terrible

So when you set an object public it can be fetched from S3 with no authentication. It can also be served over unencrypted HTTP (which is a terrible idea).

When hitting the S3 endpoint, the TLS certificate used matches the S3 endpoint hostname, which is something like s3.ap-southeast-2.amazonaws.com. Now that hostname probably has nothing to do with your business brand name, and something like files.mycompany.com may at least give some indication of affiliation of the data with your brand. But with the S3 endpoint, you have no choice.

Ignoring the unencrypted HTTP; the S3 endpoint TLS configuration for HTTPS is also rather loosely curated, as it is a public, shared endpoint with over a decade of backwards compatibility to deal with. TLS 1.0 is still enabled, which would be a breach of PCI DSS 3.2 (and TLS 1.1 is there too, which IMHO is next to useless).

Its worth noting that there are dual-stack IPv4 and IPv6 endpoints, such as s3.dualstack.ap-southeast-2.amazonaws.com.

So how can we fix this?

CloudFront + Origin Access Identity

CloudFront allows us to select a TLS policy, pre-defined by AWS, but permitting us to restrict available protocols and ciphers. This lets us remove “early crypto” and be TLS 1.2 only.

CloudFront also permits us to use a customer specific name, for SNI enabled clients for no additional cost, or a dedicated IP address (not worth it, IMHO).

Origin Access Identities give CloudFront a rolling API keypair that the service can use to access S3. Your S3 bucket then has a policy permitting this Identity access to the host.

With this access in place, you can then flick the “Block Public Access” setting account-wide, possibly on the bucket first, then the account-wide settings last.

One thing to work out is your use of URLs ending in “/”. Using Lambda@edge, we convert these to a request for “/index.html”. Similaly URL paths that end in “/foo” with no typical suffix get mapped to “/foo/index.html”.

Governance FTW?

So, have you checked if Block Public Access is enabled in your account(s). How about a sweep through right now?

If you’re not sure about this, contact me.

CloudPets security fail is not a Cloud failure

I spent several years at Amazon Web Services as the Solution Architect with a depth in Security in A/NZ. I created and presented the Security keynotes at the AWS Summits in Australia and New Zealand. I teach Advanced Security and Operations on AWS. I have run online share-trading systems for many of the banks in Australia. I help create the official Debian EC2 AMIs. I am the National Cloud Lead for AWS Partner Ajilon, and via Ajilon, I also secure the State Government Land Registry in Ec2 with Advara.

So I am reasonably familiar with configuring AWS resources to secure workloads.

Last week saw a poor security failure; the compromise of a company that makes Internet-connected plush toys for children that lets users record and playback audio via the toys: CloudPets. Coverage from Troy Hunt,  The Register, ArsTechnica.

As details emerged, a few things became obvious. But here are the highlights (low-lights, really) to me that apparently occurred:

  • A production database (MongoDB) was exposed directly to the Internet with no authentication required to query it
  • Audio files in S3 were publicly, anonymously retrievable. However, they were not listable directly (no worries, the object URLs were in that open Mondo database)
  • Non-production and production systems were co-tenanted

There’s a number of steps that should have been taken technically to secure this:

  1. Each device should have had a unique certificate or credential on each of them
  2. This certificate/credential should have been used to authenticate to an API Endpoint
  3. Each certificate/credential could then be uniquely invalidated if someone stole the the keys from it
  4. Each certificate/credential should only have been permitted access to fetch/retrieve its own recordings, not any recording from any customer
  5. The Endpoint that authenticates the certificate should have generated Presigned URLs for the referenced recordings. PreSigned URLs contain a timestamp set in the future, after which the Presigned URL is no longer valid. Each time the device (pet) would want a file, it could ask the Endpoint to generate the Presigned URL, and then fetch it from S3
  6. The Endpoint could rate limit the number of requests per certificate pre minute/hour/day. Eg, 60 per minute (for burst fetches), 200 per hour, 400 per day?

If the Endpoint for the API was an Ec2 instance (or better yet, an AutoScale Group of them), then it could itself be running in the context of an IAM Role, with permission to create these Presigned URLs. Similarly an API Gateway running a Lambda in a Role.

Indeed, that Endpoint would have been what would have used the MongoDB (privately), removing the publicly facing database.

I’ve often quoted Voltaire (or Uncle Ben from Spider Man, take your pick): “with great power comes great responsibility“. There’s no excuse from the series of failures that were conducted here; the team apparently didn’t understand security in their architecture.

Yet security is in all the publicly facing AWS customer documents (joint responsibility). It’s impossible to miss this. AWS even offers a free security fundamentals course, which I recommend as a precursor to my own teachings.

Worse is the response and lack of action from the company when they were alerted last year.

PII and PHI is stored in the cloud. Information that the economy, indeed modern civilisation depends upon. The techniques used to secure workloads are not overly costly, they mostly require knowledge and implementation.

You don’t need to be using Hardware Security Modules (HSMs) to have a good security architecture, but you do need current protocols, ciphers, authentication and authorisation. The protocols and ciphers will change over time, so IoT devices like this need to also update over time to support Protocols and Ciphers that may not exist today. It’s this constant stepping-stone approach, to continually be moving to the next implementation of transport and at-rest ciphers that is becoming a pattern.

Security architecture is not an after-thought that can be left on the shelf of unfulfilled requirements, but a core enabler of business models.

Hurricane Electric IPv6 tunnel MTU

I’ve been running an IPv6 tunnel for a long time, but occasionally I’ve been seeing traffic hang on it. It looks like it was the MTU, defaulting at 1500 bytes, causing issues when large amounts of data were being shuffled OUT from my Linux box, back to the ‘net’.

The fix is easy: /etc/network/interfaces should have an “up” line for the interface definition saying: up ip link set mtu 1280 dev henet, where henet is the name of your tunnel interface.

Easy enough to skip this line if your tunnel appears to be working OK, but interesting to track down.

New a new PC. Time for a desktop?

My 2 year old Dell Studio 1558 is doing it again: slowing to a snails pace, heating to an inferno, and then spontaneously powering off (which I think is a saftety set at CPU temperature reaching 100*C).

I had Dell come and replace parts on this laptop about 9 months ago when similar symptoms developped. I originally purchased this unit while I was in the UK, around January 2010 I think it was. I was hoping to get 3 years out of it. Sadly, at around 20 months old, I’m getting too frustrated to put up with it. I’m now living in Australia, and having any PC multi-national company honour their warranty internationally is a challenge. Heck, worse offender in this scenario is Sony, who want £20 to answer the phone!

Now that I’m no longer living in a flat with a very transient lifestyle (lots of travel having gone, and replaced by a 1 year old boy), I’m much more rooted to my home office desk. So, in light of this, I’m thinking of getting a desktop with a reasonable screen. I saw Russell Coker’s post about a 27″ whopper from Dell for AU$899 or so, and was wondering what to pair that with, or if to go for a slightly smaller screen. Then comes the questions of the all-in-ones, and the touchscreens that are around.

What I’d like is something thats got a few (2?) USB 3 ports for the next few years of my accessory usage, SATA 3 so I can throw in a fast SSD. I’d potentially run Debian on this, so possibly don’t want a Windows license.4 GB RAM minimum, possibly 8.

So looking around its a quagmire of detaisl that 15 years ago I used to thrive on. Do I care about UEFI instead of a traditional BIOS. DO I really need SATA 3 instead of 2? What about legacy (!) 1394? HDMI connector – yes please – do I still want a VGA port? What about a second HDMI? Hm. That 27″ screen’s native res is more than most on-board graphics can drive… perhaps drop to a 24″ screen. What size should this be: ATX, mini ITX, smaller?

Then comes the pre-built or custom built. Dell, pretty I’m upset about your product quality right now. HP, you’ve (a) killed my DreamScreen recently, and (b) put your entire business in up the creek with indications that the PC business is going away/sold off. Lenovo? Acer?

So I’m at a computing crossroads. I can’t be bothered to build my own PC again – I’ve been living on laptops for almost a decade now. But they are expensive, and when something goes wrong, the there’s very little to salvage. Laptops suck, but do desktops suck less. Vendors suck, but then so does the time waste on building your own? I think Tablets suck for doing lots of data input (programming). All in ones – not sure. Touchscreens – probably a gimmick.