(Previously-) Symantec Run Certificate Authority distrust is about to hit

Sometime in the next week, a large swathe of web sites around the world, from Fortune 500 companies, to governments and beyond will stop being available securely. All with the next release version of Google Chrome (version 70) and Firefox. The words “NET::ERR_CERT_SYMANTEC_LEGACY” are about to become well known.

For those wishing to look into the future, Google Chrome makes its future releases available to those interested under the labels of Beta, and prior to that, as a “Canary” (ie, in the coal mine). And if you’d cast your eyes over a few sites with these pre-release versions, you’ll see examples like that shown here (name removed).

A web site not available as the issuing certificate authority has been distrusted.

Sadly, the operators of these sites may well be looking at the embedded certificate expiry (Valid Until) date, and think this is not an issue for them. Some of these certificates may have many more years of appearing to be valid.

The case is much worse: the organisation that these web site operators obtained the certificate from — the Certificate Authority — is about to have its status revoked, having been caught acting in ways that undermine the trust instilled in it. These are all powered by Symantec’s legacy root certificates, which includes the Thawte, GeoTrust, and RapidSSL brands.
You can read plenty online about this, for example: form DigiCert, Mozilla, and Google. Here’s Scott Helm’s February 2018 post, and his follow up from a recent Alexa Top sites crawl. Several of these have since updated their certificates (Well done, Tigereair.com.au, stratco.com.au, naati.com.au; fixed it!).

So what’s actually going to happen?

Some disruption.

I’m sure a large number of these will be smeared across mainstream media for being “hacked”, or “offline”, when in reality, “oblivious” is closer to the point.

Poor service providers are going to tell vulnerable people to “ignore the security warnings”, and to “proceed” to the site regardless. This is BAD advise. If you are told this, you are better off ceasing to do business with the organisation as they do not under stand the security they are dealing with. If this is the advice of your employer, then you should consider what this means to the security of your personal HR (and other) data.

There’s far too many people operating, controlling, or otherwise “responsible” for large numbers of web sites who have no idea about what they are actually operating. It’s evident from scanning site and seeing those that still have legacy, vulnerable encryption on their HTTPS configuration, or worse, serve content over unencrypted HTTP. Just because you don’t value you’re content from modification, doesn’t mean your web visitors don’t value NOT being compromised when visiting you.

Web traffic interception happens every second of every day. In Wifi Cafes, Airports, air planes, corporate LANs. TLS (formerly SSL) is the best way we have to protect the integrity of the content across untrusted networks, but we’re in a constant capability race to ensure that services only offer ways to connect that minimise the risk of using untrusted networks.

Driven by a desire to not change things that appear to be working (or indeed, being either lazy, overworked, under resourced/funded, or unaware), organisations are not bringing up their drawbridge of security on their most vulnerable interfaces: those services that are facing the Internet, such as their web site or web services. This issue, when it breaks, will help highlight that some organisations and individuals should probably not be in charge of the services they currently operate.

Case in point: check out BankGradeSecurity.com, a ranking of financial institutions around the world and how well they have adopted modern encryption and security capabilities on their web site and Internet banking services.

It’s clear we’re constantly in the middle of technology transitions – IT Services are not simply done; they are either in-use and actively well-maintained, or they should be archived or removed. Anything else demonstrates cost cutting and under-valuation of the digital capability that allows an organisation to operate.

Organisations face a choice of two types of Managed Services providers today: those that understand service maintenance on behalf of their customers and those that do not (and are still running with the same HTTPS configuration they went live with years ago.

It’s easy to spot these services — they haven’t enabled GCM based AES block ciphers or Eliptical Curve Diffie-Hellman Ephemeral (ECDHE) key exchange mechanisms. Worse, those permitting the use of SSLv2, SSLv3, or TLS 1.0, or not yet permitting the use of TLS 1.2 (or the shiny new TLS 1.3). And unbelievably, those that don’t enable HTTPS at all.

There’s more signs of stagnation if you know what to look for; lack of HTTP/2, lack of IPv6, long TTLs on DNS records, etc, that all indicate organisations that are stifled, or don’t have capability to understand what they are doing. Sometimes its corporate direction to use 3rd party IT operations who again, use the cheapest unskilled and unqualified labour to delivery IT services, dressed up in marketing to make it look like they save the earth.

If you’re affected by this, consider attending Nephology’s Web Security training.

Gartner Magic Quadrant for Cloud 2018: Half the players dessimated

It’s as if the left hand side of the 2017 Gartner Cloud MQ just imploded! And just as interesting, most of those that were in the Visionaries quadrant are now relegated to the Niche Players sector.

Only six players remain compared to last years 14. Lets see who has gone this year:

  • Skytap
  • NTT
  • Joyent
  • Interoute
  • Fujitsu
  • Rackspace
  • CenturyLink
  • Virtustream

It’s no surprise to those close to the ground that the only real survivors here thus far are AWS and MS Azure, with Google barely making it into the top right corner, which actually looks like they just flopped over the line from last year &mdash but progress none the less. And while the gap between the top two is closing, AWS is still far above in “Ability to execute”, and slightly ahead in “Completeness of vision”.

For those that had chosen one of the departed 8 as their Cloud provider, then its time to question what their strategy is, and what yours is. Gartner may have upped its inclusion requirements resulting in some of these players being filtered out, and that may have no impact on those providers. But it may resonate poorly for their sales prospects going forward, all of which could have a long term downward trend, increased cost per customer, loss of economies of scale, etc.

Meanwhile, the rate of innovation by service improvement, refinement, or new offerings continues.

AWS VPCs: Calculating Subnets in CloudFormation

Virtual Private Cloud is a construct in AWS that gives the customer their own, er, virtual network for the deployment of network based resources such as virtual machines and more. Its been around for nearly a decade, and is a basic construct that helps provide security of those resources within an AWS Region.

CloudFormation is the (text, either YAML or JSON) templating language (service) that can take a definition of resources you would like configured, and does the execution of creating these resources for you, saving you the hassle of having to either navigate the web console for hours, or scripting up many API calls (which could be thousands of API create calls).

VPCs can be quite complex; they can specify subnets for resources, across multiple Availability Zones within a Region, define routing tables, Endpoints to create, and much more. So it probably comes as no surprise that managing a VPC via CloudFormation is a natural desire. The configuration of the virtual network for a workload needs to be as management in a CI/CD fashion as the workload that will live in there.

But there’s often been a limitation in making this simple; mathematics.
Continue reading “AWS VPCs: Calculating Subnets in CloudFormation”

AWS CloudFront launches in Perth

I moved back to Perth in 2010, having grown up here, gone to school, University and started my career here. It’s a lovely city, with the metropolitan area sprawling north and south along the blue Indian Ocean for some 50+kms. They says it’s a bit of a Mediterranean climate, normally never going below 0°C, and the heat of summer hitting mid 40°C, but with a fresh westerly coastal breeze appearing most afternoons to cool the place down.

But it is rather remote from other major population centers. The next nearest capital city, Adelaide, is 2,600 kms (1,600 miles) by road. Melbourne is 3,400 kms (2,100 miles) on the road, and Sydney is 3,900 kms (2,400 miles).  It’s a large state, some 2.5 million square kilometers of land, the size of the US Alaska and Texas states combined.

So one thing those in technology are well aware of is latency. Even with fibre to the premises (NBN in Australia), the Round Trip Time to Sydney is around 55ms – which is a similar time to Singapore. Melbourne comes in around 45ms.Latency from Perth to Singapore, Sydney, Melbourne, and New Zealand to Sydney

In 2013 I met with the AWS CloudFront team in Seattle, and was indicating the distances and population size (circa 2 million) in Perth. There’s a lot of metrics that goes in to selecting roll-out locations (Points of Presence) for caching services, with latency, population size, economic prosperity, cost of doing business, customer demand from a direct customer model, and customer demand from an end-consumer model being weighed up.

This week (1st week of January 2018) AWS CloudFront launched in Perth.

This impact on this is that all web sites that people of Perth that use CloudFront will now appear to be faster for cachable content. The latency has dropped from the 45ms (to Melbourne) to around 3ms to 5ms (from a residential NBN FTTP @ 50 Mbit/sec).

Test at 9:30pm from Perth (iiNet NBN).

In addition, the ability to upload/send data to applications (Transfer Acceleration) on-Region via the Edge (Edge Upload) may now also make a difference; with 45 ms to Melbourne, its been a largely unused feature as the acceleration hadn’t made much of a difference. There is a Transfer Acceleration test tool that shows what effect this will give you; and right now, while it shows an advantage to Singapore, just a 7% increase in performance to the AWS Sydney Region. Its not clear if TA via the Perth PoP is enabled at this point, so prehaps this will change the result over time.

And so, after several years, and with other improvements like the ability to restrict HTTPS traffic to TLS 1.2, it now makes sense to me to use CloudFront for my personal blog. In an hour, I had applied a new (additional) hostname against my origin server (a Linux box running WordPress) by editing the Apache config, symlinking the wordpress config file, and adding a Route53 CNAME for the host. I had certbot on Linux then add the new name to the Let’s Encrypt certificate on the origin. Next I applied for an Amazon Certificate Manager SSL certificate, with the hostname blog, and (if you inspect it) blog-cloudfront.james.rcpt.to. I then created a Cloudfront Distribution, with one origin, but two behaviours – one for the WordPress admin path, and one for the default paths, so that I could apply additional rules to protect the administration interface.

With this in place I could then update the DNS CNAME to move traffic to CloudFront, without any downtime. Not that downtime matters on my personal blog, but doing exercises like this you need to practice.

Welcome to Perth, CloudFront.

PS: It’s worth noting that IPv4 DNS resolution for my CloudFront distribiution is giving me 4ms RTT from Perth, but IPv6 RTT is 52ms, which indicates that IPv6 CloudFront has not yet arrived here.

AWS Certifications in Perth (II)

I wrote last year about sitting AWS Certifications in Perth. I’ve done another two AWS Certifications in the last month (Networking Specialty, and Cloud Practitioner), and a few things have changed. Gone is Kryterion as the assessment provider, and in has come PSI; this means new venues- and there’s now only one in Perth at 100 Havelock St, West Perth.

It’s a new-ish building I know well; an old friend was working on the top floor for a while, and I spoke to his teams about AWS several times (they became and AWS reference customer). Small Italian-inspired coffee shop on the ground floor (more on this later).

The booking process for exams is much the same, but now via https://aws.training/ (funky new DNS TLD). The certifications with PSI happen via their customer rigged Kiosk systems: a PC with two webcams, one mounted on the monitor facing the candidate, and one positioned on mast protruding above the screen facing the desk (down). With these two cameras, a remote monitor can view the candidate and the desk at all times to ensure there is no compromise of reference materials; and one person remotely monitoring can theoretically be proctoring multiple students in many locations simultaneously (I suspect they are listening too).

With this customer rig, there are only limited seats — in Perth, there are two. And the booking process is scheduling candidates to one of these Kiosks — literally called Kiosk 1 and Kiosk 2 — are located in a small room on the 1st floor of 100 Havelock St, looked after by the friendly Regus staff.

The exam start time is often 8:30am, and advise on the booking emails recommends turning up 15 minutes before this. By contrast, some non-AWS exams scheduled with PSI on the same Kiosks recommend arriving 30 minutes before hand. But there’s a catch; the doors on the ground floor do not unlock for access until around 8:25am, and Regus doesn’t often get staffed until 8:30am (Regus checks you in and sets you up at the Kiosk).

Unlike the Kryterion centers, this doesn’t seem to be a big problem — previously being just a few minutes late was an issue; so, if you do get there with plenty of time, the aforementioned cafe on the ground floor is open much earlier (there were open at 8:00am they day I got there early).

Photo ID is critical to have with you; a scanner mounted on the Kiosk rig is used to get an image of documents like Passports and Drivers’ Licences. You should have two forms of photo ID, but if you have bank cards or others they can supplement (just cover some of your card numbers for security’s sake). The moderator looking at the camera compares the Photo ID with the image of you sitting there in real time.

The assessment interface itself is then very similar, with the addition of a chat window to communicate with the moderator at any time. Feedback comments can be left on questions. I found one question had assumed that multi-choice answers that did not include the answer that had changed in mid-December (just a few weeks ago) so I left a commend for the AWS certification team on this and followed up by my contacts directly.

I’ve had no problem scheduling certifications with a week’s notice, but I envisage that as demand grows, the lead time to book a slot may become an issue until more Kiosks are added (or additional venues). But that’s not an issue right now.