Thoughts on the IPv6 Transition

I’ve been discussing the IPv6 transition with our customers more recently; for over 3 years we’ve been dual-stack IPv4 and IPv6 for public-facing AWS-Cloud-based solutions and services for our customers.

So what?“, you’re thinking?

It’s worth noting that from Google’s numbers, global IPv6 is now approaching 36%, while at home in Australia 27%, helped by TelCo carriers like Telstra enabling IPv6 to their mobile phone subscribers, and advanced ISPs like Aussie Broadband and Internode making IPv6 trivial to enable.

Google IPv6 Adoption, as of 12/Oct/2021

I first had an IPv6 tunnel established to Hurrican Electric in 1999 when I worked for The University of Western Australia. I championed the adoption of IPv6 as a first-class citizen in the cloud when I worked at Amazon Web Services as a Solution Architect, and these days, a large majority of AWS public-facing services already support dual-stack approaches, and more are on the way.

As the next billion people come online, the unavailability of more existing IPv4 Internet is a limiting factor. The temporary value of the IPv4 address space, being reallocated (“sold“) between assignees will eventually presumably peak when a majority of clients (people) and the services they are accessing are all on IPv6.

I have been advising a government body, who had two IPv4 “Class B” sized IPv4 subnets allocated to them. Each of these subnets is a “/16” netblock (65,535 addresses); they had only ever used a handful of /24 ranges from within their first allocation.

Most services they use, both for staff and for public-facing services, now run on the cloud, from cloud-provider address space. They’re unlikely to need all of the address blocks they currently have from the first /16 block, let alone the second.

This netblock has a current value of a couple of million dollars (AUD).

It’s likely that many public sector agencies have IPv4 address netblocks that they’re unlikely to ever use, and could also benefit from reallocating to service providers desperate for their own address space to host solutions from.

Well, desperate until most clients are using IPv6.

I’d urge any public sector organisation to review their plans for using their address space, and if they have large unused, contiguous address space, consider reallocating that. The funds raised can then help with further modernisation of workloads – including those workloads to move to IPv6 addressing.

For any managed service providers, I would urge you to “dual-stack” all public-facing Internet services. You should continue to use strong encryption in flight, modern TLS protocols, and strong authentication, regardless of the network transport protocol version.

If you are using AWS CloudFront as a CDN in front of your origin service, then enable IPv6 in the CloudFront configuration, and then publish the corresponding AAAA DNS record just as you have to the A DNS record. Similar works if using CloudFlare, Akamai, Fastly or others.

For those who use managed service providers for their corporate business networking, ask why your work Internet connection is not dual-stacked already. It’s typically a configuration question, and rarely has any actual cost associate with it. If you have a corporate proxy service, then if it is dual-stacked, the clients (on your internal corporate network) already get some benefit of being able to talk to IPv6 services.

If you have DNS services, check they not only can serve IPv6 records (AAAA), but they are reachable using IPv6. Services like AWS Route53 have done this for years (see my earlier point about getting IPv6 as a first-class citizen within AWS).

While you’re looking at DNS, have a look at creating a simple CAA record, to list the Certificate Authorities you obtain certs from.

CloudFront Functions and Security Headers

November 2021: Note there is a new way to do this natively within CloudFront, and it wont cost you a Lambda@Edge invocation.

For a long time, I’ve been using Lambda@Edge to inject various HTTP security-related headers to help browsers improve the security model of the content that they fetch and render.

I’ve been doing this as I have been using S3 as the origin (accessed via a CloudFront Origin Access Identity). S3 itself cannot add/inject many of the common security headers when it passes

These Functions execute when the origin returns the content to the CloudFront regional edge; the returned content then gets cached with the injected headers included.

The end result is getting a good rating on securityheaders.com, hardenize.com, and other public security evaluation services.

An alternate in the Lanbda@Edge execution lifecycle is to trigger on Viewer Response; in which case the cached version doesn’t have the headers injected, and every viewer request triggers the code execution. Clearly, if every viewer has the same set of headers, there’s no need to execute each view response and pay for the additional Lambda@Edge executions.

Now there’s a new option – CloudFront Functions (AWS blog post). Written entirely in JavaScript, it executes only at Viewer Request, or Viewer Response. There is no Origin Request or Origin Response option. It also executes at the CloudFront Edge, not the Regional Edge.

Thie example injects a number of headers, and would need only minor potential customisation on the Content Security Policy (and possibly Permissions Policy) to work for most sites:

function handler(event) {
    var request = event.request;
    var response = event.response;
    response.headers['strict-transport-security']= { value: 'max-age=31536000' };
    response.headers['x-xss-protection']= { value: '1'};
    response.headers['x-content-type-options']= { value: 'nosniff'};
    response.headers['x-frame-options']= { value: 'DENY'};
    response.headers['referrer-policy']= { value: 'strict-origin-when-cross-origin'};
    response.headers['expect-ct']= { value: 'enforce, max-age=86400'};
    response.headers['permissions-policy']= { value: 'geolocation=(self), midi=(), sync-xhr=(self), microphone=(), camera=(), magnetometer=(), gyroscope=(), fullscreen=(), payment=(), autoplay=(self)'};
    response.headers['content-security-policy'] = { value: "default-src: 'self'; img-src 'self' data: ; style-src 'self' 'unsafe-inline' ; frame-ancestors 'none'; form-action 'none'; base-uri 'self'; "};
    return response;
}

You may want to evaluate the cost of both Lambda@Edge and CloudFront Functions. After the first year, Functions is charged at US$0.10 per million functions. As an equivalent, Lambda@Edge for a similar Node.JS function that executes in one millisecond with 128 MB of memory would be US$0.2021 per million requests.

However, given a busy website, you may want to look at the efficiency differences between Viewer Response execution for CloudFront Functions, and Origin Response and the caching for Lambda@Edge (multiplied by the number of Edge Cache locations (13), and the cache retention rate).

If you have only a few unique URLs, and content that can be cached for a long period, and large volumes of requests, then Lambda@Edge may result in near free execution.

 Lambda@EdgeCloudFront Functions
Unique URLs100100
HTTP viewer Requests10M/month10M/month
Execution time1msN/A
Number of Regional Edges13N/A
Memory/execution128MBN/A
Execution timeOrigin ResponseViewer Response
Number of code invocations1300 (once per Regional Edge, Per Unique URL, and possibly cached for a month – depending on Edge cache expiry)10M
Possible Costs  (as at 28/Aug/2021)Duration: US$0.0000000021 * 1300 = Requests: US$0.2 * 0.0013 Total: US $0.00026273US$0.1 * 10
Total: US$1
CloudFront Functions cost uplift compared to Lambda@Edge 3,806 times more expensive

If we were using Lambda@Edge on ViewerResponse, and not caching the object with headers injected, then CloudFront Functions would be cheaper; or if the content being sent was dynamic from the origin and not suitable to be cached, in which case we wouldn’t get the efficiency savings of fewer executions.

Even if we are using Origin Response with Lambda@Edge, we can’t determine the cache expiry of the Lambda@Edge cached responses (we can influence it); the cached objects could expire and re-execute every day, so the Lambda@Edge costs could go up 30x (which would only make CloudFront functions 126 times more expensive). YMMV. TIMTOWTDI.

Browser support for FTP (another sunset)

As Hanno Böck noted in the recent Bulletproof TLS Newsletter, FTP Support in Firefox 90 has been removed. We’ve seen similar messaging from most major browser vendors over the last few years.

I’m going to make a bold prediction, and say in 10 years time we’ll be seeing the removal of (plain text) HTTP support as well. Regardless of internal or external networks (an out-dated concept aligned to the Crunch Shell of network security), the move to stronger security for all communications, backed by free TLS Certificate Authorities (such as Let’s Encrypt) means we should be doing end-to-end encryption for everything the common web browser fetches.

For some time, Firefox has had an HTTPS-only mode, with warnings when services try and dip back to unencrypted access. I’ve typically found this warning pops up when various link-shortening services are chained together, and I’m grateful for the awareness that a jump in that chain is poorly implemented.

In the meantime, the distribution of files using FTP needs to stop. If you run an FTP service then you need to think about transitioning to something that permits access using HTTPS as the transport protocol.

Another sunset in the circle of life of a protocol.

Using AWS to help secure your email domain: the MTA-STS website

I recently posted about using AWS to provide very cost-effective, Scalable, Secure Static websites. In this post, here’s a valid reason you should do this now, to publish a new website on your domain that has one, simple file on it.

Email on the Internet has used SMTP for transferring email between mail transport agents (MTAs) since 1982, on TCP port 25. The initial implementation offered only unencrypted transport of plain text messages.

It’s worth noting that people, as clients to the system, generally will send their email to their corporate mail server, not directly from their workstation to the recipient; the software on your desktop or phone is a Mail User Agent (MUA), and your MUA (client) transfers your outbound message to your MTA (mail server), which then sends the message using SMTP to your recipients MTA, and then when the user is read they sign in and read their mail with their MUA.

The focus of this article is that middle hop above – MTA to MTA, across the untrusted Internet.

SMTPS added encryption in 1997, wrapping SMTP in a TLS layer, similar to how HTTPS is HTTP in a TLS wrapper, with certificates as many are familiar with, issued by Certificate Authorities. This commonly uses TCP port 465. And while modern MTAs support both encrypted and unencrypted protocols, it’s the order and fail-over that’s important to note.

Modern Mail Servers will generally try and do an encrypted mail transfer to the target MTA, but they will seamlessly fall back to the original unencrypted SMTP if that is not available. This step is invisible to the actual person who sent the message – they’ve wandered off with their MUA, leaving the mail server the job to forward the message.

Sending an email, from left user, to right, via two MTA servers.

Now imagine an unscrupulous network provider somewhere in the path between the two mail servers, who just drop the port 465 traffic; the end result is the email server will assume that the destination does not support encrypted transfer, and will then fall back to plain text SMTP. Tat same attacker then reads your email. Easy!

If only there was a way the recipient could express a preference to not have email fall back to unencrypted SMTP for its inbound messages.

Indeed, there’s a similar situation with web sites; if how to we express that a web site should only be HTTPS and not down graded to HTTP. The answer here is the Hypertext Strict Transport Security header, which tells web browsers not to go back to unencrypted web traffic.

Well, mail systems have a similar concept, called the Mail Transport Application Strict Transport Security, or MTA-STS defined in RFC8461.

MTA-STS has a policy document, which allows the preference for how remote clients should handle connections to the mail server. It’s a simple text file, published to a well-known location on a domain. Remote mail servers may retrieve this file, and cache it for extended periods (such as a year).

In addition, there is a DNS text record (TXT), named _mta-sts.$yourdomain. The value of this for me is “v=STSv1; id=2019042901“, where the ID is effectively used as a timestamp of when the policy document was set. I can update the policy text file on the MTA-STS website, and then update the DNS id, and it should refresh on clients who talk to my mail server.

The well-known location is on a specific hostname in your domain – a new website if you will – that only has this one file being served. The site is mta-sts.$yourdomain, and the path and filename are “.well-known/mta-sts.txt‘. The document must be served from an HTTPS site, with a valid HTTPS certificate.

Here’s mine for my personal domain: https://mta-sts.james.rcpt.to/.well-known/mta-sts.txt, and here is the content at the time of writing:

version: STSv1
mode: enforce
mx: mx0.james.rcpt.to
max_age: 2592000

So an excellent place to host this MTA-STS static website, with a valid TLS certificate, that is extremely cost-effective (and possibly even cost you nothing) is the AWS Serverless approach previously posted.

You can also check for this with Hardenize.com: if you get a grey box next to MTA-STS for your domain, then you don’t have it set up.

Of course, not all MTAs out there may support MTA-STS, but for those that do, they can stop sending plain text email. Even still, don’t send sensitive information via email, like passwords or credit card information.

The MTA trying to send the message may cache the STS policy for a while (seconds as indicated in the file), so as long as TCP 443 is available at some time, and has a valid certificate (from a trusted public certificate authority), then that policy can persist even if the HTTPS MTA-STS site is unavailable later (eg, changed network).

Its worth noting that your actual email server can stay exactly where it is – on site, mass hosted elsewhere; we’re just talking about the MTA STS website and policy document being on a very simple, static web site in Amazon S3 and CloudFront.

The ASD Essential Eight in the AWS Serverless World

The Australian Signals Directorate, part of the Australian Department of Defence, has been issuing guidance to organisations to help secure their digital systems for several years. Known as the Essential Eight, it defines eight activities that help mitigate exposure to compromise or exploit.

Some of the most basic items are around patching the tech stack:

  • operating systems
  • programming runtime environments like Java, .Net, Python and more
  • software solutions that run on those run-times

Of course Multi-Factor Authentication (MFA) is a key one; and slowly our service providers are coming around to offering MFA pas part of their login services – or better yet, federation of identity to other online services hat already do this, such as Facebook, Google, etc.

But how much of this applies to your technology stack in the Serverless world of AWS? Let us begin, following the AWS guide

1. Application Control

ASD recommends organisations “prevent execution of unapproved/malicious programs including .exe, DLL, scripts (e.g. Windows Script Host, PowerShell and HTA) and installers“.

In the world of AWS Lambda, the only code that is present is our bespoke code and any libraries (layers) we’ve possibly added in. What we want to do is ensure that the code we upload is the code executing, and Lambda now allows signed code bundles (Configuring, Best Practices).

If we’re running a Serverless static web site (using S3, CloudFront, etc), then we have no executing code; only content (note you may have some Lambda@Edge or CloudFront functions to inject various Security related HTTP headers, such as HSTS, CSP, and more: see Scott Helme’s excellent securityheaders.com).

However, there are no other applications as… there is no application server per sé.

2. Patch Applications

Well, in AWS Lambda, this is where we have to update out own applications (and those layers/libraries) to ensure they are up to date. If you have abstracted those libraries and imports into Layers, then manage them and update.

Again, in a static web site deployed Serverlessly, we have no application serves to patch (again, except for any Lambda@Edge or CloudFront functions that need maintenance).

3. Configure Microsoft Office Macro Settings

Er, well, no Microsoft Office installed in Serverless, so this is a no-op. Nothing to do here, move along…

4. User Application Hardening

ASD says “Configure web browsers to block Flash (ideally uninstall it), ads and Java on the internet. Disable unneeded features in Microsoft Office (e.g. OLE), web browsers and PDF viewers.“.

We have none of this in our Serverless environments. However we should be delivering updated applications with everything we can do to support the most modern and up to date browsers; the rest of the world is auto-updating these browsers very rapidly.

For corporate environments that lock down browser updates; question why the rest of the world has better security than your corporate users you’re trying to protect!

5. Restrict Admin Privileges

Using AWS IAM, restrict who can deploy, particularly to production environments. Using CI/CD pipelines and approvals, developers should be able to write and update code and then have it deploy immediately to non-production environments, but it should require a second sign off from a separate individual (or team or people) before it gets near production. Indeed, consider that commits to a revision control repository of code being the source of truth, and that repository needs review before changes are staged ready for a CI/CD pipeline to do its delivery job.

6. Patch Operating Systems

No servers, no Operating systems. OK, Lambda will apply minor version updates to run-times to address security requirements, but its also worth updating major versions of run-times as well. Newer runtime versions have a great chance of supporting newer TLS protocols, ciphers, key exchange methods and checksums.

7. Multi-Factor Authentication

There should never be an interactive user with access that doesn’t use MFA. Not only should your access to AWS be MFA based, it should probably be federated via AWS SSO, using MFA back on your identity provider (IdP).

However, your users of your Serverless solutions may also want the option of using federated identity (SAML, etc), and with MFA implemented on their IdP as well (if you have authenticated access). Or perhaps mutual certificate authentication. If you have an open system with no authentication (publicly, anonymously available) then perhaps that’s fine too. Most web sites are, after all, publicly, anonymously available for their home page and other public content; but the ability to change that content is heavily protected.

8. Backups.

You should have Backups. You should know where your data is. If you’re using DynamoDB, then at least turn on Point In Time Recovery, and a backup schedule. Consider dumping those backups to a separate account in escrow: check out the S3 options around versioning, and retention (Life Cycle) of older versions. Consider the concept from the point of an AWS account being compromised; can an attacker than delete the backups across-account to another environment.

For your code base – is it in a revision control repository – separate to the operational runtime environment. What happens if bad code is put into your repository, and pushed through your environments – can you go back. Do you consider the code repository as a Production service, accessible for commits from developers, but managed as a Production service for them.


Summary

In summary, much – but not all – of the ASD Essential Eight evaporates from being the operator/developers responsibility, leaving you more time to concentrate on the effective implementation of the rest of those items that do remain.

This is all excellent advise, and the more that it is clearly demonstrated with easy adoption for organisations, the better we are across all sizes and types of organisations.

Going further, I am keen on is to remove the use of any unencrypted protocols, particularly HTTP. With free, globally trusted TLS certificates available, moving to HTTPS should be straight forward.

However, that’s not the end of the journey, as TLS has versions. Older versions – less than TLS 1.2 as of this time of writing – should not be used – and most browsers and crypto libraries have removed these from their technology stack to prevent them being used.

Your application – even in a Serverless environment – should verify when it establishes an outbound HTTPS connection that the details of that connection meet your minimum TLS requirements – and you should be ready to up your requirements in future. As mentioned above, sometimes that requires a newer runtime, but newer run-times often still support older TLS protocols – even if you don’t want to (or shouldn’t).

I have been recommending to organisations for some time is to start blocking corporate users from using unencrypted HTTP from their workstations. Firefox has a setting to soft-disable unencrypted HTTP as well (a warning is presented to the user). This may seem inconvenient, but its a huge step up in the security for your workers, which is a key vector into your systems.

Furthermore, stop providing convenience redirects on your services from unencrypted HTTP on port 80 to HTTPS on port 443 – for anything other than your organisations home page. Any other redirection via an unencrypted port should be a hard fail, and fixed at the source.