Well, that was busy week. It was almost impossible to keep up with the announcements; an overwhelming feeling of something akin to playing Tetris as announcements poured down faster than I could read, understand and appreciate them.
So, having got past day 1, here’s the rest of what I think of what happened next:
DynamoDB Transactions and on-demand
ACID Compliance (atomicity, consistency, isolation, and durability) was always one of the constraints that those new to NoSQL were always trying to understand. For some workloads it was OK to move this validation to the user space (app server) for others, not so much.
On-demand DynamoDB removes the need to set sharding requirements, and let DynamoDB scale (and charge) as required from usage patterns.
CloudWatch Logs Insights
When I first saw this console, it just yelled “This is Sumo Logic” at me.
Outposts
For many years, infrastructure being delivered to a Region was a pre-configured rack with all equipment ready to run. This release effectively shifts the delivery address from “AWS Region” to a customers data centre. It means there is a new channel for delivery of the equipment, and thus produces more scale, and ultimately, drives down cost further.
But who still wants to run data centres? The compliance, maintenance, physical security are all very compelling. Plus, an on-premise deployment has maintenance, and capacity limits that are way lower than the Region.
S3 Glacier Deep Archive
From Glacier with infrequent retrieval, to a deeper retention – Deep Archive requires data retention for half a year, and is a 12 hour restore. But the benefit is a huge price savings: US$1/TB/month (yes, Terabyte). That’s US$0.0009765/GB/month – so about time we changed units of measurement to the TB. Compare to Azure blob storage at US$0.002/GB/month (US$2.048/TB/month) , that’s less than half the cost .
When combined with some sensible data work-flows for backups, you’ll save a ton of money. But the biggest win will be when 3rd party backup solutions can instrument this themselves automatically. For example, the last 7 days of backup may sit on S3 Standard durability, and then get migrated to Glacier for 3 months, and then Deep Glacier after that.
Using the above tiering, lets do a 2 TB full backup once per week, and a 100 GB daily incremental. We’ll take 2.6 TB in S3 Std, then 11 weeks of S3 Glacier at 26 TB, and then 9 months of S3 Glacier Deep Archive for 93.6 TB. Sum total monthly cost is = 2.6 * 1024 * 0.025 + 26 * 1024 * 0.005 + 93.6 * 1 = 66.56 + 133.12 + 93.6 = US$293.28 / month = US$3,519.36/year assuming a one year retention.
If we had kept this all onS3 standard durability, then we would have been looking at US$37,539.84/year.
So, who’s going to make the first move? CommVault? Synology? StoreSimple? Storage Gateway VTL?
Managed Blockchain
Previously AWS had said it didn’t want to run a managed Blockchain service, saying no company should sit at the centre of this, but customer demand wins over this: and now two services filling the space: Blockchain as a Service, and the Quantum ledger database service.
Both of these are interesting to me, and I’ll be speaking with customers to see if they want us to integrate this into their solutions. Neither will replace using a relational database for temporal processing, state, etc. But for point in time authoritative signed data, they look interesting.
Textract
This one requires some testing. I’ve previously looked at Mechanical Turk for doing human-intelligence level OCR, but as a service this may be better. Any process that does text extraction should have a multi pronged approach to ensure accuracy; so perhaps a pass of Tectract, followed by a pass of Mech Turk (or other Humans), and then if there is a conflict/mismatch, flag for management inspection….
Security Hub
This is huge for me, and one I am actively getting my head around before recommending into customer environments. Its also enthused me to get back to AWS Config, which I’d previously discounted on cost.
Security Hub united several AWS security services. Each of these have had their own interface, cross-account capabilities, etc. Of course, for me, and my Public Sector customers, the lack of Macie in Australia is still a consideration here.
AWS Organisational CloudTrails
I’ve been a fan of CloudTrail since I first heard of it. The fact that it could always deliver API logs across-account – to a dedicated security account. without any fear or possibility of it being filtered or edited by the source account was a key enabler in enterprise workloads.
Its developed well since its initial launch, with multi-region support, digests files to detect tampering and more. But with all these options came the possibility of inconsistent deployments across a large fleet of accounts.
And while my perception has always been consistency, its only after circling back that you realise that not everything is consistent, with new AWS accounts being added at different times.
It is only with me starting to play with Config and Security Hub (see above) that these inconsistencies have come to light; and the new solution to this is just in time: Organisation Trails, that apply from the Billing/Organisation account, down to all dependent accounts.
An Organisation trial in a dependent account cannot be deleted or modified. They can log cross-account almost the same previous implementation – with the exception of a few new Permissions required on the destination S3 Bucket policy.
Lambda Ruby, BYO Runtime, and Firecracker
Firecracker is a strong story, but in the end, having a manage environment for it is worth it if I can do so (ie, if latency, sovereignty, etc can be met). What will be interesting is the opportunity for more eyes to review it’s source code.
FSx (Luster & Windows Fileshare)
Managed file shares sound great, but now there’s confusion between EFS and FSx (and to some degree, Storage Gateway as an NFS and CIFS file share).
And much more
I wont go into detail on the large list of other services; my interest is the vast majority of web, security and DevOps-enabling services that continue to incrementally improve. But what happens next is interesting.
Config revisted
When first launched, I got bill shock form turning Config on with just a few rules. But now its much richer, and easier to understand. As it is one of the security tools feeds across into Security Hub, its forced me to circle back to Config and start re-evaluating some of its rules. Its come a long way, and much of the tooling I have written myself in the past to do cross-account checks, which Config also does, can now feed via Security Hub back to a central (organisatoin-wide) interface for alerting and actioning.
Summary,
With some 50,000 people at re:Invent this year, the pace of innovation continues to put AWS far ahead of its competitors.