Project & Support versus DevOps and Service teams

The funding model for the majority of the worlds IT projects is fundamentally flawed, and the fall out is, over time, broken systems, lacking security and legacy systems.

It’s pretty easy to see that digital systems are the lifeblood of most organisations today. From banking, stock inventory and tracking, HR systems. And the majority of these critical operations have been deployed as “projects”, and then “migrate to support”. And it’s that “migrate to support” that is the problem.

Support roles are typically over subscribed, and under empowered. It’s a cost saving exercise to minimise the overhead, by taking the more expensive development resources and moving them to a fresh project, while more commodity problem solving labour comes along to triage operational run time issues. However, that support function has no history in the design and architecture, and often either has no access to the development and test environments to continue doing managed change, or is not empowered to do so. The end result is that Support teams use the deployed production features (eg: manually add a user to a standalone system) instead of driving incremental improvements (eg: automatically add a user base don the HR system being updated).

Contrast with a DevOps team, of dynamic size over time. The team that builds & tests & deploys & automates this more complete lifecycle, and stays with the critical line-of-business system, becomes a Service Team. Any changes they need to perform are not applied in production locally, as is often the case with “Support teams”, but in the Development environment. This then should pass automated testing and feedback loops before being promoted to a higher environment. Sounds great, yeah?

Unfortunately, economic realities are the constraint here. Both the customer, and consultancy are trying to minimise cost, not maximise capability. And navigating a procurement and legal team is something that the procurement cycle wants to do as rarely as possible, not on a continuous basis.

Contrast a Service team focus, of variable size over time, containing different capabilities over time. The cost for this team varies over time, based upon the required skill set. The team objective is to make the Best Service they can, and need to drive from metrics: Availability, Latency, Accuracy while meeting strict security requirements.

From the Service team’s perspective, they obviously need remuneration for their time, but also want to take a sense of pride in their work, and a sense of achievement.

A Support Team is not a Service Team, as they don’t have the full Software Lifecycle Management capability and/or Data Lifecycle Management capability. A Service Team should never be one person; that’s one step away from being zero people. A Service Team may look after more than one service, but not so many that they do not have crystal clear focus on any service.

S3 Public Access: Preventable SNAFUs

It’s happened again.

This time it is Facebook who left an Amazon S3 Bucket with publicly (anonymously) accessible data. 540 million breached records.

Previously, Verizon, PicketiNet, GoDaddy, Booz Allen Hamilton, Dow Jones, WWE, Time Warner, Pentagon, Accenture, and more. Large, presumably trusted names.

Let’s start with the truth: objects (files, data) uploaded to S3, with no options set on the bucket or object, are private by default.
Someone has to either set a Bucket Policy to make objects anonymously accessible, or set each object as Public ACL for objects to be shared.

Lets be clear.

These breaches are the result of someone uploading data and setting the acl:public-read, or editing a Bucket’s overriding resource policy to facilittate anonymous public access.

Having S3 accessible via authenticated http(s) is great. Having it available directly via anonymous http(s) is not, but historically that was a valid use case.

This week I have updated a client’s account, that serves a static web site hosted in S3, to have the master “Block Public Access” enabled on their entire AWS account. And I sleep easier. Their service experienced no downtime in the swap, no significant increase in cost, and the CloudFront caching CDN cannot be randomly side-stepped with requests to the S3 bucket.

Serving from S3 is terrible

So when you set an object public it can be fetched from S3 with no authentication. It can also be served over unencrypted HTTP (which is a terrible idea).

When hitting the S3 endpoint, the TLS certificate used matches the S3 endpoint hostname, which is something like s3.ap-southeast-2.amazonaws.com. Now that hostname probably has nothing to do with your business brand name, and something like files.mycompany.com may at least give some indication of affiliation of the data with your brand. But with the S3 endpoint, you have no choice.

Ignoring the unencrypted HTTP; the S3 endpoint TLS configuration for HTTPS is also rather loosely curated, as it is a public, shared endpoint with over a decade of backwards compatibility to deal with. TLS 1.0 is still enabled, which would be a breach of PCI DSS 3.2 (and TLS 1.1 is there too, which IMHO is next to useless).

Its worth noting that there are dual-stack IPv4 and IPv6 endpoints, such as s3.dualstack.ap-southeast-2.amazonaws.com.

So how can we fix this?

CloudFront + Origin Access Identity

CloudFront allows us to select a TLS policy, pre-defined by AWS, but permitting us to restrict available protocols and ciphers. This lets us remove “early crypto” and be TLS 1.2 only.

CloudFront also permits us to use a customer specific name, for SNI enabled clients for no additional cost, or a dedicated IP address (not worth it, IMHO).

Origin Access Identities give CloudFront a rolling API keypair that the service can use to access S3. Your S3 bucket then has a policy permitting this Identity access to the host.

With this access in place, you can then flick the “Block Public Access” setting account-wide, possibly on the bucket first, then the account-wide settings last.

One thing to work out is your use of URLs ending in “/”. Using Lambda@edge, we convert these to a request for “/index.html”. Similaly URL paths that end in “/foo” with no typical suffix get mapped to “/foo/index.html”.

Governance FTW?

So, have you checked if Block Public Access is enabled in your account(s). How about a sweep through right now?

If you’re not sure about this, contact me.

AWS Re:Invent: rest of the releases

Well, that was busy week. It was almost impossible to keep up with the announcements; an overwhelming feeling of something akin to playing Tetris as announcements poured down faster than I could read, understand and appreciate them.

So, having got past day 1, here’s the rest of what I think of what happened next:

DynamoDB Transactions and on-demand

ACID Compliance (atomicity, consistency, isolation, and durability) was always one of the constraints that those new to NoSQL were always trying to understand. For some workloads it was OK to move this validation to the user space (app server) for others, not so much.

On-demand DynamoDB removes the need to set sharding requirements, and let DynamoDB scale (and charge) as required from usage patterns.

CloudWatch Logs Insights

When I first saw this console, it just yelled “This is Sumo Logic” at me.

Outposts

For many years, infrastructure being delivered to a Region was a pre-configured rack with all equipment ready to run. This release effectively shifts the delivery address from “AWS Region” to a customers data centre. It means there is a new channel for delivery of the equipment, and thus produces more scale, and ultimately, drives down cost further.

But who still wants to run data centres? The compliance, maintenance, physical security are all very compelling. Plus, an on-premise deployment has maintenance, and capacity limits that are way lower than the Region.

S3 Glacier Deep Archive

From Glacier with infrequent retrieval, to a deeper retention – Deep Archive requires data retention for half a year, and is a 12 hour restore. But the benefit is a huge price savings: US$1/TB/month (yes, Terabyte). That’s US$0.0009765/GB/month – so about time we changed units of measurement to the TB. Compare to Azure blob storage at US$0.002/GB/month (US$2.048/TB/month) , that’s less than half the cost .

When combined with some sensible data work-flows for backups, you’ll save a ton of money. But the biggest win will be when 3rd party backup solutions can instrument this themselves automatically. For example, the last 7 days of backup may sit on S3 Standard durability, and then get migrated to Glacier for 3 months, and then Deep Glacier after that.

Using the above tiering, lets do a 2 TB full backup once per week, and a 100 GB daily incremental. We’ll take 2.6 TB in S3 Std, then 11 weeks of S3 Glacier at 26 TB, and then 9 months of S3 Glacier Deep Archive for 93.6 TB. Sum total monthly cost is = 2.6 * 1024 * 0.025 + 26 * 1024 * 0.005 + 93.6 * 1 = 66.56 + 133.12 + 93.6 = US$293.28 / month = US$3,519.36/year assuming a one year retention.

If we had kept this all onS3 standard durability, then we would have been looking at US$37,539.84/year.

So, who’s going to make the first move? CommVault? Synology? StoreSimple? Storage Gateway VTL?

Managed Blockchain

Previously AWS had said it didn’t want to run a managed Blockchain service, saying no company should sit at the centre of this, but customer demand wins over this: and now two services filling the space: Blockchain as a Service, and the Quantum ledger database service.

Both of these are interesting to me, and I’ll be speaking with customers to see if they want us to integrate this into their solutions. Neither will replace using a relational database for temporal processing, state, etc. But for point in time authoritative signed data, they look interesting.

Textract

This one requires some testing. I’ve previously looked at Mechanical Turk for doing human-intelligence level OCR, but as a service this may be better. Any process that does text extraction should have a multi pronged approach to ensure accuracy; so perhaps a pass of Tectract, followed by a pass of Mech Turk (or other Humans), and then if there is a conflict/mismatch, flag for management inspection….

Security Hub

This is huge for me, and one I am actively getting my head around before recommending into customer environments. Its also enthused me to get back to AWS Config, which I’d previously discounted on cost.

Security Hub united several AWS security services. Each of these have had their own interface, cross-account capabilities, etc. Of course, for me, and my Public Sector customers, the lack of Macie in Australia is still a consideration here.

AWS Organisational CloudTrails

I’ve been a fan of CloudTrail since I first heard of it. The fact that it could always deliver API logs across-account – to a dedicated security account. without any fear or possibility of it being filtered or edited by the source account was a key enabler in enterprise workloads.

Its developed well since its initial launch, with multi-region support, digests files to detect tampering and more. But with all these options came the possibility of inconsistent deployments across a large fleet of accounts.

And while my perception has always been consistency, its only after circling back that you realise that not everything is consistent, with new AWS accounts being added at different times.

It is only with me starting to play with Config and Security Hub (see above) that these inconsistencies have come to light; and the new solution to this is just in time: Organisation Trails, that apply from the Billing/Organisation account, down to all dependent accounts.

An Organisation trial in a dependent account cannot be deleted or modified. They can log cross-account almost the same previous implementation – with the exception of  a few new Permissions required on the destination S3 Bucket policy.

Lambda Ruby, BYO Runtime, and Firecracker

Firecracker is a strong story, but in the end, having a manage environment for it is worth it if I can do so (ie, if latency, sovereignty, etc can be met). What will be interesting is the opportunity for more eyes to review it’s source code.

FSx (Luster & Windows Fileshare)

Managed file shares sound great, but now there’s confusion between EFS and FSx (and to some degree, Storage Gateway as an NFS and CIFS file share).

And much more

I wont go into detail on the large list of other services; my interest is the vast majority of web, security and DevOps-enabling services that continue to incrementally improve. But what happens next is interesting.

Config revisted

When first launched, I got bill shock form turning Config on with just a few rules. But now its much richer, and easier to understand. As it is one of the security tools feeds across into Security Hub, its forced me to circle back to Config and start re-evaluating some of its rules. Its come a long way, and much of the tooling I have written myself in the past to do cross-account checks, which Config also does, can now feed via Security Hub back to a central (organisatoin-wide) interface for alerting and actioning.

Summary,

With some 50,000 people at re:Invent this year, the pace of innovation continues to put AWS far ahead of its competitors.

AWS CloudFront launches in Perth

I moved back to Perth in 2010, having grown up here, gone to school, University and started my career here. It’s a lovely city, with the metropolitan area sprawling north and south along the blue Indian Ocean for some 50+kms. They says it’s a bit of a Mediterranean climate, normally never going below 0°C, and the heat of summer hitting mid 40°C, but with a fresh westerly coastal breeze appearing most afternoons to cool the place down.

But it is rather remote from other major population centers. The next nearest capital city, Adelaide, is 2,600 kms (1,600 miles) by road. Melbourne is 3,400 kms (2,100 miles) on the road, and Sydney is 3,900 kms (2,400 miles).  It’s a large state, some 2.5 million square kilometers of land, the size of the US Alaska and Texas states combined.

So one thing those in technology are well aware of is latency. Even with fibre to the premises (NBN in Australia), the Round Trip Time to Sydney is around 55ms – which is a similar time to Singapore. Melbourne comes in around 45ms.Latency from Perth to Singapore, Sydney, Melbourne, and New Zealand to Sydney

In 2013 I met with the AWS CloudFront team in Seattle, and was indicating the distances and population size (circa 2 million) in Perth. There’s a lot of metrics that goes in to selecting roll-out locations (Points of Presence) for caching services, with latency, population size, economic prosperity, cost of doing business, customer demand from a direct customer model, and customer demand from an end-consumer model being weighed up.

This week (1st week of January 2018) AWS CloudFront launched in Perth.

This impact on this is that all web sites that people of Perth that use CloudFront will now appear to be faster for cachable content. The latency has dropped from the 45ms (to Melbourne) to around 3ms to 5ms (from a residential NBN FTTP @ 50 Mbit/sec).

Test at 9:30pm from Perth (iiNet NBN).

In addition, the ability to upload/send data to applications (Transfer Acceleration) on-Region via the Edge (Edge Upload) may now also make a difference; with 45 ms to Melbourne, its been a largely unused feature as the acceleration hadn’t made much of a difference. There is a Transfer Acceleration test tool that shows what effect this will give you; and right now, while it shows an advantage to Singapore, just a 7% increase in performance to the AWS Sydney Region. Its not clear if TA via the Perth PoP is enabled at this point, so prehaps this will change the result over time.

And so, after several years, and with other improvements like the ability to restrict HTTPS traffic to TLS 1.2, it now makes sense to me to use CloudFront for my personal blog. In an hour, I had applied a new (additional) hostname against my origin server (a Linux box running WordPress) by editing the Apache config, symlinking the wordpress config file, and adding a Route53 CNAME for the host. I had certbot on Linux then add the new name to the Let’s Encrypt certificate on the origin. Next I applied for an Amazon Certificate Manager SSL certificate, with the hostname blog, and (if you inspect it) blog-cloudfront.james.rcpt.to. I then created a Cloudfront Distribution, with one origin, but two behaviours – one for the WordPress admin path, and one for the default paths, so that I could apply additional rules to protect the administration interface.

With this in place I could then update the DNS CNAME to move traffic to CloudFront, without any downtime. Not that downtime matters on my personal blog, but doing exercises like this you need to practice.

Welcome to Perth, CloudFront.

PS: It’s worth noting that IPv4 DNS resolution for my CloudFront distribiution is giving me 4ms RTT from Perth, but IPv6 RTT is 52ms, which indicates that IPv6 CloudFront has not yet arrived here.

Staring at tomorrow: Technology Impacts on society

This started as musings on where broadband connections enter buildings. It went further.


There’s a few technical specifications that we’ve become very used to in our homes and offices for the last few decades, but they’ve not always been there. Their introduction to our lives has spurred changes in the construction of our dwellings, and our expectations of what we do in our lives. And another one is unfolding now.

Plumbing (Scheme Water, Sewerage), Electricity, and Broadcast TV have all had their impacts. Water in & out has remained largely unchanged in the last 50 years, having moved from outhouses to servicing dedicated ‘wet’ rooms in our buildings (kitchens, bathrooms, WCs). Electricity has crept from being provided for lighting purposes only, to additionally providing a single power socket per home, to multiple sockets per room at convenient places for us to leave devices semi-permanent plugged in. We’ve become accustomed to the 110v/60Hz and 230v/50Hz, and the sockets and plugs are generally down to several well known arrangements of pins and locking mechanisms.

With the dawn of broadcast TV (UK: 1936, US: 1948, Australia: 1965), and excluding set-top “rabbit ears”, we’ve had antennas on roofs with coaxial cables leading to specific rooms: family rooms, bedrooms, etc. Using the superior reception of the roof-top antenna has meant that the placement of TVs was pretty much determined by the location of Co-Axial sockets in walls. Indeed, not just which rooms, but which corners and walls of specific rooms, close to these antenna sockets. While we’ve happily added multiple power sockets to rooms for future proofing our desire to rearrange our lives, co-axial cables suffered more loss with more joins to other sockets, so it was generally avoided.

Thus our layout of our homes has been throttled by the tether to the roof top antenna.

We are well into the next revolution delivery of our video feeds over a new carrier: IP and the Internet. The age of radio frequency broadcast is declining, and with it some of our cultural norms.

One impact is building design: a time will come when we’ll no longer request TV broadcast antennas on buildings, and coaxial sockets on walls. This is replaced with either short range high speed WiFi such as 802.11ac, or wired Ethernet ports at gigabit speed (no less). Wireless has continued to increase in speed, but as with all broadcast spectrum, it is subject to interference from other wireless signals, which can impact throughput. A continual battle between ever faster wireless speeds, improved signal processing and interference handling, and cost considerations will play out to determine if our TVs will have a wired or wireless IP connection within the home.

Wireless has been through several generations, some of which are now well and truly dead. WEP encryption, once the critical protection for wireless networks, is known to be compromisable, and thus is abandoned today. Hardware devices that only implemented WEP are effectively junk. Devices with physical Ethernet socket have not suffered so – the trusty RJ45 100 base TX CSMACD works as well today as it did 20 years ago.

We’ll see people opt to use wireless for video delivery within homes for short term savings, the aggressive rate of change on encryption and signalling standards for wireless networks will find many wireless-only devices become redundant before their time. We expect to get 10 – 20 years from a TV (its often an expensive item), but should WPA2 get compromised, then those hosts are vulnerable. We’ve already seen this with the KRACK vulnerability, and those vendors that did not patch their firmwares are probably vulnerable (both the Access Point and clients need patches to address KRACK for WPA2).

Of course, the interactions that your TV makes over whatever connection should indeed be over an encrypted transport, but even the encryption that uses – the key lengths, signature algorithms, all will be refreshed and strengthened over time.  In the last 5 years we’ve seen web site certificates change in key length (1024 -> 2048 bit), signatures for asserting a chain-of-trust from a certificate authority move from what was MD5 to SHA1 to SHA256, the bulk encryption algorithms move from RC4 to AES128 to AES256 to Elliptical Curve algorithms, and message digest checksums from MD5 to SHA1, to SHA256, and not SHA384. Each of these changes needs to be applied to the software in your appliances for them to maintain the security you expect to be there.

Many manufacturers won’t bother updating this on their already-shipped units; the disposable consumer sales cycle continues unabated and effectively is helped by these changes, and customers barely make any mention of the trouble of having to replace these items.

But as we move to IP delivery of video content, we also move to more of a video-on-demand world, and away from a time-of-broadcast (playout) world where all consumers – the audience in a broadcast area – would all get the same content at the same time.

30 years ago, when I was at school, the talk in the playground was of the latest episode of some show: MacGuyver, Airwolf, The A-Team, comedies like ‘Allo ‘Allo, The D-Generation, The Comedy Company, etc. With only a few select channels to choose from (3 where I was), your peers likely saw the same content at the same time, making for a shared experience that you wove into the daily fabric of society.

Shows with catch phrases became common place, and everyone knew them. Mark Mitchell gave Australia the “Couple o’ days, Beautiful”, and everyone knew the setting and connotation.

As we move to a VOD, subscription based scenarios, this disappears. Shows on one subscription service are not on others, or appear at different times. Subscription services cross local boundaries in a way that broadcast spectrum could never do with the limitation of transmitter power. People watch content at different times, in different months, from different countries (despite the continued attempt for geo-blocking). The social background noise of modern society is changed, disjointed.

I suspect that in the next decade, broadcast TV will disintegrate. Advertising spend will continue to shift to product placement in original content, and customised pre- and mid-roll ad insertion, customised to the view, tracked and targeted.

As broadcast TV declines, it heavily modifies the stalwart of most TV stations: TV news. While news has been available form multiple media sources, local broadcast TV news was always curated to somewhat balance local, national and global current affairs. If all journalism production costs the same, then stories filed that can be replayed in multiple territories have more value per second, and the decline of local stories.

I’ve watched local news for decades, paying attention to their mix of slow news days stories (car through a fence, cat up a tree) and more interesting ones (state general or by election results, perhaps investigative journalism – but that’s on the decline). Its even interesting to sometimes see the format of content being padded out with in-content extended adverts for news clips that are scheduled later, the tweaks to the lower thirds straps, even the background animations that engage the reader in subtle ways. These things often annoy me at how much they con the audience’s emotional engagement with a story. One local broadcast TV channel has a habit of applying a cinematic dust/sparkle video filter to most human-interest stories (cat up tree). They’ve played games with putting weather forecasts showing MIN (overnight) and MAX (daytime) temperatures on top of each other, and then sometimes MAX (daytime) on top of MIN.

(While I’m ranting, the “cross to the local hospital to a reporting standing out front” seems a waste of time, as is a “cross to the next room” scenario which an anchor could have just read out and moved on from.)

One media that has withstood much change until now is broadcast radio. I suspect that because radio has had one place where its continued its existence without much challenge: vehicles. Vehicle manufacturers have been pretty slow at putting DAB radios into vehicles by default. AM and FM radios are still universal. I wrote in 1995 in the UWA student magazine Pelican about the dawn of MP3 as an audio format, and the start of Digital Audio Broadcasting (DAB). The MP3 player was born – Apple seized on it with the iPod, and the Sony Walkman receded to the annals of history. And while DAB has been around for some time, the licensing and hardware has been at an expense that did not generally warrant the improvement in delivery, and clearly not supported by vehicle manufacturers.

But while broadcast radio has been compatible with what the listener is doing in vehicles: a threat is coming to the radio’s last bastion. It’s the same threat that is coming to organisations that live off driving license fees: self-driving cars. Drivers can then do other tasks: they can then look at screens. The radio will stop becoming an in-vehicle companion to millions of single-occupant cars, as those occupants will start viewing content instead of just listening to it. Eyes will come off the road.

(And if eyes are off the road, do we have the demise of billboards and placards as an advertising medium – as no one is looking out the window any more?)

By extension to this, the music industry relies on radio for socialising its content, encouraging people to either purchase content, or become customers to performers when they tour. With broadcast music no longer curated to seed the introduction of new content over time, artists will find it difficult to get established.

Coupled with the self-driving car is the move to electric vehicles, and the eventual drop in the petroleum fuel use, and the tax (excise) that is collected from this. Governments often use this as a source of funding road building. This model will have to change, probably to a tax-per-kilometer travelled on the owner of these vehicles.

And with self-driving cars, we can finally have some backward parts of the world switch to the metric system for units of distance and speed, without the risk of the human population getting it wrong and going too fast/slow/far.

Eventually, as my friend Paul Fenwick (PJF) has spoken of, the population will move to not owning vehicles, but calling them completely on demand – a la Uber/taxi, but without the human driver. These vehicles will get corporate ownership, and will all have live mobile broadband data links to each vehicle. They’ll all have built-in dash cams, and logging of all activities in and out of the vehicles at all times. New advertising opportunities will rise up – screens in these vehicles will know the route you’re about to take to advertise products and services on or near your route, so you can chose to stop off. They’ll know your preferences, the environmental factors (warm, cold, sunny, rainy) and advertisers will bid to produce targeted advertising at you while you travel.

Next knock on effect from this is the car insurance industry. If the self-driving vehicle has less accidents, and the risk of death or major damage is less, then disruption will arrive here too. With only a few major self-driving-taxi companies that require insurance, the number of insurance companies will consolidate.

The self driving vehicle probably needs less street signs. It may require less street lighting. It may require less lane markings, cats eyes, number of lanes.

This probably sounds like a diatribe version of a Swardley map (hat tip to Simon). All these things are connected, generally by revenue streams or shared interests, and always by data and technology. As always, its all change, and resistance is futile.