The Simple Storage Service (S3, or S3) has made long term durable storage simple for the masses. The democratisation of object storage with well documented, stable APIs has been incorporated into many products. The API is part of the product.
But despite the word Simple, there are more and more advanced features: storage tiers, security policies, life-cycle policies, logging, versioning, requestor-pays, and more recently, Inventory generation and more.
S3 features prominently in long-term retention of important data due to its high durability. But today I’m diving into the another benefit: MFA Delete.
Simple CRUD
Create, Read, Update, Delete: the basics of a REST interface for sending and manipulated a data store. In AWS, IAM policy (or Bucket Policy) can permit or limit the actions that a user can perform. If you delete an Object, then it’s gone. If you overwrite an object (using the same Prefix or name), then the original is lost, as you would expect.
We can limit a calls to s3:DeleteObject, either with a explicit DENY, or carefully only permitting the fine grained controls we intend (s3:PutObject, s3:GetObject) for the role., groups or users we confer privileges to. However, we still run the change of an unintended overwrite.
Furthermore, there may be privileged users or roles that have elevated access, so while your general work-flow is protected by policy from accidental deletion, you’re not protected from accidents from other source (eg, humans with admin privs).
S3 Versioning
To help with this, S3 Versioning permits you to retain multiple revisions of the same object. When listing the bucket naturally, you see the current revision in the list. But a few API calls and you can drill into the previous revisions of the same object, helping you recover from object overwrites.
When a file is deleted from a Versioned S3 Bucket, its really just updated with a new version as a designated Delete Marker. This Marker prevents the object being included in a natural bucket listing. Without further action, the previous versions are still present, and you’re still paying for their storage.
Lifecycle Policies
I always recommend agreeing a life cycle retention policy for S3 buckets – possibly by agreed prefix – upon creation of the Bucket. It makes the creator of the data set really consider how permanent their data must be.
Lifecycle policies can change data storage tiers, but my favourite is the expiry of “previous revisions” after a customer-defined number of days. This gives me a kind of “S3 Undelete” window, and its saved my bacon on several occasions; the accidental Admin delete can easily be undone within the number of days you have specified.
But I want to go further, I want to have some buckets that I know are my “keep forever” bucket, and I want to make any kind of delete of even previous revisions difficult: enter MFA Delete.
Enabling MFA Delete
MFA delete works on Versioned S3 Buckets, and protects all revisions (including delete markers) from being deleted with a corresponding special delete command that includes a valid MFA token from an authorised user.
In my experimentation, I had an existing bucket that I had Versioning enabled. To enable this feature I had to turn to the API – this isn’t available in the AWS Console at this time. I also had to us an IAM User with MFA or the master Root identity – federated users or Ec2 Instances in IAM Roles cannot do this, as they have no MFA associated with them directly.
In this example, I created a profile for the AWS CLI called MasterUser, and had root IAM keys created (which I immediately rescinded). I had a bucket called MyVersionBucket, that I had set up just as I liked it. I also grabbed the ARN of my Virtual MFA I had for the Root user in this account (the ARN is listed as a SERIAL number in the console).
To enable MFA Delete:
aws s3api put-bucket-versioning –profile MasterUser –bucket MyVersionBucket –versioning-configuration MFADelete=Enabled,Status=Enabled –mfa ‘arn:…. 012345‘
Note: the MFA is referenced with quotes around it, as the single argument contains a space between the serial (ARN in this case) and the current value on the MFA).
To then see the configuration:
aws s3api get-bucket-versioning –profile MasterUser –bucket MyVersionBucket
With this in place it was time to test it out.
(Not really) Deleting from an MFA-Delete protected Bucket
The first thing I did was upload a file (same as normal), and then delete it. Using the “current view” of the bucket, the file vanished. In the new AWS console I could see the deleted item listed, and drilling into it, I could see the revisions there as with a regular Versioning bucket.
The next thing I tried was to “undelete” an object, an option that has just appeared in the revised S3 console, however this silently failed.
I then looked at the revisions of my sample file, and could see the delete marker sitting there. I attempted to delete the Delete Marker, but without an MFA I was blocked. This seemed to make sense: previously “undeleting” an object from S3 meant removing the delete marker, and clearly that’s just a version that I cannot really delete.
I looked at the other revisions of my sample file, and I was likewise blocked from deleting them.
Next looked at adding a Lifecycle policy to the bucket, and discovered that no Lifecycle policies can be added to an MFA protected bucket. So three’s no opportunity to move to the Infrequent Access tier of storage after a period automatically.
To truly empty the bucket, I deleted the a version of the file:
aws s3api delete-object –bucket MyVersionBucket –key sample.png –version-id Foo1234 –mfa ‘arn:… 123456‘
The VersionID was displayed to me in the list versions’ output.
Of course, I could potentially have suspended MFA delete, tidied up, and then re-enabled it.
At the end of my experiment, with MFA Delete Enabled, I could dimply delete the empty bucket as normal – there were no further challenges.
When to use MFA Delete
As MFA-Delete is a bucket-wide policy, you need to ensure that all objects that will be in this bucket are right to be considered permanent. You’ll want to limit who has access to put-version policy (perhaps your PowerUsers should have an explicit deny on this API call). If you have temporary or staging data in the bucket, or data that you want a lifecycle policy to automaticlaly clean up, then MFA Delete is not for you.