AWS CLI: searching for AMIs

I’ve been experimenting with the new Python based AWS CLI tool, and its getting to be very good indeed. It can support multiple login profiles, so its easy to switch between the various separate AWS accounts you may use.

Today I’ve been using it from Windows (!), and was searching for a specific AMI ID, and wanted to share the syntax for those who want to do similar:

C:\>aws --profile my-profile --region us-east-1 ec2 describe-images --filters name=image-id,values=ami-51ff9238
C:\>aws --profile my-profile --region us-east-1 ec2 describe-image-attribute --attribute launchPermission --image-id ami-51ff9238
C:\>aws --profile my-profile --region us-east-1 ec2 modify-image-attribute ami-51ff9238 --image-id --operation-type remove --attribute launchPermission --user-groups all
C:\>aws --profile my-profile --region us-east-1 ec2 modify-image-attribute ami-51ff9238 --image-id --operation-type add --attribute launchPermission --user-groups all




aws –profile turnkey –region ap-southeast-2 ec2 modify-image-attribute ami-fd46d6c7 –image-id –operation-type remove –attribute launchPermission –user-groups all

aws –profile turnkey –region ap-southeast-2 ec2 modify-image-attribute ami-fd46d6c7 –image-id –operation-type add –attribute launchPermission –user-groups all

AWS S3 Bucket Policies: Restricting to In-Region access

From time to time, Amazon Web Services adds new IP address ranges (it keeps growing!). These new addresses are published in the forums, such as via this post from EricS. I was creating a bucket policy to restrict access only to nonymous users who are within my region – I’m happy for the access requests, but I don’t want to pay the bandwidth charges. So here’s a small Perl script that takes the copy-and-paste text from EricS’s forum post, and creates an S3 buck policy element suitable for this:

open F, 'ips.txt' or die "Cannot read list of IPs: $!";
my @ip_conditions;
while (<F>) {
  push @ip_conditions, $1 if /^(\d+\.\d+\.\d+\.\d+\/\d+)\s/;
print "\t\"aws:SourceIp\": [" . join(",", @ip_conditions) . "]\n";

Official Debian Images on Amazon Web Services EC2

Official Debian AMIs are now on Amazon web Services

Please Note: this article is written from my personal perspective as a Debian Developer, and is not the opinion or expression of my employer.

Amazon Web Service‘s EC2 offers customers a number of Operating Systems to run. There are many Linux Distributions available, however for all this time, there has never been an ‘Official’ Debian Image – or Amazon Machine Image (AMI), created by Debian.

For some Debian users this has not been an issue as there are several solutions of creating your own personal AMI. However for the AWS Users who wanted to run a recognised image, it has been a little confusing at times; several Debian AIMs have been made available by other customers, but the source of those images has not been ‘Debian’.

In October 2012 the AWS Marketplace engaged in discussions with the Debian Project Leader, Stefano Zacchiroli. A group of Debian Developers and the wider community formed to generated a set of AMIs using Anders Ingemann’s ec2debian-build-ami script. These AMIs are published in the AWS Marketplace, and you can find the listing here:

No fees are collected for Debian for the use of these images via the AWS Marketplace; they are listed here for your convenience. This is the same AMI that you may generate yourself, but this one has been put together by Debian Developers.

If you plan to use this AMI, I suggest you read, and more explicity, SSH as the user ‘admin and then ‘sudo -i‘ to root.

Additional details

Anders Ingemann and others maintain a GitHub project called ec2debian-build-ami which generates a Debian AMI. This script supports several desired features, an was also updated to add in some new requirements. This means the generated image supports:

  • non-root SSH (use the user ‘admin)
  • secure deletion of files in the generation of the image
  • using the Eucalyptus toolchain for generation of th eimage
  • ensuring that this script and all its dependencies are DFSG compliant
  • using the redirector service in APT’s sources.list to select a reasonably ‘close’ mirror site
  • and the generated image contains only packages from ‘main’
  • plus minimal additional scripts (nuder the Apache 2.0 license as in ec2debian-build-ami) to support:
    • fetching the SSH Public Key for the ‘admin’ user (sudo -i to gain root)
    • executing UserData shell scripts (example here)

Debian Stable (Squeeze; 6.0.6 at this point in time) does not contain the cloud-init package, and neither does Debian Testing (Wheezy).

A fresh AWS account (ID 379101102735) was used for the initial generation of this image. Any Debian Developer who would like access is welcome to contact me. Minimal charges for the resource utilisation of this account (storage, some EC2 instances for testing) are being absorbed by Amazon for this. Co-ordination of this effort is held on the debian-cloud mailing list.

The current Debian stable is 6.0.6 ‘Squeeze‘, and we’re in deep freeze for the ‘Wheezy‘ release. Squeeze has a Xen kernel that works on the Parallel Virtual Machine (PVM) EC2 instance, and hence this is what we support on EC2. (HVM images are a next phase, being headed up by Yasuhiro Akarki <ar@d.o>).

Marketplace Listing

The process of listing in the AWS Marketplace was conducted as follows:

  • A 32 bit and 64 bit image was generated in US-East 1, which was AMI IDs:
    • ami-1977f070: 379101102735/debian-squeeze-i386-20121119
    • ami-8568efec: 379101102735/debian-squeeze-amd64-20121119
  • The image was shared ‘public’ with all other AWS users (as was the underlying EBS snapshot, for completeness)
  • The AWS Marketplace team duplicated these two AMIs into their AWS account
  • The AWS Marketplace team further duplicated these into other AWS Marketplace-supported Regions

This image went out on the 19th of November 2012. Additional documentation was put into the Wiki at:

A CloudFormation template may help you launch a Debian instance by containing a mapping to the relevent AMI in the region you’re using: see the wiki link above.

What’s Next

The goal is to continue stable releases as they come out. Further work is happening to support generation of Wheezy images, and HVM (which may all collapse into one effort with a Linux 3.x kernel in Wheezy). If you’re a Debian Developer and would like a login to the AWS account we’ve been using, then please drop me a line.

Further work to improve this process has come from Marcin Kulisz, who is starting to package ec2debian-build-ami into a Debian: this will complete the circle of the entire stack being in main (one day)!

Thanks goes to Stefano, Anders, Charles, and everyone who  contributed to this effort.


Load Balancing on Amazon Web Services

I’ve been using Amazon’s Elastic Load Balancing (ELB) service for about a year now, and thought I should pen some of the things I’ve had to do to make it work nicely.

Firstly, when using HTTP with Apache, you probably want to add a new log format that, instead of using the Source IP address of the connection int he first field, you use the extra header that ELB adds, X-Forwarded-For. It’s very simple, something like:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" fwd_combined

… and then wherever you’ve been using a Log statement with format “common”, just use “fwd_common”. Next, if you’re trying to use your domain name as your web server, eg “” instead (or as well) as “”, then with Amazon Route53 (DNS hosting) you’ll get a message about a conflict witht he “apex” of the domain. You get around this using the elb-associate-route53-hosted-zone command line tool, with something like:

./elb-associate-route53-hosted-zone ELB-Web --region ap-southeast-1 --hosted-zone-id Z3S76ABCFYXRX6 --rr-name --weight 100

And if you want to also use IPv6:

./elb-associate-route53-hosted-zone ELB-Web --region ap-southeast-1 --hosted-zone-id Z3S76ABCFYXRX6 --rr-name --weight 100 --rr-type AAAA

If you’re using HTTPS, then you may have an issue if you chose to pass your SSL traffic through the ELB (just as a generic TCP stream). Since the content is encrypted, the ELB cannot modify the request header to add the X-Forwarded-For. Your only option is to “terminate” the incoming HTTPS connection on the ELB, and then having it establish a new connection to the back end instance (web server). You will need to load your certificate and key into the ELB for it to correctly represent itself as the target server. This will be an overhead on the load balancer having to decrypt (and option re-encrypt to the back end), so be aware of the costs.

One of the nice things about having the ELB in place, even for a single instance web site, is that it will do health checks and push the results to CloudWatch. CloudWatch will give you pretty graphs, but also permit you to set Alerts, which may be pushed to the Amazon Notification Service – which in turn can send you an email, or call a URL to trigger some other action that you configure (send SMS, or sound a klaxon?).