Linux.conf.au 2014: LCA TV

The radio silence here on my blog has been not from lack of activity, but the inverse. Linux.conf.au chewed up the few remaining spare cycles I have had recently (after family and work), but not from organising the conference (been there, got the T-Shirt and the bag). So, let’s do a run through of what has happened…

LCA2014 – Perth – has come and gone in pretty smooth fashion. A remarkable effort from the likes of the Perth crew of Luke, Paul, Euan, Leon, Jason, Michael, and a slew of volunteers who stepped up – not to mention our interstate firends of Steve and Erin, Matthew, James I, Tim the Streaming guy… and others, and our pro organisers at Manhattan Events. It was a reasonably smooth ride: the UWA campus was beautiful, the leacture theatres were workable, and the Octogon Theatre was at its best when filled with just shy of 500 like minded people and an accomplished person gracing the stage.

What was impressive (to me, at least) was the effort of the AV team (which I was on the extreme edges of); videos of keynotes hit the Linux Australia mirror within hours of the event. Recording and live streaming of all keynotes and sessions happend almost flawlessly. Leon had built a reasonably robust video capture management system (eventstreamer – on github) to ensure that people fresh to DVswitch had nothing break so bad it didn’t automatically fix itself – and all of this was monitored from the Operations Room (called the TAVNOC, which would have been the AV NOC, but somehow a loose reference to the UWA Tavern – the Tav – crept in there).

Some 167 videos were made and uploaded – most of this was also mirrored on campus before th end of the conference so attendees could load up their laptops with plenty of content for the return trip home. Euan’s quick Blender work meant there was a nice intro and outro graphic, and Leon’s scripting ensured that Zookeepr, the LCA conference manegment software, was the source of truth in getting all videos processed and tagged correctly.

I was scheduled (and did give) a presentation at LCA 2014 – about Debian on Amazon Web Services (on Thursday), and attended as many of the sessions as possible, but my good friend Michael Davies (LCA 2004 chair, and chair of the LCA Papers Committee for a good many years) had another role for this year. We wanted to capture some of the ‘hallway track’ of Linux.conf.au that is missed in all the videos of presentations. And thus was born… LCA TV.

LCA TV consisted of the video equipment for an additional stream – mixer host, cameras, cables and switches, hooking into the same streaming framework as the rest of the sessions. We took over a corner of the registration room (UWA Undercroft), brought in a few stage lights, a couch, coffee table, seat, some extra mics, and aimed to fill the session gaps with informal chats with some of the people at Linux.conf.au – speakers, attendees, volunteers alike. And come they did. One or two interviews didn’t succeed (this was an experiment), but in the end, we’ve got over 20 interviews with some interesting people. These streamed out live to the people watching LCA from afar; those unable to make it to Perth in early January; but they were recorded too… and we can start to watch them… (see below)

I was also lucky enough to mix the video for the three keynotes as well as the opening and closing, with very capable crew around the Octogon Theatre. As the curtain came down, and the 2014 crew took to the stage to be congratulated by the attendees, I couldn’t help but feel a little bit proud and a touch nostalgic… memories from 11 years earlier when LCA 2003 came to a close in the very same venue.

So, before we head into the viewing season for LCA TV, let me thank all the volunteers who organised, the AV volunteers, the Registration volunteers, the UWA team who helped with Octogon, Networking, awesome CB Radios hooked up to the UWA repeated that worked all the way to the airport. Thanks to the Speakers who submitted proposals. The Speakers who were accepted, made the journey and took to the stage. The people who attended. The sponsors who help make this happen. All of the above helps share the knowledge, and ultimately, move the community forward. But my thanks to Luke and Paul for agreeing to stand there in the middle of all this… madness and hive of semi structured activity that just worked.

Please remember this was experimental; the noise was the buzz of the conference going on around us. There was pretty much only one person on the AV kit – my thanks to Andrew Cooks who I’ll dub as our sound editor, vision director, floor manager, and anything else. So who did we interview?

  • Alan Robertson (Assim Proj)
  • Arjen Lentz (twice – well, two topics!)
  • Daniel (A student at LCA for the first time)
  • Dave Chinner (XFS)
  • Erin Walsh (Rego desk manager)
  • Jason Nicholls (AV Director LCA 2014)
  • Jeremy Kerr (Kernel Developer)
  • Jessica Smith (Astronomy Mini Conf)
  • Jono Oxer (Audosat)
  • Karen Sandler (Gnome)
  • Keith Packard (X) and BDale Garbee (Freedom Box, Debian)
  • Kevin Vinsen (ICRAR, Square Kilometer Array)
  • Lennart Poettering (SystemD)
  • Linus Torvalds (Yet another Kernel Developer)
  • Matthew Wilcox (Another Kernel dev, and a Debian Dev as well)
  • Michael Still (OpenStack)
  • Paul Weyper (Canberra Linus Users Group)
  • Paul Wise (Debian)
  • Pia Waugh (Open Government)
  • Rusty Russel (Yet another Kernel Developer! Oh, and started LCA in 1999)

One or two talks did not work, so appologies to those that are missing. Here’s the playlist to start you off! Enjoy.

AWS CLI: searching for AMIs

I’ve been experimenting with the new Python based AWS CLI tool, and its getting to be very good indeed. It can support multiple login profiles, so its easy to switch between the various separate AWS accounts you may use.

Today I’ve been using it from Windows (!), and was searching for a specific AMI ID, and wanted to share the syntax for those who want to do similar:

C:\>aws --profile my-profile --region us-east-1 ec2 describe-images --filters name=image-id,values=ami-51ff9238
C:\>aws --profile my-profile --region us-east-1 ec2 describe-image-attribute --attribute launchPermission --image-id ami-51ff9238
C:\>aws --profile my-profile --region us-east-1 ec2 modify-image-attribute ami-51ff9238 --image-id --operation-type remove --attribute launchPermission --user-groups all
C:\>aws --profile my-profile --region us-east-1 ec2 modify-image-attribute ami-51ff9238 --image-id --operation-type add --attribute launchPermission --user-groups all

 

 

 

aws –profile turnkey –region ap-southeast-2 ec2 modify-image-attribute ami-fd46d6c7 –image-id –operation-type remove –attribute launchPermission –user-groups all

aws –profile turnkey –region ap-southeast-2 ec2 modify-image-attribute ami-fd46d6c7 –image-id –operation-type add –attribute launchPermission –user-groups all

LCA Past Organisers

LCA 2013

LCA Past Organisers

Previous core organisers of Linux.conf.au, taken at Mt Stromolo Observatory during LCA 2013 (pic by Josh Stewart); except one of these people organised CALU, and another hasn’t organised one at all!

Thanks to all the people at LCA2013 in Canberra; it was a blast! So good to see old friends and chat freely on what’s hot and happening. Radia (known for STP, TRILL), Sir Tim (the web) and old friend Bdale (Debian, SPI, Freedom Box) were inspiring. As was Robert Llewellyn (Kryten, Red Dwarf), who was a complete pleasure — he wandered back and talked for a while with the volunteer video crew.

Hats off to Pia for organising the TBL tour, to Mary Gardner for being awarded the Rusty Wrench, and to the team from PLUG (Euan, Jason, Leon, Luke) who stepped up to help with the video team – and to Paul who graciously accepted the help.

Next up – LCA2014 – Perth! Y’all come back now.. it’s been a decade.

AWS S3 Bucket Policies: Restricting to In-Region access

From time to time, Amazon Web Services adds new IP address ranges (it keeps growing!). These new addresses are published in the forums, such as via this post from EricS. I was creating a bucket policy to restrict access only to nonymous users who are within my region – I’m happy for the access requests, but I don’t want to pay the bandwidth charges. So here’s a small Perl script that takes the copy-and-paste text from EricS’s forum post, and creates an S3 buck policy element suitable for this:

#!/usr/bin/perl
open F, 'ips.txt' or die "Cannot read list of IPs: $!";
my @ip_conditions;
while (<F>) {
  push @ip_conditions, $1 if /^(\d+\.\d+\.\d+\.\d+\/\d+)\s/;
}
print "\t\"aws:SourceIp\": [" . join(",", @ip_conditions) . "]\n";
Debian GNU/Linux

Official Debian Images on Amazon Web Services EC2

Please Note: this article is written from my personal perspective as a Debian Developer, and is not the opinion or expression of my employer.

Amazon Web Service‘s EC2 offers customers a number of Operating Systems to run. There are many Linux Distributions available, however for all this time, there has never been an ‘Official’ Debian Image – or Amazon Machine Image (AMI), created by Debian.

For some Debian users this has not been an issue as there are several solutions of creating your own personal AMI. However for the AWS Users who wanted to run a recognised image, it has been a little confusing at times; several Debian AIMs have been made available by other customers, but the source of those images has not been ‘Debian’.

In October 2012 the AWS Marketplace engaged in discussions with the Debian Project Leader, Stefano Zacchiroli. A group of Debian Developers and the wider community formed to generated a set of AMIs using Anders Ingemann’s ec2debian-build-ami script. These AMIs are published in the AWS Marketplace, and you can find the listing here:

No fees are collected for Debian for the use of these images via the AWS Marketplace; they are listed here for your convenience. This is the same AMI that you may generate yourself, but this one has been put together by Debian Developers.

If you plan to use this AMI, I suggest you read http://wiki.debian.org/Cloud/AmazonEC2Image, and more explicity, SSH as the user ‘admin and then ‘sudo -i‘ to root.

Additional details

Anders Ingemann and others maintain a GitHub project called ec2debian-build-ami which generates a Debian AMI. This script supports several desired features, an was also updated to add in some new requirements. This means the generated image supports:

  • non-root SSH (use the user ‘admin)
  • secure deletion of files in the generation of the image
  • using the Eucalyptus toolchain for generation of th eimage
  • ensuring that this script and all its dependencies are DFSG compliant
  • using the http.debian.net redirector service in APT’s sources.list to select a reasonably ‘close’ mirror site
  • and the generated image contains only packages from ‘main’
  • plus minimal additional scripts (nuder the Apache 2.0 license as in ec2debian-build-ami) to support:
    • fetching the SSH Public Key for the ‘admin’ user (sudo -i to gain root)
    • executing UserData shell scripts (example here)

Debian Stable (Squeeze; 6.0.6 at this point in time) does not contain the cloud-init package, and neither does Debian Testing (Wheezy).

A fresh AWS account (ID 379101102735) was used for the initial generation of this image. Any Debian Developer who would like access is welcome to contact me. Minimal charges for the resource utilisation of this account (storage, some EC2 instances for testing) are being absorbed by Amazon for this. Co-ordination of this effort is held on the debian-cloud mailing list.

The current Debian stable is 6.0.6 ‘Squeeze‘, and we’re in deep freeze for the ‘Wheezy‘ release. Squeeze has a Xen kernel that works on the Parallel Virtual Machine (PVM) EC2 instance, and hence this is what we support on EC2. (HVM images are a next phase, being headed up by Yasuhiro Akarki <ar@d.o>).

Marketplace Listing

The process of listing in the AWS Marketplace was conducted as follows:

  • A 32 bit and 64 bit image was generated in US-East 1, which was AMI IDs:
    • ami-1977f070: 379101102735/debian-squeeze-i386-20121119
    • ami-8568efec: 379101102735/debian-squeeze-amd64-20121119
  • The image was shared ‘public’ with all other AWS users (as was the underlying EBS snapshot, for completeness)
  • The AWS Marketplace team duplicated these two AMIs into their AWS account
  • The AWS Marketplace team further duplicated these into other AWS Marketplace-supported Regions

This image went out on the 19th of November 2012. Additional documentation was put into the Wiki at: http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze

A CloudFormation template may help you launch a Debian instance by containing a mapping to the relevent AMI in the region you’re using: see the wiki link above.

What’s Next

The goal is to continue stable releases as they come out. Further work is happening to support generation of Wheezy images, and HVM (which may all collapse into one effort with a Linux 3.x kernel in Wheezy). If you’re a Debian Developer and would like a login to the AWS account we’ve been using, then please drop me a line.

Further work to improve this process has come from Marcin Kulisz, who is starting to package ec2debian-build-ami into a Debian: this will complete the circle of the entire stack being in main (one day)!

Thanks goes to Stefano, Anders, Charles, and everyone who  contributed to this effort.

Resources