Amazon Linux, one of the distributions that is recommended for Amazon EC2 customers, recently had an update — 11.09. In this there was an update to a whole raft of libraries, including the Perl LWP (libwww) library in perl-libwww-perl-5.837 (previously 5.8.33), and other related modules.
One of the changes that happened is a change of the default for “verify hostname” in the SSL protocol when using LWP::UserAgent; previously verification of the certificate to the hostname given was default disabled, and in an effort to improve security, this was turned on. You’ll see this mentioned in LWP::UserAgent documentation “The no checks behaviour was the default for libwww-perl-5.837 and earlier releases”. What’s unusual is the no-checks behaviour change is DIFFERENT in Amazon Linux’s package of 5.8.37 compared to this statement – I suspect this one line got back ported into 5.8.37 to change this default ‘in the interst of security’.
Unfortunately, this breaks a lot of scripts and other modules/libraries out there, one of which is the Amazon-issued S3 libary. S3 is the Amazon Simple Storage Service (SSS => S3), with which a user (customer) has their data arranged in “buckets”, with data in objects identified by ‘keys’ (like a file name). All data is put to, and read from the S3 service over HTTPS – it’s not locally mounted (though some cleaver fuse stuff may make that look possible – but it is still over HTTPS.
A bucket in S3 has a name, and for the example I have, the name looks like a domain name (images.foo.com). When accessing this bucket, the Amazon S3 Perl library connects to an alias hostname (CNAME) made up combining the bucket name above with “s3.amazonaws.com“, so our example here becomes “images.foo.com.s3.amazonaws.com“. This site is using a wildcard certificate for “*.s3.amazonaws.com” (you can see it as an Alternate Subject Name extension in the SSL certificate). This permits the certificate to be considered as valid for any hostname directly under the s3.amazonaws.com domain. However, subject to RFC 2818, the only thing permitted before “s3.amazonaws.com” is a single name – not a (seemingly valid) dotted domain name. So “com.s3.amazonaws.com” is OK with a wildcard certificate, but “images.foo.com.amazonaws.com” is not.
There are several solutions. The easiest is to turn off SSL certificate verification again in your script. A handy ENV environment variable may be set to do this: $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0. Alternatively, if you are using LWP directly, you can pass an initalisation parameter to LWP of ssl_options => { verify_hostname => 0}. Both effectively abandon any certificate verification.
Somewhat more complicated, you can define a custom validation callback (procedure) to further determine if the certificate is valid. This is in contravention to RFC 2818, and seems like a lot more hassle to work around.
Perhaps the easiest solution here is to avoid using period/dot/’.’ in Bucket Names in S3, thereby removing the conflict between the strict checking.
The most important thing is how lax we have been at verifying SSL certificates, and have come to rely on that just working. It is good to verify the SSL certificate matches the host in scripts: I don’t want to start communicating authentication information over an SSL channel if we can easily see we’ve been duped on the remote end. I was not familiar with wildcard certificates only being valid for one component of a domain name; this kind of reduces their effectiveness in my mind in some sense.They’ve always been more expensive than standard certificates, but being better aware of the FQDNs they will validate on is useful.
I’ve seen several other instances outside of this S3 example where invalid certificates have blindly been accepted by scripts (a CloudWatch example I saw with a redirect ‘hop’ through an SSL site); this default change from lax to legitimate certificates may actually encourage better adoption of the security that SSL can give — when we’re already paying for SSL certs — or lead us (as developers and architects) to acknowledge when we’re actively ignoring that layer of protection.
It’s early days now but as this default change filters into Linux distributions (and Perl distributions on other platforms) then we’ll start to see a lot of FAQs on this.