However, MD5s can be represented in several ways. Lets discount the binary envoding of the 128-bits (16 bytes) of data as thats rather cumbersome, and if you’re storing this in a database such as MySQL, there isn’t a 16 bytes numeric data type; BIGINT is 8 bytes, so you’d have to use two BIGINTs and do lots of horible stuff.

That brings us to the base encodings. Base 16, or hexadecimal, would require us to use a text data type to store the results – as the base16 encoding will contains the numbers 0-9, and the letters A-F (or a-f – the case is irrelevent/insensative in base 16). It would be 32 “characters” long. We can stuff that in a column with no trouble (char(32)).

We can also use a Base 64  encoding, using upper and lower case letters and a few symbols as well as numerals 0-9. This comes to 22 characters (you’ll sometimes see == appended to a Base64 to make it 24 characters). Using 22 chars as a key instead of 32 is 31.25% less data. That makes your indexes that much more compact as well as the column data.

It may not be a perfect primary key, but its possibly reasaonable. But then comes the question of converting between Base16 and base 64. Here’s one way:

#!/usr/bin/perl
use strict;
use warnings;
use Digest::MD5;
use MIME::Base64;

my $data = "foobarbasbifffoobarbasbiff";
my $md5_base64 = Digest::MD5::md5_base64($data);
printf "%s in base64 as hex: %s\n", $md5_base64, unpack('H*', MIME::Base64::decode_base64($md5_base64));

• Stackoverflow article on RDS and UTF8
• timeghost.net article on MySQL and UTF8

And a quote from the second:

[mysqld]
default-character-set=utf8
default-collation=utf8_general_ci
character-set-server=utf8
collation-server=utf8_general_ci
init-connect=’SET NAMES utf8′

[client]
default-character-set=utf8

As you’ll see in the first link above (Stackoverflow), adding params with spaces to Amazon RDS is a little tricky from Win platform – and you have no choice but to use the CLI tools for RDS to do this.

Great; I can do this with a stored function, which I did, and it worked. But it was slow. So I decided that I’d normalise these en-mass with one big INSERT statement to the normalised table (protected by a unique index constraint), and then store the resulting Title_ID.

There be dragons. As one title came through as “[Q] help me” it was duly inserted and given a Title_ID.
However, when a lower case “[q] help me” came through, it matched as a duplicate of the original and therefore was not inserted again. I then pulled the strings into a Perl hash, and of course, couldn’t find a key with “[q] help me”, only “[Q] help me”.
Turns out that the issue was my column definition. varchar(x) is not case sensative. varchar(x) binary is. The Unique index I had on here was doing its job and comparing values based upon the case in-sensative column – not its fault.

ALTER TABLE Titles CHANGE COLUMN `Title` `Title` VARCHAR(600) BINARY NULL DEFAULT NULL ;

And now I see my column as “`Title` varchar(600) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL“.

In my case, I had a table with one column of a URL’s path, and a URL’s Query String, both of which can be larger than the old 255 chars. I also have an index that cover these two plus a few other columsn – normalised protocol, normalised domain, and TCP port.

In trying to move to longer columns (1K) I had to modify my index to restrict the number of characters from these varchar columns to ensure I remained under the 767 limit – I couldn’t just change one of these columns from 255 to 1024.

I throught I’d try and simple change first – instead of increasing the oclumn, just put the restriction on the current columns as they stand – so limit the index to 255 chars (while the column still IS 255). Turns out that in at least 5.5.9, since the specified size is the current column size, it ignores this.

Lets make that clearer; you need to specify a SMALLER length for the indexed columns in order for it to stick. Once that’s done, you can then alter table to increae the column lengths.

create table FOO like BAR;

However, as I just rediscovered, any triggers on the table aren’t taken across with it. D’oh. Which reminds me, mysqldump has a specific -R flag to backup/dump routines; worth having that on too (I did).

Rule for the day: check your triggers on tables before moving/renaming.

It’s a shame that MySQL doesnt have the ability to automatically add partitions based upon date. For example, if result of a function should generate a new partition:

PARTITION BY YEARWEEK(MyDateTimeCol)


And hey presto, a bunch of new partitions would appear every time you insert new data thats in a new range. YEARWEEK() may be too course for your massive amount of data – perhaps TO_DAYS(). Perhaps it’s not that massive a data set, so just YEAR() would suffice. But having the new partitions created upon INSERT of data that matches would be great. As would deletion of partitions when a partion is empty. Nice bit of housekeeping.

select concat("0:0:0:0:0:0:", LPAD(CONV(substring_index(inet_ntoa(IPv4), '.', 1), 10, 16), 2, "0"), LPAD(CONV(SUBSTRING_INDEX(SUBSTRING_INDEX( inet_ntoa(IPv4) , '.', 2 ),'.',-1), 10, 16), 2, "0"), ":", LPAD(CONV(SUBSTRING_INDEX(SUBSTRING_INDEX( inet_ntoa(IPv4) , '.', 3 ),'.',-1), 10, 16), 2, "0"), LPAD(CONV(SUBSTRING_INDEX(SUBSTRING_INDEX( inet_ntoa(IPv4) , '.', 4 ),'.',-1), 10, 16), 2, "0")) as IPv6 from Log3NF.Access limit 1000;

Reference: Bug 15044, and WorkLog 1293.

I had been seeing this for… well, years before hand, but considered it to be “just me”. Since I filed that bug, others have come forward and said they have seen the same issue.

Now, with the Oracle/Sun merge, and Sun having shelled out to “purchase” MySQL, we see a miraculous MySQL 5.4 being drafted together. Its good; but it appears to be a simple crowd pleaser that should have happened years ago – integrate in some very popular performance improvements (notably from Google’s code repository). If code is being contributed, and it works, why wasn’t this done earlier? And if not earlier, why now? Smokescreen? Last ditch effort before the project is canned by its new owner? Of course, this has been circulating for a while; I’m not posting this later after that’s been in the news for a few weeks now.

What is interesting is the “Refactoring MySQL” project being discussed. Where does this leave MySQL 6 – a little abandoned from here. And then with MariaDB and Drizzle now forked and picking up attention (external to MySQL/Sun/Oracle), it seems the entire thing is just becoming a mess. Drizzel is supposed to be the performance version – but with the google patches being integrated, perhaps the core MySQL branded project is catching up?

What’s looking clearer is: Postgres. From my perspective, its worse for MySQL in Debian. Lenny (stable now) has MySQL 5.0. The Experimental repo has 5.1 (5.1.34), but this hasn’t even transitioned to Unstable yet!

Here’s what I’d like to see:

1. Someone picks up Bug 15044/WorkLog 1293.
2. 5.4 goes STABLE from MySQL/Sun/Oracle real soon now, with or without my pet qcache bug/feature – it’s a big enough win for multi-core systems as it stands
3. Debian moves MySQl 5.1 from experimental straight to Unstable – can the mysql-server-5.1 package can coexist with mysql-server-5.0?
4. Debian also adds, along side the MySQl 5.0 and 5.1 servers, the 5.4 into Unstable, and 6.0 into Experimental.