A few years ago, I converted my earlier idea of logging Apache request to 3rd normal form into a fully fledged Mod Perl 2.0 Log Handler – embedding this into Apache. Its essentially a very simple Handler of less than 60 lines, and a stored procedure inside MySQL that normalises the data. Its been running on my personal server since February 2009, and in that time its collected around
- 1.4 million hits
- 27,000 unique useragents
- 57,000 unique paths
- 9 basic authentiation users
- 13 HTTP methods
- 50,000 unique referrer URLs
- 13 HTTP Status
- Recorded the transfer of 192,185.9637 Megabytes of (body) data; the average body response is 144 KB.
Wow. The log data on disk is 574 MB, or around 410 bytes per request – including the indexes (this is the size of the MySQL directory containing the data).
All well and good. Now time to get it fit for IPv6, and then improve the reporting. The reporting has two phases:
- Live reporting from the 3rd normal form for data covering the last few seconds/minutes/hours/days.
- Summary reporting per day or per month, per statistic, pre-calculated
Anyway, we’re about to pull in IPv6, again storing this as efficiently as possible, and then improve the currently very basi reporting interface…. stay tuned… and see the SVN repository for code…