Fun with performance

OK, so I've been writing a web mail system for the University of Oregon for over a year now (http://alphamail.uoosl.org). I did a ton of analysis on performance and scalability of mail systems, since we have 40,000+ users, and supporting a web service for that many people is one of the primary problems with other solutions out there.

So, as we've migrated to this system, I knew how it should scale. I currently have 10,000 users, and the system was NOT scaling the way it was supposed to. I was a bit embarrassed because I had been stressing that my desktop system should be able to serve our user base without breaking a sweat, and now I was looking at telling them to buy a couple of quad-core machines with 16GB of RAM...can we say "uncomfortable"?

Anyway, I was motivated to do some optimizing, to say the least. I was in the midst of re-writing my email address parser, since I thought it was inefficient the way it stands, when I came across Apache::DProf. Ah, tools.

I found out that a library I'd written over a year ago (for a _much_ smaller project), which I'd imported into the project was calling internal things 10,000+ times a minute. Big indicator of a problem.

So, on closer inspection, I found that my session processing code from the old project was doing a really inefficient search to find out if a user was already logged in. This was fine when I was using it for 40-user systems, but obviously did not scale.

One code line later (skipping the check), and suddenly my scalability was where it was supposed to be!

It is funny, my recent move in research has been towards working in cross-cutting concerns and management of competing concerns in software. Here's a lovely example of where a concern assumption needed to be more explicit...somehow :)