Performance to the next level: memcached
I think that I’ve more or less reached the limit of my MySQL optimisation, there’s always ways that I can speed up queries here and there but I’m not going to get a 10x speed up through this path. So I am now looking at reducing the number of databases queries through a cache of some sort.
As suggested by a comment on this blog I checked out memcached. I have to say I was very impressed. I love the fact that it’s used by Slashdot (see their write up) and LiveJournal amongst other. It’s also great that it’s so simple and doesn’t even have a configuration file!
I had been thinking about doing the cache at the level of my Python application. However the problem with that is that I run multiple processes for the application and in the future I may well want to run the application on more than one physical server.
Memcache comes with a pure Python interface however I shall be using the C Python interface which is reportedly 3 times faster — because I’m in this for the speed!
I’ve not decided how to use memcached though, I could cache data structures (Python dictionaries probably) or I could cache whole pages. Following my recent MySQL optimisations/removal of awstats I’ve bought myself quite a bit of time so I won’t need to roll out memcache until traffic doubles which should give me another couple of months.
PS: I think that I may disable the MySQL query cache, I’m seeing a hit rate of 500,000 verses 14,000,000 inserts so I suspect that the cache is making performance worse. The problem is that my member table changes so frequently that the cache probably ends up being invalidated before it can be used.