FabulousEntertainment

I’ve finally got round to implementing proper bounce handling. It’s quite important to do if you don’t want to end up treated as a spammer. I’ve implemented a Variable Envelope Return Path (VERP) on all automatically generated mail, added a procmail script to sort out the bounces from other mail and then a Python script to put the bounces into a database. We’ll then mark an address as bouncing if we get three verified bounces.

I also tested our new SPF record and it does seem to be okay.

Having just finished that, I then began to notice that we were getting a lot of complaints from Hotmail/MSN addresses saying that they were not getting mail from us. It turns out that we needed to re-register the server with them, it seems to be the policy that if you have a high volume, unregistered server then they will silently delete all mail that you send.

I really wish that they would reject the mail instead to that senders would get some sort of error message to let them know what’s going on.

FOLLOW UP:
I received a response from Microsoft. The first problem that Yahoo’s spam filter put the message into my junk folder so I didn’t see it immediately. The second problem was that their response was that we should send less mail! I now need to think of something to say to the 15,000 Microsoft users on our site…

This is going to sound a bit like an advert for The Planet but I have been absolutely delighted by the speed and helpfulness of their support team over this set up stage.

The deal I got was absolutely brilliant, coming in at around half the price of where I was and to top it all the new network is running much faster too.

I also really like their DNS setup, which has allowed me to kill off my nameserver and switch to theirs.

The only negative is the choice of operating systems being very limited - surely they could offer Debian or even Ubuntu?

Over at The Planet, they had an offer that was just too good! A Dual Xeon 2.8, 2 GB memory, 2 x 80 GB drives and 2500 GB bandwidth for $177 a month!

Currently we’re paying $170 a month for a Pentium 2.4, 2 GB memory, 1 x 80 GB drive and 1500 GB bandwidth so for an extra $7 a month there is a more than doubling of processor capacity, doubling disk and nearly doubling bandwidth.

The only negative is that The Planet do not support Debian as a standard distribution, their options are CentOS and Red Hat neither of which I particularly like however this is a small price to pay.

For the future, I found out that a 100Mb unmetered pipe is $1650 although it is not available on the cheaper servers. This is a fraction over ServePath’s $1500 however it doesn’t have the one connection per customer limit — not that I will be needing more than one unmetered 100Mb connections for now!

Before the server goes live I am going to spend some time performance testing. It’s really good to have the opportunity to performance test on exactly the hardware that will be deployed.

The present server is easily doing 0.5 million page views a day in addition to email and serving all of the static assets (mainly photos) so I would like to get to at least 2 million page views a day on the new server.

California is going really well, yesterday we had more couples sign up from California than from the UK! However it’s revealing a few problems with the site:

• Interest comments are international (and in the wrong order too) so I will make it so users only see comments from their own countries
• Events and the Swingers Clubs List are national which is alright for the UK but someone in California will have no interest in clubs/events in New York

In resolving these points I will also try to address the SEO issues around the clubs list too — when Google views that page it only sees the US clubs, which isn’t ideal.

It’s coming up now for a year that I’ve been with ServePath. They have been absolutely excellent in terms of both service and reliability. However they have not been so good on price. Here’s a comparison with the also excellent Liquid Web.

To match the ServePath 250GB drive would add $15 a month but I don’t think I need that.

The main thing driving this change is bandwidth, I’m going to be on about 50% of my allocation this month and it’s rocketing up. Over the past 7 days I’ve used an average of 0.95Mbps and with a peak at 2.95Mbps so I will need a 10Mbps circuit which should do me a while (and I wouldn’t mind sometimes slowing down slightly at peak times).

For the server above with 10Mbps unmetered I’m looking at $680/month from ServePath and $369/month from Liquid Web which I’m afraid just means a change.

In the future I will need a private network when I get the inevitable second server which is cheaper at Liquid Web but when I get to the stage of needing 100Mbps connection ServePath begins to look quite good again. However according to this page there is a limit of one 100Mbps connection per customer so I guess that in the longer term I would need to move on anyway, and the bigger the site is the more difficult it is to move.

Nothing is ever easy!

UPDATE:

After sleeping on this, I’ve just worked that a 10Mbps connection has a theoretical monthly limit of 3255GB per month. Obviously 100% loading would be nonsense, so assuming a 50% load then this would be 1600GB per month. My present bandwidth limit is 1500GB so in actual fact this maybe isn’t an upgrade at all!

A Intel Xeon Dual Core Conroe 3060 from The Planet, with 250GB SATA and 2500GB bandwidth would come in at $209/month which would be virtually doubling my actual traffic and pretty much doubling the processor power too. If I need extra bandwidth I can get another 1000GB for $175/month.

My new timing code revealed that there was a problem with all of the pages on the website that generate email.

I investigated a bit more and it turns out that using the Python smtplib is many times slow than using popen2 to call the sendmail command directly.

It also means that my Apache processes aren’t tied up waiting for email to process. The downside is that I need to do a bit more checking on the validity of email addresses in my application instead of relying on the mail server.

I’ve been able to spend the whole day on Fab Swingers which has been great.

First off, I’ve written a trivial CherryPy tool which logs any slow pages. Currently “slow” is defined as 500 ms but I’ll need to cut this back a bit.

This tool has already been quite handy and I’ve spotted two missing database indexes which has already dramatically speed up the Cross-dressing page.

I’ve also implemented memcache for the homepage, the profile pages, the session system and for the new mail received prompt. As a result MySQL has dropped right down in CPU usage.

I ran vmstat today and I noticed that there was a bit of swapping going on (memory swapping that is, not rumpy-pumpy). I think the reason is that I increased the Apache max clients, but this was required because the server had ground to a halt. Also, I am about to deploy memcached with an initial 64MB so it looks like there’s a need for a bit more memory.

I’ve decided to upgrade from 1 GB to 2 GB which should keep me going for a bit. It’s a $39.99 setup and then an extra $39.99 per month from ServePath. This seems bit steep given that an extra 1 GB from Dell is a one-off $70!

I also considered upgrading the slightly elderly processor but I think that extra memory will have much more impact.

The good news is that page views and active members have continued to rocket but it’s hitting the server again. We briefly ran out of resources this afternoon and I’ve made some tweaks so I think that we should be okay until Christmas.

I really don’t want to give up on the software optimisation and just starting buying servers, so for now I’m going to park the marketing work and get back to improving the code. Currently I’m just under 8 million page views a month, I would like to get to at least 31 million page views a month on this single server. Given the results of my cache experiments I have no doubt that this is achievable.

In addition to performance there is beginning to be quite a list of other tasks that really need to be completed. There’s a lot of security improvements (such as steganography for uploaded photos), more linking between the profiles for SEO and the chatroom could use some work.

I think that this will probably be a couple of weeks work but given my schedule it will probably end up taking a month.

I’ve just done some initial benchmarking for displaying a profile using memcached.

At the moment, on my MacBook, it takes about 18 ms to display a profile. Using memcache instead this drops to 0.4 ms, in other words it’s 180 times faster.

These tests were conducted with python-memcached which is the pure-Python interface because I didn’t immediately manage to get the C interface working. However I probably don’t need to bother with this right now given the performance gain.

Of course not all of the pages on the site are suitable for memcache but I would think about half the total pageviews could be.