FabulousEntertainment

The problem du jour was that we ended up triggering Google’s Badware filter which meant that anyone visiting the site from Google got a warning that the site contained badware which might damage their computer. Needless to say pageviews dropped a bit!

I logged in the Google Webmaster Console as soon as we got complaints and checked it out. It said that the homepage was an example of a page with a badware problem. The only two external links on the homepage are to Adult Friend Finder (via my affiliate account) and a banner ad from a very reputable network ran by one of the largest companies in the world.

So I complained to Google and within an hour or so they’d removed the badware warning, so overall I’m quite happy with their response.

The interesting about Fab Swingers is that virtually all of the membership comes from word of mouth recommendation. So really the best way of getting more members now is just to work on improving the site (except for regions where we don’t yet have a critical mass of members).

The past couple of days I’ve been working on scalability. Profiles are now cached (with memcache) in a better way. I’ve also removed a whole bunch of counts in SQL and replaced them with summary tables.

Finally, I’ve also been working on a new feature. One of the things that we’ve really missed is video upload and someone requested it in the swingers forums. I’m not going to do this myself, instead I’m going to integrate with a video hosted service but it should be fairly transparent.

None of this has been deployed to the live server yet and there will be a short outage too — it’s going to take a little time to build up the summary tables.

I was just looking at the site earnings and I got a bit of a shock, from May 1 to May 4, pageviews have absolutely tanked by about half.

I went straight into Google Analytics and found that visits were holding up fine but the problem was that the bounce rate had rocketed from the usual 3.5% up to 12.5% and the page views per visitor had also fallen away.

So it looks like there was some sort of technical problem which put people off using the site for about 4 days and then everything went back to normal. It wasn’t such a critical problem that it killed the traffic completely but maybe the site was just slow or perhaps people with certain ISP’s were having a problem.

Having said that, we had no email reports of problems and usually as soon as there is the slightest glitch the support email box is filled with hundreds of complaints within minutes.

I’m not sure there is much more that I can do to investigate now so I’ll just park it for now and go back to it happens again.

I’ve just done a quick analysis of page views verses MySQL select queries:

• 16.5 select queries per second
• 7.3 page views per second

The caveat is that the page views are from Google Analytics and will be slightly under-reporting the actual page views as clients which don’t support JavaScript will not be counted. However, it is reasonable to say that we are averaging more than 2 select queries per page view.

This strikes me as quite a high number. I would like to target a ratio of 1:1 or less for selects to page views which would half the total number of SQL queries and would be a great step to increasing the performance of the site.

I think I am also going to refactor some of my code so that the cache logic is shifted out of the main application into my Python SQL methods. This will simplify the application and make it easier for me to put memcache in front of virtually every query.

PS. I also notice that the query cache hit rate is absolutely terrible! I really need to either fix this (by putting updates into a batch and doing them less frequently) or just switch the query cache off.

Recently the MySQL backup has been killing the (already loaded) server. I was doing backup using mysqldump and then piping it into bzip2. Both commands put a heavy load on the processor and in fact made the site virtually unusable in the very early morning.

As I am currently using MyISAM I can instead take advantage of mysqlhotcopy which is doing the backup in less than a hundredth of the time with virtually no load!

It is an important reminder that when you are looking at optimising the performance of a server it’s not just about the application code, it’s also about sysadmin scripts.

Reminder to me: If the site every changes to InnoDB then the backup will need to be revisited.

I have had a look at the Google App Engine and I guess my first thought was that it uses almost exactly the same architecture as Fab Swingers! The language and middleware is the same so from a technical point of view it would be a very easy thing to use for the site and would be a great way to get past the scaling problems I’ve got.

Unfortunately it’s a no go because the Program Policies would prevent me using it for Fab Swingers.

So, it’s back to look at new servers again.

The new server is beginning to grunt a little bit. It’s going 2,500,000 hits a day which included 500,000 page views. Bandwidth usage is fine at 430 GB a month out of 2,500 GB in the package (17% used). It’s the pageviews which are the problem because they generally involve quite a bit of processing.

I’m not sure if there’s a huge amount more wins on optimisation although every time I’ve looked at it I’ve found something to be done. For example, I’m not preparing any MySQL queries and there is more that could go into a cache. I also didn’t do the re-tuning exercise when we moved to MySQL 5.

On the other hand, the total cost of the server and bandwidth is as of now is going to pan out at about 5% of advertising revenues so I’m not too bothered as long as we can scale linearly from this point.

I am certainly not going to change the mail server IP again though. It took months to restore mail service everywhere after the move.

One of the spam gangs that I’ve been battling with have started using the Tor Project to attack the site. I do kinda understand where they are coming in producing a network allowing anonymous use of the internet but I do disagree with quite a few of their statements, for example Tor isn’t useful for spamming is complete nonsense nowadays.

Fortunately it is not too difficult to block Tor, there’s a list of IP addresses and I’ve written a Python script to process it and put it straight into my block list and it’s goodbye to another source of spam!

I’ve been using Qmail for the last week or so but I’ve not been very happy with it. First off, I had to edit c files in order to get it to compile and I don’t really expect to be having to do this in 2008.

Secondly, I was having difficulty mailing AOL. Again, there is a patch out there to fix it but that’s just not something I really want to be doing.

Finally, I had problems with the DomainKeys patch, it broke the internal forwarding and there were a few ISP’s who wouldn’t even accept the mails.

So I have reverted to Postfix and changed my application so that it signs mails with DomainKeys that are bound for Yahoo.

So I changed the IP address of our mail server when we moved from ServePath to The Planet. Here’s what happened with the different major email providers:

Hotmail blocked us fairly quickly after the move but after exchanging a few emails (which after the first holding mail were very promptly and professionally answered) it was all sorted out. They also have a cool feedback system where you can see how many mails you’ve sent on a particular day and what percent generated complaints (showing less than 0.1% for us).

AOL also blocked us but after a single email which was dealt with within 2 hours they removed the block so again I was pretty happy. They also have a nice feedback loop system.

Google Mail were arguably the best of all and there was no interruption during the mail server move, I guess their algorithms must have registered that none of our mail was generating spam reports so there was no need to block us despite a sudden spike in email traffic from a new IP address.

Which finally leaves Yahoo who I am afraid have been staggeringly unprofessional. I have been mailing constantly since the server move and have been unable to get anything other than a canned response. They are not interested in the number of complaints I’m getting from Yahoo users. They do have a feedback loop system but it’s taken them nearly a week to get application details to me.

Worst of all by throttling the mail they have caused huge backlog of mail to develop which has impacted our users who are on higher quality email providers. I’ve now resolved this by changing the application so it doesn’t mail Yahoo addresses but that is not ideal.

I am really now pretty stuck as to what my next step is. Maybe I need to put something on the homepage explaining that Yahoo cannot handle a high volume mailer who changes IP address and that people should use another provider but it’s not really a route I want to go down.

I’m not sure there’s much point continuing to mail Yahoo, they obviously aren’t interested in their more than 10,000 users who are unable to use our site properly.