I now have one server with MySQL and the application server on it which serves the pages of the site. The other server does the photos and outgoing mail. The photos are uploaded to the application server but are accessed via a caching proxy (Apache) on the other server.

At the moment the load on the MySQL/application server is no more than about 0.50 but the photo server is double that. Part of the problem is that I haven’t yet managed to get mod_expire working so the headers are missing.

However I am now pretty happy that this configuration will take me to at least 2 million pageviews a day. It’s also just a huge relief not to have the site on the edge of collapse!

PS. I have also signed up with rsync.net for backup. The Planet do have backup options but rsync.net is a bit cheaper and also I prefer to have the backup with a completely different provider.

This will be massively simpler than periodically rsyncing images over and will require no change to the application. The only negative is that it won’t provide an online backup but I can address that separately.

While I’m migrating the applications server to the new hardware I’ll use rinetd to forward requests while I wait for the DNS to fully propagate.

UPDATE: Rather than lighttpd or Apache I think I shall use Squid as the proxy.

Finally, after years McDonalds did open. And I really didn’t like it that much. It was, in short, a huge disappointment.

Since I started this blog I’ve been very focused on Fab Swingers’s Google rankings and finally the site is showing up high for many of the key terms I’ve been targeting. I’m sure it’s no coincidence that the sudden move up has happened at the same time as Google Analytics has shown us passing the 1 million page views a day mark.

Yet, I feel slightly let down. The effect of the Google rankings has been tiny compared with the word of mouth and even the people who are joining via Google aren’t generally the people that bring most to the site.

I’m pleased that it’s finally happened but it turns out not to be a big deal.

I do like Sausage Egg McMuffins though.

However it turns out to be a cock-up on my part. I was using rsync which is only supposed to copy over files that have changed. But if you forget to use the -t or –times option then the modification dates are not copied over which means that every single file needs to be copied over the network to find out whether or not it has changed. Whoops!

As I now have two servers at The Planet which are in different data centers (Dallas and Houston) my backup plan is just to rsync the files between then.

I am also considering using rsync to allow my applications to upload photos to the assets server, however I may use Samba instead. NFS is out though!

UPDATE: After a great deal of thought over the past few weeks I’ve reached a decision. I thought about dozens of alternatives but I needed something very secure, robust (even when there is a network outage) and asynchronous with the file uploads so that the user isn’t kept hanging around when they are uploading their images.

The solution is that photos will upload on the applications server and also be served from there. This will result in fast, reliable upload. A cronjob will then rsync the photos over to the assets server every few minutes and will update the photos table to show that the photo has now moved over so future pages will get it from the assets server. I think this should work pretty well!

It is also quite future proof, if I switched to using a Content Delivery Networks (and I note that Peer1 allow adult sites on their CDN) then this approach would work great for uploading the images to it.

Right now, Alexa is reporting 21.8% of the Adulthub’s users from that country. We have also had an increase in attempted spam from Nigeria on our site however our blanket ban on Africa has been quite effective at keeping them under control. I guess that they are just going through a phase of targeting swingers sites.

But to have over 20% of site traffic from spammers is pretty bad and must be annoying a lot of their users.

From what I’ve read, The Planet have now upgraded the bandwidth on their virtual rack product to 1 Gb. This makes it much more useful and I now plan to get two new servers in a virtual rack, one running the applications server and one running the database. I’ll leave the present server doing mail (because it is such a pain to change the IP address of a mail server) and also serving the assets.

For the new servers I’m not going to bother so much about CPU but I will be looking at memory and the speed of the hard drive as the main issue has been IO.

It is also something that is very true about marketing. Providing you have a basic proposition that people like then the key to successful promotion is just quantity; the more you do the more it works.

Fab Swingers has succeeded because of quantity. It’s not just quantity in marketing of course, but there is also a strong network effect because the more members the site has, the more valuable it is to the members. And the more likely they are to recommend it.

The only thing I’m feeling a bit negative about right now is that I’m mainly having to focus on just keeping the site going as the servers struggle to cope with the rocketing traffic. I’d be much happier if I could concentrate on marketing which is my skill.

However I’m sure that most social networking sites would be happy to have the problem of struggling to cope with traffic growth!

As I’ve studied the performance of the server generally it’s pretty clear that it is very IO bound (although things can go a bit slow sometimes, the processors are always mostly idle). So writing a 1 GB log each day isn’t going to help.

I do want some stats so I enabled the statistics module which tells me that lighttpd is averaging about 60 hits per second (although right now it’s 75 hits per second).

Since I made the change I think that performance has improved a bit but it’s not been massively significant.

I am also considering moving Lighttpd off to another server completely, although it uses almost no CPU I think that it is using up IO bandwidth. I really need another server anyway but I’ll wait until the July promotions start at The Planet.

Anyway, the problem is that this table gets very large, very quickly. My solution had simply been to run a delete every month to remove older entries. But this led to two problems:

1. The table was locked for ages while the delete happened.
2. The table was then left fragmented (which prevents concurrent inserts on my configuration)

My first thought was to use DELETE … LIMIT 100; in a loop so that the table wouldn’t be locked for too long. However that was still not an ideal solution as it was still using a lot of resource and fragmenting the table.

What I finally settled on was using a new login table for each month and then using a MyISAM merge table to join them all together. Then, I can simply drop the old tables as they go out of date and reconfigure the merge table.

I can now delete millions of rows instantly with no performance impact or fragmentation. Result!

Anyway, I identified that one of the problems was that some users just had too many messages so I’ve implemented some pruning so that users can have only 600 messages in their inbox/sent mail - this is all that the UI allowed them to see anyway so there will be no noticeable effect to them.

My improved logging also identified a missing index from my bounce table (which contains all bounced email) so I was able to speed up queries there very substantially.

I’ve also done a review of the MySQL status and it turned out I needed to increase the table cache a bit, but after boosting that from the default of 64 to 128 and then 192 I am now seeing that ‘opened_tables’ is sitting at a constant value.

The major problem that I am left with is the messaging system. It is full of joins, it gets a lot of write traffic, it’s a very large table and is the main contributor to the slow query log. Unfortunately I don’t really have a very clear idea of how to sort this out yet!