November 11th, 2008

mkdir too many links

Picture the scene: Day three of my Vegas vacation, I am a mixture of hungover, jet lagged and suffering from the Vegas culture shock. I’m so spaced out that I’m struggling to work out the tip on my meal.

Then I get the message that Fab users can’t upload photos. I stagger back to the hotel room and feeling like a technical Hunter S. Thompson I fired up ssh to investigate. It turns out that mkdir is failing with the error ‘too many links’ and a Google reveals that there is a limit of 32K directories in a directory which is way less than I thought.

Fortunately it’s not that difficult to resolve, so I write a migration script to move the album directories around so that each parent directory has no more than 10,000 directories within it. There were just a couple of helper functions that are used to display photos so I didn’t need to change that many things to get it working again.

October 17th, 2008

Swingers Site Rankings

Here’s the latest progress:

Alexa Rank PageRank Site
53 7 AdultFriendFinder
7,885 3 Sex in the UK
14,883 3 Swinging Heaven
17,181 4 SDC
38,644 2 Fab Swingers
39,704 3 The Adult Hub
67,131 2 Dogging.co.uk
99.237 1 Local Swingers
107,510 1 Swing Fans

Fab Swingers has moved up another place, overtaking The Adult Hub. The Adult Hub (incorporating Dirty David) was one of the original swingers sites and one of the biggest, so it’s great to have moved passed them.

The gap with Swinging Heaven is narrowing, I guess that we presently have about 40% of their traffic, so there’s still a good way to go. It’s a wonderful study of a conventional marketing campaign with a reasonable amount of cash behind it verses a zero-budget, grass roots operation.

I’m tempted to remove the Page Rank from this chart, I might replace it with Google Trends for Websites information instead. Fab Swingers has been enjoying its best Google performance ever since the Page Rank dropped so it just really doesn’t seem to be that relevant anymore (having said that I probably should make an effort to boost it).

The recent technical changes to the site (see previous posts) seem to have removed our performance problems for now so once again we can focus on adding features and getting bigger.

October 10th, 2008

Web accelerator problems

I noticed that the cache hits on the assets server were pretty low and as a result we were getting about 30 hits per second for images on the main applications server (what’s supposed to happen is that although the images start out on the applications server they should all end up cached on the assets server which acts as a web accelerator resulting in virtually no image hits on the apps server).

It turns out that somehow I’d messed up the Cache-Control headers so they were missing which I guess made the cache virtually useless. My problem was that I’d forgotten the ‘ExpiresActive on’ directive which resulted in all my ExpiresByType directives failing silently.

Note to self: lynx -head http://www.fabswingers.com/ is a great way to verify that the headers are correct.

While I was at it, I also adjusted the settings for mod_deflate to make sure that all the right pages were being zipped. On Debian, the default is:

AddOutputFilterByType DEFLATE text/html text/plain text/xml

Which I changed to:

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css

So my (fairly lengthy) stylesheets are now compressed too, which should make a fractional difference to speed and bandwidth.

October 6th, 2008

Fab Swingers is Two

Fab Swingers is now two years, but like a much older person a birthday is a time for some introspection.

As of today the Alexa Traffic Rank is 39,713 which is a whisker away from The Adult Hub (and on the basis of the shorter term data The Adult Hub are already behind us in terms of traffic).

It’s pretty clear that unless something goes horribly wrong that Fab will overtake Swinging Heaven in the next twelve months to become the largest UK swingers site.

The only slight disappointment is that the USA has not done very well but that’s simply because it’s not had enough time spent on it. The problem is that my other website is now doing more than $250,000 per month whereas Fab Swingers is doing less than 1% of that so it’s very hard for me to spend the time on the site that it deserves.

On the positive side though it does mean that I can afford to invest a bit more financially than I have done in the past.

Also, my experience in other businesses has been that as you begin to threaten the market leaders they will typically get the lawyers on the case (on any spurious grounds) so I am now beginning to build up a contingency fund to make sure we can give as good as we get.

A final remark: After two years of struggle we are finally getting great placement on Google for various swinger related search terms BUT this has exactly coincided with our page rank (as reported by the Google Toolbar) dropping a notch. Proof positive that page rank is not that important anymore. I am tempted to drop it from the benchmarking reports.

August 31st, 2008

UK Swingers Site Rankings for August 2008

I’ve left it a little late to do the rankings, but here they are at last:

Alexa Rank PageRank Site
44 7 AdultFriendFinder
4,978 4 Sex in the UK
14,626 3 Swinging Heaven
16,966 4 SDC
35,772 3 The Adult Hub
48,322 3 Fab Swingers
67,381 2 Dogging.co.uk
94,524 2 Local Swingers
106,523 1 Swing Fans

There’s not that much news this time. Swinging Heaven has declined, Fab Swingers has moved up and there haven’t been any very dramatic changes.

The gap between Fab and Adult Hub has narrowed looking at the more recent data and you can see this graphically on Google Trends for Websites. Looking at users online Fab has 1016 right now verses 753 at the Hub and the Alexa weekly rankings show Fab well ahead so personally I think that we have already overtaken them but the stats have yet to catch up.

August 17th, 2008

Server move complete (more or less)

I’ve finished the migration more or less. The great news is that the new configuration has fixed the performance problems and it looks like there is now plenty of headroom for growth.

I now have one server with MySQL and the application server on it which serves the pages of the site. The other server does the photos and outgoing mail. The photos are uploaded to the application server but are accessed via a caching proxy (Apache) on the other server.

At the moment the load on the MySQL/application server is no more than about 0.50 but the photo server is double that. Part of the problem is that I haven’t yet managed to get mod_expire working so the headers are missing.

However I am now pretty happy that this configuration will take me to at least 2 million pageviews a day. It’s also just a huge relief not to have the site on the edge of collapse!

PS. I have also signed up with rsync.net for backup. The Planet do have backup options but rsync.net is a bit cheaper and also I prefer to have the backup with a completely different provider.

August 4th, 2008

Faster photos

I’ve had some more thoughts about how to handle the photo/assets server. Rather than trying to rsync the images from the applications server I’m going to set up a caching proxy (probably using lighttpd but I may use Apache instead).

This will be massively simpler than periodically rsyncing images over and will require no change to the application. The only negative is that it won’t provide an online backup but I can address that separately.

While I’m migrating the applications server to the new hardware I’ll use rinetd to forward requests while I wait for the DNS to fully propagate.

UPDATE: Rather than lighttpd or Apache I think I shall use Squid as the proxy.

July 26th, 2008

Google Rankings

When I was a kid we didn’t have a McDonalds in my home town. I heard lots of stories about McDonalds, how there was every type of burger you could imagine, the slick American service, the French fries which were totally different to what we called “chips”.

Finally, after years McDonalds did open. And I really didn’t like it that much. It was, in short, a huge disappointment.

Since I started this blog I’ve been very focused on Fab Swingers’s Google rankings and finally the site is showing up high for many of the key terms I’ve been targeting. I’m sure it’s no coincidence that the sudden move up has happened at the same time as Google Analytics has shown us passing the 1 million page views a day mark.

Yet, I feel slightly let down. The effect of the Google rankings has been tiny compared with the word of mouth and even the people who are joining via Google aren’t generally the people that bring most to the site.

I’m pleased that it’s finally happened but it turns out not to be a big deal.

I do like Sausage Egg McMuffins though.

July 20th, 2008

Rsync problems

I’ve had some problems with backup running very slowly — in fact by the time backup of the images has finished, it’s pretty much time to begin again.

However it turns out to be a cock-up on my part. I was using rsync which is only supposed to copy over files that have changed. But if you forget to use the -t or –times option then the modification dates are not copied over which means that every single file needs to be copied over the network to find out whether or not it has changed. Whoops!

As I now have two servers at The Planet which are in different data centers (Dallas and Houston) my backup plan is just to rsync the files between then.

I am also considering using rsync to allow my applications to upload photos to the assets server, however I may use Samba instead. NFS is out though!

UPDATE: After a great deal of thought over the past few weeks I’ve reached a decision. I thought about dozens of alternatives but I needed something very secure, robust (even when there is a network outage) and asynchronous with the file uploads so that the user isn’t kept hanging around when they are uploading their images.

The solution is that photos will upload on the applications server and also be served from there. This will result in fast, reliable upload. A cronjob will then rsync the photos over to the assets server every few minutes and will update the photos table to show that the photo has now moved over so future pages will get it from the assets server. I think this should work pretty well!

It is also quite future proof, if I switched to using a Content Delivery Networks (and I note that Peer1 allow adult sites on their CDN) then this approach would work great for uploading the images to it.

July 14th, 2008

Nigerian spammers targeting UK swingers sites?

I was just taking a look at the Alexa page for The Adulthub and I noticed that they had recently had a big traffic spike - from Nigeria!

Right now, Alexa is reporting 21.8% of the Adulthub’s users from that country. We have also had an increase in attempted spam from Nigeria on our site however our blanket ban on Africa has been quite effective at keeping them under control. I guess that they are just going through a phase of targeting swingers sites.

But to have over 20% of site traffic from spammers is pretty bad and must be annoying a lot of their users.

Adult Hub Nigerian Spam