Rsync problems
I’ve had some problems with backup running very slowly — in fact by the time backup of the images has finished, it’s pretty much time to begin again.
However it turns out to be a cock-up on my part. I was using rsync which is only supposed to copy over files that have changed. But if you forget to use the -t or –times option then the modification dates are not copied over which means that every single file needs to be copied over the network to find out whether or not it has changed. Whoops!
As I now have two servers at The Planet which are in different data centers (Dallas and Houston) my backup plan is just to rsync the files between then.
I am also considering using rsync to allow my applications to upload photos to the assets server, however I may use Samba instead. NFS is out though!
UPDATE: After a great deal of thought over the past few weeks I’ve reached a decision. I thought about dozens of alternatives but I needed something very secure, robust (even when there is a network outage) and asynchronous with the file uploads so that the user isn’t kept hanging around when they are uploading their images.
The solution is that photos will upload on the applications server and also be served from there. This will result in fast, reliable upload. A cronjob will then rsync the photos over to the assets server every few minutes and will update the photos table to show that the photo has now moved over so future pages will get it from the assets server. I think this should work pretty well!
It is also quite future proof, if I switched to using a Content Delivery Networks (and I note that Peer1 allow adult sites on their CDN) then this approach would work great for uploading the images to it.