Thursday, March 08, 2007

Sweet Sweet Amazon S3

Another gigantic hurdle overcome! Last night I woke up around 4am and laid there thinking about all of the sweet simplifications possible. I've just signed up with Amazon S3 (Simple Storage Solution) and I think it just lowered my blood pressure a bit. The Coding Horror blog, which is fast becoming a favorite, details their efforts at reducing server load by putting static images on S3. I've heard about S3 since it has been out but thought it too complicated and didn't want to be a first adoptor, but now I'm sold.

I run two separate servers for EarthBrowser, one for the website and one as the dedicated data server for all of the EarthBrowser instances out there. Whenever I put out a release, my web server would get slammed and become unresponsive with thousands of people each downloading the 10 megabyte installer file. I have actually written and implemented, in PHP, a distributed, caching data server toolset that would allow me to rent several cheap $7 dreamhost setups and plop it in to increase my capacity. They would talk to each other and update their internal datasets for more frequently requested items. I was even considering finding some European or Australian hosting sites to get closer to the rest of the world in anticipation of EarthBrowser v3 coming out later this year. It was working pretty nicely but the administrative and logistical overhead was getting to be a bit overwhelming. Now I don't have to worry about storing and managing a group of servers to distribute EarthBrowser v3 data. With Amazon S3 I have virtually limitless capacity and limitless download bandwidth for about $0.20 a Gigabyte.

Now all that work can be thrown away, which makes me both cringe and sigh in relief at the same time. Often I work for a week or two on a technology and wind up throwing it away when something better comes along. Sometimes only through the act of creating and exploring a problem space can better alternatives be discovered. That is why product specs should never be static documents. This time the better solution came along and whacked me on the head.

It is a little tricky to get working well since it uses a SOAP or REST interface with some quirks. I wrote a python script that works with the excellent boto module to allow anyone with a S3 account to create/delete/update objects and buckets from the command line. I'm making it open source and downloadable for anyone who wants it. If you have any suggestions, let me know. There are some issues if you are using Python 2.3 or below, contact me and I'll tell you how to fix them.

2 comments:

marko said...

Here are a portion of the normal inquiries we get got some information about underwriter home credits. For more nitty gritty data about any of the accompanying, don't waver to get in contact. Car Title Loans

marko said...

Regardless of what kind of advance sharks you get stirred up with, removing yourself from their grip might be one of the greatest difficulties you will ever confront. Credit sharks, both the legitimate kind and the back rear way kind, are famous for pursuing down the cash they are owed. Cash Advance