Monday, October 06, 2008

EarthBrowser and Amazon cloud computing

First a quick note: EarthBrowser has been named a semi-finalist for the Adobe MAX awards. It would be great to get to the finals but I'm just honored to have EarthBrowser be recognized by Adobe as a great use of their amazing Flash technology.

As you may already know, EarthBrowser is a virtual globe that runs on the desktop but that is only half of the story. The other half of the EarthBrowser service is running on a set of servers that manage the website, licenses and data services. Data ingest, processing and management is a pretty big task, but is perhaps one of the most fun tasks that I do and is also the best way to create great content for EarthBrowser. EarthBrowser can read most KML files but unfortunately there aren't many great dynamic KML datasets out there. Perhaps because KML was designed for static content.

My favorite dataset in EarthBrowser is the precipitation forecast model, especially when paired with the tropical storms dataset. It gives a really interesting idea of the extent of the storm and also what is going on inside of it.

Richard Stallman of GNU fame believes that cloud computing is a "trap" but he is using a very narrow definition of the term. Cloud computing seems to be thrown out there to describe anything from web based applications to software as a service, the differences are obscure but important. At least for me, what cloud computing means is that it takes what was once a very time consuming administrative task and turns it into a programming problem.

Managing the software stack, configuration and uptime of these servers is cumbersome task to say the least. Designing an algorithm that can launch and maintain a fault-tolerant clustered server setup with no user interaction is an extremely attractive business advantage and also a very cool programming problem. I have been doing some extensive investigation of Amazon Web Services and am now planning on deploying my entire server side infrastructure on the AWS platform. I have been using the Amazon S3 storage service for quite some time because it is simple and just makes so much sense. With the recent release of the Elastic Block Store and Elastic IP Addresses, Amazon has a complete and very powerful solution for self-configuring web services that can also easily save and restore state in the event of a system crash.

So now in addition to EarthBrowser, I am working on a side project, I'm calling it FireAnt, that will create a server setup that can scale from one small instance to a full blown cluster of servers. It will scale up by adding new servers and as the load increases and scale back down as it decreases saving you money in the process. The base server is running a modified version of GeoDjango with Apache and PostGIS. Pound is used for web server proxying when scaling up and each server can take over for any of the others in the event of a system failure. I may release the code as open source, but I am a little worried about the time it would take me to manage an open source project. There is Scalr which is an open source project that will do something similar but it has a minimum of 4 server instances and it is based on PHP. Yuck.


Johan said...


For getting to the semis in the Adobe MAX awards!!!!

Now let's hope you get all the way to the final,
that would be great for you :-)

However, I'm very happy for your success so far.

Cheers, Johan

Justin Bronn said...

I'm curious as to how you modified GeoDjango. If I had to hazard a guess it would be to bolt on some sort of multi-database support. FWIW, this is something that is planned for Django as a whole in 1.1 (due in March).

If your changes are useful to other users in general, then it could be integrated with GeoDjango and it would be one less maintenance burden for you to worry about.

linda1porter said...

Excellent advices for future planning. I think everyone should plan for retirement. To save money in a young man is rational. I also try to set aside some of the money to the winners. I playing here and every time I try to save 10% on retirement.

Ida Wallace said...

Thanks for talking a time for sharing such kind of blog on EarthBrowser and amazon cloud computing and making aware us about EarthBrowser that is earth reproduction software. And this blog will be so much helpful to students who are taking marketing assignment help - from the academic experts of Australia’s top leading assignment assistance provider consultancy Assignment Help Folks as it contains very useful information about iPhone version of EarthBrowser and Amazon Web Services.