EarthBrowser

EarthBrowser on the iPhone/iPad/iPod Touch

2010-10-05T10:19:00.000-07:00

It is hard to believe that I haven't posted to this blog for over a year. I'm not dead and neither is EarthBrowser. After a long hiatus where I've gained valuable skills (and salary) I have begun work on a new version of EarthBrowser. Working nights and weekends when possible.

Progress is coming along quite nicely since I am generating so much realtime content already however I have no release date yet. It's fun writing the shaders and being able to do coordinate projections right in the graphics hardware. I have to say that the limitations of the iOS are actually a blessing. Being able to focus on a small core set of features rather than trying to create the be-all software product is fun. It works amazingly fast on my iPhone 4 but the iPad is proving to be a bit slower than I'd like. I think it's a fill rate problem due to the large screen area.

Anyway, here's a screen shot:

If anyone still reads this old blog, please post your thoughts, wishes or any constructive ideas for what you'd like in the iPhone version of EarthBrowser. Now is the time to get your wishes out in the open.

Zoltar Available on iTunes

2009-05-14T15:32:00.000-07:00

Actually it's been available for about a week now but Apple didn't list it. I've run into a big problem with the iTunes store that a lot of other developers have been having. The result is that I've only had a few downloads, and Zoltar is much higher quality than most apps I see out there.

I hate to ask, but for anyone who follows this blog and has an iPhone, please go to the iTunes store and give Zoltar a great review. I'll buy you a beer the next time I see you!

SQLite C++ wrapper, so you don't have to

2009-05-14T13:45:00.000-07:00

SQLite is great. It is small, fast and easy to set up. However, if you are writing software that needs it, you're in for a bit of learning. You will need to learn what API calls to make and when to make them, usually by going over example code and using trial and error. I am introducing an open source SQLite C++ wrapper that will allow you to use the database without having to learn the SQLite API. It may save you several days of work.

The wrapper is just two C++ files, has a MIT style license and is platform independent. You will still need to link in SQLite 3.0 and learn the quirky SQLite SQL syntax.

You can download it here.

Creating a database and table

#include "sdsqlite.h"

void create_db(void)
{
   sd::sqlite database("mydb.db");
   database << "create table if not exists work (first_name text, last_name text, hours real)";
}

Database insertion

#include "sdsqlite.h"

struct work_data { char* first; char* last; float hours; };

work_data wdata[] = {
   {"Joe", "Smith", 2.5},
   ...
};

void insert_rows(void)
{
   try
   {
      sd::sqlite database("mydb.db");   // open the db with the table already created
      
      sd::sql insert_query(database);   // build an sql query
      insert_query << "insert into work (first_name, last_name, hours) VALUES(?, ?, ?)";
      
      database << "begin transaction";// create a transaction for speed
      
      // insert data (sdsqlite will auto-detect data type and execure query)
      for(int i=0;i<sizeof(wdata)/sizeof(work_data);++i)
         insert_query << wdata[i].first << wdata[i].last << wdata[i].hours;
         
      database << "commit transaction";// complete transaction
   }
   catch(sd::db_error& err)
   {
      // do something with error
   }
}

Database extraction

#include "sdsqlite.h"

void extract_name(const std::string& name)
{
   try
   {
      sd::sqlite database("mydb.db");   // open the db with the table already created
      
      // select all names that begin with the contents of the "name" variable
      sd::sql selquery(database);
      selquery << "select first_name, last_name, hours from work where first_name like ?" << name+"%";
      
      // extract the matching rows
      float hours;
      std::string first, last;
      while(selquery.step())
      {
         selquery >>   first >> last >> hours;
         
         // do something with the data
      }   
   }
   catch(sd::db_error& err)
   {
      // do something with error
   }
}

Zoltar the iPhone Fortune Teller

2009-04-30T09:00:00.000-07:00

I've been taking a break from EarthBrowser for the past couple of months and working on an iPhone app. The result is Zoltar the Fortune Teller.

Apple is now reviewing it for release in the iTunes store. I've heard it takes anywhere from a couple of days to several weeks if they don't send it back for fixes. Hopefully it will be available in the next week or so. I'll do another post when it is up.

You select a card from one of five categories and Zoltar will do a divination on his magic crystal ball. He then will speak your fortune to you from a set of over 100 fortunes based in part on the minor arcana of the Tarot deck. You can rotate him around and zoom in close with a swipe or pinch on the iPhone screen. I created the music and am the voice of Zoltar and put some spooky echo effects so it sounds pretty cool.

It was a really fun project which allowed me to test the limits of the iPhone hardware. The model of Zoltar was created by my cousin in 3d Studio Max and has over 10,000 polygons. Even so it renders at about 30 frames per second which is really good for such a small processor.

The 3d game engine that powers EarthBrowser was originally ported from C++/OpenGL into ActionScript. Since then I have made many improvements and I've re-ported it back to C++. Have I said lately how much I like OpenGL since I don't have to write my own 3d graphics library like I do in Flash? I built bindings into the Lua scripting language for ease of development. Lua has a strange syntax but it is lightweight and fast and gets the job done.

iPhone OS 3.0

2009-03-12T11:55:00.001-07:00

Engadget is reporting that version 3.0 of the iPhone OS will be announced on March 17th.

I'm excited to hear what is going to come out in the new OS, but rumors are that there will be a micropayments model, which if true will be huge. However it seems to me that typing in your iTunes password for each micropayment would be a bit too unpleasant.

iPhone vs. Flash Player smackdown

2009-03-11T14:30:00.000-07:00

Developing for the closed iPhone ecosystem is similar in many ways to Flash development. The graphics capabilities alone make it a superior platform. It doesn't have to be that way. I'm proposing three bold moves that would enable Flash to extend it's current lead in the RIA space and attract even more developers to the platform.

1) Explicit 3D graphics card support
Adobe can sandbox it all they like, but access to native high speed 3D rendering is one of the things that is making the iPhone such a hot platform. All that is really needed is to be able to pass on projection matrices, interleaved vertex buffer data, lighting and material settings to a graphics card and let it do it's thing. Make it detectable so developers can have a fallback if it's not available. Instantly web games become 1000% more possible.

2) A Javascript bridge a mile wide
ExternalInterface can get you where you need to go, with a lot of effort. Can you make it easier for us Adobe? I've taken the FABridge code, fixed some bugs (try passing the string \" from JS), and enabled the EarthBrowser plugin to be completely scriptable in Javascript. If Adobe could sit a few people down and nail down support for all browsers (including using VBScript to fake property getters and setters in IE7) then you could in effect create any Flash app completely in Javascript. This would open up a whole new world of possibilities for developers to create on the fly code that wouldn't have to be precompiled into a SWF. This would be HUGE.

3) An app store like iTunes
Along with great hardware, sleek design and serious developer tools the iPhone has a way for developers to monetize their creations. Adobe should do this for AIR apps and Flash plugins. Make a marketplace of website widgets and AIR apps that can be purchased for a few bucks. I have to administer my own servers and integrate a purchasing system with credit card companies, renew and manage a https certificates and deal with a lot of technical support not directly related to my software. It is a very small percentage of developers that are willing and able to spend the kind of time and money that it takes in order to sell stuff online. Apple has managed to create a system that does all of the heavy lifting of e-commerce and takes a healthy 30% cut for itself, but it's worth every percentage point.

Adobe has a spectacular platform that has a lot of room to grow. Flash market penetration is unreal and it is time for Adobe to get out of their overly cautious mindset and really create something new to help us developers create. Hire me, I'll direct the development.

Turned down for Where 2.0

2009-03-03T08:54:00.000-08:00

I'm a little disappointed that my Where 2.0 proposal was rejected. I thought that demonstrating a virtual globe that supports KML and Shapefiles, can do coordinate projections on large datasets and be completely customized with Javascript and embedded on any website would be somewhat interesting. I find it interesting.

They have some interesting talks planned to be sure, I suppose if I were a sponsoring corporation they might have taken me more seriously. At least they offered me a 25% discount on the $1000+ registration fee to attend, but I don't think I'll take them up on that.

Oh well, back to my iPhone app that has nothing to do with geospatial.

C++ vs AS3

2009-02-10T12:02:00.000-08:00

I've been working on an iPhone app for the past week or so and I'm pretty happy to be coding in C++ again I must say. C++ is not perfect by any means, in fact strong binding is a constant thorn in my side after the free and easy ways of Actionscript. Here are a few pros and cons that immediately stood out in moving between the two.

Pro C++:

Variable declaration ("int x" is much better than "var x:int")
Enums
Templates in general and template meta-programming specifically
#defines and conditional compilation
Speed of execution

Pro Actionscript 3:

Dynamic typing
Static constants of any type
Built in associative arrays
Built in UTF8 string support
Size of compiled code
Platform neutrality

In general I prefer C++. Perhaps because it's closer to the hardware and more things are possible, but you really have to know what you are doing to not spin your wheels too much. I *really* miss built in associative arrays and the awesome set of standard objects available in Flash however.

New KML extensions?

2009-02-03T09:30:00.000-08:00

There appears to be a few new KML extensions with the new Google Earth 5.0 release. As someone who has to keep up with my own implementation of the KML standard, this is of great interest to me. The new extensions seem to mostly be geared toward creating animated tours, a laudable goal. I think that is a great improvement in providing the ability for everyone to create more eye candy using GE. For some reason they just added a new namespace "gx" rather than putting out a new version of the spec. I guess this makes sense since the core functionality didn't change much and they can roll the new elements into the main spec in a future version.

The <gx:TimeSpan> and <gx:TimeStamp> tags are what most interest me. The spec quotes that they are merely copies of their standard KML namesakes, but that they "allow for the inclusion of time values in AbstractViews" and "Time values are used to control historical imagery, sunlight, and visibility of time-stamped Features". I don't know why they had to add copies in a different namespace when they could have just allowed the non gx versions to be included in the <Camera> and <LookAt> tags. What I think this all means is that you can now control the timeline of visible features using the <Camera> tag. A huge improvement. however I've spent a year or two going down the road of trying to create dynamic animations using static control values to hardwired algorithms. It will get you down the road a little ways, but it is a totally inflexible and brittle way of defining 3D animation.

I am a little disappointed that they didn't include time coordinates as I suggested last summer. I think that I mentioned at the time that I had no interest in spending time getting that feature through the appropriate committees to make it happen, but I think that at this point I probably would, even though I have less time available now. The time coordinates I proposed, based on modified julian date, are tremendously useful.

As an example, the new EarthBrowser Site Tracker uses them to define the animated visitor hits. Basically each visitor hit looks like this:

<Placemark>
    <name>...</name>
    <LineString>
        <coordinates>coord0 coord1 coord2 ...</coordinates>
        <ebr:time>mjd0 mjd1 mjd2 ...</ebr:time>
        <ebr:fade>.0625</ebr:fade>
    </LineString>
</Placemark>

The interesting parts are the <ebr:time> and <ebr:fade> tags. What they specify are the modified julian date time value at which each coordinate in the LineString occurs. This provides an easy way to animate a single placemark by using linear interpolation to calculate intermediate locations for smooth animation and also provides the ability to show a cool time based fade effect. The <ebr:fade> specifies that after 1.5 hours the line trace should fade. I use these two KML extensions with the Satellite orbits in the latest version of EarthBrowser and also in the site tracker tool. To do something roughly equivalent in current KML, you would have to provide thousands of Placemarks each with their own TimeSpan and geometry, there would be no smooth animation or fade effect and it would take many megabytes for each placemark. With these two simple extensions I can display some really interesting features with effects in a single placemark with a very small footprint. If anyone from the OGC or Google is interested in contacting me about helping these extensions get through to the next spec, please post a comment or send me an email.

Interesting EarthBrowser Site Tracker discoveries

2009-02-03T08:20:00.000-08:00

Last week I released the EarthBrowser Site Tracker, a free tool that gives an animated 3D global view of traffic to any website. I've been monitoring my own site traffic with it and have noticed some very interesting trends about my site, and perhaps some insight into the people who visit. It seems that I get slightly more European visitors than US visitors. Perhaps that is because EarthBrowser is in about 15 different languages and a lot of software is only in English. More people seem to visit a few hours after daybreak, perhaps when they get to their jobs, but in EuropeI've noticed that there seems to be a burst of visitors about an hour after nightfall.

The code was pretty simple and written in roughly 100 lines of Javascript which basically puts up the time slider, zoom scroller and loads the data file. I have some plans on improving the actual analysis and visualization of the data. I'd like to do trends such as visitors per country, browser language preferences, top countries and states and other fairly simple breakdowns of generic statistical data. But that will have to wait for a month or so, as will the open Javascript/Flex EarthBrowser API.

I am learning yet another new platform to program on. I got myself an iPod Touch, which is like an iPhone without the phone. I've just started porting some of my OpenGL framework from a mothballed version of EarthBrowser to OpenGL ES for a couple of new projects, one I'm doing as a contract. The new apps are really cool and a lot of people are going to love them. I'm not so impressed with most of the iPhone apps I've seen so far sadly, Google Earth is by far the best. Perhaps it is just hard to find good apps when there are tens of thousands out there, hopefully mine will be compelling enough to get some attention.

Free EarthBrowser 3D website traffic visualizer

2009-01-30T06:00:00.000-08:00

Introducing the EarthBrowser Site Tracker. It shows where and when people visit your site from all over the world. Play your traffic back and forth with the timeline slider, zoom in and out and see aggregated website stats that are updated every 10 minutes.

It's free to put on any website, just customize it and copy/paste some text into your HTML and you are up and running, no signup necessary.

Here's a live version:

This is one of many new free tools that I'm going to be releasing that showcase the upcoming EarthBrowser Flash plugin. I'll be putting out another nice embed tool that will be customizable to show any set of orbiting satellites and will zoom from one to the next.

Is anyone else tired of the GIS Inauguration tie-ins?

2009-01-23T14:45:00.000-08:00

Microsoft, Google, GeoEye, DigitalGlobe, AEgis, CNN and more have all jumped on the bandwagon. Shameless event tie-in marketing. Let's think of some other tremendously useful things to do with GIS.

What the world needs now are more Santa trackers.

Simple solution for MOUSE_WHEEL events on Mac

2009-01-22T12:45:00.000-08:00

Flash does not support the MOUSE_WHEEL event for Macs. I was hoping it would be supported in version 9, and again in Flash 10 but I'm still waiting. There have been a number of solutions out there, and the best one I've found is Gabriel Bucknall's MacMouseWheel code. It consists of an actionscript class and some javascript code that will work together to have the browser pass the mouse wheel events to your Flash code.

After reading an awesome article by Peter McBride called JavaScript and VBScript Injection in ActionScript 3, I've put together a single as3 class that fixes the MOUSE_WHEEL problem without the need for any external javascript code. It works on Safari, FireFox and Opera and probably Chrome when it becomes available on the Mac.

The trick was to inject the mouse handling javascript code using ExternalInterface. The code finds the DOM node for itself, intercepts and passes the mouse wheel events which are then disributed to the current InteractiveObject under the cursor.

An example of initialization would be:


class awesomeapp extends Sprite {
  public function awesomeapp() {
    addEventListener(Event.ADDED_TO_STAGE, stage_init);
  }
  private function stage_init(e:Event):void {
    MacMouseWheelHandler.init(stage);
    addEventListener(MouseEvent.MOUSE_WHEEL, wheel_event);
  }
}

Download the source (MIT license).

A 3D virtual globe for your website

2009-01-16T11:07:00.000-08:00

It is nearly impossible to get people to download and install software, I know from experience. Even Google Earth is only installed on a few million computers. Installing software or browser plugins is not allowed in most corporate and government environments. If you want to display geospatial content on your website and not exclude most users, your only choice until now has been flat maps.

With EarthBrowser, you can have a true 3D virtual globe that is self-branded with a custom interface that fits your special data requirements right on your website. There is no installation required since Flash 9 has about 95% penetration. With just a few lines of javascript (or as part of a Flex app) you can make your site really stand out.

</marketing>

The first example of the EarthBrowser Flash plugin is up at earthbrowser.com with a simple interface that gives a hint at what is in the desktop AIR application and an easy way to download and install it. Download the free trial AIR app and take a look at all the great features. Chip in a few bucks if you like it and want to support my efforts to provide an alternative to the multi-billion dollar behemoths out there.

A small javascript program (about 350 lines) controls what data is visible and various aspects of the globe on that site. It lets you control the clock and camera elevation and toggle night shadows, clouds, doppler radar, earthquakes, snow depth and a set of 7 popular satellites which are all updated in realtime. Click on a satellite and it will zoom to it and smoothly animate the earth underneath in real-time using very accurate orbital elements (I've got permission to re-distribute satellite positional data from the US Space Command).

In my last post I mentioned that the demo app would not work on Internet Explorer due to problems with the JScript engine. Since Javascript is very close to Actionscript, I just made a few changes to the javascript code (adding ":*" after variable and function declarations) to make it compile into the Flash preloader. Now it works in Explorer too. However the special handling of mouse-wheel events isn't as nice so I'll have to hurry up on my VB getter/setter workaround I mentioned earlier.

The EarthBrowser plugin can read KML files and KMZ archives. It can also read Shapefiles and do coordinate projections using a ported version of the open source proj.4 library. There is a full set of GUI components including buttons, text input boxes, sliders, popups and scrolling lists that make creating a fully featured app simple. The desktop version is a pretty complex application but is written in under 4,000 lines of javascript. I'm working on the API documentation and am making a code repository so people can create and share their own components for use on the web or in the desktop version. As an example, adding one of the Free ESRI map layers to the globe takes about 3 lines of code.

I haven't mentioned it yet since I've not blogged in a while, but EarthBrowser did not win at the Adobe MAX show. Scrapblog won, they have a pretty cool app with a very well defined and active niche. It was a great experience and I met some really cool people. By the way, Ben Forta is *way* more impressive in person than his blog picture suggests.

EarthBrowser 3.1 written in Javascript

2009-01-14T09:45:00.000-08:00

The new version of EarthBrowser which will be released later today has been written in Javascript. Can that be right? Yes and no. The entire control structure of the application has been re-written in less than 4000 lines of Javascript code using the new EarthBrowser plugin. However, all of the fast rendering and calculation code is still written in Flash (Actionscript 3).

Desktop Version
Version 3.1 has had a major facelift, is about twice as fast as the previous version and has some pretty cool new datasets like real-time satellites. This is a free update for all version 3 users of course, just starting up EarthBrowser will allow a seamless update once I put it out.

Online Version
The online version API is still not quite ready for release (I'm only one guy). However, later today when I release, please check out the EarthBrowser home page to see the first public demo of the online plugin. Right now it only works in Safari and Firefox. I've created a really simple 600 line demo application by simply copying some Javascript code from the desktop version and putting it up on the home page.

Writing your own modules
In November I alluded to being able to customize EarthBrowser with Javascript. I am working on creating a code repository with many examples on how to extend and customize EarthBrowser with your own data and user interface elements. The code will be portable and can be developed and tested on the desktop and then, if you like, posted to the web for the same functionality on your website.

I still need to fully document the API. Also I am working on support for Internet Explorer, which is tricky because Explorer does not have getter/setter methods. It uses it's own lame and non-standard JScript engine. However there is a trick I found in the Classical inheritance for Javascript code which uses Visual Basic to provide getter/setter functionality. A clever but lame hack, but it should work until Explorer 8 comes out and becomes mainstream.

Customized web/desktop hybrid apps with EarthBrowser and AIR 1.5

2008-11-17T10:05:00.000-08:00

Adobe AIR 1.5 is being released. It uses the SquirrelFish Javascript accelerator which is really cool. With the upcoming EarthBrowser Javascript API, having a fast Javascript engine will make it much easier to create not only customizable web applications with the EarthBrowser virtual globe embedded, but also a customizable desktop app using your same code base with the ability to work offline and have hooks into an embedded database.

What the? Creating your own AJAX style web app with the EarthBrowser API will be just a small step away from creating your own desktop application with your geo-content.

More on this later.

Live blogging Adobe MAX show

2008-11-17T09:41:00.000-08:00

For some reason I'm off the Adobe feed aggregator, which is a bit of a bummer but I'll post about what's going on anyway.

Adobe Flash Catalyst? Don't know what it is yet but it's just been announced alongside Flash and Flex Builder.

Announcing an association with U2's Red initiative called Red Wire. Five bucks a month seems like a lot, at least for me, for an online news magazine. Goes to a very good cause though.

So far my favorite part was getting the high score on the free Asteroids machine in the lobby (19,710). Old School.

Showing off EarthBrowser Web Plugin at MAX

2008-11-14T15:50:00.000-08:00

EarthBrowser was chosen as one of the finalists for Adobe Rich Internet Application of the year, I am very honored by the selection.

Please vote for EarthBrowser for the People's Choice Award (near the bottom):
http://adobemax08.com/na/experience/#?s=5&p=3

I've been working very hard to get the EarthBrowser Javascript API ready to show off at Adobe MAX next week. The API is nearly done and it is much more powerful than I first imagined it. With it you can put EarthBrowser directly on your website and control every aspect of the user interface, globe, camera, clock from a HTML script tag. It understands KML, JSON, XML, AMF and even Shapefiles with projected coordinates (I ported the proj.4 library to Actionscript). There is also the ability to smoothly animate features based on time coordinates for which I gave a Google "tech talk" about in July.

I've put up a small with the demo that I plan on showing off at MAX.
Please take a look here

All of that demo is written in Javascript in a couple of days. I'll be releasing the API documents and a free non-commercial license keys in the next few weeks when I've put the final polish on it.

Special thanks to Lucian Plesea for the help with the OnEarth WMS layer help and a big thanks to Peter Minton for his great work on the island shapefile dataset.

EarthBrowser at the Adobe MAX awards

2008-10-15T11:30:00.000-07:00

EarthBrowser has been selected as a finalist in the Rich Internet Application category of the Adobe MAX awards.

If you are going to be at MAX, please stop by and say hello. I believe I'll be showing off the latest version of EarthBrowser from 6:30 - 8:30pm in the Moscone center on Monday Nov 17th.

I have to say that I'm pretty excited to be attending Adobe MAX this year. Flash/AIR is a very impressive platform for applications development, I'm pretty sure that I'll never go back to doing a GUI application in C++. The coding team for Flash Player and AIR have my highest respect for enabling such a powerful tool.

I hope to have the EarthBrowser website plugin API finalized and ready for the public by the time MAX starts. With it you will be able to create your own customized and self-branded virtual globe on your own website. Several companies have already contacted me about developing solutions for their geospatial offerings based on EarthBrowser. I also just finished a project for Autodesk to have EarthBrowser running on a kiosk with a 40 foot projection wall in their new San Francisco Customer Care Center. I hope to drop by and check it out when I'm down there, but the MAX show is keeping me pretty busy...

EarthBrowser and Amazon cloud computing

2008-10-06T09:30:00.000-07:00

First a quick note: EarthBrowser has been named a semi-finalist for the Adobe MAX awards. It would be great to get to the finals but I'm just honored to have EarthBrowser be recognized by Adobe as a great use of their amazing Flash technology.

As you may already know, EarthBrowser is a virtual globe that runs on the desktop but that is only half of the story. The other half of the EarthBrowser service is running on a set of servers that manage the website, licenses and data services. Data ingest, processing and management is a pretty big task, but is perhaps one of the most fun tasks that I do and is also the best way to create great content for EarthBrowser. EarthBrowser can read most KML files but unfortunately there aren't many great dynamic KML datasets out there. Perhaps because KML was designed for static content.

My favorite dataset in EarthBrowser is the precipitation forecast model, especially when paired with the tropical storms dataset. It gives a really interesting idea of the extent of the storm and also what is going on inside of it.

Richard Stallman of GNU fame believes that cloud computing is a "trap" but he is using a very narrow definition of the term. Cloud computing seems to be thrown out there to describe anything from web based applications to software as a service, the differences are obscure but important. At least for me, what cloud computing means is that it takes what was once a very time consuming administrative task and turns it into a programming problem.

Managing the software stack, configuration and uptime of these servers is cumbersome task to say the least. Designing an algorithm that can launch and maintain a fault-tolerant clustered server setup with no user interaction is an extremely attractive business advantage and also a very cool programming problem. I have been doing some extensive investigation of Amazon Web Services and am now planning on deploying my entire server side infrastructure on the AWS platform. I have been using the Amazon S3 storage service for quite some time because it is simple and just makes so much sense. With the recent release of the Elastic Block Store and Elastic IP Addresses, Amazon has a complete and very powerful solution for self-configuring web services that can also easily save and restore state in the event of a system crash.

So now in addition to EarthBrowser, I am working on a side project, I'm calling it FireAnt, that will create a server setup that can scale from one small instance to a full blown cluster of servers. It will scale up by adding new servers and as the load increases and scale back down as it decreases saving you money in the process. The base server is running a modified version of GeoDjango with Apache and PostGIS. Pound is used for web server proxying when scaling up and each server can take over for any of the others in the event of a system failure. I may release the code as open source, but I am a little worried about the time it would take me to manage an open source project. There is Scalr which is an open source project that will do something similar but it has a minimum of 4 server instances and it is based on PHP. Yuck.

EarthBrowser web plugin satellite example

2008-08-21T12:10:00.000-07:00

Good news! The US Space Command has just granted my request to redistribute satellite orbit data. Expect to see satellites coming to EarthBrowser soon.

Here's the first public example of the EarthBrowser plugin, it requires Flash 9. it shows the current position of the International Space Station. Use your mouse wheel if you have one, or click the globe and use arrow keys to zoom out and see the Hubble Space Telescope and the TOPEX/Poseidon satellites.

This is a simple example with no extra datasets or interface elements and limited zooming. I hope to have the Javascript API available sometime next month.

Imagine a zero-install, completely programmable virtual globe that understands KML on your website.

My iTunes Pirate Album

2008-08-06T15:45:00.000-07:00

My alter-ego has been revealed...

I'm the drummer first mate on the pirate rock band HuckleScary Finn captained by my best friend Kevin Hendrickson. I'm afraid that I can't take credit for the huge resurgence in pirate themed activities here in the Portland area, but I'd like to think I got in on the ground floor.

Notable Portland Oregon Pirate links:
• Portland Pirate Festival
• Captain Bogg & Salty (best pirate band in the world).
• International talk like a pirate day
• Captain Henry's Pirate Store (a pirate merchandise only store)

Arrrrrrrrr

What's in the EarthBrowser pipeline?

2008-07-23T10:40:00.000-07:00

If you aren't interested in wading through my sleep inducing 30 minute Google presentation, I'm planning on detailing much of what I said in a series of blog posts.

Contrary to almost every bit of advice I've been given I'm going to lay most of my cards out on the table about the future of EarthBrowser. For a single guy trying to go up against multi-billion dollar corporations, I need as much attention as I can get. However I'm not going to reveal everything, I've got some pretty cool things under wraps for release later this year.

Flash 10: I'm very excited about Flash 10 and believe that it will open the door for real-time 3D in the web browser. For me the most important aspects of F10 are native matrix operations, hardware accelerated rendering and a shading language. This will enable EarthBrowser to have true 3D navigable terrain with mountains, valleys and even underwater exploration, right in your web browser or in the desktop authoring app. I am hoping to get a 50-100% increase in rendering speed and be able to do really pretty things like normal mapping and atmospheric scattering.

3D Collada models: I'm not really a big fan of the obsession by Google and MS to put accurate 3D buildings on the earth, but if you have the data it isn't really hard to do. I've got more interesting uses for 3D models which will become more apparent in future versions.

GPS track importing: A very easy feature to add for people to edit and share.

iPhone version: I still hold out hope for Apple and Adobe to get Flash running on the iPhone so I haven't allocated much of my bandwith to this project yet. However after some exploration of the problem space, I estimate it will be relatively simple to port from my current codebase due to the nicely abstracted rendering backend.

My next blog post will detail the geometry / time coordinate extension that I hope will make it into the KML standard at some point. Time coordinates are designed for use in feature animation, which is an obvious use but was never a design consideration for KML. I don't really have the time or desire to submit and shepherd a proposal through an OGC standards committee. Perhaps there could be something like the Boost Library for KML where proposed extensions are selected to become part of the standard after being tested and refined by use in the community. Seems like this would be a great place for me and the WorldWind developers to start taking a little initiative and advance the state of the art for virtual globes.

EarthBrowser presentation at Google

2008-07-21T12:50:00.000-07:00

Last Tuesday I gave a 30 minute presentation at Google about the upcoming EarthBrowser Flash plugin. I also presented an enhancement to improve the expressiveness and utility of KML along with a demonstration with orbiting satellites. Many thanks to Michael Weiss-Malik and the great guys at Google for inviting me to talk about whatever I wanted to!

If you have problems with the embedded video you can see it on youtube.

In the presentation I go through some of the features and benefits of the EarthBrowser Flash plugin. Then I talk about giving time it's own coordinate in the geometry primitives in order to reduce KML file size and complexity as well as add new functionality. Toward the end I talk about future enhancements to the EarthBrowser desktop/web platform. I also mention that I'm developing an iPhone version.

Wow! C/C++, Python and Ruby running in Flash Player

2008-07-09T08:40:00.000-07:00

Earlier this year I saw the demo of Quake running in Flash directly translated from C++. It was very impressive to say the least, Adobe has some great coders and the amazing advantage of having their code running on almost every computer with a web browser.

I just came across this post that talks about an open source backend to the LLVM that will enable C/C++, Python, Ruby, Perl, Lua and other languages to be targeted to ActionScript bytecode (the language the flash player runs). It sounds like it will need some more work and support of a new Flash player version but the possibilities are extremely exciting.

More detailed information on the initial LLVM work by Ryan Stewart here.

KML Extension, Julian Time

2008-06-19T14:30:00.000-07:00

Creating a virtual globe that can view KML files as well as creating numerous datasets that have to use KML has given me what I feel is a unique perspective on the strengths and weaknesses of the language. It also gives me a really unique opportunity to enact some features in the language that I feel are needed. Hopefully at some point the OGC and Google will consider some of my suggestions once they are more formalized.

I've just come up with a *really* nice extension to KML that can drastically reduce the size of some datasets (e.g.: a 1.2 MB dataset is reduced to 60K) and at the same time enable functionality that is sorely missing. I'm not going to spill the beans on it quite yet since I've got another couple of enhancements that will enable even more great functionality. I'll present this extension and others when I give my talk at Google on the 15th of July. I'll post a video of that talk on this blog once its available.

Julian Time:
One of the less fortunate aspects of KML, in my opinion, is it's use of the ISO 8601 Date and Time Format. I understand why they did it; because it is human readable and a standard to hang their hat on. The processing overhead on parsing the date is significant, but not a deal breaker so why do I care?

Clearly the ISO time format should be left in, but there should be another option for specifying it as Julian Time. There are so many nice properties of julian time that I would be very surprised if Google Earth, Virtual Earth, ESRI and any other virtual globes don't all use it for their internal representation of time. It is a floating point number (usually a 64 bit double) that can be manipulated with standard mathematical functions. Anyone who has ever written a date library the naive way (as I have) has come to dislike all of the little details like days in a month, leap years and worst of all... time zones. Why not standardize on a time format that does away with all of that dreck and can be operated on like a regular number? You can add 3.5 days, subtract an hour and do simple comparison operations like greater than, less than and equal. I can't imagine ever using another internal representation.

Here's a conversion routine in Python:


# gdate is (year, month [1-12], day, hour, minute, second)
def julian_date(gdate):
 if gdate[1] < 3:
  M = gdate[1] + 12
  Y = gdate[0] - 1
 else:
  M = gdate[1]
  Y = gdate[0]
 D = gdate[2] + (gdate[3] / 24.0) + (gdate[4] / 1440.0) + (gdate[5] / 86400.0)
 A = math.floor(Y/100.0)
 RV = math.floor(365.25*(Y+4716.0)) + math.floor(30.6001*(M+1.0))
 return RV + D + (2.0-A+math.floor(A/4.0)) - 1524.5

Google open sources Gears

2008-06-13T06:31:00.001-07:00

Google has open sourced it's web plugin Gears under a modified BSD license. Normally I wouldn't comment on something like this because it seems unrelated to what I'm doing with EarthBrowser. However I believe that it could be very important to the future direction of EarthBrowser. A web plugin that supports all major browsers on all major platforms is a *very* large project, one I would never attempt myself. I am kind of in shock that Google would make it so easy for others to make web plugins by giving this away under such a permissive license.

A really long time ago, way back when in the days of Netscape before Internet Explorer even came around, I had a version of EarthBrowser working as a web plugin. It had too many problems with memory limits on the old MacOS 7 and 8 so I canned the project and went with a desktop app. So now there is an open source web plugin code base that I can use to port my OpenGL version of EarthBrowser right into a website. I probably won't tackle that particular project until next year, but it is good to know that it is now possible.

I am very impressed with the release of this code, this is such a different way of doing business. I don't see any benefit for them in doing this, but there isn't any harm either. Way to share knowledge Google!

KML to JSKML translator now available

2008-06-12T12:34:00.000-07:00

It's now available at the JSKML Site run though the Google App Engine.

I've created a KML parser that will transform a KML document into the JSKML format, or alternately a JSON formatted JSKML document if that is what you require. It's a simple copy and paste translator similar to Dean Edwards excellent Javascript Packer.

It currently works on the KML 2.2 standard and could use a little optimization but it is functional and easy to use. If you give it a try and find an error, please let me know.

jskml: Javascript KML Dialect

2008-06-09T15:15:00.000-07:00

KML data structures need to be represented in other formats than just XML. With virtual globes moving into the web browser, the need for an alternate representation that does not require parsing to go from XML text into Javascript objects is an important step, in my opinion, in simplifying web based scripting of online mapping.

I have created a simple static website describing the new dialect at:
http://www.jskml.org.

I used Google App Engine and the Django framework to create this site. I've got another exciting app or two planned for deployment using this platform and this website was a good way for me to orient myself as well as put out a format that I feel should be supported.

Thoughts on the Google earth plugin API

2008-06-06T09:59:00.000-07:00

Good job Google earth/maps team.

While it feels like they rushed the introduction of the earth plugin a bit, leaving pretty spotty documentation of the API and only a Windows version, it is indeed a great first step. Putting javascript tendrils deep into the control structure of the plugin was inspired and will be transformative.

However, the programmer in me can't help but be a little offended by the size and verbosity of the interface. Am I to understand that the very first example of creating a placemark is what will be needed for such a simple task?

var placemark = ge.createPlacemark('');
placemark.setName("You are at Google");
ge.getFeatures().appendChild(placemark);

// Create style map for placemark
var normal = ge.createIcon('');
normal.setHref('http://maps.google.com/mapfiles/kml/paddle/red-circle.png');
var iconNormal = ge.createStyle('');
iconNormal.getIconStyle().setIcon(normal);
var highlight = ge.createIcon('');
highlight.setHref('http://maps.google.com/mapfiles/kml/paddle/red-circle.png');
var iconHighlight = ge.createStyle('');
iconHighlight.getIconStyle().setIcon(highlight);
var styleMap = ge.createStyleMap('');
styleMap.setNormalStyle(iconNormal);
styleMap.setHighlightStyle(iconHighlight);
placemark.setStyleSelector(styleMap);

// Create point
var la = ge.getView().copyAsLookAt(ge.ALTITUDE_RELATIVE_TO_GROUND);
var point = ge.createPoint('');
point.setLatitude(la.getLatitude());
point.setLongitude(la.getLongitude());
placemark.setGeometry(point);

Admittedly it uses the super-annoying StyleMap which I feel should be abolished. Don't you think it is too much to ask to perform this kind of operation for every placemark? I suppose someone could create a wrapper library for this to simplify common tasks.

How about implementing this in your API instead:

ge.addPlacemark({
    name:'You are at Google',
    Style:{IconStyle:{Icon:{href:'http://maps.google.com/mapfiles/kml/paddle/red-circle.png'}}},
    Point:{coordinates:{lon,lat,alt}}
});

Could the maps/earth team create the nicer interface wrapper?

EarthBrowser is Adobe site of the day, Google talk

2008-06-04T21:30:00.000-07:00

I don't have the marketing megaphone that a Google has but EarthBrowser is starting to get a little attention in the Flash world. I was just notified today that earthbrowser.com was selected as the Flash site of the day. There was also a nice article recently on RIApedia.com titled EarthBrowser - 3D AIR App in Flash.

Google, via Michael Weiss-Malik, has invited me to come down and deliver a presentation about EarthBrowser, KML and whatever else I want to talk about. I guess it will be something like the Google Tech Talks where they make a YouTube video of the presentation. I've been consolidating things on version 3 for the past month so I haven't set a date yet but I'm pretty sure it will be mid July if that is available for them. Hopefully those guys aren't too miffed about my KML criticisms, or my out of date Google Earth criticisms. Perhaps I should bring a rotten tomato shield.

I've got a little project I'm hoping to unveil at or before the talk, but I'll leave everyone in suspense about what that could be...

Cyclone Nargis and GDACS

2008-05-06T11:00:00.000-07:00

I've added a few new datasets to EarthBrowser's "Featured External Layers", which is below the main dataset element in the EarthBrowser control window.

I'm temporarily putting in a high resolution MODIS overlay of Cyclone Nargis from earthobservatory.nasa.gov. It's a really large image so be patient, it takes a little while to download. Due to a bug in Flash for large images scaled to be small, it doesn't look very good far away, but looks better as you zoom in. Hopefully they've got a fix for this in the works.

I've also added the Global Disaster Alert and Coordination System kml dataset which is updated with real-time information for ongoing disasters and recovery around the globe. To view the information for each disaster, you have to navigate through the list of elements in the control window, I wish they would put in placemarks to view the great information they have more easily.

Finally, the Europe Media Monitor layer is a news aggregation service that plots global news stories on a map and provides an overview and link to the geo-located story.

I'm also hoping to have an update out later this week with a timezone fix for sunrise/sunsets and also a global grid overlay feature.

EarthBrowser 3.0, 5 days in

2008-05-05T09:00:00.000-07:00

Version 3 is doing very well over the past 5 days. It's selling better than I'd hoped for, the django based system I am using for the main website and for the separate data server have held up incredibly well with just a few hiccups. Amazon S3 really helps me to take the load off of my servers. I've got a handle on the tech support emails over the weekend, there are still a lot left however. I try to answer each and every person with a question or problem with a personal message. I think that kind of respect and care for each customer has really helped the business over the years. My goal is to have no unhappy customers, if I can't fix your problem or you are still not satisfied, you get a refund, it's pretty simple.

Thanks for the mention James, Mickey, Lxnyce and Bull.

Some people are wondering why would they should support EarthBrowser as shareware when they can get Google Earth, Microsoft's Virtual Earth and NASA WorldWind for free. Well you get EarthBrowser for free too, you are just reminded to help support the future development of the shareware if you like it enough to use it. Google Earth is great but it isn't your computer's operating system, you CAN use more than one virtual globe at the same time.

EarthBrowser may not have 2 inch resolution datasets, but the weather layers really blow away anything the other globes offer, in fact the weather forecast layers blow away most of what I've seen from the major weather providers offer. I'm talking with CustomWeather to see if we can do some sort of exchange of data for perhaps exclusive use of EarthBrowser on their website. I think that would really make them stand out from their competitors.

KML is great! New datasets are being added to EarthBrowser all the time, if you only know where to find them. Today I noticed from one of my favorite blogs Frank Taylor's Google Earth Blog that there is a new dataset available from NASA showing the total electron count in the earth's ionosphere. I like it so much I think I'll add it to the Featured Datasets in EarthBrowser. The red areas indicate a large concentration of electrons which may cause problems with communications. It is neat to see that the electrons build up on the daytime side of the earth and disperse shortly after darknes.

If you want to see the animated link of the ionosphere for the past 24 hours in EarthBrowser just drag this link onto the EarthBrowser window. Or if you are reading this blog with EarthBrowser's embedded web-browser, just click on the link. There is also a light weight version of this file that just shows the current condition.

EarthBrowser 3.0

2008-04-30T14:00:00.000-07:00

EarthBrowser 3.0

Finally, it's out. Thank you to all of my wonderful supporters!!!

I transferred over the domain name to the development server and within about 20 minutes VersionTracker picked it up and I'm getting sales already.

Steve Wozniak was the 11th person to register version 3. He's been great to me and has bought several site licenses for schools. Maybe someday I'll be able to talk with the guy...

I have to give a tip of my hat to the Papervision 3D guys. They were my initial inspiration for the flash version, and although I wound up writing my own specialized rendering code, they deserve a lot of credit for what they have done! I'll go into details about how I did some things in flash in future posts, it has never been easy or fast to do software rendering. I'll also talk about some of my plans for the future.

Right now I'm sipping on a Paulaner Salvator Dopple Bock and basking in the euphoria of 4.5 years of hard work coming to fruition.

EarthBrowser 3.0 goes live tomorrow

2008-04-29T15:30:00.000-07:00

Hopefully people will like it.

It's shareware, which means you can use it for free for as long as you like. However, I'm funding it out of my own pocket, which is pretty much empty at this point. So if you want all the features and don't want to be annoyed by reminders to register please pay the modest fee and help me to make it better and better. Upgrades discounted for current users of course. Oh, by the way, registered users will be able to embed EarthBrowser with their own content on their personal websites within the next several weeks.

I just wanted to say thank you to everyone who has been so encouraging to me over the past few years; my wife, children and friends. I couldn't have done it without you.

Do I seem frustrated and bitter?

2008-04-27T08:16:00.000-07:00

I was just called out on the tone of my blog posts by someone I've known and respected for a long time. He said that he was a little surprised by my writing after reading my blog for the first time. I seemed bitter and frustrated, perhaps mostly at Google, and that isn't how he knows me. I guess I'm a little bitter, Google pretty much put me out of business years ago by releasing Google Earth for free. Hopefully EarthBrowser 3.0 will attract enough customers to keep my family stocked up on food, diapers and other necessities.

I have taken a confrontational approach to many of my posts, some of this tone is conscious, some isn't. I actually try to be a little controversial and opinionated on the blog because I'm just some joe-schmo and nobody wants to read some random guy's opinions that are mushy and congenial. That's not exciting or even interesting.

Unfortunately when writing in the blog some of my gigantic ego shows through which generally wouldn't happen in a conversation with me. I think that most programmers have really big egos, but mostly about their own coding abilities. In a self-preservation kind of way, I have to think that I'm a good programmer to tackle such a large problem by myself. I think that is just the nature of programming, no confidence = no code.

That said, I'm going to try to tone it down. Sorry libKML guys, hope I haven't offended anyone, I realize that it is just an alpha version.

KML, libkml and the "standard" mistake

2008-04-14T16:51:00.000-07:00

Passing off KML to the OGC so it could become a "standard" was a big mistake for Google.

I consider myself well versed in the KML format since I just implemented most of it's features, and found them to be needing much more in the way of styling. KML will now change at a glacial rate due to the standards process, right when it needs to change the most! Google Earth's feature set will now become dictated by an outside entity, with their input of course, but that's no way to develop software! Why would they cripple Google Earth like that?

Also libkml has been released and it was exactly what I thought it would be, a glorified xml validation script for the kml dialect. I predict that no significant software will choose to link that library in.

I have to say the lamest thing about KML is the whole Style/StyleMap tag collection that enables one to set a separate style on an icon for mouse-over events. It's a great way to have a non-standardized interface since everyone rolls their own mouse-over effect for each placemark style!

I've implemented a few of my own extensions to the KML format which I will go into detail about later. They are mostly aimed at visibility and styling extensions. One of the most useful extensions is the tag which allows one to control the visibility of a Feature element (Placemark or GroundOverlay) globally based on camera elevation. This is much easier than setting up one of those elements, especially for a single placemark.

EarthBrowser 3.0 Imminent

2008-04-08T16:10:00.000-07:00

EarthBrowser 3.0 is currently in beta testing and I think that it might live up to my prediction to revolutionize virtual globes. It should be released within the next two weeks.

Floundering In The Past

I have been working on version 3 of EarthBrowser since 2004 and it is finally coming to fruition. Up until about a year ago, I was still working on a version that was based on OpenGL with a Python scripting engine. It was basically a new platform that would do seamless version updates, enable users to extend the interface programmatically, do extremely fast raster and vector map projections on multi-gigabyte data files using JPEG-2000 incremental decompression. Use a hexagonal dataset grid to feed the fragment shader based clipmap engine for smooth panning around the poles with no perspective distortion. I was also downloading, pan-sharpening and color-space correcting the Landsat 15m dataset. Bit-torrent distributed datasets were integrated, relief mapping and atmospheric diffusion were supported. It really is a masterpiece, but then Google Earth was given away for free and sales of version 2 began to dry up as I was digging myself deeper and deeper.

The Turning Point

Then along came Modest Maps which introduced me to Flash programming. Once again plunging into yet another new technology, I decided to make a quick sliding map with a few datasets for people to put on their websites for free advertising. Then I saw Papervision3D and realized that I could do a globe. It was a little too slow for my needs so I ported my C++ game engine kernel over to Flash. I kept adding features expecting to hit the limit of what was possible with Flash and I never did. Around June I put the OpenGL version on the back burner and began working full time on a web version of EarthBrowser.

Unexpected Windfall

This February Adobe released Adobe AIR which enabled me to bring my EarthBrowser back to the desktop. AIR has some *really* nice features that make EarthBrowser much more powerful than I imagined it could be in my last post. There is an integrated web browser, right within EarthBrowser 3.0 now. That is huge and you won't really understand how easy it makes things until you browse some geo-websites and drag and drop KML links right into your placemarks folder or look at the Wikipedia page for a city or country.

Weather Datasets

There are a lot of other nice features that you won't see in Google Earth or Microsoft's Virtual Earth. I've been creating a lot of real-time datasets generated from the NOAA Forecast Models to give regions of rainfall, snowfall, humidity, temperature and many other measurements which are all animated across the globe with an intuitive time slider. Continental US doppler radar, earthquakes, volcanoes, webcams and many more datasets are all there and animated too.

Finally: KML Integration

KML support in EarthBrowser 3.0 is really nice and intuitive. There is a Panarimio KML file that allows you to see pictures from all over the world as you zoom in closer. You can download a KML or KMZ file directly within the embedded browser or drag and drop it from your desktop. EarthBrowser even has some extensions to the KML format that I felt were missing and greatly enhance the expressiveness of the format. But I'll save that for another post.

KML Mashup Tool

EarthBrowser 3.0 has really been designed to be a KML authoring and mashup tool. If you want to save a single feature from a dataset, just drag a placemark icon from the globe right into your placemarks folder and it will make a copy. You can drag out any combination of items in your placemark folder onto a text editor to make a custom KML mashup file to post it on your own website or share it with your friends. Version 3.1 will extend EarthBrowser from your desktop to your website. It will be easy since it is based on Flash technology. A simple and seamless way of creating and distributing your personal or corporate geospatial content.

I'm Not a Machine, Or Am I?

Is it possible for one man to compete with Google and Microsoft? Not really, and I'm not trying to. I want to make EarthBrowser an excellent tool for education, weather watching and KML authoring and have it be just enjoyable to use. Google and Microsoft are working hard to... hmmm... put buildings in... add more resolution... see star texture tiles and some other stuff. It takes a team of coders and a multi-billion dollar company to give this kind of software away for free. That's why I have to charge for it. I've spent countless sleepless nights and many evenings and weekends away from my family. I've gone into debt while sales of version 2 have dropped to almost nothing, trying desperately to get this new version done. I really have to thank my wife and children for their patience in allowing me to pursue the dream of owning my own business making virtual globes. Once sales pick up again with version 3, crossing fingers, I can focus on adding even more great features, but I'll talk about those later.

EarthBrowser 3.0 to be released soon

2008-01-10T11:39:00.000-08:00

It's been hard keeping quiet about it for so many months, I'm really excited about all the great new features.

Some of the major new features are:
- Supports 90% of KML
- US Doppler radar
- Rain/Snow/Temperature forecasts

Version 3 is not just another 3d globe, but will revolutionize virtual globes for reason's that will be obvious when you first run it.

If you are a customer and would like to participate in the Beta program, please drop me an email and state any relevant testing experience (not necessarily required, but helpful).

Here is a small preview:

That's all for now.

Fast Steradian Intersection Correction

2007-09-20T16:41:00.001-07:00

I just wanted to go back and mention that if the sum of the steradian angles is greater than 180 degrees, then due to the sine and cosine functions, the simple formula breaks down. However, we know that if the sum of the angles is greater than 180 degrees then by definition they intersect. Due to this, we should also store the steradian angle along with the angle sine and cosine.
So the revised intersection test algorithm becomes:

angle_a + angle_b > pi or dot(normal_a, normal_b) >= cos_a*cos_b - sin_a*sin_b

Not as pretty. A speedup for large coverages :-) but a slowdown for everything else... :-(

Simplified GeoJSON proposal

2007-09-20T16:40:00.000-07:00

I've come up an extremely simplified GeoJSON example which can do everything that the current GeoJSON specification can do but with only 4 core elements. Every element is a feature and can contain a list of points, lines and polygons or other features. Everything else in the GeoJSON spec is pretty much taken from some OGC standard or another. It's important not to get caught up in the past we are at the beginning of a potential standard, as Sean said in his post: "there's only one first chance to get a standard right." The only thing that I think might be questionably useful about this would be the nesting of features.

{
    "points":[
            [x0,y0], [x1,y1], ..., [xn,yn]
        ],
    "lines":[
            [x0,y0, x1,y1, ..., xn,yn],
            ...
        ],
    "polygons":[
        [
            [x0,y0, x1,y1, ..., xn,yn],
            ...
        ],
        [
            [x0,y0, x1,y1, ..., xn,yn],
            ...
        ],
        ...
    ],
    "features":[
        {
            "points":[
               [x0,y0], [x1,y1], ..., [xn,yn]
            ],
            "lines":[
                ...
            ]
        },
        ...
    ]
}

Also an optional "crs" coordinate reference system can contain an EPSG code, ESRI WKT (Well Known Text) or a PROJ.4 projection string or all 3. If none is specified, the default is decimal degrees in the WGS84 datum. The coordinate reference system of the parent cascades down the to all children until a child specifies one.

"crs":{
    "epsg":"4326",
    "wkt":"COMPD_CS["OSGB36 / British National Grid...",
    "proj4":"+proj=utm +zone=15 +ellps=GRS80 +datum=NAD83 +units=m +no_defs"
}

Another useful item would be an optional "bounds" element that specifies the bounding envelope in the element's crs.

"bounds":[min_lon, min_lat, max_lon, max_lat]

Minimalistic and precise.

GeoJSON redux

2007-09-20T11:05:00.000-07:00

Sean Gilles has a post responding to my call for a simplified coordinate representation in GeoJSON. His argument is that clarity of representation is more important that implementation overhead which I agree with to some extent.

However, it seems to me that the reason that JSON is so much better than XML for many purposes is exactly that it *does* take into account implementation overhead, thereby making it easier to exchange internal data structures without the overhead of XML.

For any sane implementation of JSON, the following must be extracted as a list of lists:


[ [x1, y1, z1], ..., [xn, yn, zn] ]

So a list of 10,000 coordinates will give you 10,000 lists of 3 floating points each. Or 10,000 list structures and 30,000 floats. Each list takes time and memory to create and address and extract the elements out of. Whereas:


[ x1, y1, z1, ..., xn, yn, zn ]

gives you the same 30,000 floats but only one list. A big win for any standardized JSON reading algorithm. Creating a custom, context sensitive JSON parser to ignore the structure is more than a little implementation detail.

Strangely, after Sean argues against removing context information to improve clarity, he then requests just that. He suggests that the "type" element that describes what kind of GeoJSON element you are looking at be removed since it should be obvious from the type of request that you made to receive the GeoJSON content. What if one were to receive a set of files with various different sets of data, some single features, some feature sets and perhaps some just geometry elements? If you remove the type field from the geometry elements, how would you know what kind of geometry you have? You couldn't tell a Point from a Polygon without the type field.

All that said, I do have another simplification for GeoJSON that is unrelated to the previous issues. How about we do away with the 'Point', 'LineString' and 'Polygon' geometry types. No really. They are just special cases of 'MultiPoint', 'MultiLineString' and 'MultiPolygon' with one element. I find myself writing code to take the special case single element entities and put them into the more general multi-element entities. That is code I would much rather not write since it introduces complexity and potential bugs. The only real difference in the two is the lack of a "Multi" and an extra set of an enclosing brackets.

Fast Steradian Intersection Testing

2007-09-19T09:40:00.000-07:00

I've come up with a pretty neat inclusion testing algorithm and I wanted to share it. The need is to test for inclusion of datasets within a camera view. Datasets are normally constrained to a certain area on the globe, but can cover the entire globe in some cases.

The simple way to do this is to make a bounding box with longitude and latitude coordinates and then just do intersection testing from a view bounding box. However as you get farther from the equator, the bounding boxes get progressively more distorted and you can be either over or under inclusive in your datasets.

My solution is to use steradians, or solid angles. All you need is a normal vector and an angle, four floating point numbers which is the same as a bounding box, actually I add another to speed up calculation but I'll explain that later. It performs uniformly anywhere on the sphere, and to specify global coverage you just set the angle to 180 degrees.

Now all you need to do is to calculate the steradian coverage for a dataset, and one for the visible camera volume and you can do inclusion testing with just five multiplies, three adds and a compare. It may be a little more work than a bounding box intersection test, but not much more and I think that it is an acceptable tradeoff for the properties that it provides.
It's simple really, the dot product between steradian normals gives you cosine of the angle between them. Intersection is indicated by an angle less than the sum of each steradian's coverage angle, or:

dot(normal_a, normal_b) >= cos(angle_a + angle_b)

There is a nice formula for the cosine of a sum:

cos(a + b) = cos(a)cos(b) - sin(a)sin(b)

So if you store the sine and cosine of the angle rather than the angle you get:

dot(normal_a, normal_b) >= cos_a*cos_b - sin_a*sin_b

Simple, fast and effective. I haven't found a non O(N^2) algorithm to determine the minimum steradian on a dataset yet. It is fast and easy to use lat/lon bounding boxes, but you are going to have the same problems in higher and lower latitudes with over-coverage so it is best to calculate the minimum steradian for each dataset. At least you only have to do it once on static data.

libkml: wtf?

2007-09-19T08:51:00.000-07:00

So I saw mention that Google is soon to be releasing an open source library for the kml format.

Google will be releasing an open-source KML library in C++ that implements and tracks the standard as it progresses.

I can see two intended audiences for this library; kml content creators and content consumers. I just don't think it makes sense for either of them. For kml creators, why would you need to interface with a C++ library in order to create kml files. The answer is, you shouldn't need to. It's kind of like my post on Dreamweaver, if you know what you are doing it just gets in the way and if you don't know what you are doing it is way too complex. A C++ library seems like overkill to write out some xml text. I guess it could keep track of external and document-wide styles? Big deal.

If you are a kml consumer then it makes a little more sense to use a library in C++, but not much more. Using external libraries requires you to build a bridge between your code and the library concepts. So the libkml will be dictating what types of entities you support and how they are interrelated within your code. This is restrictive on how you would develop your internal classes by forcing you to make a class structure identical to the libkml structure or you could try to build a conceptual bridge between your internal structure and the libkml structure in order to be compatible. Once you have either one, why would you need an external library just to parse the XML entities.

By providing a reference library it allows developers to more easily keep up to date with KML without having to maintain their own library and track standards changes.

So developers won't have to support any changes in the standard if they are using libkml? I guess it sounds more like it's for the kml content producers.
I guess an alternate explanation is that they are trying hard to make it seem like the standard will be truly open. Of course I'll take a look at it when it comes out to see what it's all about, but the whole concept seems like an exercise in futility.

JSON beats XML, some comments on GeoJSON

2007-09-17T10:15:00.000-07:00

I've been setting up some new data services and after experimenting with formatting my data in JSON (Javascript Object Notation) and I'm hooked. Programming mostly in C++ has made XML the easiest data format to use but I'm very unhappy with the hurdles that one must go through to go from XML to the internal representation of the data. I've been doing a lot of Javascript and Python programming lately and am just blown away by how easy it is to create, maintain and share data through the JSON format.

For XML, on the server end one would take an internal data representation in Python and create custom classes to format each data item with the appropriate XML tag, convert it to a string and output the XML tree. Not too hard if it is a relatively shallow dataset without too much data nesting. On the client end, you would write classes that would decode the tree structure of the tags with full knowledge of each tag's data type (string, integer, floating point, etc...) and create a new data structure to mirror the server side data.

For JSON, you take your data structure, usually a dictionary structure with a name associated with a value, dump it directly to JSON with a single procedure call and save it to a file. On the client end, you load the data directly into a similar data structure with a single call. No special decoding classes needed, no XML tag data types (string, integer, float) to have foreknowledge of.

I've been looking at the GeoJSON specification and it looks pretty nice and but with a few caveats. Say you have a LineString feature type, the coordinates are a list of lists which contain 2 coordinate elements (longitude and latitude). This is extremely space and processing time inefficient, which is very important for me with large datasets. I suggest using the standard single list of coordinates like is done in all other formats like GML and KML. Also it would be nice if there were some sort of feature style information specified like color, line width, line style and placemark icons.

I'll definitely be using my modified GeoJSON format when I return to C++ programing for EarthBrowser v3 soon. Sorry XML, please don't feel bad. It's not you, it's me. I just feel like we need a little space.

The Economist snubs EarthBrowser

2007-09-11T08:58:00.000-07:00

Amongst yet another "gee whiz" article about the "geoweb," it is claimed that: 'Keyhole, an American firm, released the first commercial “geobrowser” in 2001.'

I guess it's too much to ask for a magazine to research each claim made in each article, but EarthBrowser, then called Planet Earth, was the first commercial "geobrowser." It predated Keyhole's existence by several years. In fact someone from Keyhole contacted me back then and inquired about purchasing the earthbrowser.com domain name, the EarthBrowser trademark and my customer list. The amount they offered was laughably low considering EarthBrowser had over 2 million downloads and sold over 20,000 copies by then.

On a different note, several months ago I alluded to working on a new project that was a bit of a diversion from the next great OpenGL version of EarthBrowser. Project Kraken is nearing completion now and it has become better than I'd imagined it could be. It's been hard not blabbing all about it on the #worldwind or #planetgeospatial irc rooms to get useful feedback, but I've restrained myself.

Here is some newly declassified information about Project Kraken:

It will be released within the next 30 days
There will be a free version
It is easily customizable
It will compete for mindshare with Google Earth

Reviews: Django, CSS, jQuery and DreamWeaver oh joy

2007-07-18T06:40:00.000-07:00

One saying from my youth that has stuck with me was from my drumming instructor: "It's good for your beat to be ballsy, but the balls have to have hair on them." I took the meaning to be that a great idea isn't enough, it must be executed with finesse.

I am near completion of the secret project that I undertook a few months ago. With luck it will be unveiled in the next few weeks. Right now I am rewriting earthbrowser.com, which hasn't had a major update since 2003. I've had to update my skill set in the web design arena, since I have to do everything in this little business and I've learned some little nuggets that may be of use to someone. On to the tool review...

Let me just say right now that I don't understand why Dreamweaver exists. It has a very complex interface and set of features, so it looks like it is for professional web designers right? After trying to use it and spending all my time in the incredibly weak text editor to edit the HTML code, I just started using TextMate. I conclude that any expert web designer will just be editing the raw html source. Beginners will be overwhelmed by the options and difficulty of changing settings through the dialog box interface to the raw code. Again, who is the target market for Dreamweaver?

On to the server logic. I've looked at several CMS (Content Management System) packages such as Drupal, Joomla, ExpressionEngine, TurboGears and Django. I've decided that I'm pretty much done with PHP because it is so lame. Does anyone use Perl anymore? That leaves me with my preferred scripting language of Python and Django won the initial test phase hands down.

Django is actually very simple, it basically consists of models, views, url parsing and a templating system which are all nicely intertwined. You server parses the incoming url request and sends all the request info to a view function that you define. From there you can choose what models (if any) are involved in the request along with any other data, perform operations on that data like sorting and filtering, then pass it to the templating system where it can fill in subsections of your pre-defined HTML template. It took about 5 minutes to make a webpage that listed all of the recent earthquakes over magnitude 4, sorted by most recent and provide a link with a relevant title for each. Of course that doesn't include getting that quake information updating into the database, that is a whole other story. There are also nice little modules to do RSS or Atom feeds, blog posts and many other neat features. I'm now rewriting my purchasing system in Python instead of PHP and I couldn't be happier about it. If you are in search of a CMS for your site and don't have your heart set on PHP, you must try Django.

I've decided that giving my site that Web 2.0 feel isn't enough, I'm going with web 2.1. Learning CSS is now mandatory, it isn't that complex, just the ability to set up some inherited styling options for HTML elements. You just have to know what tags you are using and what styling options look best for what you want your site to look like. jQuery is fast becoming my friend and I haven't even done much with it yet. It is a set of Javascript functions that lets you alter the structure of your site when certain events happen, like the page loads or a button or link is clicked. It abstracts away the XMLHttpRequest, handles JSON data, animates elements and a lot more. If you design websites, use it!

In conclusion, if someone were to ask me what they would use to create a fairly complex website I would say set up a dedicated server with Apache, MySQL running the Django framework for the server side logic, edit your HTML and CSS with a nice text editor like TextMate and use jQuery for the client side logic. Don't bother with Dreamweaver.

Chewing Tobacco with Bill Gates

2007-07-11T08:32:00.001-07:00

I've seen some famous people in my time, but have talked to very few. Perhaps meeting Linus Pauling, who was a personal hero, when I made a complete fool of myself has made me a bit shy. Mr Pauling gave a talk to my first year chemistry class at Reed College in 1985. After the lecture, I approached him and he gave me his full attention with everyone watching the exchange. I could tell that everyone was expecting a thought provoking question with an insightful and wise response from him. I kind of froze and asked a confused, misinformed clarification about his position on vitamin C. He corrected me politely and moved on leaving me groaning inside at my lack of having a prepared question.

In 1989 during summer break, I worked for Microsoft and was quite the wiseass. The marketing folks asked for quotes from the CalTech summer hires to put in the school newspaper the next year and I thought up some ultra-geeky quote that had a hidden, and very rude, message in it and everyone had a chuckle over the fact that the marketing people thought it was real. Toward the end of the summer, all of the interns were invited to Bill Gates house for an afternoon barbeque. At that time in my life I thought I was being a rebel by smoking nasty convenience store cigars and chewing tobacco. So I'm there at Mr. Gates house and I had perhaps my second "wad of chew" ever in my mouth, it was my last. I went to the bathroom to spit it out as it was making me feel sick but there was a line about 6 people deep. I look behind me and Bill himself is behind me in line and he strikes up a conversation with me about what I'm doing this summer outside of work. I tell him that I'm taking windsurfing lessons and ask him about the garage for boats he has near the lake edge. I realize that I'd flubbed another chance to ask someone famous a probing and insightful question. Feeling sicker and sicker as the nicotine leeched into my bloodstream through my raw gums, it was finally my turn and I made sure there was no residue for Mr. Gates to find as I spit it in the sink. The fact that one of the richest men in the world was waiting in line to use his own bathroom so he could talk to his young employees is amazing. For that, I will always have a great deal of respect for him.

I not even going to go into what happened when I met Vatos, the drummer from Oingo Boingo.

Google getting into GPU/multicore programming

2007-06-05T21:39:00.000-07:00

No one can say that they are dumb. I've written about GPU programming and it's enormous potential in previous posts. However, the news that Google is purchasing PeakStream is still a little surprising.

Personally I prefer the open source libsh library to any commercial offerings. However I don't know if it is being actively developed any longer since McCool started up RapidMind, the biggest PeakStream competitor. Both RapidMind and PeakStream are very lame company names by the way. Where does this leave NVIDIA'a CUDA?

PeakStream's and RapidMind's technologies are probably more geared toward multi-core processors since that is where the high end bucks are, which is usually where smaller companies aim. However it seems to me that the whole stream processing market will eventually be dominated by the graphics card makers since they comprise the low end and ubiquitous commodity market. Either that or multi-core and graphics chipsets will merge eventually, which is probably the most likely scenario.

Blue Moon Tonight

2007-05-31T12:18:00.000-07:00

See a rare blue moon tonight.

A blue moon is the second full moon within the same calendar month. The moon will not actually be blue...

New version of Blue Marble Next Generation

2007-05-31T11:27:00.000-07:00

Just got word from Reto Stockli that there's a new version of BMNG out. It hasn't hit the mirrors yet, but it should be available shortly.

Changes include:

1) Replacement of the standard BMNG Antarctica by use of a colorized version of the high quality MODIS Mosaic of Antarctica by Terry Haran and Ted Scambos (NSIDC)
2) Fix of Southern Hemisphere sea floor topography errors (GEBCO vesion 1.02, and bug in my code)
3) Fix of Arctic 80N-90N land mask inconsistencies between the MOD12Q1 and GEBCO sea floor topography. Remaining 80N-90N inconsistencies are due to the fact that there are no MOD09A1 land surface reflectances in this area and it was painted with permanent snow in BMNG during all months.

I'm excited for the new dataset! It isn't mentioned, but I hope that they fix the hillshading in southern Crete.

Dynamic Relief Mapping

2007-05-17T10:20:00.000-07:00

EarthBrowser 3.0 is starting to shape up nicely. I've created a normal map generator using a Sobel filter and SRTM data. Normal maps are just a texture respresenting compressed normal directions and they enable one to create the illusion of very detailed geometry with the help of hardware accelerated shaders.

Africa Relief

Asia Relief

Using the mosaic and raster classes that I described in my previous post, I generate a normal map in the gnomonic projection and put that together with the texture map and a wgs84 ellipsoid and it is starting to look pretty nice. The great thing about it is the shadows move where they should when the light source moves. Someday I'll make a youtube video to demonstrate that effect.

Hexagonal Dataset Rundown

2007-05-10T16:57:00.000-07:00

The Icosahedral Hexagonal grid system for the next version of EarthBrowser is nearly complete. This was no trivial task and it required several helper technologies in order to be possible. One of the primary motivations of this technology was to be able to represent the northern and southern latitudes as accurately any other area on the globe, which it does seamlessly.

Unfortunately the size of the datasets are not significantly smaller as I had hoped. The size of the BMNG (Blue Marble Next Generation) dataset in the generic Plate Carée projection is just over 6 Gigabytes uncompressed. In the Hexagonal grid format it comes in roughly the same. The full 500 meter per pixel hex grid consists of 162 tiles roughly 4096x4096 in size. Each hex tile image is a square in the Gnomonic Projection centered on the hexagon tile center. A similar resolution tile set in Plate Carée would be 128 tiles of size 5700x5700. Theoretically the dataset size should be much smaller, but I added about 5% padding on each edge, also each corner of the square image has redundant data that is represented in adjacent hexagons.

Compressed into Jpeg2000 format, I can get the whole dataset down to around 250 Megabytes without noticeable compression artifacts. I think that will be better than you can get from Plate Carée since land is over-represented and less compressible than ocean areas.

mosaic
In order to make all of this work, I needed a nice little tool I was using to build my Landsat 15 meter dataset. The class is called mosaic and it will take a set of geo-referenced datasets in any projection and build a new geo-referenced image in any other projection. This has made creating the hexagonal tile dataset very simple. The mosaic class can read Jpeg2000, MrSid, GeoTIFF, ECW, jpeg, png or even raw images. It also take advantage of the ability of Jpeg2000 and MrSid formats to supply reduced resolution subsets of images in order to speed up the processing. In future versions I'll add netCDF and some other neat formats out there. It's all very fast too in optimized C++ code, much faster than gdal_warp.

raster
The mosaic class is possible due to another class that has become the very heart of the new EarthBrowser program. The raster class is useful not only in importing and exporting data from image files, but also in the graphics "game" engine for vertex buffers. Raster can be used represent any block of data based on it's height, width, depth and storage type (8-bit, 16-bit unsigned, float, double, unsigned 64-bit, etc.). It can also have an arbitrary interleaved ordering: interleaved by pixel, by line or by plane (bip, bil and bsq). It can be geo-referenced with a supplied origin, resolution, rotation, projection and datum. It can also be subsampled with various sampling kernels like bilinear, cubic convolution, cubic spline and even nearest neighbor! I tried Lanczos but failed and gave up.

Due to the flexibility of the mosaic and raster classes, I build the Hexagonal dataset from BMNG and SRTM Plus (fused elevation and bathymetry) with the very same set of calls, with just a different source dataset name and output raster format. I decided to use bilinear on SRTM and cubic convolution on BMNG too. Of course all of this functionality will be available in EarthBrowser 3.0 from the Python console or from your own imported Python scripts.

EarthBrowser 3 is going to be a quantum leap from version 2. Now if I can just nail down the rights to use the i-cubed 15 meter Landsat dataset I won't have to waste my time on that again!

Thanks for the (indirect) mention Mr. Hanke

2007-05-10T08:29:00.000-07:00

The much awaited blog from the Google earth/maps team has now arrived, and it doesn't disappoint. I look forward to hearing more about many things Google is doing or planning, especially details of the collaboration with NASA. I sure hope that NASA puts any generated earth data in the public domain. John Hanke kicks it off with his perception of today's online geo-referenced world:

I don't think that there is agreement on what the geoweb is, but I think there is a lot of enthusiasm and energy across many fronts to make it happen. I expect the "it" will evolve substantially over the next few months and years as we (the geo ecosystem on the web) collectively figure out how "earth browsers," embedded maps, local search, geo-tagged photos, blogs, the traditional GIS world, wikis, and other user-generated geo content all interrelate.

EarthBrowser, the original "earth browser," on the web will continue to be a part of the evolving "geoweb", for lack of a better euphemism. I'll be posting an update later today on some of the cool new technologies coming out in EarthBrowser 3.0.

Project Kraken and EarthBrowser circa 1998

2007-03-20T14:33:00.000-07:00

For the past two weeks I've been working on a secret project (codename Kraken); and it will be the perfect compliment to the next version of EarthBrowser. I'll give out some hints and details in the coming weeks about it's potential uses and capabilities. However before I reveal the future, I wanted to share the earliest snapshot of EarthBrowser caught in the wild...

I haven't kept many records from the time, but I looked up my old website from Dec 6, 1998 in the wayback machine. It featured the first version of EarthBrowser which was called Planet Earth at the time. I couldn't get the domain name for that so I changed the name to EarthBrowser at the end of 1999. Anyone who is interested can take a look in the wayback machine here. Unfortunately there are no images saved, they were pretty cheesy anyway. Once Planet Earth started selling, I quit my job and have been working on it full time ever since.

Here's the main text:

Planet Earth 1.0
Have you ever wanted to see the Earth from space? Now you can see what it looks like right on your desktop. Planet Earth displays a realtime 3-dimensional model of the Earth with the current cloud information downloaded directly over the internet. Night and day shadows are updated continuously and you can rotate it to view any spot on the Earth.

Features

Cloud information updated every 6 hours.
Areas of night and day are updated continuously.
Real 3-dimensional model can be rotated to reveal any part of the earth.
Zoom in or out
Position viewing location over any spot on Earth
Rotates freely or stays in fixed position

Requirements

Macintosh with PowerPC processor
Internet connection (for clouds)
System 7.5 or above
1 MB Disk
4 MB Ram

Mac System 7.5, that's old!! I remember trying to get it to work on the Motorolla 68000 processors but they were just too under powered in floating point...

15 meters of Landsat

2007-03-19T13:00:00.000-07:00

I've been working on a new Landsat 15 meter per pixel base map for version 3.0 of EarthBrowser which is due out later this year. Tasks like this make it painfully obvious that I am crazy to be doing this all by myself. Here is where I'm at with that...

A couple of years ago I decided that I needed a royalty free base map for EarthBrowser that looks better than the one available through the OnEarth WMS service which EarthBrowser currently uses. My only option at the time was to download the raw data and process it myself. I wrote a script that downloaded the raw tiles from Landsat.org slowly so as not to overtax their servers. I was looking at a long, uphill battle to get what I wanted. Now I have all of the raw data, but it needed a lot of pre-processing to be a seamless mosaic that wouldn't have the abrupt changes in contrast and quality from tile to tile.

Then in November of last year I found just what I was looking for: a set of pre-mosaiced 15 meter Landsat data from NASA John C. Stennis Space Center. Two of the color channels were in the infrared spectrum, which is great because many of the visible color channels have haze in them which makes it difficult to get a seamless looking image, but the infrared cuts through the haze. However this means that it looks strange because it isn't a true color image. More image processing needed...

I keep going and began taking scenes in different regions of the world and re-coloring them by hand and using the differences in the two images to "train" an algorithm to color the entire dataset in a more realistic color scheme. First I tried using principal component analysis to reduce the dimensionality of the problem. This didn't work too well since the principal components changed from region to region. I settled finally on using a RGB to HSV conversion to separate the color components. I then use a fairly simple piecewise polynomial approximation using Jacobian matrix decomposition in three dimensions, after which I re-project it back into RGB space and it doesn't look too bad!

However I am now running up against a problem that I didn't think about until I got to this step (doesn't it always work that way). This dataset has been compressed using a "lossy" wavelet compression algorithm. That means that my polynomial approximations will work for one trained tile but won't transfer well to other areas of the mosaic due to the incosistancy of color values. I'm getting little yellow dots appearing randomly and big swaths of green where it should be dark blue due to compression artifacts. Arrrrgh!

As an example just recently I was contacted by an environmental organization from the Greek island of Lesbos asking me if I could donate a high resolution image of Lesbos for a mural at their center. Happy to help an environmental organization and also as a first real test of some of the new capabilities of EarthBrowser, I made a high res image to his specifications and unfortunately had to do a lot of hand editing in order to make it look good. Here is a before and after showing my color transform algorithm in operation.

Unprocessed NASA Landsat Lesbos

EarthBrowser Processed NASA Landsat Lesbos

Hand Edited EarthBrowser processed Lesbos

It is clear from this test that I need the data in an uncompressed format in order for my algorithms to work properly. After inquiring around, I have finally found someone within NASA that can sell me the uncompressed data on hard drives but it is $16,000 and they are not quite clear on whether it is the exact dataset that I am referencing. Needless to say, I can't afford that. So I'm kind of stuck.

There is some hope though, just recently I read on the excellent Bull's Rambles Blog that a fantastic new Landsat 15m dataset has been "donated... to the public domain." One would assume that the use of NASA resources to produce this dataset would have a prerequisite of the product being in the public domain. I've been in contact with the WorldWind project lead Patrick Hogan to inquire about it's availability, and they understandably want to have it "premiered" by WorldWind. This leads me to believe that it will be generally available at some point, but there has been no difinitive answer yet.

I'm keeping my fingers crossed because I'd really rather spend my time programming.

Sweet Sweet Amazon S3

2007-03-08T11:21:00.000-08:00

Another gigantic hurdle overcome! Last night I woke up around 4am and laid there thinking about all of the sweet simplifications possible. I've just signed up with Amazon S3 (Simple Storage Solution) and I think it just lowered my blood pressure a bit. The Coding Horror blog, which is fast becoming a favorite, details their efforts at reducing server load by putting static images on S3. I've heard about S3 since it has been out but thought it too complicated and didn't want to be a first adoptor, but now I'm sold.

I run two separate servers for EarthBrowser, one for the website and one as the dedicated data server for all of the EarthBrowser instances out there. Whenever I put out a release, my web server would get slammed and become unresponsive with thousands of people each downloading the 10 megabyte installer file. I have actually written and implemented, in PHP, a distributed, caching data server toolset that would allow me to rent several cheap $7 dreamhost setups and plop it in to increase my capacity. They would talk to each other and update their internal datasets for more frequently requested items. I was even considering finding some European or Australian hosting sites to get closer to the rest of the world in anticipation of EarthBrowser v3 coming out later this year. It was working pretty nicely but the administrative and logistical overhead was getting to be a bit overwhelming. Now I don't have to worry about storing and managing a group of servers to distribute EarthBrowser v3 data. With Amazon S3 I have virtually limitless capacity and limitless download bandwidth for about $0.20 a Gigabyte.

Now all that work can be thrown away, which makes me both cringe and sigh in relief at the same time. Often I work for a week or two on a technology and wind up throwing it away when something better comes along. Sometimes only through the act of creating and exploring a problem space can better alternatives be discovered. That is why product specs should never be static documents. This time the better solution came along and whacked me on the head.

It is a little tricky to get working well since it uses a SOAP or REST interface with some quirks. I wrote a python script that works with the excellent boto module to allow anyone with a S3 account to create/delete/update objects and buckets from the command line. I'm making it open source and downloadable for anyone who wants it. If you have any suggestions, let me know. There are some issues if you are using Python 2.3 or below, contact me and I'll tell you how to fix them.

The virtual globe as a software platform

2007-03-05T22:42:00.000-08:00

Web browsers are a very interesting and instructive case of a software platform that is completely dependent upon the network but lives on the desktop, like virtual globes should. Browsers have a language that controls the content (HTML), a language that can provide functional logic (Javascript) and a plugin architecture. The magnitude of the world wide web is a testament to just how powerful a platform the web browser is. But what elements make it so successful? A display language is a given since the browser would not exist without it, but that is not sufficient. The network is where the full power becomes apparent. The ability to algorithmically define content for local display on remote computers is where the real power comes from. However server side control of content has many limitations. HTTP being a stateless connection has been the headache of many web developers. We are now starting to see some of the real potential of the web browser become unleashed with the new "Web 2.0" hype surrounding the Javascript language. What we are really seeing is the control of the display being performed by the client rather than a remote server. Imagine that, client side control of an application, we're back to the 1980s and it's about time!

I have no doubt that each of the development teams of the major virtual globes want their creation to become the default platform for geospatial content. Why shoot for anything less? Let's take a look at where things stand for the big guys and give a little update on where EarthBrowser is headed.

Google Earth: The KML format gives the ability for novice users to add their own simple content to Google Earth, much like a static web page. The network link provides the ability to get dynamic, server based content into GE, it is very "Web 1.0." with no client side processing. They are currently at the top of the game. They have a tremendous asset with their enormous and growing dataset. However as I mentioned in a previous post, they will be running up against some limitations of their design if they don't change course, but their biggest problem right now is how to make money from it.

WorldWind: This product has advanced the farthest in the key platform categories. It supports KML and has a plugin architecture as well. There have been some recent developments which might allow one to use Python for client side processing. Their early decision to use .NET has delayed it becoming cross platform, however a Java port is on the way. I am generally underwhelmed by Java applications but am hopeful they will pull it off. Being from a government agency, they don't have the same pressures as a commercial entity. This, and being open source, gives them an advantage in the platform building arena. WorldWind has the best shot at becoming the platform of choice for researchers, if they can just convince them to write in Java.

Arc!GIS Explorer: It seems to me the primary purpose for ESRI putting this out is to keep their customers from migrating to Google Earth for certain tasks. I understand that they are licensing the engine from another company (Skyline Globe or GeoFusion?) and are therefore not in control of their own product. Apparently they allow some sort of scripting (a restricted ArcScript?) functionality. It has the potential to be something great, but ESRI doesn't really have the DNA to be a mass market product leader. Future enhancements are clouded by not owning the source code.

Microsoft's Virtual Earth: VE is a strange bird as it is an ActiveX browser plugin and not a full fledged application, but I see little difference in the installation procedure. Like ESRI, it seems they are trying to stave off Google Earth and keep the market open by providing an alternative. However they want to lock you in to using Windows and Internet Explorer at the same time. It has some really neat features, it is scriptable, and I'm certain it will be pushing Google to come up with bigger and better solutions. They, like Google, are going to have a hard time monetizing this product, the whole billboard thing is kind of stupid and belongs in something like Second Life. Perhaps that is what they are ultimately shooting for. The battle for the mass consumer geospatial market will be between Microsoft and Google.

Where is EarthBrowser in this list? Right now it is not in the same league as the other offerings. It does have some nice features that are just now being added to some of the other offerings like live cloud animations, earthquake data, sea surface temperature animations and weather forecasts. Version 3 is a complete re-write that is years in the making. It will have all of the features that are currently in version 2 integrated into a full hardware accelerated 3D environment. You will be able to smoothly fly through mountains, into and out of the atmosphere. It will support most of the popular raster and vector formats and enable you to link to sources of data on the internet. There are quite a few other neat little tricks in store that I'll reveal as it gets closer to the release date.

Global Grid: The Icosahedral Hexagonal Grid

2007-02-21T09:15:00.000-08:00

I've been working with high resolution earth images for almost a decade now in various forms. In the late '90s I flew down to visit Tom Van Sant who is creator of the GeoSphere Project and the first person, to my knowlege, to have created a high resolution satellite composite of the entire earth without cloud cover. Early versions of EarthBrowser used the GeoSphere base map and was used in several museum kiosks around the world as part of the GeoSphere project. Since then EarthBrowser has moved on to higher resolution datasets, but the basic limitations that were present then remain.

The need to represent a sphere with a series of rectangles has offended my aesthetic senses for a long time. It seems simple enough, images are two dimensional and the surface of an ellipsoid is also two dimensional. The basic problem is that a planar representation of an ellipsoid requires distortion, generally around the poles. This isn't a huge problem since you can re-project the image to eliminate this distortion. The problem for me is that there is an unacceptable amount of waste. More data bins (pixels) have to be created via a resampling stretch near the poles only to be resampled down again when used. Yuck!

I've been obsessing about this for the past year and came up with what I thought was an elegant solution, and later discovered that others had been there before (of course!). The basic idea is that the plane can be tiled by only three of the fundamental polygons: the equilateral triangle, the square and the hexagon.

The basis of the EarthBrowser version 3 dataset will be the dodecahedron. From the dodecahedron you can surround each of the 12 pentagons with hexagons to generate a soccer ball-like solid (sometimes refered to as a bucky-ball). Higher and higher numbers of hexagons can be filled in between the 12 pentagons to give you a set of seamless spherical hexagons, plus the original 12 pentagons. This allows you to cover the globe at varying resolutions.

This solution is a very elegant one and is already beginning to be used in several areas like climate and ocean modeling. Getting away from the rectangular grid creates some real complexity issues with indexing, dataset management and data projection. However the benefits of minimizing cell to cell distortion often outweigh the extra complexity for certain uses.

For global earth viewers like EarthBrowser, Google Earth and World Wind, I'm not sure the extra complexity would normally be worth the savings in data size. The data tiles have to be in special map projections and are hard to edit and manage. However in EarthBrowser v3 I have created some tools and techniques within EarthBrowser that compliment it's hexagonal grid in such a way as to eliminate many of the problems and turn several negatives into positives. I'll save discussion of those components for future posts. Having a this global grid technology makes EarthBrowser very flexible for future enhancements that I hope will include the ability to perform data modeling on global datasets.

But best of all it satisfies my aesthetic sensibilities.

More on GPU processing

2007-02-20T12:24:00.000-08:00

Things are moving rapidly in the Graphics Processing Unit (GPU) programming arena. NVIDIA just released CUDA a GPU compiler that will allow applications to offload work to the GPU. Most GPUs run about 10 times faster than their host CPU and are ideal for running uniform mathematical operations on large sets of data.

NVIDIA is adding a nice compiler that does the hard work of translating your math into GPU assembly code. All you have to do is provide the source code of the program, compile it and upload the program and data to the graphics processor and get the results. ATI is apparently going with a much more barebones approach and allowing you to program the GPU in assembly code directly. That is more flexible, but requires someone else to develop and support the toolset. I like NVIDIA's solution better, but I don't think that they allow you direct access to assembly so it is a little less flexible.

GPU programming is going to become a big deal in scientific computing in the next few years.

Faster than light travel

2006-10-04T17:12:00.000-07:00

We know that light travels "c" in a vacuum and slower in other mediums, measured with the index of refraction.

We know that scientists have slowed down light to a very small percentage of c.

Is there another state of matter in which the speed of light is greater than c.

Could quantum enganglement be such a state?

GPU processing for GIS is coming!

2006-09-20T12:59:00.000-07:00

There is a very interesting article over at The Register that has some "insider" info on a new product from AMD/ATI. ATI has created a server product that will make the raw processing power of the GPU available to general applications. This has incredible implications for the GIS software field.

The advent of programmable vertex and shader processing has opened up a reverse-pandora's box in my opinion. A whole lot of good things are coming out of the ability to access the highly optimized floating point processing power available on modern GPUs. However, like the external floating point processors from the '80s there are some significant issues with using this technology.

Stream processing isn't something that is applicable to general software development, but scientific computing can use it to great advantage for problems that fit within it's scope. It is most useful for taking a large block of homogeneous data and applying fixed transforms (guided by variable parameters) on it and storing the results in a similar block of data, that resulting data can then be further processed.

For anyone in the field, the work done by McCool, et. all, at the Univ. of Waterloo on the Sh metaprogramming language has given a glimpse of what was to come. When I first read the book describing Sh, I wanted to use it immediately but found it a bit limiting due to it's static nature and the inability to treat shading programs as dynamic assets as the code you write is basically hardwired into your executable. There is, however, an intermediate format that could possibly be used as a game asset, but it is just a little too funky for use as a general purpose shader generator. This programming model fits scientific (GIS) applications much better. I'm looking forward to seeing what is coming out of AMD/ATI. We are just scratching the surface right now, I think that there needs to be more evolution in the field.

I am planning on making EarthBrowser plugins that make use of this kind of functionality in the future. All the pieces are in place right now for version 3.0, but each plugin has to be written for a specific problem domain.

Goodbye Xena, Hello Eris

2006-09-15T10:59:00.000-07:00

The planet formerly nicknamed "Xena" has officially been renamed Eris, which was suggested by it's discoverer Michael Brown. Eris is the Greek goddess of strife, which is what it's discovery has caused. The commotion ultimately led to the demotion of the smaller Pluto to the status of dwarf planet. Eris has one known satellite and that is now named Dysnomia formerly nicknamed "Gabrielle."

Pluto has always been an interesting object for many reasons and many people are upset that it is no longer considered a planet. However I agree with the new designation, it's orbit was too wild, it is too small and the alternative of adding the other newly discovered Kupier belt objects to the list of official planets is not really an acceptable outcome. Pluto has received it's minor planet number of 134340.

Here is the official IAU designation.

Don't be evil

2006-09-14T09:39:00.000-07:00

Well, I guess the other shoe has dropped. Google has now hired the most evil lobbying shop in the country. I recall that they previously acquired their earth imagery with a restrictive license not allowing other companies to license the same data. I can't recall the company this was with, but suffice it to say that the "don't be evil" ruse that has given them the benefit of many a coder's doubt is now inoperative. Too bad, I wanted to believe that a corporation could actually try to do good for the world.

Some people will say that they are just paying them as a form of protection money from the Republican juggernaut. Some will say that it is the responsibility of a publically traded company to maximize it's shareholder's profits. Some will say that maybe they won't have them pull any of the dirty tricks that is the only reason to hire DCI. Sorry, but that is just too many ifs and maybes, as should be, you are judged by the company you keep. Corporations do not need to be sociopaths. Brin and Page should be ashamed.

Google fanboys, join the Republican fanboys. Sadly, you are now the same.

GML & JP2 a simple concept with real impact

2006-09-10T11:59:00.000-07:00

Funny how little it takes to make programmers happy, just make our jobs easier. I mentioned in my last post that AJAX does something fairly obvious and it is a quantum leap in the web browser world. Sadly most people still think of the web browser as the only interface to the internet. Browsers are so needlessly restrictive, I suppose security issues have a lot to do with it, but I think it is mostly a lack of imagination.

That brings me to GML in JP2. Jeff Thurston over at Vector One has a post up which goes into just enough detail to let you know that it is a good idea. Not brilliant, not even tricky in the least. The designers of the JPEG 2000 standard saw fit to allow the addition of text metadata along with the picture in their format. That is all that is needed to add a simple xml file conforming to the GML specification which will allow great things like telling you how the JP2 image(s) are to be georeferenced, you can even add other features like point and polygon features. TIFF allows this as well, but anyone who has looked at the guts of libtiff or seen the contortions need to make an image conform to the GeoTIFF "well known text" spec knows that it is anything but easy.

Why is it a big deal? It is and it isn't. JPEG has comment tags, but I'm not sure if they have a length limit, and I don't really care to check. PNG? same thing. The reason it could be a big deal is because people are talking about it and building a de-facto standard that will be presumably supported in future GIS software. Much like the "well known text" popularized by the ubiquitous GDAL and PROJ.4 packages, a tacit agreement that you check your JP2 images for GML metadata is all that is needed for this to make a significant impact.

But what is the impact of GML & JP2 exactly? By combining the image with the metadata you make distribution, storage, reference and cataloging of such datafiles an order of magnitude simpler. Like my previous post about the "seduction of the one" having one anything makes it so much easier in so many ways. Just think of the all of the possible states inherent in the horrid shapefile format, you have the .shp, the .dbx and possibly the .shx file just to describe some vector data. Right there you have a lot of code to manage all of the possible states of the files. You also have to have data transport functionality that can send and request data from more than one file if needed. It is about an order of magnitude harder to handle a multi-file format than it is a single file one. Not only that, but if you do support a multi-file format you are boxed into not doing something really slick that is only possible from a one file format.

As an example, EarthBrowser v3.0 has a really neat unified data stream architecture internally which I can attach arbitrary metadata to each stream. A simple but powerful concept that can simplify object data interfaces across the board. If I had to make a unified data stream structure that incorporated multiple streams as different parts of the same data, well no thanks. If you really had to do support something like that, you would probably want to pre-process the data to an internal format when reading and just bite the bullet and code it up if you ever had to write that format out.

There you have it; GML and JP2. Highly compressed raster data along with descriptive metadata all in one package. I give it a big thumbs up, at least someone is looking out for us poor overworked programmers.

KML needs an AJAX style upgrade

2006-09-10T07:09:00.000-07:00

The cranky older programmer is back for another session detailing the failings of today's "best of breed" applications. Take AJAX, the acronym for a simple concept that has brought about a new excitement to web development, dubbed "Web 2.0." I've read so many excited articles and posts about how great this new capability is and how it is going to bring about a resurgence of the dot com boom. That may be the case, but let me say something about the underlying technology of AJAX.

Cutting away all the marketing hype, the two keys to the whole thing are 1) the ability for a script to download data asynchronously and receive a callback when it has arrived. 2) the ability to change the contents of a web page without the entire page refreshing. Of course there are a lot of third party packages springing up around these capabilities but they all rely on these foundations. From the perspective of a programmer, my question is: what took the web browsers so long? Gee, let's let them specify their own download URLs and process them with a scripting language rather than forcing static urls and the fixed processing of interface elements by the web browser itself. Guess what else follows that outdated content model, yep, it's KML and Google Earth. At least the first few halting steps are being taken in the web development world thanks to Javascript and XMLDom.

Now, why is KML lame? For a simple example, you can write a KML file to show a set of georeferenced data points. That's great, but what if you have a set of points with different attributes that you want to publish and view, but not all at once. Perhaps you want to enable some selection criterion for the data points, such as earthquakes during an arbitrary time frame? A hokey solution could perhaps be ginned up using the network link and having a separate server script parse the link and return the appropriate data. However, how are you going to provide an interface for them to make the arbitrary selection? With KML? At least HTML has Javascript and GUI elements, KML and Google Earth have no easy client side data capture or processing capabilities that aren't hard coded into GE itself. Please correct me if I'm wrong and I'll eat a heapin' helpin' o' crow.

I look at the spectrum of software development going on in various industries right now and am somewhat stunned by the isolation each has. The game industry is doing some fantastic work, the best in the industry, but the GIS field doesn't seem to pay any attention. Web development is a huge chunk of the market but is completely isolated by the web browser. Everyone has web browsers but writing a plugin for all browsers is a hurculean task. I heard that the Opera web browser will include a BitTorrent client. Hurrah, someone is getting it!

I must say again that I think that Google Earth is the best earth explorer out there right now. However there really needs to be some serious innovation and insight done in the software industry in general. I've taken a look at many open source projects out there and am shocked by how many are written in C. Of those that are written in C++ (the only language for serious library and application development work) fewer still incorporate the STL libraries. I've only come across two that use the boost libraries, yikes! I realize it takes a lot of time and effort to keep up with the state of the art and evolve along with the other industry segment's innovations. Perhaps the corporations putting out software today are just unwilling to support the continuing education of their primary product creation assets, the programmers. The programmers themselves need to take responsibility for their own continuing education during non-work hours if they want the be marketable in the future. Finally product and project managers need to have some real programming experience (at least 5-7 years) to be considered competent in my opinion. It's too easy for an inexperienced programmer to bamboozle someone who doesn't know about development work, it is also too easy for a forceful manager to impose unrealistic expectations on a development team.

Let's all celebrate the advent of AJAX, the ability for client side control of what to download and how to process and display it. It only took a decade, at least it's a start. Google does it with Google Maps, think guys.

Google Earth and KML are outdated

2006-09-03T08:17:00.000-07:00

The developers of Google Earth boast that it works well with older graphics cards, that is due to the fact that is based on an aging codebase and was designed with those components in mind. EarthBrowser (as primitive as it was) had the virtual globe market all to itself from 1998 to about 2002 when I first heard about Keyhole EarthViewer when someone from Keyhole offered to buy earthbrowser.com and my customer list. The code is probably still based on their original code developed back then. I can see that they have added more and more on top of that old code. Believe me, you can really pile new features on top of an old foundation but it just gets harder and harder. It gets harder to add new things and it gets harder to change old ones. Just look at Windows or any product that has been out there for any period of time. EarthBrowser itself has been rewritten twice already and I'm on my third complete re-write. A complete re-write is time consuming, difficult and almost never done for a commercial product due to the time, cost and difficulties in such an undertaking of an established product. However, the benefits of such a re-write are immense, for an example, look at the transformation of MacOS since Steve Jobs came back.

Back to Google Earth. Google seems good at publishing APIs for their web services, like Google Maps. Unfortunately for people who use them, they reserve the right to yank the rug out from under you at any time with their licensing terms. The Google Earth API is effectively KML, perhaps they are going to introduce something new after the acquisition of Sketchup which has a reportedly nice Ruby API, but for now it is just static data with the weakest possibility of time based animation through some kludgy use of the "network link." By the way, what kind of hyperlink is not a "network" link? Just wondering.

I don't want to help Google with any of it's problems since they are the main competitive threat to EarthBrowser. I tried writing a file format for EarthBrowser back in 2000 (version 1) that would use static files to control actions in EarthBrowser. It was a very difficult to thing to keep up with. Any new action had to be first coded into the program, then an interface to that code provided and access through the file format. Tweaking that was a nightmare so I tried making my own tiny programming language to embed in the files to provide more flexibility. It worked OK, but that just added another layer of complexity along with the file format and the hand coded feature linkings. You can see the dilema they are facing by adding more and more features into KML that need to be supported in current and future GE versions. They need a rewrite, but guess who will have a new virtual globe out soon (besides EarthBrowser)... Microsoft. I don't know what MS has in store for us but from the looks of Flight Simulator X, it will blow GE out of the water.

Here is discussion of one of the extensions someone has made for Google Earth using KML. It is just so kludgy...

I've learned the lesson of trying to make a static file format control a virtual globe. The next version of EarthBrowser will feature a fully scritable game engine which users and developers can add their own data types and visualizations directly on the globe. As an example, it will be possible to code a module that will download raw data from any source (e.g. NASA) and process it for display on the globe. From this scriptable code base it will be easy to create a KML converter and easily update it with new versions of KML with few changes to the actual code base.

I'll leave you with an early screen shot of EarthBrowser under development:

The seduction of The One

2006-07-11T19:06:00.000-07:00

As a programmer, the notion of The One is very tempting to me. Let me explain.

When designing code, you come across many different elements that have to be coordinated, manipulated and routed. Data and state information may need to be transmitted to other parts of your code, other programs on your system and sometimes even remote systems. Usually you come up with a model of how these different parts will interact with each other and you can make simplifications in the code that will enable enormous flexibility and scalability. For me it also gives me a good feeling inside knowing that I've just created a quality tool that will make the project easier in the future. I don't know much about eastern philosophy, but perhaps this is a Zen or Tao feeling of "rightness" in the code. Anyone who has spent much time programming will know this feeling.

Having an abstraction that provides a single interface from many code state sources to many state consumer destinations is something that, when done right, reduces the complexity of the code by an order of magnitude. This is "The One." A single representation of an idea that interoperates with all or most of your code making state changes nearly effortless.

However in real coding, things are never that simple. There are always problems with dependencies, synchronization and sometimes it is like trying to fit a square peg into a round hole. There is a saying attributed to Einstein along the lines of make things as simple as possible, but no simpler. This rings true again and again when coding. I have wasted countless days, weeks, even months trying to create an abstract superset of functionality that the project would just fit in nicely and have plenty of room to expand, wouldn't that be nice? To go from being an expert programmer to a master code craftsman, one must learn to avoid this pitfall at all costs. Nothing eats up more time than writing code that winds up never being used. We all throw away big blocks of code when a better replacement comes along, that is unaviodable, but in the planning stage of a project is where an over-enthusiastic programmer can really mess things up with a "simplification." There are local maxima and minima in programming and going over a little hill of work will sometimes put you in a state where things are much easier. More often however, doing a little foundation work to smooth the interface out will leave you where you started or even worse, make things more complex.

To tie this to my recent post about the shapedb format, the ability to add raster data to the shapedb is certainly nice and simplifies distribution for related data. However the need that brings rise to the shapedb format is not a convenient repository for data, but the processing overhead required in extracting and converting data into something useful from shapefiles. Now that the madness Hopefully I've just saved myself a few days of trying to make a nice "geodatabase" format that fits all sizes, I'll just focus on vector data for now.

Shapefiles considered harmful

2006-07-05T09:29:00.000-07:00

There were a couple of posts about the usefulness of the shapefile going into the future. Jeff Thurston posed the question in his Moving Beyond the Shapefile post recently. A responding post by the Drkside of gis took an appropriate opposition position to the idea of using personal geodatabases as a replacement. Closed formats are no longer suitable in such an important data market. They reduce compatibility and introduce a vendor lock in on data that should, and in many cases, is in the public domain.

As a GIS file format discussion, this is near and dear to my heart. I agree wholeheartedly with the premise that the shapefile's days are and should be numbered. The question is, who will be big enough to put out a competitor? The .shp, .dbx and .shx troika to describe a single set of data reeks of '80s design. A single unit of data should reside in a single file, the complexities of opening 3 files and coordinating the shared representation of data between them would be a non-starter if it were designed today. I understand the convenience of de-coupling the index and attributes from the data when adding records to a shapefile (or should I say shapefiles) since you can just write to the end of each file rather than shuffle things around in a single file. Sorry, but that just isn't a good enough excuse anymore.

This isn't just a helpless rant about a data format that has outlived it's time. I am proposing a real alternative, a non-closed geodatabase, which I will call a "shapedb" that will work with the open source database Sqlite. If you haven't seen it yet, it is a cross platform, open source embedded database engine. It can be built into your commercial or non-commercial product with it's very non-restrictive license. I have been using it in EarthBrowser 2.8 onward and am relying on it heavily in version 3. I can't tell you how easy it makes things to have an embedded database. The ability to manipulate data with SQL statements to extract just the information you need is a quantum leap from the old dumb file format. Sqlite files are cross platform compatible so those endian issues between Motorola and Intel aren't an issue (believe me, that's a big issue usually, even for shapefiles). You can add as much data as you like to the file and index it however you like. Not only that, you can have raster data as well as vector data, gml data, kml data or whatever your requirements are.

Just saying put it in a cross platform database file and all your problems are solved isn't really a complete solution to the problem. There must be standards for how the data is organized and formatted, just like the shapefile. I propose a small group of GIS programmers and a user or two (or perhaps just me if nobody is interested) provide a standard template for each data type. As the simplest example I can think of just to illustrate the point. How about something like this:

Tables describing shapefile, shape objects and attributes:


create table shp (type integer,
xmin real, ymin real, zmin real, mmin real,
xmax real, ymax real, zmax real, mmax real);

create table shp_atts(id integer primary key,
 ...user defined attributes...);

create table shp_object (id integer primary key,
atts_id integer, shp_order integer,
shp_type integer, nvertices integer,
xmin real, ymin real, zmin real, mmin real,
xmax real, ymax real, zmax real, mmax real,
vertices blob);

The need for a shx file can be eliminated with the shp_order field, just use a select with an "order by shp_order". Another problem that I can't stand is removed as well. Each shape object in a shapefile has to have a corresponding entry in the dbx attributes, which is an ugly redundancy for me. Also you have to manually group shapes by checking against a key in the attributes that is not known beforehand and is different for each shapefile. With the atts_id field, you can group all shapes that are common to a particular entity. You could do a nice query like the following:


select * from shp_object as o, shp_atts as a where
 a.state='OR' and a.id=o.atts_id order by shp_order;

You now have the vector outline of Oregon. Amazingly simple!
You can run with that idea and say you put in an attribute as to whether the vector is on the shoreline and all of a sudden you can do something like:


select * from shp_object as o, shp_atts as a where
 a.state='OR' and a.shoreline=0 and a.id=o.atts_id
 order by shp_order;

Now you have the outline of Oregon without the shoreline portion.

With the right configuration of attributes, you could even put multiple shapefiles into one shapedb, in fact that would very simple and make a lot of sense. Not only that, you could include many different raster formats in the same shapedb with your shape objects. That would create a neat little package to distribute a set of data that belongs together anyway. No more unpacking zip and tar files and getting the file paths correct. Just dump it all into one shapedb and send it out to your customers.

The raster format could be even more simple. You could just have an identifier, format information (like 'image/jpg') and an image blob. Why not just throw the well-known-text projection information into a field as well, or some gml data. The problem with many image formats is the all too common restriction in decompression libraries for the data to reside in a file. For problems like this, you could just dump the data to a temp file which is deleted upon completion of the operation. Using the shapedb format for a 1GB MrSid or ECW file would probably not be the best use of the format anyway for performance reasons. However a set of relatively small tif, png, jpg, jp2 files (a few megabytes each) would work fine, you could include multiple resolution levels, a system of image tiles or whatever application specific use you can think of. However one of the pre-defined table templates should be adhered to if you wanted the shapedb to be compatible with other applications using the shapedb format.

An important consideration is how easy it would be to convert current shapefiles to a shapedb. The short answer is that it would be almost trivial. The long answer is depending on how complex you decided to make the table setup, it could require a little more logic. Just a straight read of the .shp, .dbx and .shx file and inserting each shape object and attribute list into the respective tables, using the ordering of the shx file would do the trick. You could get slightly more tricky and collapse identical dbx records so they were unique and then index off of those in the shp_object table which would improve the speed of your select statements.

So in summary I propose a new format, called shapedb, as a new open format for interchange of GIS data which:
- based on the sqlite database file format
- can be shared cross platform
- store data from many shapefiles simultaneously
- store multiple raster files
- store application/vendor specific data
- data can be accessed and operated upon with SQL statements
- new formats possible by conforming to a table structure "template"

I am currently using this as my data format for EarthBrowser v3 and am considering spinning off an open source library to support the format in other applications.

Let me know what you think!

NASA confirms global warming

2006-03-14T12:48:00.000-08:00

Satellite surveys of Greenland show melting of the ice sheets is accelerating. What is unique is that NASA is now confirming that this is being caused by global warming.

In the press release, NASA states:
"If the trends we're seeing continue and climate warming continues as predicted, the polar ice sheets could change dramatically. The Greenland ice sheet could be facing an irreversible decline by the end of the century."

Meltwater flowing into a moulin in the Greenland ice sheet.
Photo by Roger J. Braithwaite, The University of Manchester, UK.

Not to be alarmist, but we're in deep trouble in the next century.

It is beyond time for us all to take out heads out of the sand and do something positive to help mitigate the effects of this unavoidable catastrophe. At least in your own personal lives, please do something to reduce your reliance on fossil fuels, purchase fluorescent lights for your home (you can find them for about $1 each), bicycle to work, start a garden, vote Democrat.

New version of CosmoSaver

2006-03-14T11:59:00.000-08:00

I've just released CosmoSaver 1.52 for anyone who may be reading this little blog.

CosmoSaver is a 3D screen saver that tours our solar system and 29 of the major moons. All of the planet and moon positions are within 1 arcsecond of accuracy. The rotations are supposed to be accurate too, at least they are for the earth, for the many moons does it really matter?

You can try out a free demo version at cosmosaver.com

I am planning on incorporating the solar system into the next version of EarthBrowser with much higher resolution versions of earth, mars and the moon. If I am a programming god, I'll make a cool sun model with flares and everything, but that is yet to be seen.

Is EarthBrowser a mashup?

2006-03-12T09:36:00.000-08:00

EarthBrowser takes data from disparate sources and displays them on a 3D globe. I'm not sure what the exact definition of a "mashup" is, if there even is one. I think it is something like a service that draws together one or more sources of data and uses a mapping API to display them. Does the definition of a newly coined word like that really matter, or is it the underlying concept that is being labeled what is really important.

EarthBrowser certainly doesn't use an external mapping api, however it's internal program structure provides it's api. Are those KML files created independently by people defined as a mashup? My guess is they are not, but why not? Does a mashup require display on a web page, i.e. a web browser as a display mechanism? That seems overly restrictive. Perhaps it requires the logic being applied to the data through server scripts preparing it for final display. Google Earth with KML generally just displays static data, but it certainly can allow that through KML via a network link. Perhaps that is why so many in the geospatial community are excited about the network link in KML. It allows Google Earth to display live data, albeit in a pretty hacked up way.

I submit that the current definition of a "mashup" is a somewhat misleading goalpost for the amateur GIS community. The underlying concept of displaying location based information is much broader than what is contained in the current crop of mapping APIs. Also the restriction of server based processing of data is one that isn't an inherent restriction, there just aren't any tools to do general geoprocessing on the client side. Yet.

Update: I forgot about ArcExplorer which is supposed to have a pretty decent set of client side geoprocessing tools. That's what I get for trying to post when looking after a toddler and 6 week old...

What's the big deal about KML format?

2006-02-01T07:02:00.000-08:00

Looking over some different blog posts about the wonders of the KML format, I am left scratching my head as to what is so great about it. Certainly, it is nice to have a way to specify certain geospatial features, camera values, overlays, etc.. along with network links to other information in the same file in XML format. But each of these features are pretty obvious in themselves.

It's always the user interface that is the make or break issue of any software application. KML is pretty good at specifying what you are going to see and from where you will be looking. However, there is a disconnect in how this information is used by the underlying graphics engine and how those files and their contents can be managed and manipulated by the interface. The My Places/Temporary Places link area is very klunky and limited in my opinion.

As a format, GML is more precise and complete and would be a better basis for a feature format. One is a very nice number, having all of the information you need in a single file is very compelling from a standpoint of simplicity. The network link allows data from various sources to be cobbled together, very powerful, just like HTML. Could this be improved upon?

The idea I have for EarthBrowser is of a self contained data package that contains different elements such as features, camera settings, overlays, image files of various formats, shapefiles, engine scripts in separate files (or together if it makes sense) all packaged together in a zip file (or my favorite a bzip2'ed tar file). This reduces the number of network connections needed, consolidates the data so you won't have a situation where a critical part of the data is missing due to a download failure and eliminates the program states where only partial data is available. The engine script can use the a pre-existing user interface system to decide how it will display itself in the interface. For very large data packages, it becomes feasable to share them as a .torrent file amongst other EarthBrowser users, reducing the server load for popular packages and improving availability for end-users. One negative is that since you need the whole package, there is no way to display information as it arrives like a web page. However for 3D environments it may not make much sense, or be possible, to display partial information like HTML.

WMAP high resolution cosmic background imagery

2006-02-01T06:32:00.000-08:00

The Wilkinsin Microwave Anisotropy Probe (WMAP) has some spectacular new imagery of the cosmic background radiation. The cosmic background radiation is the leftover "glow" of energy from the big bang. The lighter colors represent hotter areas in the background of the visible universe.

WMAP microwave sky image

A spherical mapping would be a nice background for EarthBrowser

Images courtesy of the NASA/WMAP Science Team

There are a number of microwave bands that could be used (K, Ka, Q, V, W). Now I just need to find the time to reproject them and, the hard part, put in a selection interface. Things like this will be so much easier to add to version 3.

Thanks to ccablog for the find!

More on Microsoft's geometry clipmap patent

2006-01-20T09:21:00.000-08:00

Here is the text of the geometry clipmap patent application titled "Terrain rendering using nested regular grids."

The researchers who have developed this idea (Losasso & Hoppe) published an article on their technique in NVidia's "GPU GEMS 2." That seems like a pretty lame thing to do, given that they applied for the patent in early 2004. I've come across a discussion thread at gamedev where the general consensus is that it is a defensive patent and they wouldn't defend it against others who would use it. While this may be the case for most games developed, the virtual globe market is red hot and I would assume that they would use this against Google, ESRI and any other virtual globe product out there that uses it to give their offering an edge.

Current EarthBrowser status

2006-01-19T19:32:00.000-08:00

EarthBrowser version 2.9 is coming out soon. I am abandoning CodeWarrior as a development environment. It was a lifesaver in the 90s, but with XCode and Visual Studio, I understand why Motorola has them focusing on embedded developers.

I got a new iMac Intel (is that what they are calling them?) and have been preparing the Universal Binary release. Version 2.9 will have some major speed boosts due to better hardware optimizations. It will also feature a completely new database engine which is faster, portable and open source. Apple has expressed interest in distributing EarthBrowser as part of their school software bundle program, but we haven't signed any contracts yet.

It's like running through water to be developing the OpenGL based version 3 and have to keep going back and updating the old code base. At some point I hope I can focus on the new version exclusively. Speaking of the new version, I had YAMS (yet another major setback) on that front. I had been trying to develop my own clipmap and geometry clipmap classes for the past few months. After googling for some more recent info on the geometry clipmap, I came upon Microsoft's patent application for the method. Very disappointing, but I'm certain they will get granted many of their claims. I can't take the risk of implementing patented code, so guess I'll be moving to geomipmaps, they are simpler anyway. All those guys over at gamedev.net who are putting it into their games should be made aware of it, but I am not a member there, just a lurker.

I've been working on version 3 off and on for the past year and a half and the foundation is very stable, scriptable and extensible. Once my second son is safely getting accustomed to this world after a few weeks, I hope to dig in and really bring the version 2 and 3 functionality closer together.

Blog Purpose

2006-01-18T11:42:00.000-08:00

I envision the Earth Browser blog as a place to discuss issues and ideas about software development, earth science, GIS and social implications of all of the above.

I produce a software product called EarthBrowser that I have been developing since 1996 in graduate school. It first went on sale in 1998 as a Macintosh shareware product called Planet Earth. It is, as far as I know, the first commercially available product to display dynamically updated earth data (just clouds).

EarthBrowser now has many competitors, most notably Google Earth (formerly Keyhole), ESRI has just put out ArcGIS Exporer and Microsoft is rumored to be putting out it's own 3D earth product. Each of these products will have major infrastructure support and are being given away for free. Even with these challenges I believe that EarthBrowser will flourish due to it's superior user experience, relevant real-time data, excellent support and focused, dedicated and small (just me) development staff.

Welcome to the Earth Browser blog.