Wednesday, September 20, 2006

GPU processing for GIS is coming!

There is a very interesting article over at The Register that has some "insider" info on a new product from AMD/ATI. ATI has created a server product that will make the raw processing power of the GPU available to general applications. This has incredible implications for the GIS software field.

The advent of programmable vertex and shader processing has opened up a reverse-pandora's box in my opinion. A whole lot of good things are coming out of the ability to access the highly optimized floating point processing power available on modern GPUs. However, like the external floating point processors from the '80s there are some significant issues with using this technology.

Stream processing isn't something that is applicable to general software development, but scientific computing can use it to great advantage for problems that fit within it's scope. It is most useful for taking a large block of homogeneous data and applying fixed transforms (guided by variable parameters) on it and storing the results in a similar block of data, that resulting data can then be further processed.

For anyone in the field, the work done by McCool, et. all, at the Univ. of Waterloo on the Sh metaprogramming language has given a glimpse of what was to come. When I first read the book describing Sh, I wanted to use it immediately but found it a bit limiting due to it's static nature and the inability to treat shading programs as dynamic assets as the code you write is basically hardwired into your executable. There is, however, an intermediate format that could possibly be used as a game asset, but it is just a little too funky for use as a general purpose shader generator. This programming model fits scientific (GIS) applications much better. I'm looking forward to seeing what is coming out of AMD/ATI. We are just scratching the surface right now, I think that there needs to be more evolution in the field.

I am planning on making EarthBrowser plugins that make use of this kind of functionality in the future. All the pieces are in place right now for version 3.0, but each plugin has to be written for a specific problem domain.

Friday, September 15, 2006

Goodbye Xena, Hello Eris

The planet formerly nicknamed "Xena" has officially been renamed Eris, which was suggested by it's discoverer Michael Brown. Eris is the Greek goddess of strife, which is what it's discovery has caused. The commotion ultimately led to the demotion of the smaller Pluto to the status of dwarf planet. Eris has one known satellite and that is now named Dysnomia formerly nicknamed "Gabrielle."

Pluto has always been an interesting object for many reasons and many people are upset that it is no longer considered a planet. However I agree with the new designation, it's orbit was too wild, it is too small and the alternative of adding the other newly discovered Kupier belt objects to the list of official planets is not really an acceptable outcome. Pluto has received it's minor planet number of 134340.

Here is the official IAU designation.

Thursday, September 14, 2006

Don't be evil

Well, I guess the other shoe has dropped. Google has now hired the most evil lobbying shop in the country. I recall that they previously acquired their earth imagery with a restrictive license not allowing other companies to license the same data. I can't recall the company this was with, but suffice it to say that the "don't be evil" ruse that has given them the benefit of many a coder's doubt is now inoperative. Too bad, I wanted to believe that a corporation could actually try to do good for the world.

Some people will say that they are just paying them as a form of protection money from the Republican juggernaut. Some will say that it is the responsibility of a publically traded company to maximize it's shareholder's profits. Some will say that maybe they won't have them pull any of the dirty tricks that is the only reason to hire DCI. Sorry, but that is just too many ifs and maybes, as should be, you are judged by the company you keep. Corporations do not need to be sociopaths. Brin and Page should be ashamed.

Google fanboys, join the Republican fanboys. Sadly, you are now the same.

Sunday, September 10, 2006

GML & JP2 a simple concept with real impact

Funny how little it takes to make programmers happy, just make our jobs easier. I mentioned in my last post that AJAX does something fairly obvious and it is a quantum leap in the web browser world. Sadly most people still think of the web browser as the only interface to the internet. Browsers are so needlessly restrictive, I suppose security issues have a lot to do with it, but I think it is mostly a lack of imagination.

That brings me to GML in JP2. Jeff Thurston over at Vector One has a post up which goes into just enough detail to let you know that it is a good idea. Not brilliant, not even tricky in the least. The designers of the JPEG 2000 standard saw fit to allow the addition of text metadata along with the picture in their format. That is all that is needed to add a simple xml file conforming to the GML specification which will allow great things like telling you how the JP2 image(s) are to be georeferenced, you can even add other features like point and polygon features. TIFF allows this as well, but anyone who has looked at the guts of libtiff or seen the contortions need to make an image conform to the GeoTIFF "well known text" spec knows that it is anything but easy.

Why is it a big deal? It is and it isn't. JPEG has comment tags, but I'm not sure if they have a length limit, and I don't really care to check. PNG? same thing. The reason it could be a big deal is because people are talking about it and building a de-facto standard that will be presumably supported in future GIS software. Much like the "well known text" popularized by the ubiquitous GDAL and PROJ.4 packages, a tacit agreement that you check your JP2 images for GML metadata is all that is needed for this to make a significant impact.

But what is the impact of GML & JP2 exactly? By combining the image with the metadata you make distribution, storage, reference and cataloging of such datafiles an order of magnitude simpler. Like my previous post about the "seduction of the one" having one anything makes it so much easier in so many ways. Just think of the all of the possible states inherent in the horrid shapefile format, you have the .shp, the .dbx and possibly the .shx file just to describe some vector data. Right there you have a lot of code to manage all of the possible states of the files. You also have to have data transport functionality that can send and request data from more than one file if needed. It is about an order of magnitude harder to handle a multi-file format than it is a single file one. Not only that, but if you do support a multi-file format you are boxed into not doing something really slick that is only possible from a one file format.

As an example, EarthBrowser v3.0 has a really neat unified data stream architecture internally which I can attach arbitrary metadata to each stream. A simple but powerful concept that can simplify object data interfaces across the board. If I had to make a unified data stream structure that incorporated multiple streams as different parts of the same data, well no thanks. If you really had to do support something like that, you would probably want to pre-process the data to an internal format when reading and just bite the bullet and code it up if you ever had to write that format out.

There you have it; GML and JP2. Highly compressed raster data along with descriptive metadata all in one package. I give it a big thumbs up, at least someone is looking out for us poor overworked programmers.

KML needs an AJAX style upgrade

The cranky older programmer is back for another session detailing the failings of today's "best of breed" applications. Take AJAX, the acronym for a simple concept that has brought about a new excitement to web development, dubbed "Web 2.0." I've read so many excited articles and posts about how great this new capability is and how it is going to bring about a resurgence of the dot com boom. That may be the case, but let me say something about the underlying technology of AJAX.

Cutting away all the marketing hype, the two keys to the whole thing are 1) the ability for a script to download data asynchronously and receive a callback when it has arrived. 2) the ability to change the contents of a web page without the entire page refreshing. Of course there are a lot of third party packages springing up around these capabilities but they all rely on these foundations. From the perspective of a programmer, my question is: what took the web browsers so long? Gee, let's let them specify their own download URLs and process them with a scripting language rather than forcing static urls and the fixed processing of interface elements by the web browser itself. Guess what else follows that outdated content model, yep, it's KML and Google Earth. At least the first few halting steps are being taken in the web development world thanks to Javascript and XMLDom.

Now, why is KML lame? For a simple example, you can write a KML file to show a set of georeferenced data points. That's great, but what if you have a set of points with different attributes that you want to publish and view, but not all at once. Perhaps you want to enable some selection criterion for the data points, such as earthquakes during an arbitrary time frame? A hokey solution could perhaps be ginned up using the network link and having a separate server script parse the link and return the appropriate data. However, how are you going to provide an interface for them to make the arbitrary selection? With KML? At least HTML has Javascript and GUI elements, KML and Google Earth have no easy client side data capture or processing capabilities that aren't hard coded into GE itself. Please correct me if I'm wrong and I'll eat a heapin' helpin' o' crow.

I look at the spectrum of software development going on in various industries right now and am somewhat stunned by the isolation each has. The game industry is doing some fantastic work, the best in the industry, but the GIS field doesn't seem to pay any attention. Web development is a huge chunk of the market but is completely isolated by the web browser. Everyone has web browsers but writing a plugin for all browsers is a hurculean task. I heard that the Opera web browser will include a BitTorrent client. Hurrah, someone is getting it!

I must say again that I think that Google Earth is the best earth explorer out there right now. However there really needs to be some serious innovation and insight done in the software industry in general. I've taken a look at many open source projects out there and am shocked by how many are written in C. Of those that are written in C++ (the only language for serious library and application development work) fewer still incorporate the STL libraries. I've only come across two that use the boost libraries, yikes! I realize it takes a lot of time and effort to keep up with the state of the art and evolve along with the other industry segment's innovations. Perhaps the corporations putting out software today are just unwilling to support the continuing education of their primary product creation assets, the programmers. The programmers themselves need to take responsibility for their own continuing education during non-work hours if they want the be marketable in the future. Finally product and project managers need to have some real programming experience (at least 5-7 years) to be considered competent in my opinion. It's too easy for an inexperienced programmer to bamboozle someone who doesn't know about development work, it is also too easy for a forceful manager to impose unrealistic expectations on a development team.

Let's all celebrate the advent of AJAX, the ability for client side control of what to download and how to process and display it. It only took a decade, at least it's a start. Google does it with Google Maps, think guys.

Sunday, September 03, 2006

Google Earth and KML are outdated

The developers of Google Earth boast that it works well with older graphics cards, that is due to the fact that is based on an aging codebase and was designed with those components in mind. EarthBrowser (as primitive as it was) had the virtual globe market all to itself from 1998 to about 2002 when I first heard about Keyhole EarthViewer when someone from Keyhole offered to buy earthbrowser.com and my customer list. The code is probably still based on their original code developed back then. I can see that they have added more and more on top of that old code. Believe me, you can really pile new features on top of an old foundation but it just gets harder and harder. It gets harder to add new things and it gets harder to change old ones. Just look at Windows or any product that has been out there for any period of time. EarthBrowser itself has been rewritten twice already and I'm on my third complete re-write. A complete re-write is time consuming, difficult and almost never done for a commercial product due to the time, cost and difficulties in such an undertaking of an established product. However, the benefits of such a re-write are immense, for an example, look at the transformation of MacOS since Steve Jobs came back.

Back to Google Earth. Google seems good at publishing APIs for their web services, like Google Maps. Unfortunately for people who use them, they reserve the right to yank the rug out from under you at any time with their licensing terms. The Google Earth API is effectively KML, perhaps they are going to introduce something new after the acquisition of Sketchup which has a reportedly nice Ruby API, but for now it is just static data with the weakest possibility of time based animation through some kludgy use of the "network link." By the way, what kind of hyperlink is not a "network" link? Just wondering.

I don't want to help Google with any of it's problems since they are the main competitive threat to EarthBrowser. I tried writing a file format for EarthBrowser back in 2000 (version 1) that would use static files to control actions in EarthBrowser. It was a very difficult to thing to keep up with. Any new action had to be first coded into the program, then an interface to that code provided and access through the file format. Tweaking that was a nightmare so I tried making my own tiny programming language to embed in the files to provide more flexibility. It worked OK, but that just added another layer of complexity along with the file format and the hand coded feature linkings. You can see the dilema they are facing by adding more and more features into KML that need to be supported in current and future GE versions. They need a rewrite, but guess who will have a new virtual globe out soon (besides EarthBrowser)... Microsoft. I don't know what MS has in store for us but from the looks of Flight Simulator X, it will blow GE out of the water.

Here is discussion of one of the extensions someone has made for Google Earth using KML. It is just so kludgy...

I've learned the lesson of trying to make a static file format control a virtual globe. The next version of EarthBrowser will feature a fully scritable game engine which users and developers can add their own data types and visualizations directly on the globe. As an example, it will be possible to code a module that will download raw data from any source (e.g. NASA) and process it for display on the globe. From this scriptable code base it will be easy to create a KML converter and easily update it with new versions of KML with few changes to the actual code base.

I'll leave you with an early screen shot of EarthBrowser under development: