Monday, March 19, 2007

15 meters of Landsat

I've been working on a new Landsat 15 meter per pixel base map for version 3.0 of EarthBrowser which is due out later this year. Tasks like this make it painfully obvious that I am crazy to be doing this all by myself. Here is where I'm at with that...

A couple of years ago I decided that I needed a royalty free base map for EarthBrowser that looks better than the one available through the OnEarth WMS service which EarthBrowser currently uses. My only option at the time was to download the raw data and process it myself. I wrote a script that downloaded the raw tiles from Landsat.org slowly so as not to overtax their servers. I was looking at a long, uphill battle to get what I wanted. Now I have all of the raw data, but it needed a lot of pre-processing to be a seamless mosaic that wouldn't have the abrupt changes in contrast and quality from tile to tile.

Then in November of last year I found just what I was looking for: a set of pre-mosaiced 15 meter Landsat data from NASA John C. Stennis Space Center. Two of the color channels were in the infrared spectrum, which is great because many of the visible color channels have haze in them which makes it difficult to get a seamless looking image, but the infrared cuts through the haze. However this means that it looks strange because it isn't a true color image. More image processing needed...

I keep going and began taking scenes in different regions of the world and re-coloring them by hand and using the differences in the two images to "train" an algorithm to color the entire dataset in a more realistic color scheme. First I tried using principal component analysis to reduce the dimensionality of the problem. This didn't work too well since the principal components changed from region to region. I settled finally on using a RGB to HSV conversion to separate the color components. I then use a fairly simple piecewise polynomial approximation using Jacobian matrix decomposition in three dimensions, after which I re-project it back into RGB space and it doesn't look too bad!

However I am now running up against a problem that I didn't think about until I got to this step (doesn't it always work that way). This dataset has been compressed using a "lossy" wavelet compression algorithm. That means that my polynomial approximations will work for one trained tile but won't transfer well to other areas of the mosaic due to the incosistancy of color values. I'm getting little yellow dots appearing randomly and big swaths of green where it should be dark blue due to compression artifacts. Arrrrgh!

As an example just recently I was contacted by an environmental organization from the Greek island of Lesbos asking me if I could donate a high resolution image of Lesbos for a mural at their center. Happy to help an environmental organization and also as a first real test of some of the new capabilities of EarthBrowser, I made a high res image to his specifications and unfortunately had to do a lot of hand editing in order to make it look good. Here is a before and after showing my color transform algorithm in operation.



Unprocessed NASA Landsat Lesbos

EarthBrowser Processed NASA Landsat Lesbos

Hand Edited EarthBrowser processed Lesbos



It is clear from this test that I need the data in an uncompressed format in order for my algorithms to work properly. After inquiring around, I have finally found someone within NASA that can sell me the uncompressed data on hard drives but it is $16,000 and they are not quite clear on whether it is the exact dataset that I am referencing. Needless to say, I can't afford that. So I'm kind of stuck.

There is some hope though, just recently I read on the excellent Bull's Rambles Blog that a fantastic new Landsat 15m dataset has been "donated... to the public domain." One would assume that the use of NASA resources to produce this dataset would have a prerequisite of the product being in the public domain. I've been in contact with the WorldWind project lead Patrick Hogan to inquire about it's availability, and they understandably want to have it "premiered" by WorldWind. This leads me to believe that it will be generally available at some point, but there has been no difinitive answer yet.

I'm keeping my fingers crossed because I'd really rather spend my time programming.

12 comments:

Bull_[UK] said...

getting a firm date from NASA is neary imposible, my guess would be they want to prmiere it in World Wind Java, so once thats out it should be available. Your version does look nice though.

matt_giger said...

Thanks! It doesn't look too bad from a distance. Close up the yellow speckles start showing.

Dave Smith said...

Any updates? I am currently searching for free/low cost imagery of Cameroon, for a volunteer Engineers Without Borders project to supply rural villages with water/wastewater infrastructure...

matt_giger said...

I understand that madmappers.com has a lot of African data and from a quick look they seem to have some Cameroon rasters. You could also use a carefully crafted WMS URL to get what you want from onearth.jpl.nasa.gov.

Bull_[UK] said...

I asked Maurizio from Madmappers to get in touch with you Dave, hopefully he can help.

Anonymous said...

Is this wat you are looking for...?
http://www.geotorrent.org/details.php?id=43

matt_giger said...

Actually I did download and try out that torrent a year ago. It is compressed to such an extent that I knew right off the bat that it wouldn't work. The whole 1.5 terabyte dataset compressed down to 26GB!

However, thanks for the pointer!

Erik said...

FWIW, it sounds like i-cubed processed the on their own (no NASA resources involved) and then made it PD and gave it to NASA. I'm sure the raw data will be available once everyone involved has had a chance to milk the PR for all it's worth.

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/03-20-2007/0004549264&EDATE=

matt_giger said...

Thanks for the link Eric, it was an excellent read. From what little I was able to find previously it was hard to understand exactly what was going on. I'm still a bit confused, the article makes it sound like the i-cubed processing was done from within WorldWind appliation itself. That would be a bit suprising, but not out of the realm of possibility I suppose.

That i-cubed made such a valuable resource public domain after devoting their own time and resources to it is quite astonishing. I applaud them! Similar datasets are being sold right now for almost $200K (i.e. the NaturalVue and TerraColor datasets)

Erik said...

Yeah, it sounds like the actual processing know-how and horsepower was provided by i-cubed (a geospatial data provider) and Isilon (who sells big storage hardware). Then they gave it to NASA who made it available as a WW layer while all the parties worked together to write a 100% confusing press release.

matt_giger said...

Hilarious... :-)

Thomas pedro said...

These proportions can be surpassed to some degree with repaying variables. Candidates for credits may have a pay of up to 115% of the middle pay for the region. Families must be without satisfactory lodging, yet have the capacity to bear the cost of the home loan installments, including charges and protection. Cash Advance San-diego