Massive performance improvements in converting MrSID files

I finally decided that there is probably no really fast method for distributing 280+ mb MrSID files over the internet without any manipulation. Mapserver worked pretty well locally but really bogged down to an unusable level over the internet. Open Source MapGuide serves MrSIDs as well but was even slower for these large county wide mosaics than Mapserver. Putting a tile server in front of either of these products may have helped but attempts to do so were unsuccessful using IIS and Win XP Pro as my test server.

The main problem I’ve had is that these large files did not convert into a better format very easily. MrSidDecode just decommpresses them to formats which would take several gigs per county. gdal_translate will directly convert from MrSID to another compressed format. However, the performance on these large images was truly terrible. So I thought I would use either MrSidDecode or gdal_translate to convert chunks of the larger image and then convert those images to ecw or jpeg2000. I kept getting an error with gdal_merge and was looking for help on that when I ran across this hidden gem of a response by Frank Warmerdam.

[Gdal-dev] gdal_translate versus mrsiddecode

http://lists.maptools.org/pipermail/gdal-dev/2005-July/006051.html

Setting GDAL_FORCE_CACHING = YES made an enormous difference in the performance of gdal_translate and allowed me to create 10×10 grids of ecw “tiles” from the county wide imagery with a single command per “tile”.

Previously when I tried this it took ~8hrs to convert 1 of the 100 “tiles” from the large MrSID to a more reasonably sized ECW file. This was just too slow for me. With the correct cache setting it now took less than 8min per “tile”. An over 60 times better performance.

This setting seems to work best with small request or files (in apprently chooses this automatically on the small end) or very large requests or files. I tried this same setting on the 2004 NAIP imagery which I had in individual Quarter-Quads already and the performance difference was only about a 10% decrease in the processing time.

I created a tile lookup file for MapServer and used it instead of the MrSid file as the data source and the performance was great over the internet & locally. Now if I can get a tile server in front of this it should be screaming fast.

UMN Mapserver + MS4W + MrSID = Happy Again

I saw on the Mapserver documentation of supported formats that MrSID and a number of other raster formats were also supported through the use of gdal as a 3rd party component. Well, I had read this to be about the same as Manifold saying that it supported MrSID through 3rd party software. Which was not very well, if at all.

You can preach to me about how uncool MrSID is until you are blue in the face, but the fact is that is the way most of our imagery data comes to us unless we want to pay more or wait for publicly available quarter quads in different formats. And these are mostly large full county mosaics, so decoding or gdal-translating them is a pretty inefficient process. So that’s what I’m stuck with unless I absolutely have to change.

On a lark, I decided to use one of the many sample Mapserver map files and edit it to point to one of the full county sid’s. Bam! Just like that, I now had a WMS image server. I was able to immediately hook it up to UDig and with a bit of experimentation, also hook it into ArcGIS Explorer. This is a ~250mb .sid file and it was being served out by my test server in the same intranet with equal or greater speed than AGX could read the file locally. UDig was even better, it actually displayed the raster faster from Mapserver’s WMS than ArcMap did reading it directly from the disk.

So, for the moment, I’m going to cool off about Manifold (I really didn’t want to have to do that data conversion right now anyway). And I’m going forward with Open Source. I’ll use Mapserver installed from the MS4W package to serve the imagery at least. The data will probably also be from Mapserver but I’ll be using SLD as the cartographic options for polygons and various fill types aren’t directly available in map files alone.

I like ka-Map or MapGuide OS as web-based clients. I’m most excited about using AGX and it’s .NET based custom tasks for a desktop based client.

My only gripe about Mapserver is that there doesn’t seem to be a good way to avoid hand coding some of the map file, unless you use MapScript. Since I installed PHP 5, MapLab doesn’t work very well, and you have to fill all that stuff by hand anyway.

If I have the time (read, not anytime soon), I’d like to make a simple multi-file select data uploader & styler that then writes the actual map file for you. And will automatically attempt to read the projection info and spatial extents of your data.

I probably won’t do much with this until the end of the week, but if I getting a working demo, I’ll defiantly post with pictures about it.