Massive performance improvements in converting MrSID files
9 01 2007I finally decided that there is probably no really fast method for distributing 280+ mb MrSID files over the internet without any manipulation. Mapserver worked pretty well locally but really bogged down to an unusable level over the internet. Open Source MapGuide serves MrSIDs as well but was even slower for these large county wide mosaics than Mapserver. Putting a tile server in front of either of these products may have helped but attempts to do so were unsuccessful using IIS and Win XP Pro as my test server.
The main problem I’ve had is that these large files did not convert into a better format very easily. MrSidDecode just decommpresses them to formats which would take several gigs per county. gdal_translate will directly convert from MrSID to another compressed format. However, the performance on these large images was truly terrible. So I thought I would use either MrSidDecode or gdal_translate to convert chunks of the larger image and then convert those images to ecw or jpeg2000. I kept getting an error with gdal_merge and was looking for help on that when I ran across this hidden gem of a response by Frank Warmerdam.
[Gdal-dev] gdal_translate versus mrsiddecode
http://lists.maptools.org/pipermail/gdal-dev/2005-July/006051.html
Setting GDAL_FORCE_CACHING = YES made an enormous difference in the performance of gdal_translate and allowed me to create 10×10 grids of ecw “tiles” from the county wide imagery with a single command per “tile”.
Previously when I tried this it took ~8hrs to convert 1 of the 100 “tiles” from the large MrSID to a more reasonably sized ECW file. This was just too slow for me. With the correct cache setting it now took less than 8min per “tile”. An over 60 times better performance.
This setting seems to work best with small request or files (in apprently chooses this automatically on the small end) or very large requests or files. I tried this same setting on the 2004 NAIP imagery which I had in individual Quarter-Quads already and the performance difference was only about a 10% decrease in the processing time.
I created a tile lookup file for MapServer and used it instead of the MrSid file as the data source and the performance was great over the internet & locally. Now if I can get a tile server in front of this it should be screaming fast.
Hi Matt,
thank you for this interesting post. Could you make an example of the command you write to produce 10×10 grids of ecw “tiles” from the county wide imagery?
Thank you again,
andrea
Well that is actually a series of commands (100 to be exact) to create this 10×10 grid. I wrote a quick & dirty batch file creator in VB.net to auto-create all these commands. I really should have written it in Python or some other scripting language, but I know vb.net & c# better and had already written a similar program over a year ago so it was easy to reuse. I then had to monkey with the batch file quite a bit, once I really understood what it was that I wanted gdal_translate to do. I’d be happy to post that batch file creator here if you or anyone else is interested.
Hi priour,
thank you for your reply. I’m interested in your batch file. Only a question. Which command I must write in my shell If I would use gdal_translate to convert one mrsid file in ecw format using GDAL_FORCE_CACHING parameter?
Have a nice day
This sounds very intriguing. At work we simply place the mr. sid files in a folder but if we could break them into tiles and somehow serve them up I am sure things would speed up. I would be very interested in your code.