Why I love living in the sub-tropics

Moved to Personal Post Archives

Massive performance improvements in converting MrSID files

I finally decided that there is probably no really fast method for distributing 280+ mb MrSID files over the internet without any manipulation. Mapserver worked pretty well locally but really bogged down to an unusable level over the internet. Open Source MapGuide serves MrSIDs as well but was even slower for these large county wide mosaics than Mapserver. Putting a tile server in front of either of these products may have helped but attempts to do so were unsuccessful using IIS and Win XP Pro as my test server.

The main problem I’ve had is that these large files did not convert into a better format very easily. MrSidDecode just decommpresses them to formats which would take several gigs per county. gdal_translate will directly convert from MrSID to another compressed format. However, the performance on these large images was truly terrible. So I thought I would use either MrSidDecode or gdal_translate to convert chunks of the larger image and then convert those images to ecw or jpeg2000. I kept getting an error with gdal_merge and was looking for help on that when I ran across this hidden gem of a response by Frank Warmerdam.

[Gdal-dev] gdal_translate versus mrsiddecode

http://lists.maptools.org/pipermail/gdal-dev/2005-July/006051.html

Setting GDAL_FORCE_CACHING = YES made an enormous difference in the performance of gdal_translate and allowed me to create 10×10 grids of ecw “tiles” from the county wide imagery with a single command per “tile”.

Previously when I tried this it took ~8hrs to convert 1 of the 100 “tiles” from the large MrSID to a more reasonably sized ECW file. This was just too slow for me. With the correct cache setting it now took less than 8min per “tile”. An over 60 times better performance.

This setting seems to work best with small request or files (in apprently chooses this automatically on the small end) or very large requests or files. I tried this same setting on the 2004 NAIP imagery which I had in individual Quarter-Quads already and the performance difference was only about a 10% decrease in the processing time.

I created a tile lookup file for MapServer and used it instead of the MrSid file as the data source and the performance was great over the internet & locally. Now if I can get a tile server in front of this it should be screaming fast.

5 Things You Don’t Know About Me

Just catching up on some blog reading after taking a break from my computer for the holidays, and I noticed I’ve been tagged by Steve. So here it goes.

1. You would never know it now, but I was a very fat child & adolescent. When I entered my sophomore year of high school at 15, I weighed 285 lbs at 5 ft 9 in. I wore a size 38 pant & only those really baggy cut jeans fit me. By the time I turned 16, I weighed 210 at 6 ft 1 in. I hit a growth spurt, played football, and started weight training and they all combined to completely reshape my body. It took many years before I stopped thinking of myself as “the fat kid”. I now weigh even less and a few years ago got down to the lowest weight since 5th grade while training for my first (& only, thus far) marathon.
2. I am very much a cat person and don’t really like dogs all that much. I’ve never owned a dog & maybe one day will, but I’m in no hurry. However, I couldn’t imagine life without cats, and have always missed having them around whenever I didn’t own any cats.
3. I can’t remember ever actually believing in Santa Claus. I figured it out at age 4 after seeing 3-4 mall Santas in one day. I couldn’t figure out which one was the real one and why my parents where trying so hard to find some gifts for my baby sisters that they said Santa was bringing them. I asked my mom point blank and she gave me an honest answer. I then promptly told several of my friends and was on their parents “naughty” list.
4. I have a serious problem with getting addicted to video/computer games. So much so, that I won’t even play minesweeper or solitaire on my computer. If it is a computer based game, no matter how childish, stupid, simple, or boring it is, you can bet that if I start playing it I won’t be able to stop. And if it is even a little bit good, I’ll ignore work, eating and sleeping to play it. I was even addicted to a PC port of some Nintendo based Pokemon trading game in Japanese. I stayed up 3 nights in a row and ignored work for nearly a week to solve that game. Just in case you were wondering, “Yes, I was WAY to old to be interested in Pokemon” and “No, I don’t speak Japanese, and had no idea what any of the characters were saying”. I avoid video games like the plague and that is the only way to handle that problem for me.
5. I am the super “un-handy” man. If you have something that works and you want it broken, just let me work on it a bit. One of the worst jobs of “fixing up” something I ever did is on display in our living room. I was given a small side-table height chest of drawers. It was the perfect size for our TV which was way too big for our old stand. So I though, wouldn’t it be cool if I made the bottom drawer into a VCR/DVD player compartment with a hinged door. That way we could open it when we used it and keep it hidden at other times. Well, that didn’t quite happen…

I’ll pass it on to: Tim (aka Sabin21), Jeff Thurston, Brian Flood