UMN Mapserver + MS4W + MrSID = Happy Again

I saw on the Mapserver documentation of supported formats that MrSID and a number of other raster formats were also supported through the use of gdal as a 3rd party component. Well, I had read this to be about the same as Manifold saying that it supported MrSID through 3rd party software. Which was not very well, if at all.

You can preach to me about how uncool MrSID is until you are blue in the face, but the fact is that is the way most of our imagery data comes to us unless we want to pay more or wait for publicly available quarter quads in different formats. And these are mostly large full county mosaics, so decoding or gdal-translating them is a pretty inefficient process. So that’s what I’m stuck with unless I absolutely have to change.

On a lark, I decided to use one of the many sample Mapserver map files and edit it to point to one of the full county sid’s. Bam! Just like that, I now had a WMS image server. I was able to immediately hook it up to UDig and with a bit of experimentation, also hook it into ArcGIS Explorer. This is a ~250mb .sid file and it was being served out by my test server in the same intranet with equal or greater speed than AGX could read the file locally. UDig was even better, it actually displayed the raster faster from Mapserver’s WMS than ArcMap did reading it directly from the disk.

So, for the moment, I’m going to cool off about Manifold (I really didn’t want to have to do that data conversion right now anyway). And I’m going forward with Open Source. I’ll use Mapserver installed from the MS4W package to serve the imagery at least. The data will probably also be from Mapserver but I’ll be using SLD as the cartographic options for polygons and various fill types aren’t directly available in map files alone.

I like ka-Map or MapGuide OS as web-based clients. I’m most excited about using AGX and it’s .NET based custom tasks for a desktop based client.

My only gripe about Mapserver is that there doesn’t seem to be a good way to avoid hand coding some of the map file, unless you use MapScript. Since I installed PHP 5, MapLab doesn’t work very well, and you have to fill all that stuff by hand anyway.

If I have the time (read, not anytime soon), I’d like to make a simple multi-file select data uploader & styler that then writes the actual map file for you. And will automatically attempt to read the projection info and spatial extents of your data.

I probably won’t do much with this until the end of the week, but if I getting a working demo, I’ll defiantly post with pictures about it.

Manifold IMS Redux

In which I eat a small bite of crow, and then discover all this head banging is fairly irrelevant.

My previous post really took on a new life of it’s own when James Fee linked to it in his blog. I have recieved several emails from Manifold users & a number of comments pointing me it the right direction to get Manifold IMS working properly.

The gist of it is that I merely skimmed the Optimizing Performance document, as I thought it was just that (squezzing every thing you can from a service), and not an ESSENTIAL document on setting up Manifold map files for serving over the web. The main thrust of this document is that you need to minimize the size of the .map file in relation to your server’s RAM and that if you are connecting to a database of anykind, use the best driver for that connection.

With only 512kb of RAM in a AMD 1.8ghz machine running Windows XP as my home test server, a 1.4GB .map file is definatley NOT optimized in relation to the available RAM. So for the image server componentI exported my 256 quarter-quad ecw’s to a single even further compressed ecw. Then I created a new project and linked the single image, created a map and exported it as an ASP.NET webpage with support for WMS. I ended up with a 130kb .map file. I was able to view the imagery on the server and link to it in another application on the server. However, there are obviously some security permission issues that are preventing me from getting to the data on any remote machine. If I decide to pursue the Manifold IMS stuff further, I’ll post to the forums and get that figured out.

So the image server train wreck was fully my fault. Now as for the actual mapdata IMS project, that did not go as well as the image server. Even after linking the shapefiles and PGDB’s, the data was served very slowly and draw incompletely or not at all. I think the bottom line on that one is that I need to use a RDBM to store the spatial data in. The ESRI formats are just not efficient to work with on this scale in Manifold. I’d had already planned on using a RDBM for this stuff anyway, but I’d rather go with PostGIS, thus giving me spatial SQL accessable from a wide variety of clients.

Now, why didn’t I link this stuff to begin with. Well actually, I did, when setting up the desktop project. I figured that linking would be a much more efficient way to deal with data after reading some of the intial documentation and tutorials. However, every time I tried to pan, zoom or resize my Map with either the imagery and/or spatial data files linked, Manifold crashed. Now this was not on that crummy old test server but on an AMD Athlon 64 3400+ with 1GB RAM and a pretty good video card. ArcMap 8.3, UDig, QGIS, & the TatukGIS Viewer were all able to handle the same level of non-imagery data or much more without any severe problems.

So I’d like to apologize for fully flaming Manifold before at least trying to get a solution from Manifold first. However, I was just really tired of fighting with this software. There were the crashes I talked about above and I’m still pissed about the MrSID support. If you don’t really offer it, then say so. It would have been pretty easy to hyperlink the “with 3rd party decoder” statement next to it in the Supported Formats section over to the “Why we don’t really do MrSID” page of the online help manual. The proported support for this format, which I have a TON of data in and continue to recieve data in, is the main reason for purchasing Manifold in the first place. I’ve felt somewhat cheated ever sinceI realized that this was not really supported. So I had a lot of pent up frustration that just finnally boiled over when the promise of a “click-click-done” IMS solution didn’t work out as planned.

For why I think this all irrelevant now, see next post.