ArcGIS REST & JS API, The Real Deal?

Well, shut my mouth!

After complaining & proclaiming loudly that there was no way anything coming out of Redlands was going to be good semantic REST or true http goodness, I seem to have been proven dead wrong.

The REST & JS API examples released by ESRI today really look good. I’m impressed by what I’ve seen so far. I’m not alone as James Fee & Sean Gillies have already written up very approving posts as well.

I got my 9.3 private beta invite last night and am looking forward to taking these once sorely missing pieces out for a test drive.

Where does that leave the ArcDeveloper.Net REST for AGS project? I can’t speak for the others involved, but I for one would still like to proceed ahead. Not for the primary reasons stated before, frustration with the Web ADF & lack of faith for ESRI to produce semantic REST, but mainly for having .Net versions of TileCache & FeatureServer combination. The intention never was to couple it very tightly with AGS. Therefore having this program available when Python under IIS is just not an option, would still be a great benefit to the community. Additionally, for those not upgrading to AGS 9.3 anytime soon, such a program could allow them the lightweight interactions missing from 9.2.

Just Wait It Might Be Great

UPDATE: Well it looks like it just might be great. I hope it is.

That is what James Fee is seeming to say. He seems to thinking that building our own RESTful interface to ArcGIS Server might be they way to go but not until we at least see what ESRI has developed.

Well I respectfully disagree. I would be willing to bet that whatever ESRI cooks up will NOT be semantic REST but rather some RPC-REST hybrid at best. Trust me, Sean Gillies is going to want to burn the letters E, I, S & R in effigy once their marketing machine gets going. Your going to see REST API all over the place and to add insult to injury, that incorrect term will be used to describe something that is barely RESTful & defiantly not semantic REST  I think we as a community could probably do better than that. (disclaimer:I have absolutely no idea what the REST interface to AGS will look like, I’m purely speculating)

Who among us doesn’t think that a Community Driven, Standards Based Open Source project would do a better job of serving our needs and be more responsive and flexible than what ESRI is cooking up. Waiting for ESRI is like “Waiting for Godot”. If want to directly use the various feature storage formats out there, raster data, or WMS/WFS then there are good open source tools which allow you to do that. However, what if you really want to harness the power of the ArcGIS Server stack and just get it to listen & talk in a wider and more flexible array of languages. Make it so that you can plug in AGS into just about anything out there. AGS is a very powerful tool that should power backend solutions, fire & forget geoprocessing, etc..For this I think a community supported open source solution is going to be the best. I say “strike while the iron is hot”, if we can direct the anger & frustration of the WebADF that is boiling over right now into a more useful purpose then let’s do it now rather than later.

The Web ADF, may it REST in peace

There has been a lot of blowback against the ESRI ArcGIS Server WebADF in the last week. Dave’s complaint against an obvious short coming in the Web ADF was echoed by Doron Yaacoby and the steam really started to to vent in the comments of James’ post. Dave has since made a call to action, to paraphrase: “stop whining, or inventing your own secret personal version, and let’s get together and build something!” I could not agree more. In fact it is exactly what I’ve been turning over in my head for a while, but it didn’t seem like something that could be properly done unilaterally

The Basic Question

Why do most people want to use the Web ADF? They want to publish their content & let users interact with it. The Google/Yahoo/Microsoft map API’s & even more so, the OpenLayers & FeatureServer combination has shown us that it doesn’t require the bloated, overly chatty, buggy, difficult, and EXPENSIVE Web ADF solution. You could do more, so much more, easily with a good common-sense standards based approach that allows easy integration into your favorite Web Mapping display engine.

Where to Start

A well thought out RESTful interface with AGS would be a great start down this path. There are really only 2 resource types required to make web maps.

• Tiles (raster data or pre-rendered cartographic output)
• Vectors, of which points are somewhat of a special case

TileCache and FeatureServer are great examples of how to access and emit this data in an extensible, lightweight, standards based manner. While there a few things I would like to change about FeatureServer, I think this is a great model to start from.

True RESTful URL’s

There has been a lot of discussion in the last year about what constitutes a proper RESTful URL. The most vocal geobloggers on this topic have easily been Sean Gillies (the Grumpy Guardian of GeoREST) and Sebastian Good, though others have chimed in as well. Is it just a URL that will point to the same resource regardless of it’s structure (i.e.. Permalink & RPC/REST hybrids) or is it a URL with limited or no query string and each directory of the URL contains a standardized semantic meaning. I am strongly in favor of the second option and think that is what we should shoot for.

Vectors

Single Feature:
http://<server>/<service or map resource>/<layer>/<id>.<format>

Vertices within a Single Feature:
http://<server>/<service or map resource>/<layer>/<id>/<vertex index>.<format>
Feature Query (preferably using Open Search specs):
http://<server>/<service or map resource>/<layer>/q?<query string>

Often there are times when you only want the geometry or you only want the attributes so these should be separable
Single Feature, Geometry Only:
http://<server>/<service or map resource>/<layer>/<id>/geom.<format>
Single Feature, Attributes Only:
http://<server>/<service or map resource>/<layer>/<id>/attrib.<format>

Notice I didn’t put a format for the query return? That is because here is where I would diverge from FeatureServer. Instead of returning the actual features that match that query I would return resource links. This is because you have no idea of knowing wether or not such a request is truly a cacheable resource identifier or not before hand. However, you do know that the feature URL’s are and should be cached. In the REmapper.com maps, I cache the response to a feature URL on both the server & the client for as long as possible. Depending on how static or dynamic your data is, this may not be appropriate for you, but it works very well for us.

Tiles:
Tiles can come from many sources and may very well be remote and neither owned nor controlled by you. These also might be something that is created & cached on the fly. Additionally, you may not even want a server side cache to persist longer than a user’s session. Here is where something acting in the same way as TileCache could be very helpful. A tile request handler, configured via a config file, could intercept these requests and then reformat them and pass them on to the real resource and return & cache the resource in whatever manner you’ve instructed it to (disk, memory, Amazon S3, raster data catalogue, etc..).

The main drawback with TileCache is that it only accepts WMS-C, TMS, or WorldWind requests. The tile request handler should be flexible enough to take regular WMS requests and translate (normalize) them into WMS-C or TMS requests and cache a standardized tile, which could be returned the next time any arbitrary bounding box query came in that contained that tile. OpenLayers does that request translation on the client side but you could connect to many more client types if you did it on the server side. Additionally our tile request handler should understand Google, Yahoo, Virtual Earth, etc tile requests and translate those into a standardized cacheable request as well. (by the way TileCache IS flexible enough to handle all types of requests, but no one has written a provider for the types it doesn’t).

So for tiles the premise is accept just about any valid request, fetch or create & cache the requested tile, then return a reference to the resource or the resource itself.

The Plumbing

Following the lead of FeatureServer & TileCache yo have the basic processes:

For Vectors:

Request -> Translator -> Query Object -> Provider -> Common Feature Object -> Serializer -> Response

For Tiles:

Request -> Translator -> Storage Provider -> (Layer Type Provider, if not found by Storage Provider) -> Response

Using this method you can plug in any provider you want. You can interact with ArcGIS Server via SOAP API calls, directly connect to ArcObjects, ArcSDE, Spatial RDBMS, use the OGR library, whatever you want as long as you can translate the feature into some lingua franca that can then be serialized in to any available format. That format could be any of the wire formats (GeoJSON, KML, GML, GeoRSS, etc..) but it could also be a shapefile or other binary format. You just need to plug in the right provider for that “serialization”.

On the client side it would probably be helpful to provide some helper JavaScript functions that can help translate vector wire formats into map objects and vice versa.

So we have the basic blueprint, all the copper we need, and a workspace. Let’s get plumbing!

FWTools & MySQL leaves ArcGIS geoprocessor in the dust

I wanted to know what block groups where included in each of the Zillow neighborhoods. So I needed to:

• Merge all the Zillow neighborhoods into a single datasource
• Intersect the block group data source with the neighborhoods

I had just done a similar process for the block groups and all US Census “populated places”, so I knew that the spatial intersecting in the ArcGIS geoprocessor was going to take a bit of time. I already had a MySQL data source with the block groups, so I thought I’d do a comparison.

FWTools & the Geoprocessor merged all the shapefiles into a single shapefile in about the same amount of time. In case you don’t know, here is how you do that using the ogr2ogr utility & DOS/Windows BATCH syntax.

FOR %f IN (*.shp) DO ogr2ogr -update -append -f "ESRI Shapefile" zillow_hoods.shp %f -nln zillow_hoods

Now that I had a combined shapefile, I loaded into the MySQL database,  using the ogr2ogr command for MySQL. This was executed in less than a few seconds.

Finally, I wrote a SQL query to intersect the 2 datasources and create a new table with the results.

CREATE TABLE remap.bghoods SELECT b.bg_num,z.* FROM remap.bg_data b, remap.zhoods z WHERE INTERSECTS(b.SHAPE,z.SHAPE)

The final result

ArcGIS Geoprocessor : 1hr 2min

MySQL : 5min 29sec

Creating ArcGIS Image Catalog From USB Stick

I like using ArcGIS Table Based Image Catalogs when working with large numbers of rasters in ArcMap. The old style table based Image Catalogs work for clients that still have ArcGIS 8.x or even ArcView 3.x. While a TileCache/Mapserver WMS image server is faster, it is sometimes not a real option for certain clients & situations.

However, lately I’ve been running into a number of situations in which the Image Catalog creator scripts that I’ve always used : USGS Image Catalog Builder or This One aren’t working. On Vista, the user extensions won’t even register and thus are of no use. I’ve also had troubles with large numbers of the newest MrSID type imagery giving "Automation Error" errors & the script stops working.

So I needed another way to create image catalogs that was easy and wouldn’t require me installing additional software on clients’ computers, etc.. While, I could have used Python or some other script, I wanted something that people without any scripting or coding experience could also do.

Software Used:

• FWTools (with a few environment variables modified in the startup batch file so that it would run properly from a stick)
• PSPad

Steps:

1. Open FWTools Command Line
2. Run gdaltindex on the folder containing the imagery.

   gdaltindex naip_05_index.shp 2005_NAIP_NC\*.sid

3. Use ogr2ogr to create a BNA format file from the tile index shapefile created in the previous step.

   ogr2ogr -f BNA naip05index.txt naip_05_index.shp

   This gives you a file with this sample output:

   "2005_NAIP_NC\e2997_01_1.sid","",5
   
   596140.0000000000,3319570.0000000000

   602854.0000000000,3319570.0000000000

   602854.0000000000,3311970.0000000000

   596140.0000000000,3311970.0000000000

   596140.0000000000,3319570.0000000000

4. Open the file with PSPad
5. Record a macro in PSPad that  flattens this to a single line.
6. Go to Tools->Macros->Marco Manager and select "Run X Times" and have it run for as many files as you created the tile index from.
7. Final step is to label the columns correctly
   • You need to add IMAGE, XMIN, YMIN, XMAX, & YMAX column headers
   • The 2nd coordinate group is the XMAX/YMAX group & the 2nd to last (4th) coordinate group is the XMIN/YMIN group.
   • You can call the other columns whatever you want, they will be ignored. However you MUST label them or delete them. I use:

     IMAGE,A,B,C,D,XMAX,YMAX,E,F,XMIN,YMIN,G,H

THAT’S IT!

Now you have a CSV based text file table that you can use as an Image Catalog.

The only additional step you may need to take is to convert it to a DBF file to work properly on older versions of ArcGIS or ArcView. This can easily be accomplished in the ArcCatalog, ArcToolbox or by loading the table into a map and exporting it to a DBF. Alternatively you can just use Excel, Open Office, or DBF Explorer to convert it as well.

It actually took me much longer to write this post then it did for me to create several image catalogs this way.

Dev Summit Final Day

Day 3 Thurs Mar 22

I’m finally getting a chance to blog about this now. Steve, I, and nearly 2 dozen other attendees got totally screwed by the storms in Phoenix. However, at least we didn’t have to sit on the tarmack for nearly 3 hours like the other group waiting at the same gate. Though, by the time I got home 25 hours after leaveing the conference, sleeping in the Phoenix airport and eating only pretzals and snack mix; that wait seemed pretty trivial. I hope everyone else also eventually got home safely.

General Thoughts

This was a short day. Only 2 sessions and then the lunch & closing session. It seemed that about 25-30% of the folks had already left and a number of the ESRI devs had left as well. The sessions seemed a bit less technical that those on Wensday.

Sessions Attended

Building & Using ArcGIS Server Map Caches (Best Practices)

The MapCache is a tile cache created by rasterization & tile cutting of the data you have presented in your MXD. The main take home points of this talk were:

• Understand what a tiling schema is 1st
• Design your tiling schema BEFORE trying to build on for production use
• Design a SINGLE schema that will serve the entire organization and all your data layers. In other words: Level 10 = the same scale for regardless of which layer you are building from; Tile sizes should be equal across various data sources; Use same origin point for all caches.
• Client can rapidly reproject data to any projection system so use one that makes the most sense for both your data and web services methods (ideally, Geographic).
• DO NOT use png24. PNG8 best if you need transparency, JPG best if you don’t need transparencey.
• Smaller tile sizes (128×128) do not neccessarily result in faster performance over the web, while they take 5 times as long to produce & take up significantly more storage space than large ones (512×512). Test to make sure performance improvement is worth the extra effort. Use the largeest tiel size you can.
• Fuse as many layers together as possible to create your base map caches. Therefore instead of Intermitten Streams, Permanent Streams, Rivers, & Lakes all being separate caches, they would be a single Hydrography cache that would appear as a single layer in the map service.
• Each zoom level should be cartographically distinct
• Forcing an ExportMapImage call from a web service can be a big performance hit. Understand what would make a client do that and avoid it as much as possible.
• Building large many leveled caches can be VERY time consuming (days-weeks for worldwide layers). Pick a small area, TEST-TEST-TEST and then TEST before doing the full export, instead of having to do it over & over.
• Cache will build over the extent of your MXD if the layer(s) you are building a cache for don’t have data in the full extent, background color tiles will be created. A waste of CPU time & storage space.
• Dynamic cache building (a la TileCache) is planned for future releases.

Creating Windows & Web Apps with WPF

This was not at all about GIS. It was strictly on WPF.

WPF is very powerful & can provide you with a lot of UI design patterns.

Security & Permissions can be tightly controlled. Each app runs in a security sandbox. App origin, app settings, and client settings all interact to chose the most restrictive security level for the app.

You can do stuff with this technology TODAY. But, it’s real power won’t be realized until Visual Studio ORCAS is released and Vista or .Net 3 runtimes are more widely accepted. The 1 code for web & desktop paradigm is only for IE6+ with .Net 3 runtime right now. Must use WPF/E for other browser support.

Closing Session

Open Q&A after lunch. No really hard questions were asked, but Steve & other continued to point out that the extra cost of putting the ADF on a separate server is just not reasonable, and that the overall cost of ArcGIS Server is too high.

Dev Summit Continues

Day 2 , Wed Mar 21

I was really hanging this morning, for someone who has a beer or glass of wine only occasionally, going beer for beer with professional buisness travelers is never a good idea. But it was free and pretty decent beer so I can’t complain too much.

General Thoughts

I got a whole lot more out of today’s sessions and meetings than the 1st day. The talks I attended went into more depth, the .Net SIG was good, and I took Brian & James’s advice and spent more time talking with devs in the Community center.

A few interesting things I learned in these informal conversations

• ArcGIS Explorer (AGX) will NOT have a custom skining option. You can mess around with some config files and the AGX document to change color themes, but that is about it. So a custom branded AGX that looked completely like your product but was actually AGX under the art, won’t be easily done.
• AGX has no inherent editing capabilities, no area measurement tool and no way to discover any attributes about data sources other than local vector sources. You have to capture mouse events and do that work yourself.
• AGX has no way of directly interperting ArcGIS geometeries, you must provide a map/globe service that is spitting out rasterized data or convert that geometery to E2 geometery yourself.
• ArcWeb Services can add an ArcIMS service & soon an ArcGIS Server service to maps, but GeoRSS feeds, PostGIS, WMS, OGC, local/server stored vectors (shapefiles,PGDB, GDB, SDE, etc) all have to be progamatically adding in by transforming and adding the geometery to the map yourself. The only other way to get those alternate sources in is to intergrate them with ArcIMS or ArcGIS Server. If you do have a WMS you could but it through the OGC/ArcIMS translator servlet and leverage it that way.
• A large number of people have compalined about the highly fragmented documentation, poor indexing, incomplete docs, and lack of community comments on EDN & other ESRI help sites. It may just be conference talk but it looks like at least a few people at ESRI are begining to listen. Time will tell.

Sessions Attended

• Programing Custom Tasks for ArcGIS Explorer
• The task framework is not direct & intuitive. RTFM on that or you will really be lost. Not directly supported formats such as GeoRSS are not terribly difficult to transform into E2 geometeries and create content from a vast array of data sources. Just don’t expect AGX to do that for you. Deploying custom tasks is SUPER easy. Deploying a malicous task is also possible so watch what you download. They are looking for ways to patch that which won’t create Admin required installs. AGX2 is in the works and the more feedback you give them on the forums and directly, the better it will be.
• Special Interest Group: .Net
• A brief marketing talk about MS Visual Studio Team Foundation Server was followed up by Dave Bouwman’s great talk about wrapping feature classes, SDE, & GDB in .Net abstractions using well known code gen techinques from non-spatial DB. It pulled together some of his posts on the subject and made it all look like such a better way to code. I’ve actually used the same techinques on shapefiles using RapTier and either the Jet OleDb or the ESRI OleDb drivers. Shapefiles are not as ammenable to being edited in this manner as PGDB or other real database storage formats are. I can read the data very well and databind on it but updating the gives rather unpredictable results.
• I would strongly recommend visiting Dave’s blog and checking out his talk and an resources it is an exicting concept. It also makes you wonder, why they don’t just offer Dave a position or short-term contract at ESRI to incorporate all they modern coding techniques in at least ArcGIS 10 if not 9.3. When you see simple object property assignments rather than the AO way of doing it and the self policing code it creates, you wonder why in the world would you want to do it the old way.
• James Fee hit on the craziness that is the 9.2 server licensing & the unreasonable expectation to pay 1/2 of the 1st license cost (of ~20k at retail) to put the Web ADF on another machine. He was told, “We know people don’t like it, but that is the business model for today. We may be changing it in future releases but don’t hold your breath” (paraphrasing, not a direct quote).
• Building & Deploying Enterprise Solutions with ArcGIS Server
• Not a lot of good tech meat here, just an eye opening talk about all the hoops & roadblocks that deploying ESRI server products in lerge organizations that don’t conform to ESRI’s way of doing things can cause.
• Developing Custom Web Tasks using the .NET Web ADF
• A lot of technical meat in this session. It went a long way to demonstrate that doing interesting things using this framework is not going to be so easy. You really have to jump through a lot of hoops to make cool things happen, and you have to jump through too many hoops to make super simple things happen. They covered a lot of techniques, tips, & tricks for working with this and I highly recomend veiwing this session on the post-conference web site if available. Especially the code that way demonstrated at the last 3rd of the talk.
• ArcGIS Mobile SDK
• A super energetic talk was presented by Jeff & Mike. They have really worked hard to abstract all the underlying native C++ code into a 1st class .NET SDK. They have built a really nice toolkit for VisStudio & using it and the Mobile 5 libaries, you can get some nice applictions working with a very minimal amount of code. It relies heavily on MapCaches produced by ArcGIS Server and they didn’t go into creating those or pushing them out but I’m hoping to get that info in a later session. This was my favorite talk by ESRI thus far.

Thoughts on Dev Summit

Day 1 (for me) Tues, Mar 20

Due to airline maintance problems, I didn’t get to the Summit until after the Plenary session was over. So I can’t comment on that although several others can.

Rob Elkins – 1, 2, 4, 5
Dave Bouwman
The Steve0

General Thoughts

I was deeply disapointed with the 1st day of Tech Sessions that I attended. I figured that the Best Practices sessions would be general overviews, but the Deep Dives I attended weren’t very deep. This was probably made worse by the fact that they couldn’t get internet connections to work during the presentations. I should have followed James & Brian’s advice to just hang out in the Community Center and listen to Tech Talks and speak with the ESRI Devs themselves. The highlight of that day was getting to meet somany of the other geo bloggers and members of the ESRI team whose names I knew but had never seen or spoken to before.

Sessions Attended

Leveraging ArcWeb Services in GIS (Best Practices)

This was a nearly utter waste of time. They basically just walked through the docs, info & help from ArcWeb Services home page. This site is in fact far more informative than the talk was. The only truly interesting bit of info from this is that AWX v2 will be coming out later in the year. It is currently in alpha right now. This will allow direct connection to ArcGIS Server and possibly OGC web services.

Building AJAX-Based Web Apps with Server & .Net (Deep Dive)

Slight snorkel would have been appropiate. Again, you could learn as much and more from the SDK docs. The main take home message from this was that the out-of-the-box toolkit and base classes have ICallbackEvent handlers pre-cooked into them. Thus you don’t have to directly handle implementing the request & response callback functions in JavaScript. You just pass and recieve delimited text serializations. They delimited text was 30-40% faster than XML. JSON adds another 12-15% and will be directly supported soon (probably 9.3).

Deploying & Tuning ArcGIS Server / Leveraging GeoProces Framework in ArcGIS Engine .Net

The Server deployment was well attended and was also the subject of several very in depth Tech Talks. Grumblings about the licensing model and its pricing structure that actually discourges implementing the best practices models continue.

I caught the last half of the GP in Engine talk. That was rather interesting. Some compliants have been made about sending big tasks off to the GP only to have them fail or crash the coputer with no ryhme or reason. Appearently you can implement a listner that will talk to you about the status of your job as it is processing. That technique is not easily discoered through help docs.

COMMENTS CLOSED DUE TO COMMENT SPAM

Common GIS Tasks in Manifold & ArcGIS

Here is a link to a great paper from Cornell comparing the performance of common GIS tasks in Manifold 6 & ArcGIS 8.3

The information is slightly out of date now that Manifold is on version 7 & the release of ArcGIS 9.2 is immenent.

I haven’t had a chance to do as much comparing & testing this week as I had hoped. Hopefully this should answer some questions for those who are curious about what Manifold is really like but don’t want to spend ~$250 just to find out.