ArcGIS REST & JS API, The Real Deal?

Well, shut my mouth!

After complaining & proclaiming loudly that there was no way anything coming out of Redlands was going to be good semantic REST or true http goodness, I seem to have been proven dead wrong.

The REST & JS API examples released by ESRI today really look good. I’m impressed by what I’ve seen so far. I’m not alone as James Fee & Sean Gillies have already written up very approving posts as well.

I got my 9.3 private beta invite last night and am looking forward to taking these once sorely missing pieces out for a test drive.

Where does that leave the ArcDeveloper.Net REST for AGS project? I can’t speak for the others involved, but I for one would still like to proceed ahead. Not for the primary reasons stated before, frustration with the Web ADF & lack of faith for ESRI to produce semantic REST, but mainly for having .Net versions of TileCache & FeatureServer combination. The intention never was to couple it very tightly with AGS. Therefore having this program available when Python under IIS is just not an option, would still be a great benefit to the community. Additionally, for those not upgrading to AGS 9.3 anytime soon, such a program could allow them the lightweight interactions missing from 9.2.

Just Wait It Might Be Great

UPDATE: Well it looks like it just might be great. I hope it is.

That is what James Fee is seeming to say. He seems to thinking that building our own RESTful interface to ArcGIS Server might be they way to go but not until we at least see what ESRI has developed.

Well I respectfully disagree. I would be willing to bet that whatever ESRI cooks up will NOT be semantic REST but rather some RPC-REST hybrid at best. Trust me, Sean Gillies is going to want to burn the letters E, I, S & R in effigy once their marketing machine gets going. Your going to see REST API all over the place and to add insult to injury, that incorrect term will be used to describe something that is barely RESTful & defiantly not semantic REST  I think we as a community could probably do better than that. (disclaimer:I have absolutely no idea what the REST interface to AGS will look like, I’m purely speculating)

Who among us doesn’t think that a Community Driven, Standards Based Open Source project would do a better job of serving our needs and be more responsive and flexible than what ESRI is cooking up. Waiting for ESRI is like “Waiting for Godot”. If want to directly use the various feature storage formats out there, raster data, or WMS/WFS then there are good open source tools which allow you to do that. However, what if you really want to harness the power of the ArcGIS Server stack and just get it to listen & talk in a wider and more flexible array of languages. Make it so that you can plug in AGS into just about anything out there. AGS is a very powerful tool that should power backend solutions, fire & forget geoprocessing, etc..For this I think a community supported open source solution is going to be the best. I say “strike while the iron is hot”, if we can direct the anger & frustration of the WebADF that is boiling over right now into a more useful purpose then let’s do it now rather than later.

The Web ADF, may it REST in peace

There has been a lot of blowback against the ESRI ArcGIS Server WebADF in the last week. Dave’s complaint against an obvious short coming in the Web ADF was echoed by Doron Yaacoby and the steam really started to to vent in the comments of James’ post. Dave has since made a call to action, to paraphrase: “stop whining, or inventing your own secret personal version, and let’s get together and build something!” I could not agree more. In fact it is exactly what I’ve been turning over in my head for a while, but it didn’t seem like something that could be properly done unilaterally

The Basic Question

Why do most people want to use the Web ADF? They want to publish their content & let users interact with it. The Google/Yahoo/Microsoft map API’s & even more so, the OpenLayers & FeatureServer combination has shown us that it doesn’t require the bloated, overly chatty, buggy, difficult, and EXPENSIVE Web ADF solution. You could do more, so much more, easily with a good common-sense standards based approach that allows easy integration into your favorite Web Mapping display engine.

Where to Start

A well thought out RESTful interface with AGS would be a great start down this path. There are really only 2 resource types required to make web maps.

• Tiles (raster data or pre-rendered cartographic output)
• Vectors, of which points are somewhat of a special case

TileCache and FeatureServer are great examples of how to access and emit this data in an extensible, lightweight, standards based manner. While there a few things I would like to change about FeatureServer, I think this is a great model to start from.

True RESTful URL’s

There has been a lot of discussion in the last year about what constitutes a proper RESTful URL. The most vocal geobloggers on this topic have easily been Sean Gillies (the Grumpy Guardian of GeoREST) and Sebastian Good, though others have chimed in as well. Is it just a URL that will point to the same resource regardless of it’s structure (i.e.. Permalink & RPC/REST hybrids) or is it a URL with limited or no query string and each directory of the URL contains a standardized semantic meaning. I am strongly in favor of the second option and think that is what we should shoot for.

Vectors

Single Feature:
http://<server>/<service or map resource>/<layer>/<id>.<format>

Vertices within a Single Feature:
http://<server>/<service or map resource>/<layer>/<id>/<vertex index>.<format>
Feature Query (preferably using Open Search specs):
http://<server>/<service or map resource>/<layer>/q?<query string>

Often there are times when you only want the geometry or you only want the attributes so these should be separable
Single Feature, Geometry Only:
http://<server>/<service or map resource>/<layer>/<id>/geom.<format>
Single Feature, Attributes Only:
http://<server>/<service or map resource>/<layer>/<id>/attrib.<format>

Notice I didn’t put a format for the query return? That is because here is where I would diverge from FeatureServer. Instead of returning the actual features that match that query I would return resource links. This is because you have no idea of knowing wether or not such a request is truly a cacheable resource identifier or not before hand. However, you do know that the feature URL’s are and should be cached. In the REmapper.com maps, I cache the response to a feature URL on both the server & the client for as long as possible. Depending on how static or dynamic your data is, this may not be appropriate for you, but it works very well for us.

Tiles:
Tiles can come from many sources and may very well be remote and neither owned nor controlled by you. These also might be something that is created & cached on the fly. Additionally, you may not even want a server side cache to persist longer than a user’s session. Here is where something acting in the same way as TileCache could be very helpful. A tile request handler, configured via a config file, could intercept these requests and then reformat them and pass them on to the real resource and return & cache the resource in whatever manner you’ve instructed it to (disk, memory, Amazon S3, raster data catalogue, etc..).

The main drawback with TileCache is that it only accepts WMS-C, TMS, or WorldWind requests. The tile request handler should be flexible enough to take regular WMS requests and translate (normalize) them into WMS-C or TMS requests and cache a standardized tile, which could be returned the next time any arbitrary bounding box query came in that contained that tile. OpenLayers does that request translation on the client side but you could connect to many more client types if you did it on the server side. Additionally our tile request handler should understand Google, Yahoo, Virtual Earth, etc tile requests and translate those into a standardized cacheable request as well. (by the way TileCache IS flexible enough to handle all types of requests, but no one has written a provider for the types it doesn’t).

So for tiles the premise is accept just about any valid request, fetch or create & cache the requested tile, then return a reference to the resource or the resource itself.

The Plumbing

Following the lead of FeatureServer & TileCache yo have the basic processes:

For Vectors:

Request -> Translator -> Query Object -> Provider -> Common Feature Object -> Serializer -> Response

For Tiles:

Request -> Translator -> Storage Provider -> (Layer Type Provider, if not found by Storage Provider) -> Response

Using this method you can plug in any provider you want. You can interact with ArcGIS Server via SOAP API calls, directly connect to ArcObjects, ArcSDE, Spatial RDBMS, use the OGR library, whatever you want as long as you can translate the feature into some lingua franca that can then be serialized in to any available format. That format could be any of the wire formats (GeoJSON, KML, GML, GeoRSS, etc..) but it could also be a shapefile or other binary format. You just need to plug in the right provider for that “serialization”.

On the client side it would probably be helpful to provide some helper JavaScript functions that can help translate vector wire formats into map objects and vice versa.

So we have the basic blueprint, all the copper we need, and a workspace. Let’s get plumbing!