Comparing Mashup Platforms Using JSON & MySQL - Part 2 - Process JSON

30 03 2007

Technologies Used: PHP 5, MySQL 4.1.2, JavaScript, JSON

Software Used: PHP Designer, Aptana, Firebug, Firefox

There are a wide variety of ways to handle JSON that is returned from a server. One of these methods is to provide a callback function pass it to the JSON emiting webservice via REST. When used with static or dynamic script tags, you are no longer restricted to Cross Site Scripting (XSS) limits requireing that the source be on the same server as your webpage. The XSS limit seems particularly ill-suited for mash-ups. The callback function method is what I have choosen for my examples. You will also see a static script tag for these examples. That is to simplify them and make them easier to read. The general preference is to use dynamic script tags rather than static ones.

Actual JSON string returned:

handleJSON({”Observations”:[{"Station":{"number":"1","lat":"29.5762","lon":"-98.7041","total_sp":"9","Species":[{"common":"Mourning Dove","number":"1","code":"MODO","scientific":"Zenaida macroura"},{"common":"Eastern Phoebe","number":"1","code":"EAPH","scientific":"Sayornis phoebe"},{"common":"Carolina Chickadee","number":"1","code":"CACH","scientific":"Poecile carolinensis"},{"common":"Black-crested Titmouse","number":"3","code":"BCTI","scientific":"Baeolophus atricristatus"},{"common":"Carolina Wren","number":"1","code":"CARW","scientific":"Thryothorus ludovicianus"},{"common":"Bewick's Wren","number":"2","code":"BEWR","scientific":"Thryomanes bewickii"},{"common":"Rufous-crowned Sparrow","number":"1","code":"RCSP","scientific":"Aimophila ruficeps"},{"common":"Northern Cardinal","number":"3","code":"NOCA","scientific":"Cardinalis cardinalis"},{"common":"Brown-headed Cowbird","number":"1","code":"BHCO","scientific":"Molothrus ater"}]}},{”Station”:{”number”:”2″,”lat”:”29.574″,”lon”:”-98.7036″,”total_sp”:”5″,”Species”:[{"common":"Black-crested Titmouse","number":"1","code":"BCTI","scientific":"Baeolophus atricristatus"},{"common":"Carolina Wren","number":"1","code":"CARW","scientific":"Thryothorus ludovicianus"},{"common":"Ruby-crowned Kinglet","number":"1","code":"RCKI","scientific":"Regulus calendula"},{"common":"Northern Cardinal","number":"1","code":"NOCA","scientific":"Cardinalis cardinalis"},{"common":"Brown-headed Cowbird","number":"1","code":"BHCO","scientific":"Molothrus ater"}]}}]});

It is not very human readable, but it is highly machine readable.

Below is an extremely simple example of handling JSON and doing something with it.

<html>
<head>
<meta http-equiv=“Content-Type” content=“text/html; charset=iso-8859-1″ />
<title>Simple JSON Handling</title>

<script type=”text/javascript” charset=”utf-8″>
function showCoords(jsd){
var lat = jsd.Observations[0].Station.lat;
var lon = jsd.Observations[0].Station.lon;
var msg = ‘Lat:’ + lat + ‘, Lon:’ + lon;
alert(msg);
}
</script>
</head>
<body>
<script src=http://www.plateauwildlife.com/bbc-mgmt/getstations.php?action=getdata&cid=2&year=2006&func=showCoords type=”text/javascript” charset=”utf-8″></script>
</body>
</html>


See It In Action

The above example is not particulalry usefull for anything other than demostration purposes. We want to actually DO something with our JSON to move us closer to creating the actual mashup. The number thing which I intially strugled with when using JSON & callback functions was that you MUST define the callback function BEFORE your dynamic or static script tag.

Google Maps, Google Earth, Yahoo Maps, and Virtual Earth all take HTML for the contents of the info window when you rollover or click on a point.

ArcWeb Explorer (AWX), however doesn’t take HTML as info window content. AWX does take styled text, videos, picture, audio, & swf for info window content. To this end, if you want to embed rich non-HTML content, AWX allows for some extremelly interesting content to be blended together and presented with great ease. The documentation for text styling is lacking, so creating simple content is actually more dificult in this platform than the others.

I’ve created a javascript file which we can reference in any of the HTML docs that actually embed the mashup.

This file contains

  1. The main callback function
  2. A function which builds an array of HTML tables containing the formated results from each station
  3. A function which builds an array of jscript strings containg the weakly formated results from each station

See The HTML Builder In Action

    //A global variable to assign the parsed JSON to

var jsobj;

 

//Main Callback handler.

//simple assignment to a global variable allows me to reuse and pass

//around the object without any server trips

function handleJSON(reply){

    jsobj=reply;

}

 

    function buildHTML(Observations){

    var info_win = new Array();

    // Build a table element with Station number & total species observed

    for(var i=0;i<Observations.length;i++){

        str = ‘<table border=”1″><tbody><tr class=”station”>’;

        str += ‘<td colspan=”2″>Station ‘+ Observations[i].Station.number + ‘</td></tr>’;

        str += ‘<tr class=”sta_total”><td colspan=”2″>Total Species - ‘ + Observations[i].Station.total_sp + ‘</td></tr>’;

       str += ‘<tr class=”obs_header”><td>Species</td><td>Number</td></tr>’;

        var details = “”;

        var arr = new Array();

        // assign each Species array to a local variable to reduce typing & increase readibility

        //build an string of <tr> elements containing the details of species observed

        arr = Observations[i].Station.Species;

            for (var y=0;y<arr.length;y++){

            details+=‘<tr class=”obs_detail”><td><a href=”http://www.google.com/search?q=%22′;

            details += arr[y].scientific.replace(/\s/,“+”);

            details += ‘%22″>’ + arr[y].common + ‘ (’ + arr[y].code + ‘)</a></td>’;

            details += ‘<td>’ + arr[y].number + ‘</td></tr>’;

            }

        str +=    details;

        str += ‘</table></tr></tbody></table>’;

        //add table element to array of table element html strings

        info_win[i]=str;

    }

return info_win;

}

 

function buildAWXtxt(Observations){

        var info_win = new Array();

    // Build a formated text list for each Station number & total species observed

    for(var i=0;i<Observations.length;i++){

        str = ‘Station ‘ + Observations[i].Station.number + ‘\n’;

        str += ‘Total Species - ‘ + Observations[i].Station.total_sp + ‘\n’;

       str += ‘Species          Number’;

        var details = new Array();

        // assign each Species array to a local variable to reduce typing & increase readibility

        //build an array formatted text data elements containing the details of species observed

        //using this convention, we can assign a url property to each species line

        //through .data{elements[]} in the properties for each marker

        arr = Observations[i].Station.Species;

            for (var y=0;y<arr.length;y++){

            details[y] = arr[y].common + ‘ (’ + arr[y].code + ‘) - ‘ + arr[y].number + ‘\n’;

            }

        var obs_info = new Array([str,details]);

        //

        //add table element to array of table element html strings

        info_win[i]=obs_info;

    }

return info_win;

}

 

Previous Parts

1. Emit JSON

 

Next Parts

3. Arcweb Explorer Mashup

4. Yahoo Maps Mashup

5. Virtual Earth Mashup

6. Google Maps Mashup




Dev Summit Continues

22 03 2007

Day 2 , Wed Mar 21

I was really hanging this morning, for someone who has a beer or glass of wine only occasionally, going beer for beer with professional buisness travelers is never a good idea. But it was free and pretty decent beer so I can’t complain too much.

General Thoughts

I got a whole lot more out of today’s sessions and meetings than the 1st day. The talks I attended went into more depth, the .Net SIG was good, and I took Brian & James’s advice and spent more time talking with devs in the Community center.

A few interesting things I learned in these informal conversations

  • ArcGIS Explorer (AGX) will NOT have a custom skining option. You can mess around with some config files and the AGX document to change color themes, but that is about it. So a custom branded AGX that looked completely like your product but was actually AGX under the art, won’t be easily done.
  • AGX has no inherent editing capabilities, no area measurement tool and no way to discover any attributes about data sources other than local vector sources. You have to capture mouse events and do that work yourself.
  • AGX has no way of directly interperting ArcGIS geometeries, you must provide a map/globe service that is spitting out rasterized data or convert that geometery to E2 geometery yourself.
  • ArcWeb Services can add an ArcIMS service & soon an ArcGIS Server service to maps, but GeoRSS feeds, PostGIS, WMS, OGC, local/server stored vectors (shapefiles,PGDB, GDB, SDE, etc) all have to be progamatically adding in by transforming and adding the geometery to the map yourself. The only other way to get those alternate sources in is to intergrate them with ArcIMS or ArcGIS Server. If you do have a WMS you could but it through the OGC/ArcIMS translator servlet and leverage it that way.
  • A large number of people have compalined about the highly fragmented documentation, poor indexing, incomplete docs, and lack of community comments on EDN & other ESRI help sites. It may just be conference talk but it looks like at least a few people at ESRI are begining to listen. Time will tell.

Sessions Attended

  • Programing Custom Tasks for ArcGIS Explorer
    • The task framework is not direct & intuitive. RTFM on that or you will really be lost. Not directly supported formats such as GeoRSS are not terribly difficult to transform into E2 geometeries and create content from a vast array of data sources. Just don’t expect AGX to do that for you. Deploying custom tasks is SUPER easy. Deploying a malicous task is also possible so watch what you download. They are looking for ways to patch that which won’t create Admin required installs. AGX2 is in the works and the more feedback you give them on the forums and directly, the better it will be.
  • Special Interest Group: .Net
    • A brief marketing talk about MS Visual Studio Team Foundation Server was followed up by Dave Bouwman’s great talk about wrapping feature classes, SDE, & GDB in .Net abstractions using well known code gen techinques from non-spatial DB. It pulled together some of his posts on the subject and made it all look like such a better way to code. I’ve actually used the same techinques on shapefiles using RapTier and either the Jet OleDb or the ESRI OleDb drivers. Shapefiles are not as ammenable to being edited in this manner as PGDB or other real database storage formats are. I can read the data very well and databind on it but updating the gives rather unpredictable results.
    • I would strongly recommend visiting Dave’s blog and checking out his talk and an resources it is an exicting concept. It also makes you wonder, why they don’t just offer Dave a position or short-term contract at ESRI to incorporate all they modern coding techniques in at least ArcGIS 10 if not 9.3. When you see simple object property assignments rather than the AO way of doing it and the self policing code it creates, you wonder why in the world would you want to do it the old way.
    • James Fee hit on the craziness that is the 9.2 server licensing & the unreasonable expectation to pay 1/2 of the 1st license cost (of ~20k at retail) to put the Web ADF on another machine. He was told, “We know people don’t like it, but that is the business model for today. We may be changing it in future releases but don’t hold your breath” (paraphrasing, not a direct quote).
  • Building & Deploying Enterprise Solutions with ArcGIS Server
    • Not a lot of good tech meat here, just an eye opening talk about all the hoops & roadblocks that deploying ESRI server products in lerge organizations that don’t conform to ESRI’s way of doing things can cause.
  • Developing Custom Web Tasks using the .NET Web ADF
    • A lot of technical meat in this session. It went a long way to demonstrate that doing interesting things using this framework is not going to be so easy. You really have to jump through a lot of hoops to make cool things happen, and you have to jump through too many hoops to make super simple things happen. They covered a lot of techniques, tips, & tricks for working with this and I highly recomend veiwing this session on the post-conference web site if available. Especially the code that way demonstrated at the last 3rd of the talk.
  • ArcGIS Mobile SDK
    • A super energetic talk was presented by Jeff & Mike. They have really worked hard to abstract all the underlying native C++ code into a 1st class .NET SDK. They have built a really nice toolkit for VisStudio & using it and the Mobile 5 libaries, you can get some nice applictions working with a very minimal amount of code. It relies heavily on MapCaches produced by ArcGIS Server and they didn’t go into creating those or pushing them out but I’m hoping to get that info in a later session. This was my favorite talk by ESRI thus far.



AGXcellent? Not Quite Yet.

29 11 2006

I was like a kid in a candy store last night and stayed up way later than I should have when I found out that ArcGIS Explorer was available for download. I’ve really been waiting for this since rumors of it’s existance began.

What I like:

Aggregation of a variety of services and formats.

Native support for MrSID

  • I have nearly 150gigs of SIDs for Texas. Various years of NAIP, DOQQ, and Municipality, County, Regionally collected aerial photos. Along with some private source imagery. All of the stuff we use is 2 years or less old. Data quality ranges from 2m CIR to 0.5ft TrueColor. It is all better than what you can get from any of the free web maps and nearly all of it is better than ArcWeb’s DigitalGlobe Standard imagery. We also use some ArcIMS services for some county’s annually updated super high-quality aerial photos. The last thing I want to do is convert all this into ECW or JPG2k. We get updates pretty regularly of different areas so continual data conversion is not something I really want to do either. We have NO budget for ArcIMS or ArcGIS, so finding a way to use our imagery in its existing format has been our biggest challange to distributting our data over the web.

Creation of custom tasks and geoproccessing models

  • We want a few very simple editing and feature creation tasks. I don’t need a full suite of GIS tasks. Keep it simple, keep it focused.

Fully skinable

Simple but powerful navigation and interaction methods

All of the above, except the tasks without programming

What needs to improve:

The STREAMING SUCKS!!

  • Come on! I’m testing all of this on the same machine and it takes 3-5 times longer to load base imagery than ArcMap 8.3, and guess what, those are being read of the same disc as AGX and AGX is cacheing them. After going to one area, seeing the imagery of it again should be lighting fast, but it is SLOW!
  • The ArcGIS Online servers are definately not handling anything nearly as quickly as GE, VE, or Yahoo. That said, even if they were, would we know it? When even local imagery loads slowly, imagery from external sources is like watching a maple syrup race in January.

No easy way to change the styling of imported data sources once they are in.

  • Should offer to read .lyr files in addition to shape files. All the styling info it needs, including visiblity scaling has already been recorded there. Why should I have to go through all that again, using unfamilar scales of km or mi above earth’s surface. (By the way to help with that you can use the formula provided in an earlier post)
  • Even if you don’t read .lyr files, then you should at least give an easy way to restyle a layer that you have added from a local or external data source. I understand not restyling layers from GIS server, WMS, or KML as they should have all the styling info they need in them. But importing a shape file doesn’t come with any styling info and trial and error of import, delete, re-import just isn’t cool with me.

Label support is poor and much more difficult to manage than in GE, GMap, Yahoo or VE.

Bottom line is that this is NOT going to truly challange GE/VE. This could be a good 3D enabled simple GIS client that is highly customizable without the use restrictions of GE. The general population doesn’t need a true GIS. They want a great GeoVisualization System with the ability to see data they are interested in draped over 3d enabled imagery. AGX is CLEARLY not trying to be this. It is a ArcGIS Server client that can be repurposed for some other uses for those who are above the general masses in thier GIS needs but don’t need a fully functional professional GIS.

That is exactly what I’ve been looking for, but I don’t think there are that many people in this same position. This is going to be a niche product. If they can get the speed up to some reasonable level, I think it will a really good one.

I’m going to see how well this works with Manifold WMS tonight. I’ll definately be posting more about this in the next days and weeks.




GIS Data Distribution & Visualization Methods

14 08 2006

I am attempting to put together a list of the various methods by which one can distribute GIS data AND the ability to visualize it to non-GIS users. Below is my 1st pass at making such a list. My main focus is the ability to distribute private data in a controlled manner. The end user for this system is typically someone with limited computer skills and no previous GIS use. They would need to search for & identify features, turn layers on/off, and print or export maps. Some users may also want to create their own custom layer by selecting a sub-set of an existing layer and applying different symbolization to it. They do not need a fully featured GIS, nor do they want to need much if any training to use the system.

I will be using this list in a presentation & proposal to help us decide the best method for distributing our multi-county project to outside of the 4 related companies that I developed it for.

I am further expanding on this list to include pros, cons, & costs related to each method along with links to the software & examples of implementations. The expanded list can be found at www.kestrelcomputer.com/GIS-Distribution.htm. I will post the fully expanded list on Friday Aug 18.

Please let me know if you think I have left any important method, software, or technology out.

GIS Data Visualization & Distribution Methods

Physical
Data & Viewer Reside on Client Computer or Removable Storage

  • DVD - Tatuk Viewer, ArcView Project, and Data & Imagery Files
  • DVD - ArcExplorer, ArcPress Document, and Data & Imagery Files
  • CD - Either of Above Options

Web
Data & Viewer Reside on Internet or Intranet

  • ArcIMS Viewer
  • ArcWeb Services Viewer
  • ArcWeb Explorer (ASP.Net or PHP Custom Version)
  • Google Maps (ASP.Net or PHP Custom Version)
  • SharpMap (ASP.Net Custom Implementation)
  • MapServer
  • MapBuilder
  • MapXtreme 2005 .NET
  • GeoServer
  • DEMIS WebMap Server

Hybrid
Viewer Resides on Client Computer & Data Is Served From Internet/Intranet

  • ArcGIS Explorer
  • Google Earth
  • WorldWind (with customizations)
  • uDig (with customizations)
  • SharpMap (Windows App Version)
  • WorldWind (with customizations)
  • MapDotNet Server 2007
  • MapWindow GIS
  • Dapple (based on WorldWind)



Customize ArcWeb Explorer & Pay For It

12 06 2006

When I previously posted about the ArcWeb Explorer (AWX) JavaScript API, there was no credit cost for embeding AWX in a custom webpage. This seemed like quite a deal, since you could hook up almost all of the ArcWeb Services. I wondered how long this free-ride would last. Well the answer is "Not long". You must now register your webpage with ArcWeb Services and recieve a AWX API key for custom implementations. There is nothing explicitly dealing with Public Services users on the AWX QuickStart Page or on the AWX registration page. So I don't know what level of access Public Service users get.




ArcWeb Explorer Javascript API Available

6 05 2006

ESRI Logo

http://www2.arcwebservices.com/v2006/labs/awx_lab.do

Sorry if this is old news, it was the 1st I'd seen of it. I was bumping around on the site to look for some documentation for a "Proof-of-Concept" project I'm doing for a client and noticed that they now have a link availble. I'm going to see if I do a custom AWX project right now.