ArcGIS REST & JS API, The Real Deal?

7 02 2008

Well, shut my mouth!

After complaining & proclaiming loudly that there was no way anything coming out of Redlands was going to be good semantic REST or true http goodness, I seem to have been proven dead wrong.

The REST & JS API examples released by ESRI today really look good. I’m impressed by what I’ve seen so far. I’m not alone as James Fee & Sean Gillies have already written up very approving posts as well.

I got my 9.3 private beta invite last night and am looking forward to taking these once sorely missing pieces out for a test drive.

Where does that leave the ArcDeveloper.Net REST for AGS project? I can’t speak for the others involved, but I for one would still like to proceed ahead. Not for the primary reasons stated before, frustration with the Web ADF & lack of faith for ESRI to produce semantic REST, but mainly for having .Net versions of TileCache & FeatureServer combination. The intention never was to couple it very tightly with AGS. Therefore having this program available when Python under IIS is just not an option, would still be a great benefit to the community. Additionally, for those not upgrading to AGS 9.3 anytime soon, such a program could allow them the lightweight interactions missing from 9.2.




Just Wait It Might Be Great

1 02 2008

UPDATE: Well it looks like it just might be great. I hope it is.

That is what James Fee is seeming to say. He seems to thinking that building our own RESTful interface to ArcGIS Server might be they way to go but not until we at least see what ESRI has developed.

Well I respectfully disagree. I would be willing to bet that whatever ESRI cooks up will NOT be semantic REST but rather some RPC-REST hybrid at best. Trust me, Sean Gillies is going to want to burn the letters E, I, S & R in effigy once their marketing machine gets going. Your going to see REST API all over the place and to add insult to injury, that incorrect term will be used to describe something that is barely RESTful & defiantly not semantic REST  I think we as a community could probably do better than that. (disclaimer:I have absolutely no idea what the REST interface to AGS will look like, I’m purely speculating)

Who among us doesn’t think that a Community Driven, Standards Based Open Source project would do a better job of serving our needs and be more responsive and flexible than what ESRI is cooking up. Waiting for ESRI is like “Waiting for Godot”. If want to directly use the various feature storage formats out there, raster data, or WMS/WFS then there are good open source tools which allow you to do that. However, what if you really want to harness the power of the ArcGIS Server stack and just get it to listen & talk in a wider and more flexible array of languages. Make it so that you can plug in AGS into just about anything out there. AGS is a very powerful tool that should power backend solutions, fire & forget geoprocessing, etc..For this I think a community supported open source solution is going to be the best. I say “strike while the iron is hot”, if we can direct the anger & frustration of the WebADF that is boiling over right now into a more useful purpose then let’s do it now rather than later.




The Web ADF, may it REST in peace

1 02 2008

There has been a lot of blowback against the ESRI ArcGIS Server WebADF in the last week. Dave’s complaint against an obvious short coming in the Web ADF was echoed by Doron Yaacoby and the steam really started to to vent in the comments of James’ post. Dave has since made a call to action, to paraphrase: “stop whining, or inventing your own secret personal version, and let’s get together and build something!” I could not agree more. In fact it is exactly what I’ve been turning over in my head for a while, but it didn’t seem like something that could be properly done unilaterally

The Basic Question

Why do most people want to use the Web ADF? They want to publish their content & let users interact with it. The Google/Yahoo/Microsoft map API’s & even more so, the OpenLayers & FeatureServer combination has shown us that it doesn’t require the bloated, overly chatty, buggy, difficult, and EXPENSIVE Web ADF solution. You could do more, so much more, easily with a good common-sense standards based approach that allows easy integration into your favorite Web Mapping display engine.

Where to Start

A well thought out RESTful interface with AGS would be a great start down this path. There are really only 2 resource types required to make web maps.

  • Tiles (raster data or pre-rendered cartographic output)
  • Vectors, of which points are somewhat of a special case

TileCache and FeatureServer are great examples of how to access and emit this data in an extensible, lightweight, standards based manner. While there a few things I would like to change about FeatureServer, I think this is a great model to start from.

True RESTful URL’s

There has been a lot of discussion in the last year about what constitutes a proper RESTful URL. The most vocal geobloggers on this topic have easily been Sean Gillies (the Grumpy Guardian of GeoREST) and Sebastian Good, though others have chimed in as well. Is it just a URL that will point to the same resource regardless of it’s structure (i.e.. Permalink & RPC/REST hybrids) or is it a URL with limited or no query string and each directory of the URL contains a standardized semantic meaning. I am strongly in favor of the second option and think that is what we should shoot for.

Vectors

Single Feature:
http://<server>/<service or map resource>/<layer>/<id>.<format>

Vertices within a Single Feature:
http://<server>/<service or map resource>/<layer>/<id>/<vertex index>.<format>
Feature Query (preferably using Open Search specs):
http://<server>/<service or map resource>/<layer>/q?<query string>

Often there are times when you only want the geometry or you only want the attributes so these should be separable
Single Feature, Geometry Only:
http://<server>/<service or map resource>/<layer>/<id>/geom.<format>
Single Feature, Attributes Only:
http://<server>/<service or map resource>/<layer>/<id>/attrib.<format>

Notice I didn’t put a format for the query return? That is because here is where I would diverge from FeatureServer. Instead of returning the actual features that match that query I would return resource links. This is because you have no idea of knowing wether or not such a request is truly a cacheable resource identifier or not before hand. However, you do know that the feature URL’s are and should be cached. In the REmapper.com maps, I cache the response to a feature URL on both the server & the client for as long as possible. Depending on how static or dynamic your data is, this may not be appropriate for you, but it works very well for us.

Tiles:
Tiles can come from many sources and may very well be remote and neither owned nor controlled by you. These also might be something that is created & cached on the fly. Additionally, you may not even want a server side cache to persist longer than a user’s session. Here is where something acting in the same way as TileCache could be very helpful. A tile request handler, configured via a config file, could intercept these requests and then reformat them and pass them on to the real resource and return & cache the resource in whatever manner you’ve instructed it to (disk, memory, Amazon S3, raster data catalogue, etc..).

The main drawback with TileCache is that it only accepts WMS-C, TMS, or WorldWind requests. The tile request handler should be flexible enough to take regular WMS requests and translate (normalize) them into WMS-C or TMS requests and cache a standardized tile, which could be returned the next time any arbitrary bounding box query came in that contained that tile. OpenLayers does that request translation on the client side but you could connect to many more client types if you did it on the server side. Additionally our tile request handler should understand Google, Yahoo, Virtual Earth, etc tile requests and translate those into a standardized cacheable request as well. (by the way TileCache IS flexible enough to handle all types of requests, but no one has written a provider for the types it doesn’t).

So for tiles the premise is accept just about any valid request, fetch or create & cache the requested tile, then return a reference to the resource or the resource itself.

The Plumbing

Following the lead of FeatureServer & TileCache yo have the basic processes:

For Vectors:

Request -> Translator -> Query Object -> Provider -> Common Feature Object -> Serializer -> Response

For Tiles:

Request -> Translator -> Storage Provider -> (Layer Type Provider, if not found by Storage Provider) -> Response

Using this method you can plug in any provider you want. You can interact with ArcGIS Server via SOAP API calls, directly connect to ArcObjects, ArcSDE, Spatial RDBMS, use the OGR library, whatever you want as long as you can translate the feature into some lingua franca that can then be serialized in to any available format. That format could be any of the wire formats (GeoJSON, KML, GML, GeoRSS, etc..) but it could also be a shapefile or other binary format. You just need to plug in the right provider for that “serialization”.

On the client side it would probably be helpful to provide some helper JavaScript functions that can help translate vector wire formats into map objects and vice versa.

So we have the basic blueprint, all the copper we need, and a workspace. Let’s get plumbing!




FWTools & MySQL leaves ArcGIS geoprocessor in the dust

22 01 2008

I wanted to know what block groups where included in each of the Zillow neighborhoods. So I needed to:

  • Merge all the Zillow neighborhoods into a single datasource
  • Intersect the block group data source with the neighborhoods

I had just done a similar process for the block groups and all US Census “populated places”, so I knew that the spatial intersecting in the ArcGIS geoprocessor was going to take a bit of time. I already had a MySQL data source with the block groups, so I thought I’d do a comparison.

FWTools & the Geoprocessor merged all the shapefiles into a single shapefile in about the same amount of time. In case you don’t know, here is how you do that using the ogr2ogr utility & DOS/Windows BATCH syntax.

FOR %f IN (*.shp) DO ogr2ogr -update -append -f "ESRI Shapefile" zillow_hoods.shp %f -nln zillow_hoods

Now that I had a combined shapefile, I loaded into the MySQL database,  using the ogr2ogr command for MySQL. This was executed in less than a few seconds.

Finally, I wrote a SQL query to intersect the 2 datasources and create a new table with the results.

CREATE TABLE remap.bghoods SELECT b.bg_num,z.* FROM remap.bg_data b, remap.zhoods z WHERE INTERSECTS(b.SHAPE,z.SHAPE)

The final result

ArcGIS Geoprocessor : 1hr 2min

MySQL : 5min 29sec




Creating ArcGIS Image Catalog From USB Stick

20 01 2008

I like using ArcGIS Table Based Image Catalogs when working with large numbers of rasters in ArcMap. The old style table based Image Catalogs work for clients that still have ArcGIS 8.x or even ArcView 3.x. While a TileCache/Mapserver WMS image server is faster, it is sometimes not a real option for certain clients & situations.

However, lately I’ve been running into a number of situations in which the Image Catalog creator scripts that I’ve always used : USGS Image Catalog Builder or This One aren’t working. On Vista, the user extensions won’t even register and thus are of no use. I’ve also had troubles with large numbers of the newest MrSID type imagery giving "Automation Error" errors & the script stops working.

So I needed another way to create image catalogs that was easy and wouldn’t require me installing additional software on clients’ computers, etc.. While, I could have used Python or some other script, I wanted something that people without any scripting or coding experience could also do.

Software Used:

  • FWTools (with a few environment variables modified in the startup batch file so that it would run properly from a stick)
  • PSPad

Steps:

  1. Open FWTools Command Line
  2. Run gdaltindex on the folder containing the imagery.
    gdaltindex naip_05_index.shp 2005_NAIP_NC\*.sid
  3. Use ogr2ogr to create a BNA format file from the tile index shapefile created in the previous step.
    ogr2ogr -f BNA naip05index.txt naip_05_index.shp

    This gives you a file with this sample output:

    "2005_NAIP_NC\e2997_01_1.sid","",5

    596140.0000000000,3319570.0000000000

    602854.0000000000,3319570.0000000000

    602854.0000000000,3311970.0000000000

    596140.0000000000,3311970.0000000000

    596140.0000000000,3319570.0000000000

  4. Open the file with PSPad
  5. Record a macro in PSPad that  flattens this to a single line.
  6. Go to Tools->Macros->Marco Manager and select "Run X Times" and have it run for as many files as you created the tile index from.
  7. Final step is to label the columns correctly
    • You need to add IMAGE, XMIN, YMIN, XMAX, & YMAX column headers
    • The 2nd coordinate group is the XMAX/YMAX group & the 2nd to last (4th) coordinate group is the XMIN/YMIN group.
    • You can call the other columns whatever you want, they will be ignored. However you MUST label them or delete them. I use:
      IMAGE,A,B,C,D,XMAX,YMAX,E,F,XMIN,YMIN,G,H

THAT’S IT!

Now you have a CSV based text file table that you can use as an Image Catalog.

The only additional step you may need to take is to convert it to a DBF file to work properly on older versions of ArcGIS or ArcView. This can easily be accomplished in the ArcCatalog, ArcToolbox or by loading the table into a map and exporting it to a DBF. Alternatively you can just use Excel, Open Office, or DBF Explorer to convert it as well.

It actually took me much longer to write this post then it did for me to create several image catalogs this way.




Part Timer Goes Full

13 01 2008

Three months ago, in October, I quit my regular salaried job and became a full-time freelance developer. My son was born in July and I took significant time off of work. During that time I started aggressively pursuing work for Kestrel Computer. I made a rookie mistake of worrying about not having a regular paycheck and said yes to every project I could. I ended up working much longer hours and harder than I had ever worked for my previous employer with just barely more pay. The reduced pay was due in large part to me GROSSLY under estimating my time to complete projects. When you are used to stealing time in evenings & weekends to do freelance work and then see entire weeks of possible work time in front of you, I think it is natural to assume you could do exponentially more work that ever before.

However, even with the large amount of family & interaction with my newborn son that I missed out on, I knew I would rather be doing full time custom development work than working for my previous employer.

I had always bristled at any sales targets, quotas, or billable work targets in previous jobs. I am not primarily motivated by money and such things generally struck me as something “dirty” & un-pure. I am mostly motivated by the challenge of solving problems and by food :) . However, now there is a direct correlation between my billable hours and food on our table. Suddenly tracking, targeting, and making goals for billable hours/work no longer seems cheap & tawdry but rather vital & important. To ensure that I am accountable, my wife is helping set my billable work goals & tracking them. To motivate us, if I hit my billable work goals every week in the month, we get a date night. With a 6 month old & a 4 year old, that’s pretty good motivation.

I’ve promised myself that in the new year I would also be contributing more regularly (at least once a week) with my blog. The future post will be more technical than personal. I’ve gotten the opportunity to work on some interesting things and would like to share them with the community.




The FREE Rosetta Stone of GIS Data Formats

9 04 2007

Could DNR Garmin be the Rosetta Stone of GIS Data?

It will now read from and/or save to the following file formats:

  • Shapefiles
  • ESRI File-based Geodatabase (9.2 only & only if you have it installed on your system. It apparently uses ArcObjects)
  • KML
  • GPX
  • Text files (comma or semi-colon delimited)
  • DBF

 

The Save As file options (I don’t have ArcGIS 9.2 installed on this computer so the Geodatabase option doesn’t show up).

In addition to saving as & reading from files, it will also read from & save directly into ArcMap or Google Earth.

A new version (5.2 beta) was released on March 12, 2007, with several new & important features. This program was already on my “Must Have” list for anyone working with GPS units. It is invaluble for the field work we do at Plateau Land & Wildlife.

Another great feature that it has but only some of us can use is the Image Hotlink/Geocode feature. This works essentially the same as GPicSync (a Google Code project). But DNR Garmin has been doing it for several years now.

This feature works based on the time stamps of your waypoints or tracks and the time stamp of your photos. It can automatically calculate an offset if your clocks aren’t syncronized.

Notice the “Project and Unproject Coordinates” menu items above. The software uses the Proj.4 projection libary to allow reprojection into a vast array of coordinate systems & datums very rapidly.

I’m fairly certian that you could use the libaries used by this program to make a utility that worked with any and all attributes included in your data rather than just the GPS centric fields that this works with by default. To be fair, though if you have some standard attributes, you can cutomize the fields for waypoints, tracks, routes, and real-time data to include whatever fields you want.
Update - Price for all of this : Free

I’m sure you can think of plenty of innovative ways to use this software but let me share an example with you.

As I’ve posted about before we do a lot of bird surveys in the spring. Many of these are done by sub-contractors who don’t have access to a GIS system. Let’s say we have a new client for whom we have never done a survey before. We send the sub-contractor a shapefile of the client property. She then uses DNR Garmin to convert that shapefile directly into KML & into her Google Earth My Places file. She opens up GE & there the property is with aerial photo, roads, & elevation. She adds placemarks for each of the proposed survey stations which she picks based on the info visible in GE. If she wants to colaborate with the biologist familar with the property they can easily share the info to each other. Then she opens DNR Garmin, loads those placemarks directly from GE to her GPS unit. Gets driving directions from her house to the client property through GE and goes and does the survey. If we wanted her to, she could send us back a shapefile of the census stations by saving the actual station locations from her GPS unit into a shapefile via DNR Garmin.




Comparing Mashup Platforms Using JSON & MySQL - Part 2 - Process JSON

30 03 2007

Technologies Used: PHP 5, MySQL 4.1.2, JavaScript, JSON

Software Used: PHP Designer, Aptana, Firebug, Firefox

There are a wide variety of ways to handle JSON that is returned from a server. One of these methods is to provide a callback function pass it to the JSON emiting webservice via REST. When used with static or dynamic script tags, you are no longer restricted to Cross Site Scripting (XSS) limits requireing that the source be on the same server as your webpage. The XSS limit seems particularly ill-suited for mash-ups. The callback function method is what I have choosen for my examples. You will also see a static script tag for these examples. That is to simplify them and make them easier to read. The general preference is to use dynamic script tags rather than static ones.

Actual JSON string returned:

handleJSON({”Observations”:[{"Station":{"number":"1","lat":"29.5762","lon":"-98.7041","total_sp":"9","Species":[{"common":"Mourning Dove","number":"1","code":"MODO","scientific":"Zenaida macroura"},{"common":"Eastern Phoebe","number":"1","code":"EAPH","scientific":"Sayornis phoebe"},{"common":"Carolina Chickadee","number":"1","code":"CACH","scientific":"Poecile carolinensis"},{"common":"Black-crested Titmouse","number":"3","code":"BCTI","scientific":"Baeolophus atricristatus"},{"common":"Carolina Wren","number":"1","code":"CARW","scientific":"Thryothorus ludovicianus"},{"common":"Bewick's Wren","number":"2","code":"BEWR","scientific":"Thryomanes bewickii"},{"common":"Rufous-crowned Sparrow","number":"1","code":"RCSP","scientific":"Aimophila ruficeps"},{"common":"Northern Cardinal","number":"3","code":"NOCA","scientific":"Cardinalis cardinalis"},{"common":"Brown-headed Cowbird","number":"1","code":"BHCO","scientific":"Molothrus ater"}]}},{”Station”:{”number”:”2″,”lat”:”29.574″,”lon”:”-98.7036″,”total_sp”:”5″,”Species”:[{"common":"Black-crested Titmouse","number":"1","code":"BCTI","scientific":"Baeolophus atricristatus"},{"common":"Carolina Wren","number":"1","code":"CARW","scientific":"Thryothorus ludovicianus"},{"common":"Ruby-crowned Kinglet","number":"1","code":"RCKI","scientific":"Regulus calendula"},{"common":"Northern Cardinal","number":"1","code":"NOCA","scientific":"Cardinalis cardinalis"},{"common":"Brown-headed Cowbird","number":"1","code":"BHCO","scientific":"Molothrus ater"}]}}]});

It is not very human readable, but it is highly machine readable.

Below is an extremely simple example of handling JSON and doing something with it.

<html>
<head>
<meta http-equiv=“Content-Type” content=“text/html; charset=iso-8859-1″ />
<title>Simple JSON Handling</title>

<script type=”text/javascript” charset=”utf-8″>
function showCoords(jsd){
var lat = jsd.Observations[0].Station.lat;
var lon = jsd.Observations[0].Station.lon;
var msg = ‘Lat:’ + lat + ‘, Lon:’ + lon;
alert(msg);
}
</script>
</head>
<body>
<script src=http://www.plateauwildlife.com/bbc-mgmt/getstations.php?action=getdata&cid=2&year=2006&func=showCoords type=”text/javascript” charset=”utf-8″></script>
</body>
</html>


See It In Action

The above example is not particulalry usefull for anything other than demostration purposes. We want to actually DO something with our JSON to move us closer to creating the actual mashup. The number thing which I intially strugled with when using JSON & callback functions was that you MUST define the callback function BEFORE your dynamic or static script tag.

Google Maps, Google Earth, Yahoo Maps, and Virtual Earth all take HTML for the contents of the info window when you rollover or click on a point.

ArcWeb Explorer (AWX), however doesn’t take HTML as info window content. AWX does take styled text, videos, picture, audio, & swf for info window content. To this end, if you want to embed rich non-HTML content, AWX allows for some extremelly interesting content to be blended together and presented with great ease. The documentation for text styling is lacking, so creating simple content is actually more dificult in this platform than the others.

I’ve created a javascript file which we can reference in any of the HTML docs that actually embed the mashup.

This file contains

  1. The main callback function
  2. A function which builds an array of HTML tables containing the formated results from each station
  3. A function which builds an array of jscript strings containg the weakly formated results from each station

See The HTML Builder In Action

    //A global variable to assign the parsed JSON to

var jsobj;

 

//Main Callback handler.

//simple assignment to a global variable allows me to reuse and pass

//around the object without any server trips

function handleJSON(reply){

    jsobj=reply;

}

 

    function buildHTML(Observations){

    var info_win = new Array();

    // Build a table element with Station number & total species observed

    for(var i=0;i<Observations.length;i++){

        str = ‘<table border=”1″><tbody><tr class=”station”>’;

        str += ‘<td colspan=”2″>Station ‘+ Observations[i].Station.number + ‘</td></tr>’;

        str += ‘<tr class=”sta_total”><td colspan=”2″>Total Species - ‘ + Observations[i].Station.total_sp + ‘</td></tr>’;

       str += ‘<tr class=”obs_header”><td>Species</td><td>Number</td></tr>’;

        var details = “”;

        var arr = new Array();

        // assign each Species array to a local variable to reduce typing & increase readibility

        //build an string of <tr> elements containing the details of species observed

        arr = Observations[i].Station.Species;

            for (var y=0;y<arr.length;y++){

            details+=‘<tr class=”obs_detail”><td><a href=”http://www.google.com/search?q=%22′;

            details += arr[y].scientific.replace(/\s/,“+”);

            details += ‘%22″>’ + arr[y].common + ‘ (’ + arr[y].code + ‘)</a></td>’;

            details += ‘<td>’ + arr[y].number + ‘</td></tr>’;

            }

        str +=    details;

        str += ‘</table></tr></tbody></table>’;

        //add table element to array of table element html strings

        info_win[i]=str;

    }

return info_win;

}

 

function buildAWXtxt(Observations){

        var info_win = new Array();

    // Build a formated text list for each Station number & total species observed

    for(var i=0;i<Observations.length;i++){

        str = ‘Station ‘ + Observations[i].Station.number + ‘\n’;

        str += ‘Total Species - ‘ + Observations[i].Station.total_sp + ‘\n’;

       str += ‘Species          Number’;

        var details = new Array();

        // assign each Species array to a local variable to reduce typing & increase readibility

        //build an array formatted text data elements containing the details of species observed

        //using this convention, we can assign a url property to each species line

        //through .data{elements[]} in the properties for each marker

        arr = Observations[i].Station.Species;

            for (var y=0;y<arr.length;y++){

            details[y] = arr[y].common + ‘ (’ + arr[y].code + ‘) - ‘ + arr[y].number + ‘\n’;

            }

        var obs_info = new Array([str,details]);

        //

        //add table element to array of table element html strings

        info_win[i]=obs_info;

    }

return info_win;

}

 

Previous Parts

1. Emit JSON

 

Next Parts

3. Arcweb Explorer Mashup

4. Yahoo Maps Mashup

5. Virtual Earth Mashup

6. Google Maps Mashup




JetDbConnect - a class providing a simplified interface to JET OleDb compatible datasources using ADO.Net

26 03 2007

I needed a way to run SQL queries on structured text files & Excel files.

Download links available at end of post

For the last few months, I have been toiling away in the highly arcane world of fixed width text files. Very large ones at that (250-500mb). These come from text file data dumps of appraisal district databases. I need only a subset of these records. From there they go into Excel where they are cleaned up on a nearly line by line basis that can’t be handled consistently by RegEx since data is recorded so differently by each county. From there I want to summarize that data on a single household basis. I could go in Excel & do:

  1. Advanced filter, Unique Values (based on a unique household identifier)
  2. Use vlookup & sumif functions to copy and/or sum the data for each column

However, this is rather time consuming since different data is provided by in different order and level of detail for each county. Also, for rolls with more than 25k records and over 100 fields these Excel formulas can take ~30min to complete. You then need to copy & paste values or you’ll have to wait a long time anytime a recalc is performed. Needed a programmatic way to deal with this data and to minimize the human time to just records which required actual human eyes to modify.

Admittedly, creating an OleDb connection, command, reader & even datatable using ADO.Net methods is a pretty simple thing.

Dim connect As New OleDb.OleDbConnection
Dim strConnect As String = “Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\MyExcel.xls;Extended Properties=”"Excel 8.0;HDR=Yes;IMEX=1″”"
connect.ConnectionString = strConnect
Dim cmd As New OleDb.OleDbCommand(”SELECT * FROM [Sheet1$] WHERE City=’Austin’ AND LandValue > 30000″, connect)
Dim reader As OleDb.OleDbDataReader = cmd.ExecuteReader()
Dim dt As New DataTable
dt.Load(reader)

But I wanted something even simpler & significantly more flexible than that. I wanted an object that could handle any potential Jet datasource, field list, target table, and SQL Select statement. I wanted something that could be reused in a variety of different applications and not require any hard coding of these parameters or make me implement different ways to do this in each application. I needed a good object oriented design with high reusability. To this end I created the JetDbConnect.dll

Structure

Properties

  • Closed - OleDbConnection closed or open
  • ConnectionString (ReadOnly)
  • DataSource - the file path to your Jet compatible datasource. The ONLY property you must set either by assignment or through instantiation.
  • DbConnection (ReadOnly)
  • ExtendedProperties - if you need to use a Jet datasource that is not directly supported, here is where you would give the connection string parameters that would allow you to use it
  • StorageMethod - one of the 4 enums of the file types this handles directly (Text, Excel, Access, Dbase). Should never need to be set unless you have a really messed up filename.
  • TablesList - List of Strings of available tables in the datasource
  • TargetTable - The table from which to query data, only needed in Excel & Access. If not given for those then it picks 1st table on table list
  • TextFormat - enum: FixedWidth or Delimited. If not set assumes delimited with default delimiter set in your registry settings

Methods

  • New
    • () - Default
    • (DataSource as string, TextFormat Enum)
      • This is used ONLY with text-based datasources
      • DataSource = path to datasource
      • TextFormat Enum is one of the 2 members of this Enum
    • (DataSource as string, [TargetTable] as string)
      • This can be used with any datasource type.
      • If you use it with a text-based datasource, and it is not delimited with your default delimiter, then you will also have to set the TextFormat enum elsewhere BEFORE using one of the Command,Reader, or Records methods
  • Close() - manually close the connection. Not required but can be useful
  • CommandGetAll, ReaderGetAll, RecordsGetAll ([DataSource] as string, [TargetTable] as string)
    • All 3 methods take the same parameters, an optional datasource path and an optional target table name
    • These methods allow you to easily retrieve an OleDbCommand, OleDbReader, or DataTable containing all the records in your target table of your datasource
    • If you don’t set DataSource here and it has not been set elsewhere, this method will fail
    • If you don’t set TargetTable here and it is an Access or Excel datasource and TargetTable is not set elsewhere, then “Sheet1″ or the first member of TablesList will be used for Excel or Access datasources respectively
  • CommandGetSql, ReaderGetSql, RecordsGetSql (sqlWhere as string, [fields] as string, [DataSource] as string, [TargetTable] as string)
    • All 3 methods take the same parameters:
      • Required: sqlWhere string, ex. “zipcode=’78258′ AND land_value>10000″
      • Optional: fields. A comma delimited list of fields to return with the query. ex. “owner_name, zipcode, land_value”
      • Optional: DataSource & TargetTable - see above for how the default values are handled for these 2 parameters
    • These methods allow you to easily retrieve an OleDbCommand, OleDbReader, or DataTable containing just the records in your target table of your datasource which meet your SQL criteria and optionally to return only the specified fields
    • If you want ALL the fields, pass Nothing or Void for fields. An empty string will cause an error.
  • SchemaTable
    • Returns a Schema DataTable for the DataSource not a TargetTable.
    • If you want a Schema DataTable for a table then get it from an OleDbReader on that table
  • FromXml, ToXml
    • Provides methods for serializing & de-serializing the object.
    • Since only DataSource is required for text & Dbase files and DataSource & TargetTable for Access and Excel, I don’t see these methods as being particularly useful. However, I always like to implement them just in case I need to persist an object or compare an object to a previous state.

Usage

Examples:

Get an OleDbReader for all records in an Excel file from a named sheet

dim jetdb as new JetDbConnect(”C:\myfiles\ExcelFile.xls”, “Hays County”)

dim reader as OleDbReader = jetdb.ReaderGetAll()

-or-

dim jetdb as new JetDbConnect

dim reader as OleDbReader = jetdb.ReaderGetAll(”C:\myfiles\ExcelFile.xls”, “Hays County”)

note: the class automatically adds the required decorations so that Hays County becomes [Hays County$] as required for SQL statements on an Excel datasource

Get a DataTable with select records & fields from delimited text file

dim jetdb as new JetDbConnect(”C:\myfiles\some text file.txt”)

dim dt as DataTable = jetdb.RecordsGetSql(”zipcode=’78258′ AND land_value>10000″,”owner_name, zipcode, land_value”)

note: I did not need to ever specify the TargetTable attribute since it is automatically set to “some text file.txt” and the Data Source is actually set to “C:\myfiles”

Get a list of available Sheets and Ranges from an Excel file without the need for any Office libaries or applications

dim jetdb as new JetDbConnect(”C:\myfiles\ExcelFile.xls”)

dim tables as List(of string) = jetdb.TablesList

Downloads

Just the DLL

The Visual Studio 2005 project




Dev Summit Final Day

24 03 2007

Day 3 Thurs Mar 22

I’m finally getting a chance to blog about this now. Steve, I, and nearly 2 dozen other attendees got totally screwed by the storms in Phoenix. However, at least we didn’t have to sit on the tarmack for nearly 3 hours like the other group waiting at the same gate. Though, by the time I got home 25 hours after leaveing the conference, sleeping in the Phoenix airport and eating only pretzals and snack mix; that wait seemed pretty trivial. I hope everyone else also eventually got home safely.

General Thoughts

This was a short day. Only 2 sessions and then the lunch & closing session. It seemed that about 25-30% of the folks had already left and a number of the ESRI devs had left as well. The sessions seemed a bit less technical that those on Wensday.

Sessions Attended

Building & Using ArcGIS Server Map Caches (Best Practices)

The MapCache is a tile cache created by rasterization & tile cutting of the data you have presented in your MXD. The main take home points of this talk were:

  • Understand what a tiling schema is 1st
  • Design your tiling schema BEFORE trying to build on for production use
  • Design a SINGLE schema that will serve the entire organization and all your data layers. In other words: Level 10 = the same scale for regardless of which layer you are building from; Tile sizes should be equal across various data sources; Use same origin point for all caches.
  • Client can rapidly reproject data to any projection system so use one that makes the most sense for both your data and web services methods (ideally, Geographic).
  • DO NOT use png24. PNG8 best if you need transparency, JPG best if you don’t need transparencey.
  • Smaller tile sizes (128×12 8) do not neccessarily result in faster performance over the web, while they take 5 times as long to produce & take up significantly more storage space than large ones (512×512). Test to make sure performance improvement is worth the extra effort. Use the largeest tiel size you can.
  • Fuse as many layers together as possible to create your base map caches. Therefore instead of Intermitten Streams, Permanent Streams, Rivers, & Lakes all being separate caches, they would be a single Hydrography cache that would appear as a single layer in the map service.
  • Each zoom level should be cartographically distinct
  • Forcing an ExportMapImage call from a web service can be a big performance hit. Understand what would make a client do that and avoid it as much as possible.
  • Building large many leveled caches can be VERY time consuming (days-weeks for worldwide layers). Pick a small area, TEST-TEST-TEST and then TEST before doing the full export, instead of having to do it over & over.
  • Cache will build over the extent of your MXD if the layer(s) you are building a cache for don’t have data in the full extent, background color tiles will be created. A waste of CPU time & storage space.
  • Dynamic cache building (a la TileCache) is planned for future releases.

Creating Windows & Web Apps with WPF

This was not at all about GIS. It was strictly on WPF.

WPF is very powerful & can provide you with a lot of UI design patterns.

Security & Permissions can be tightly controlled. Each app runs in a security sandbox. App origin, app settings, and client settings all interact to chose the most restrictive security level for the app.

You can do stuff with this technology TODAY. But, it’s real power won’t be realized until Visual Studio ORCAS is released and Vista or .Net 3 runtimes are more widely accepted. The 1 code for web & desktop paradigm is only for IE6+ with .Net 3 runtime right now. Must use WPF/E for other browser support.

Closing Session

Open Q&A after lunch. No really hard questions were asked, but Steve & other continued to point out that the extra cost of putting the ADF on a separate server is just not reasonable, and that the overall cost of ArcGIS Server is too high.