Discussing Cricket with an Alaskan

Whilst this post might have more to do with a game played with leather and willow to a guy who lives in a part of the world that could only accommodate ice cricket for most of the year, well to be honest the UK isn’t much better for cricketing weather either but that’s beside the point, it’s actually about this years ESRI’s Developers Summit which I was lucky enough to attend this year in Palm Springs.

To ArcGIS 10 and beyond!

image The 2010 #DevSummit was a cornucopia of new technologies and online systems rolling out towards the release of ArcGIS 10. Even though I work for a distributor there was still lots of new stuff to see especially around the online world through use of Amazon for hosted ArcGIS Server (pricing to be confirmed), you can see an overview imagefrom the Business Partners Conference here and the development of the aforementioned arcgis.com (currently not accessible, but will be real soon and has the new ArcGIS Explorer Online) which you can see two videos of here (BPC) and here (DS).

The iPhone SDK got quite a lot of slide-time in both the plenary here as well as a number of sessions. I wanted to make time to have a look, I have an iPhone but no Mac, but there were so many other interesting sessions that I never had a chance, although I can now catch up online.

Online all the time.

Due to the nature of today’s internet the videos of the developers summit were online before my flight had dropped me back at Heathrow. So you too can follow mostly the same schedule that I followed, minus the face to face discussions with Development leads, Service Leads, Product Managers and other Distributors that make conferences like these invaluable and allow you to plan your development strategies for months to come. Some of the sessions I say were as follows.

  • The plenary : lots of videos here. Digs into all the different areas of the upcoming product.
  • Using the ArcGIS Server REST API’s : I can’t get enough about the REST API, the variety of new geometry functions provided
  • An Overview of the ArcGIS API for Microsoft Silverlight/WPF : I’ve been doing lots of Silverlight development recently and whilst many of the cool kids went to the Flex sessions, I checked that I wasn’t doing anything insane by going to some of the Silverlight ones.
  • Python Scripting for Map Automation in ArcGIS 10 : I like the way you can analyze map documents using arcpy! In fact I like most of the stuff you can do with arcpy, now if only someone would write me an avenue wrapper for it and I’d be good.
  • Advanced Map Caching Topics : Lot’s of tips about how to create efficient caches, how caches will change in ArcGIS 10 and the new mixed mode image format, watch and cache.

Unfortunately some of the interesting ones aren’t up there yet, especially the one from the prototype lab with Augmented reality and Microsoft surface demo’s which looked quite slick (and had one of the biggest overhead camera’s I had ever seen), David Chappell’s keynote going through Cloud Platforms (a must watch if you haven’t heard him go through this before) and finally there was another Silverlight session where they showed Windows Phone 7 integration with the Silverlight API. If I see them post them online I’ll update the list above.

So what’s with the title?image

Anyway the title of this piece is all about the benefits about attending a conference far away from home, the opportunity to talk to people face to face about technology, something that is not possible when your watching the talks over the internet. In this case, we were out for dinner one night, when a colleague turns up with some poor unsuspecting fellow he’s imagemet at the conference (@jdoneill from Girdwood, AK), as is the way with conversations with me the subject turned on to cricket and then the floodgates opened.

Although we never did discuss the merits of leg theory I did try and cover all the other aspects of the game, he even managed to seem interested for most of the conversation as well, he was a baseball fan so the compare / contrast, mutual appreciation of each others stat laden sport ensued. If you ever want to image read such a comparison from an English point of view then I’ll point you to this book.

As an aside we won’t mention about the guy he turned up with the next night. Remember, what happened in Palm Springs, stays in Palm Sp
rings, well apart from the technical information that is.

Watermarking, WMS and maybe other things beginning with W.

I should caveat this post with the often used phrase, ‘don’t try this at home kids’. When tinkering with the guts of any system and modifying the information being sent to and from a service by ‘hacking’ into the request pipeline of a message your opening up a whole can of performance and stability worms that need a great deal of testing under load to understand the direct effect on the scalability of any site.

A Simple Questionimage

This post is based upon a question I had with a customer at our recent DeveloperHub conference in Birmingham. He asked how it would be possible to watermark an image that had been served from a request to a WMS service. Ed has given an excellent overview about why a customer might want to watermark their images here and some methods to do it. For this query though there was a need to do the watermarking at the server level and not the client, as you don’t want to restrict access to only those systems that have been modified to adopt whatever watermark solution you have adopted.

With WMS you also to need to make sure you don’t force a client process a response that is not compliant with the OGC WMS specification. So you end up with the need to do some invisible modification of the request or response in order to handle the addition of a watermark without any client realising anything has happened.

The WMS Request / Response Cycle

It’s worth taking a brief aside here to look at the WMS request / response cycle. WMS Services can be simply called from a URL as stated in the OGC specification (here) using a HTTP GET. Depending upon the type of operation you are trying to perform the parameters for the URL will vary. In our case we are most interested in the operation that requests maps, unsurprisingly enough the GetMap request. This uses parameters to control the location of the area to return a map of, the layers to be displayed and the format of the image to be returned.

Once the request has been processed the response is in the format of a image is returned as binary to the client. In a web browser that gets placed into an image for display. It’s this binary image that we are able to edit as it goes through to watermark.

Why not burn it in? image

One question that needs to be answered is that of why you don’t just create a tile cache with an image already burnt in. This would be the most ‘performant’ solution as it front loads any of the processing away from the actual request by the user, this increases the response time but leaves a cache that can possibly only be used for one task. Indeed with more than one client requiring more than one type of copyright notice, or image overlay, then each would possibly need their own set of tiles, or own service.

Alternatively, you could have a map service with a layer which contains any of the watermarking details. You can search the WMS request string for the inclusion of this layer, if it’s not there then you can always add it later. This is fine, but it means you are actually messing with the request that’s being made by the client, which could possibly cause for bugs to be introduced into any application making the requests.

A more flexible solution, albeit possibly less performing, would be to handle the addition of any information over the top of the image at a stage of the request where it can be applied post the actual creation of any map. In terms of ArcGIS this would be after the request has come in, been processed by the map service and then return the clear image, as a binary image object.

A Pipeline Solution

The last method was the answer I actually gave at the event. This was to intercept the response to the WMS service and stamp the returned image with the required watermark, either textural or image based. But how to achieve this, the serving of the image from ArcGIS is handled deep within the SOC process which was untouchable, what wasn’t untouchable was the request/response pipeline in the web server, in my case IIS. In the past this might have required the writing of some sort of ISAPI filter to hook into this pipeline, but since .NET came along it became possible to write a HTTPModule to do the same.

The HTTP Module allows you to hook into public events in the request / response pipeline. Specifically the BeginRequest and EndRequest events, which allow you to check the content of a request before it’s forwarded on to ArcGIS Server and process the returned image that is the result from ArcGIS server, before it’s returned to the client. This pipeline can be simply shown in the following diagram.

 image

Bringing it all together

In order to get the application to run, and to be able to debug it (especially in IIS6 which can only work vsprojectwith files processed by the asp.net worker process) you need to create a handler that maps to the arcgisservices directory within your ArcGIS install (see why I say don’t do this at home!). The easy way of doing this is creating a visual studio project within that directory (as you an see in from the VS 2008 project to the right).

Once the solution is in the right place you can update the existing web.config within that directory. It will already contain the ESRI Handler and Module details that are needed for the operation of the ArcGIS server services, by placing an entry for a new module after the existing ones will allow you to hook into the pipeline before and after the ESRI modules (remember this could seriously damage your ArcGIS Server health, use with caution on a test machine before letting it anywhere near production). The entry would be similar to that given below.

wbcnfgOnce we have these elements in place we can add our class with the IHttpModule interface. You can see how to do this in the example at the MSDN site for the creation of a custom HTTP module.

Hooking into the ArcGIS Server Requestimage

In order to perform the watermarking task, its necessary to perform a number of steps, before and after th e request. Where we get involved is in the BeginRequest event handler, this gets fired once a request is made to ArcGIS. In any system it’s good to only do processing of requests when needed, therefore being able to test that a request to ArcGIS Server is for a WMS map is necessary. This can be done by converting the incoming request stream to a string and parsing that, code to be found here.

At this point our watermarking service could perform no end of housekeeping, checking the type of watermark to apply to a specific service or if indeed it is to be applied at all. At this point it also might be good to read any image to be applied to the response and add it into a cache layer (if we do this for lots of images we don’t want any disk access slowing us down more than once). We are now set to let the request filter down the stack to ArcGIS for processing and we can wait for the EndRequest event handler to fire and for use to get down and dirty with the WMS response.

In next week’s episode – Hooking into the ArcGIS Server Response

image It’s at this point I realise that I’ve written another 1000+ words on something that started as a simple question and that having to read much more in one go might cause you to slowly lose the will to live. In order to save you at this point I’ll save the next part, taking the response and applying the watermark till another post. Probably after the ESRI Developers Summit where I know doubt will be shown a better way of doing this.

PS: What no code?

So you might be thinking where my sample is, how you can get access to it. Well, whilst I’ve provided all the tools to write this application, they haven’t been tested especially for use at scale. Modifying the pipeline of the ArcGIS is not to be taken lightly. The amount of work to actually do this isn’t very hard, it’s almost all provided with samples from MSDN and like I did, I would start by reading the custom HTTPModule section on that site and good luck!

Install as I say not as I do.

betaAs we all know the pace of change in technology shows no sign of abating for good or ill. In software terms it’s a continual moving walkway of new patches, version and features, usually for the better sometime not so. I’m both lucky and cursed to be able to install a wide variety of new software where I work and at this moment installing a beta of ArcGIS 9.4 (or 10 as it will soon be) onto a new copy of Windows Server 2008 R2. I’ll soon be downloading and installing a copy of Visual Studio 2010 onto that virtual machine as well. Lucky eh? Well yes and no, lucky because I get to try out new technology as it comes out, unlucky as I’m sure there will be a whole host of frustrations about bugs and workflow changes that will eat time along the way.

This is good right?

When you see a new technology being released, usually as part of an existing product you use it can be tempting to upgrade as soon as possible. When you’ve been working on that technology for a while, at the cutting edge so to speak, you want to tell people how good it is. The problem comes around when the technology you use is not actually supported for the applications running on it. Sure it might work, even if you have to spend all night tinkering with he registry, but without support you’re on your own (or at the very best, it’s you and a forum of people!).

There is also a propagation of new and cool, as people install the newest and shiniest new software others also do, as successes increase people believe that because it works it is also support, this is definitively not the case, especially in the case of server software.image

Windows Server 2008 R2 and ArcGIS 9.3.1

I like Windows 2008 R2 in the same way as I like Windows 7, they have the same heritage, the main one being that they are not based upon the same core as Vista. Where possible I’ve upgraded all of my servers to this release, all of those servers I mean that do not run ArcGIS 9.3.1. Why if it so good, well because it’s not on the magic list. “What magic list?” I hear you ask; this one. The image below shows the list of platforms that is supported by ArcGIS 9.3.1. Look through it, notice no R2.

image

Now there are people who have no choice than to install on a new system such as R2, where purchases or machine suppliers can’t give you a copy of non-R2 or Windows 2003, in these cases, such as given here, the time for installing can be a lot greater than it should have been given a supported operating system, even if it seems that some people have an easier time installing it than others.

Now I have quite a lot of questions about which of the Microsoft operating systems are the best to install ArcGIS Server on, I used to say Windows 2003 as I felt at home in IIS6 manager and used to get lost in imagethe new IIS7 manager, but now I have my head around it I stick with recommending Windows 2008. I never recommend the use of desktop systems for anything more than brief testing (I do development against ArcGIS Server from a desktop machine, in my case Windows 7, I try and never install server software on my development machine if I can help it). Doing this gets you into good habits and doesn’t lead you to the problem of serving out large caches of data to an organisation using Windows XP’s crippled IIS5.1 (yes I have seen it happen, and no I don’t encourage de-crippling through registry hacking).image

Remember by its very name ArcGIS Server is a server not a desktop product and friends don’t let friends install servers on desktops. Until I here otherwise from places like here and here, I for one won’t be recommending R2 for ArcGIS 9.3.1 (and nor should other people be encouraging it!). If you have to, then  good luck, I’ll try not to sail in you boat.

Install as I say not as I do

imageSo to sum up, it’s easy to think that as people blog about software working together they are often only giving their opinion about how it has worked for them. They might be able to give you advice about how it might work for you, but when your production system goes down in the middle of the night because your versions were not certified I can guarantee that they probably won’t be coming round to explain the short comings to your boss.

When I say here that I’ve seen ArcGIS Server 9.3.1 running on Windows Server 2008 R2 don’t assume that it’s supported when the Support site says that 2008 R2 isn’t supported for 9.3.1, if you want to go ahead and do it, it’s a free country, but don’t expect the support department to lose sleep over your downtime.

The CleanerSure it’s nice to
try new software out once in a while and even install beta products to work out how they tick, but when money is on the line, take some advice from someone who has been there before and be conservative with your software installs, if it’s a production system then play it safe. So you don’t need to employ a ‘cleaner’ to remove the mess.

image

Anyway I’m off my installs are done and I have beta software to make work.

The web world is (mostly) flat.

It might come as a shock to many of you, but often on the internet the world is flat. Yes I know that you thought this whole debate had gone out with the ark (or actually a little later), but after years coming to imageterms of the world being a sphere, cartographers everywhere needed a method of putting that world  down onto paper.

Now this was fine for many years until 2nd May 2000 the US decided to turn off Selective Availability and whole world seemingly brought into WGS84. The difficulty here though is the fact that WGS84 and Longitude and Latitude coordinates are an approximate representation of the position in the real world, geographic coordinates, many maps paper and web based are representations of a flat world using Cartesian coordinates.  

A world of BNG

Obviously as a GIS professional you know all this, although come to think about it, a lot of people in the UK probably haven’t. Why, I hear you cry? Because the Ordnance Survey in its role of national owner of all things spatial in the UK decided that we needed a more accurate representation of the surface of the United Kingdom (fortunately we are not a large county, well unless you measure it in ego image terms, so it can work quite well for us). In doing this they created the British National Grid (BNG) which allows them to produce all of their excellent paper maps (and not paper globes which would be inconvenient for packing up in your rucksack).

Now why do I mention this, well for the first three years of my work at ESRI(UK) I touched nothing else but data in the BNG projection, it was only when I needed to implement a solution using GPS data for tacking refuse trucks that I came up against the need to occur re-projecting data between two projection systems WGS84 (for GPS) and BNG (to plot on the web map).

Whilst BNG is the preferred coordinate system for many users of OS data, amongst many web developers who have come across mapping via the various web offerings of Microsoft, Yahoo and Google (or indeed in the past with ArcWeb Services) the coordinate system they use will be Web Mercator. In fact in the UK there is a chance now that more people use the Web Mercator projection than the BNG projection, a fact that shouldn’t be lost on people. So where was I? Ah yes setting the stage for my problem.

What was I doing?

The reason why I came to this post was the fact I was implementing a little demo for a colleague to show a GeoRSS feed of Earth Quakes (what I term a ‘classic feed’ as it appears in every demo) with the Bing map service, both within the ArcGIS Silverlight API. I thought that this would be no problem as I knew of two existing ESRI demo’s with the functionality I required in. Firstly the GeoRSS layer sample from Morten and the Bing sample from the ArcGIS Silverlight concepts section.image

Now using my advanced skills in copy / paste I managed to get a map that looks like this (image right). The classic, “everything is off the equator near africa” map. Hmm, I think, when i was using the old ArcGIS Services map (as per the sample) I could get an image like this (see below).

imageAh-ha I think, something is up in the state of projections. How can I request the GeoRSS feed in another projection, in this case Web Mercator, rather than the Long/Lat WGS projection that was given as standard.

The answer to my ponderings was that GeoRSS defines the returned format as having to be WGS84 and in order to place it on top of my Bing map I would have to re-project the data myself. Fine I thought, I know how to do that, ArcGIS Server doesn’t have a Geometry Service for no reason you know.

Chatting with the Geometry Service

With the implementation of the Geometry service within ArcGIS it’s been very easy to re-project data between coordinates, you can find the documentation to do this here. This is good as it allows you to project between many different coordinate systems as it’s all server bound you don’t need to pollute your client with any algorithms.

It should be noted here, that over time there have been a number of SRS’s used to define Web Mercator. An email that I’ve seen on the boards which explains the differences between the numbers used can be found here (see the post by Melita Kennedy).

The problem you have is that it can be rather inefficient to get points into the client and then send them to another service to project, get the data back and then have to display that in the map. This solution might be the most flexible (as it could possibly handle any projection required) but it leaves a bad architectural taste in the mouth.

I put it to the back of my mind in the folder entitled, architectures to use when other methods fail and went on thinking about how it might done, back to math (+s for UK readers).

An Algorithm

I’ve always wondered what the actual algorithm used for doing the projections between WGS84 and Web Mercator, this was a demo so I didn’t have to be too careful about the accuracy. I once again used my Google brain to come up with the following link which contained a Python script which contained the algorithm a required. Which when converted to C# (not too hard) looks like so:

image

When I placed this into my GeoRssLoader.cs class file (see the GeoRSS Silverlight sample again) then I managed to get the points placed into the correct position, see map below.

imageSorted I thought, but then I got thinking. Surely this is an amazing common task that everyone is doing, GeoRSS is very popular and there are one or two ESRI developers out there (I know I have 250+ of them coming to see me present at our Developer Hub Conference next week).

If you’re interested in converting between WGS84 and OSGB36 then this link should be handy, if I get time I’ll knock up a C# class doing it and post it on this site somewhere.

The Easy Way

So I was showing my solution to another colleague of mine all impressed that I could do it using the power of math when he said that the JavaScript API had a function for it built in called esri.geometry.geographicToWebMercator(). Gah! I thought, all these JavaScript dudes have it so easy, no one ever creates one of these for us poor Silverlight chumps. Well actually they do.

Hidden away in the ESRI.ArcGIS.Client.Bing.Transform class are the following two methods:

imageBoth of which much simpler than including your own projection algorithm within your code, why reinvent the wheel after all. As we can see there are always many ways to do things with technology. It’s often the simplest one that can avoid notice when your thinking through a problem, although it should be said that there are merits to all approaches due to flexibility (REST Service), transparency (algorithm) or simplicity (existing class), the choice as they say, is yours.

Adventures through the Silverlight

imageOver the last few weeks I’ve been having a few adventures in the world of Silverlight. A bit like Alice I’ve been following white rabbits down holes and through looking glasses. What I’ve discovered is that having an IDE doesn’t always make things easier, especially when the error is occurring somewhere between the chair and the keyboard, a place which is notoriously hard to debug.

Brain don’t fail me now

One issue I have is with my brain. If you start thinking as if the development environment is going to help you, then when it doesn’t it can completely throw the processes you use to figure stuff out, if indeed you can figure it out. Strangely enough (well for me anyway) if I’m in an environment where there is little help, read ‘no intellisense’, then my brain rewires itself for self help. This can often be easier on the development as I tend to check the code more and be more robust my development methodologies (i.e. checking my environment is setup correctly for one thing). Usually I find the differences in coding for Silverlight or using Dojo to follow these patterns (I often still use VS for Dojo, but obviously get precious little help!).

With Silverlight my development is all done in Visual Studio 2008 (with some design done in Expression Blend of course). Now 2008 is quite helpful when checking the syntax of C# code, but it can come off the imagerails with the XAML syntax, so once your done in blend and are hacking around with the mark-up you can often come unstuck. Many the time I’ve spent at the top of the page.xaml tinkering with the namespaces, wondering why the code wont work when I’ve copied it straight from the ArcGIS Silverlight API samples (note: always check the breaking changes in any of the API release notes like here as the samples sometimes lag behind the releases and don’t always correspond).

Other times the IDE just goes a bit spooky on you, such as when I added a new class to the top of my page.cs file (don’t ask me why I wasn’t refactoring things into different places, I was prototyping it’s allowed). Now I figured that this wouldn’t have been a problem, I’ve often slung a class at the top of a file and had no problem (or none I can remember!), but whilst the compilation and run of the application had no problem, the actual linking up of the page.xaml and the page.cs seem to have been b0rked.

Every time I needed to add a new event handler, or to navigate to an existing handler from the XAML to the code I would get the following error:

To generate an event handler the class ‘page’ must be the first class in the file.

Now I didn’t believe what my eyes were reading at first, of course I was using Visual Studio so my brain had partially shut down, therefore using my outsourced brain (read Google) I spent a few minutes imagetrawling the interweb in the hope of finding a solution. I did find it buried deep down in the following thread, where it spelt out the reason for my ‘code fail’ to be the fact that my new class was first in the code behind file, move the class to the end and hey-presto everything was tickety-boo.

Now this serves to highlight both my initial problem of my brain expecting simple issues like this to be sorted out by the wonder that is Visual Studio 2008 and secondly highlighting the fact that whilst an order of magnitude better than Visual Studio 2005 for its integration with Silverlight, it wont be 2010 that Microsoft will have a true development environment for it. Note to self, better get installing the 2010 RC when it’s released next month to check life will be peachy.

A Dash or Two

Where Silverlight (or any RIA environment, Flash, HTML 5) really excels is in the delivery of dashboards that allow for the easy cognitive processing of information without the clutter of hardcore GIS tools that are often prevalent in some internet mapping applications.

Indeed in my opinion (not necessarily anyone else’s though) if your using Silverlight to just deliver Y.A.M.A. (Yet Another Mapping Application) then unless you really need it to be rotating on a flying cube surrounded by dancing leprechauns (however tempting it might be) then you probably need to be using a more standard HTML/JavaScript based client, such as Dojo which doesn’t have the plug-in overhead.image

The ability to present and link multiple maps together, all updating in real time, with graphs and reports can really help sell the benefit of GIS to upper management, who often don’t get excited about data formats, tile caching and the different API’s. Show them the ability to visualise all their assets and modify their assignment in real time allowing for visual modelling of costs, then you might be on to a winner, show them the common operating picture of an unfolding disaster then you almost certainly are, especially if it can save money in the long run. Sure it might use all of the cool technologies under the hood, but most people who make decisions don’t care, they want simple tools that can leverage powerful geoprocessing tasks without even noticing. With good design and the interactivity given by Silverlight (and other RIA’s) then the move of GIS from allowing people to make niche decisions to impacting throughout the business should become a whole lot easier to show and use.

imageLet me at it – hold the white rabbit

Hopefully you can see not only the wonderland that can be offered by Silverlight but also the frustration that can occur if you blindly follow white rabbits around whilst developing.  If you don’t have time to build your own Silverlight client from scratch, you can always get a head start from one of the example applications fro
m the community section of the Silverlight resource center here or some nice examples of dashboards and UI can be seen on the ESRI North East Africa site here.

Which date is it?

image

I have to admit it I have a problem with dates. I always seem to draw a blank when trying to remember them; I’ve outsourced much of my anniversary notification to outlook and my iPhone, or work with them in applications. It has been ironic this week as the last decade ends and the next decade begins that the first thing that stumps me is a little date problem.

A choice of dates

I’ve been doing some development using the Silverlight API and I came to the part of the project where I needed to filter data using a date range and display on the map and in graph (or chart) form. Now querying with dates with any system is always tricky, especially with dates. Getting the right format can be the difference between date success and date fail! The right positioning of brackets or hash (US pound) symbols has often been the bane of my life.

As this is the Silverlight API I’m backing onto the ESRI REST API for my heavy lifting.  So I need to know the right REST syntax in order to create a correct query for use both as the ‘where filter’ for the feature layer as well as the ‘where filter’ for the query task. Fortunately the REST API provides a nice query form with which you can enter these parameters to your heart’s content (sample server example link).

imageNow, at this point you might think my work here is done, that this form and the collective knowledge of the internet (by that I mean my Google search box, to which I have outsourced my ability to remember syntax), would be able to get me to my goal of a working filter and in some ways it did, but then I found that there was a choice.

Choices Choices

Looking at the help page for the REST API here we can see the query layer section which gives the imagepossible filters for the whereas ‘any legal SQL where clause operating on the fields in the layer is allowed’. Now in the project that I was working on the dates being queried were in two different layers, one which was being filtered and the one providing the data for plotting on the graph. Both the layers contained one or more fields of the type “esriFieldTypeDate”, so I thought the same query would work on both, wouldn’t it?

Well it turns out that the answer is no. I initially started with one layer and was provided with the date format of #2009/01/01 00:00:00# which worked fine for querying a single field.

In usage this would give a query in the format of

"FIELDNAME > #2009/01/01 00:00:00#";

But when I applied this to another layer where I was filtering data by two date fields I got the strange error of:

“Unable to perform query. Please check your parameters.”

Hmm I thought, but it worked on the other layer (which is synonymous to the cry ‘but it worked on my machine’) so doing a bit of googling I managed to turn up a link on the forums where someone had had a similar problem and a suggestion to use the following syntax in the where clause:

"FIELDNAME > DATE ‘2009/01/01 00:00:00’";

This worked for both sets of queries. Whether it will work for all databases or data sets will require more imagetesting, but hopefully if your using dates in your application then this way of formatting a query for dates will hopefully work. If I come up with any more definitive answers for which format to use and where or a list of any other date formats that might not work with all datasets then i will endeavour to update this post!

Pointless Predictions

image As the first post of the year you might be expecting a string of pointless predictions about cloud computing, three screens and maybe a slate. Unfortunately I’ve yet to obtain the job title of ‘Futurologist’ so I’ll leave that to those sort of people who probably should really be out there doing a real job, like policeman, soldier or plumber, hmm I think I might need to shut up now and slink off with my Solution Architect business cards before someone outs me as a fraud.

Anyway enough of my blathering and hope you have had a happy new year up till now and your date searches will forever be successful.

The Local JavaScript API for Local People

I was giving a presentation at an ESRI gathering hosted in reading at Microsoft this week, talking about all things Silverlight and MapIt, and very little unsurprisingly about the Flex API. During a presentation imageabout the Web API’s I mentioned JavaScript and Silverlight as two offerings that whilst can do similar things, need to be considered carefully in relation to the audience in questioning and the tasks they wish to perform. Now I like the Silverlight API and as I mentioned in the talk it can really allow for the avoidance of browser support pain by abstracting away the layer between your application  and the browser hosting it.

Often though you don’t want a plug-in to preclude people accessing your site, this might be down to a number of reasons but unless your going to start hand cranking simple code for map access (people who have to disable JavaScript within your browser should stop reading now) then you have the JavaScript API. Usually this is delivered to you from somewhere in the cloud (http://serverapi.arcgisonline.com specifically), but often there are a number of scenarios that might occur that will mean you have to install it locally.

Why Local?

If you are an ArcGIS server subscriber you are able to request the JavaScript API from ESRI or your local imagedistributor as explained here on the support site. There are a number of reasons why you might wish to do this:

You are super secure and think that access to the internet is a potential security risk  and therefore getting access to the online API is not possible. Many organisations have yet to embrace the concept of Content Delivery Networks (CDN’s), such as that used for hosting the ESRI JavaScript API. Whilst in a consumer space this is a common architectural practice as it places the download on the users connection for the CDN and not on the hosts (saving bandwidth allowing that to be used for serving images and functionality). Often within the firewall of organisations it is not possible to access external files as the network is secure.  This is also the case where you have a slow external connection and you wish to limit the amount of information that is downloaded from the ‘cloud’.

imageYou want to create a custom build of the API to reduce the number of files downloaded, improving browser response. The classic number one rule from High Performance Web Sites of making fewer HTTP  requests asks for front end developers to minimise the number of files they download from a site, when building on top of the JavaScript API it is common to build your application into many separate files to separate functionality or Dijits and to allow the usual spaghetti of development that can happen with JavaScript to be controlled.

A concurrent dilemma

The problem with this is that each JavaScript library you add to the application the longer it will take to download due to the limited number of concurrent files a browser can download at once. This is the same if you include new CSS files and images as well. This is down to the way HTTP 1.1 was implemented and setting a limit of no more than two files concurrently per hostname. BrowserScope explains it nicely as:

When HTTP/1.1 was introduced with persistent connections enabled by default, the suggestion was that browsers open only two connections per hostname. Pages that had 10 or 20 resources served from a single hostname loaded slowly because the resources were downloaded two-at-a-time. Browsers have been increasing the number of connections opened per hostname, for example, IE went from 2 in IE7 to 6 in IE8.

image This often leads to developers hosting parts of their application on different hostnames one starting with a URL such as images.site.com and another such as scripts.site.com, to get around such limitations. With large JavaScript libraries the time it takes to download these on slower connections can leave an application imageunresponsive and give a very bad user experience.

You can see from their tests (see image left) that modern browsers have upped this limit to 8 connections, this can have a dramatic affect on perceived speed as more of the application can be downloaded at once and when partitioned correctly (using a lazy loading pattern preferably) then users can see something happening in the browser much earlier than they could with a plug-in such as Silverlight or Flash (beware the loading circle of doom).

Of course with external (public facing) applications you will always have some users using IE6 and the limitations of 2 connections per hostname. This is where the ability to package your JavaScript files up into as small package as possible and as few packages as possible becomes more important and the need for a copy of the local JavaScript API imperative for use in a custom Dojo build.image

One build for all?

Unfortunately each build is different for each project and therefore there is no magic bullet for this. Fortunately there are a number of key pieces of information and discussions about how to perform this task. There is a thread on the ESRI support forums going through the process an excellent Geocortex post upon why they did a custom build here and finally a link to the Dojo documentation outlining the process in full. Whilst it might be a complex procedure, for people who are interesting in getting their application to load in the fastest time within the browser, it should be a procedure worth looking at.

IE 6, 7 and 8 and other browsers. Support them all.

As you can see I’m not one for short posts, but sometimes a great article needs to be pimped. Smashing imageMagazine has lots of those and whilst I’m mostly a Web Developer and not a Web Designer (I do have an iPhone so there may be hope for me) it’s good to keep an eye on the current design trends for web applications.

Smashing, super, great, well not so great with IE6.

Their current article “CSS Differences in Internet Explorer 6, 7 and 8” is a change from the normal rant at Microsoft fare, but aims at understanding all the differences that occur in the Microsoft browser segment of the market around, in this case, their different handling of CSS. The main fact of importance is the imagecontinuing need to support all of the browser due to their market share, each browser holds almost an equal share of the 65%+ of the market (as seen on Market Share).

In fact if you break it down even more we see that IE8 compatibility mode beats Chrome and that really for public sites you should be testing for Safari as well.

Browser Market Share
Microsoft Internet Explorer 6.0 24.42%
Microsoft Internet Explorer 7.0 19.39%
Microsoft Internet Explorer 8.0 16.84%
Firefox 3.5 12.65%
Firefox 3.0 9.62%
Safari 4 2.92%
Microsoft Internet Explorer 8.0 (Compatibility) 2.30%
Chrome 2.0 1.74%

Like any large company, Microsoft has a great deal of legacy to support, people who have brought their products, people unlike myself who don’t want to upgrade at the first opportunity, who are happy with the way something works (even if it is IE6). Especially where a browser still holds such a large part of the market within the enterprise space, the cost to upgrade all of the browsers (which work for all the applications they currently use by the way!) could be prohibitive in the current economic climate.

Testing Painimage

All this leaves is a massive amount of legacy testing pain for development teams, especially when developing applications for use within an enterprise environment. It’s all very well for people on the social side of web development to trumpet the call to the new, but when part of your market is stuck in 2001, so to a certain extent are you.

At this point it’s interesting to look at the support level of browser for ArcGIS Server here. The supported browsers are Firefox 2,3 and IE 6,7,8. That’s not to say stuff won’t work in Chrome (it’s does, I’ve tested it) or Opera (I don’t know, sorry), but the official line is what is shown on the site, so when developing applications for use with ArcGIS Server it’s always good to remember what’s official supported and choose that as a baseline.

In fact many people are tempted to throw the browser out completely and fall back into the loving arms of the plug-in. Flash and Silverlight take much of the worry about browser support of the nuances of CSS imageand JavaScript but providing a nice sandbox for the application to play in. Unfortunately they are no true panacea. Often the enterprises that are still on IE6 are probably on Flash 5, if they have Flash at all and might never have considered Silverlight (although Microsoft’s positioning gives Silverlight a more enterprise-y feel allowing it to make inroads sooner within the firewall). This leaves you back at the beginning needing to support your browser based application in IE6, there is no escape, accept it and test accordingly.

And finally.

One great quote from the comments is this “You could power every house on Earth for a few hours– just from the sheer RAGE MSIE elicits from Web Designers.”. It’s not just IE, the lack of standards amongst browser has always been a frustration, Microsoft might be the current football to be kicked around, but in its day Netscape was the villain. image

Whilst most people, probably even Microsoft, want IE6 to go away, it wont soon, people image like www.ie6nomore.com have a funny campaign, but the only thing that will change that is when all the big companies have gone through a technology refresh and the
re are only a few copies of IE6 lingering left. In the non-corporate world, well as Amy Barzdukas (Microsoft IE general manager) says "Friends do not let friends use IE6”.

Where are you from? A journey into WCF with Dojo.

To celebrate the upgrading of the JavaScript API to 1.5 I decided to have a little play with adding a where are you from page to the site (you can get too it from the page link above). This is a little routine to take your IP address and geo-locate where you are. It’s a very simple application in the style of Al Pascual’s imageMap Stats Silverlight application that you can get here. In fact it’s almost the same, except it works with Dojo uses a different IP geolocator from IPInfoDB as the one used by Al didn’t know places like New Zealand existed.

It was an interesting application to write, not only did it let me get acquainted again with the JS API, but also accessing simple WCF services from Dojo (easier than you might think) and working with the .NET JSON data contract serializer’s. Once you have the basics of those then the world is your oyster in terms of getting functionality into the browser. The Dojo bits were easy, a few bits of JavaScript on the page a call to a service using xhrGet and were cooking on gas, all we need now is the working WCF service. Hmm now that should be easy yes?

WCF Learning Pains

I’ve always been amazed about how Microsoft took a nice simple thing such as Web Services and made it so much harder with WCF. I do understand that you can to so much more, but with the Visual Studio IDE imageyou just used the template to create the webservice (add new!) and there it was, easy as the proverbial pie. Now with WCF it’s all about message contracts and whether they right information has been dumped into your web.config file. Now I do understand the power of data contracts and how they can be extremely powerful, but the fact that after creating a new WCF service, I had to make sure everything was decorated correctly down to specifying the right binding on my endpoint (if you don’t know what I mean then don’t worry it’s not really worth knowing, if you do I think you should get out more). Why do I have to do this? I think I’ve become lazy when developing with Microsoft products, when the tools fail me I have to start relying on my brain again (ok I use Google’s brain, but haven’t we all outsourced our thinking to Google already?), I think I last used that was when I was developing Java applications (or come to think of it debugging JavaScript!). So my new checklist for getting WCF services ready to be used with Dojo is:

  • Is your endpoints binding set to webHttpBinding in your web.config, mine wasn’t it caused me grief.
  • Have you set your AspNetCompatibilityRequirements to allowed in your service attributes? I think this matters, stuff works once it’s been added in (lots more info about WCF and ASP.NET can be gained here, a must read).
  • Does your operating contract (contained in your interface file if you didn’t know) specify the right parameter names for the Dojo request (mine didn’t it took me a while to realise why nothing was actually being passed in, classic brain-fail!). image
    Now I can actually see that these things are going to help me in the long run, I just wished that stuff I could have knocked up in about 5 minutes in the old way takes me so much longer and a great deal more fiddling to do. It’s like having to do real coding again, not just concerning myself with actually getting the applications to do what the customer wants, yeah I remember those days, they weren’t good in my opinion!

serviceHostingEnvironment what’s this then?

Once I had the code working there was one final journey into the unknown depths of the servicemodel section of the web.config file. This had to do with deploying my application onto my host, which has a number of URL’s that are assigned to my domain. This confuses WCF and gives a strange error when it cannot seem to resolve the correct one. So it was back to the web.config file to set the baseAddressPrefixFilters element within the serviceHostingEnvironment tag. Fortunatley my Googlebrain was able to help me out again and I managed to find this post (ironically from my host!) although the simple solution is contained in the comments.

I am not Spock.

Yeah I know you think I’m making it all up, I wish I was. When designing this stuff they should have read Jol Spolsky’s duct tape programmer post to understand whilst many people can get software to work and be delivered on time, were not all Spock.  Nor do we wish to devote out life to being Spock, I have games to play and wine to drink (and if I have any time left over a family to be with, well maybe I can spend a little more time coding!).

Anyway after all this I managed to get my simple, yeah it’s not very complicated once this bits fall into place, location application working. I’ve got a few more plans for it (integration with Firefox location, addition of Tweets in your area, <insert additional social API integration here>, you get the idea.

Once complete I’ll post some of the code up and if your really interested, sometime I’ll go through how I wrote a WCF class to consume the ArcGIS Server REST API, yeah I know you can’t wait. In the meantime I’ll put the existing code up soon but head over to the page to find out where you are (if you don’t already know!) and remember as you develop all this ninja code the following adage…

 

image

MapIt system requirements and the Developer Hub.

image

I had a query today about the system requirements of MapIt. For those of you who don’t know what MapIt is yet I point towards the link above, it’s a great product for a certain market, especially those Microsoft partners who want to make use of their data within SQL Server 2008 with all of the spatial bells and whistles available. Jithen gives a good overview of what you get out of the box here including the system requirements on the ESRI wiki site.

Many people will hopefully be buying or evaluating MapIt over the next few months and should keep an eye on a number of the requirements. Specifically:

  • It requires IIS 6 or IIS7 so no XP for the server.
  • It runs on 32 and 64 but Vista, Windows Server 2003 and 2008, which is obvious given the above requirements but check the SP’s supported for each.
  • It requires Visual Studio 2008 SP1, Silverlight 2 or 3 and .NET 3.5 SP1. Mixing Silverlight versions, especially toolkits, on a single machine can be a world of hurt so tread carefully. Development can be done on Windows XP of course as long as the requirements for Visual Studio 2008 are met.
  • It supports all version of SQL Server Express 2008, including Express!

    See and hear it in action!

    Coupling MapIT with the new Microsoft WebsiteSpark initiative would allow companies of all sizes to build one powerful internet mapping solution and when you’re ready you can move up to a full ArcGIS Server package for a more comprehensive GI services and data management capability.

image For people in the UK were holding a seminar with Microsoft on the 6th October about ESRI and Microsoft integration, MapIt and use of Bing maps. Go over to the ESRI(UK) site here if you’re interested to register. I’ll be speaking about MapIt and integrating it with Enterprise data and Bing maps there, so hope you can come along.

For those people who can’t make it there are some excellent webinars and links given on the new ESRI(UK) Developer Hub. Especially Sarah and Mark on the ArcGIS WebAPI’s what a team!

image