Discussing Cricket with an Alaskan

Whilst this post might have more to do with a game played with leather and willow to a guy who lives in a part of the world that could only accommodate ice cricket for most of the year, well to be honest the UK isn’t much better for cricketing weather either but that’s beside the point, it’s actually about this years ESRI’s Developers Summit which I was lucky enough to attend this year in Palm Springs.

To ArcGIS 10 and beyond!

image The 2010 #DevSummit was a cornucopia of new technologies and online systems rolling out towards the release of ArcGIS 10. Even though I work for a distributor there was still lots of new stuff to see especially around the online world through use of Amazon for hosted ArcGIS Server (pricing to be confirmed), you can see an overview imagefrom the Business Partners Conference here and the development of the aforementioned arcgis.com (currently not accessible, but will be real soon and has the new ArcGIS Explorer Online) which you can see two videos of here (BPC) and here (DS).

The iPhone SDK got quite a lot of slide-time in both the plenary here as well as a number of sessions. I wanted to make time to have a look, I have an iPhone but no Mac, but there were so many other interesting sessions that I never had a chance, although I can now catch up online.

Online all the time.

Due to the nature of today’s internet the videos of the developers summit were online before my flight had dropped me back at Heathrow. So you too can follow mostly the same schedule that I followed, minus the face to face discussions with Development leads, Service Leads, Product Managers and other Distributors that make conferences like these invaluable and allow you to plan your development strategies for months to come. Some of the sessions I say were as follows.

  • The plenary : lots of videos here. Digs into all the different areas of the upcoming product.
  • Using the ArcGIS Server REST API’s : I can’t get enough about the REST API, the variety of new geometry functions provided
  • An Overview of the ArcGIS API for Microsoft Silverlight/WPF : I’ve been doing lots of Silverlight development recently and whilst many of the cool kids went to the Flex sessions, I checked that I wasn’t doing anything insane by going to some of the Silverlight ones.
  • Python Scripting for Map Automation in ArcGIS 10 : I like the way you can analyze map documents using arcpy! In fact I like most of the stuff you can do with arcpy, now if only someone would write me an avenue wrapper for it and I’d be good.
  • Advanced Map Caching Topics : Lot’s of tips about how to create efficient caches, how caches will change in ArcGIS 10 and the new mixed mode image format, watch and cache.

Unfortunately some of the interesting ones aren’t up there yet, especially the one from the prototype lab with Augmented reality and Microsoft surface demo’s which looked quite slick (and had one of the biggest overhead camera’s I had ever seen), David Chappell’s keynote going through Cloud Platforms (a must watch if you haven’t heard him go through this before) and finally there was another Silverlight session where they showed Windows Phone 7 integration with the Silverlight API. If I see them post them online I’ll update the list above.

So what’s with the title?image

Anyway the title of this piece is all about the benefits about attending a conference far away from home, the opportunity to talk to people face to face about technology, something that is not possible when your watching the talks over the internet. In this case, we were out for dinner one night, when a colleague turns up with some poor unsuspecting fellow he’s imagemet at the conference (@jdoneill from Girdwood, AK), as is the way with conversations with me the subject turned on to cricket and then the floodgates opened.

Although we never did discuss the merits of leg theory I did try and cover all the other aspects of the game, he even managed to seem interested for most of the conversation as well, he was a baseball fan so the compare / contrast, mutual appreciation of each others stat laden sport ensued. If you ever want to image read such a comparison from an English point of view then I’ll point you to this book.

As an aside we won’t mention about the guy he turned up with the next night. Remember, what happened in Palm Springs, stays in Palm Sp
rings, well apart from the technical information that is.

Watermarking, WMS and maybe other things beginning with W.

I should caveat this post with the often used phrase, ‘don’t try this at home kids’. When tinkering with the guts of any system and modifying the information being sent to and from a service by ‘hacking’ into the request pipeline of a message your opening up a whole can of performance and stability worms that need a great deal of testing under load to understand the direct effect on the scalability of any site.

A Simple Questionimage

This post is based upon a question I had with a customer at our recent DeveloperHub conference in Birmingham. He asked how it would be possible to watermark an image that had been served from a request to a WMS service. Ed has given an excellent overview about why a customer might want to watermark their images here and some methods to do it. For this query though there was a need to do the watermarking at the server level and not the client, as you don’t want to restrict access to only those systems that have been modified to adopt whatever watermark solution you have adopted.

With WMS you also to need to make sure you don’t force a client process a response that is not compliant with the OGC WMS specification. So you end up with the need to do some invisible modification of the request or response in order to handle the addition of a watermark without any client realising anything has happened.

The WMS Request / Response Cycle

It’s worth taking a brief aside here to look at the WMS request / response cycle. WMS Services can be simply called from a URL as stated in the OGC specification (here) using a HTTP GET. Depending upon the type of operation you are trying to perform the parameters for the URL will vary. In our case we are most interested in the operation that requests maps, unsurprisingly enough the GetMap request. This uses parameters to control the location of the area to return a map of, the layers to be displayed and the format of the image to be returned.

Once the request has been processed the response is in the format of a image is returned as binary to the client. In a web browser that gets placed into an image for display. It’s this binary image that we are able to edit as it goes through to watermark.

Why not burn it in? image

One question that needs to be answered is that of why you don’t just create a tile cache with an image already burnt in. This would be the most ‘performant’ solution as it front loads any of the processing away from the actual request by the user, this increases the response time but leaves a cache that can possibly only be used for one task. Indeed with more than one client requiring more than one type of copyright notice, or image overlay, then each would possibly need their own set of tiles, or own service.

Alternatively, you could have a map service with a layer which contains any of the watermarking details. You can search the WMS request string for the inclusion of this layer, if it’s not there then you can always add it later. This is fine, but it means you are actually messing with the request that’s being made by the client, which could possibly cause for bugs to be introduced into any application making the requests.

A more flexible solution, albeit possibly less performing, would be to handle the addition of any information over the top of the image at a stage of the request where it can be applied post the actual creation of any map. In terms of ArcGIS this would be after the request has come in, been processed by the map service and then return the clear image, as a binary image object.

A Pipeline Solution

The last method was the answer I actually gave at the event. This was to intercept the response to the WMS service and stamp the returned image with the required watermark, either textural or image based. But how to achieve this, the serving of the image from ArcGIS is handled deep within the SOC process which was untouchable, what wasn’t untouchable was the request/response pipeline in the web server, in my case IIS. In the past this might have required the writing of some sort of ISAPI filter to hook into this pipeline, but since .NET came along it became possible to write a HTTPModule to do the same.

The HTTP Module allows you to hook into public events in the request / response pipeline. Specifically the BeginRequest and EndRequest events, which allow you to check the content of a request before it’s forwarded on to ArcGIS Server and process the returned image that is the result from ArcGIS server, before it’s returned to the client. This pipeline can be simply shown in the following diagram.


Bringing it all together

In order to get the application to run, and to be able to debug it (especially in IIS6 which can only work vsprojectwith files processed by the asp.net worker process) you need to create a handler that maps to the arcgisservices directory within your ArcGIS install (see why I say don’t do this at home!). The easy way of doing this is creating a visual studio project within that directory (as you an see in from the VS 2008 project to the right).

Once the solution is in the right place you can update the existing web.config within that directory. It will already contain the ESRI Handler and Module details that are needed for the operation of the ArcGIS server services, by placing an entry for a new module after the existing ones will allow you to hook into the pipeline before and after the ESRI modules (remember this could seriously damage your ArcGIS Server health, use with caution on a test machine before letting it anywhere near production). The entry would be similar to that given below.

wbcnfgOnce we have these elements in place we can add our class with the IHttpModule interface. You can see how to do this in the example at the MSDN site for the creation of a custom HTTP module.

Hooking into the ArcGIS Server Requestimage

In order to perform the watermarking task, its necessary to perform a number of steps, before and after th e request. Where we get involved is in the BeginRequest event handler, this gets fired once a request is made to ArcGIS. In any system it’s good to only do processing of requests when needed, therefore being able to test that a request to ArcGIS Server is for a WMS map is necessary. This can be done by converting the incoming request stream to a string and parsing that, code to be found here.

At this point our watermarking service could perform no end of housekeeping, checking the type of watermark to apply to a specific service or if indeed it is to be applied at all. At this point it also might be good to read any image to be applied to the response and add it into a cache layer (if we do this for lots of images we don’t want any disk access slowing us down more than once). We are now set to let the request filter down the stack to ArcGIS for processing and we can wait for the EndRequest event handler to fire and for use to get down and dirty with the WMS response.

In next week’s episode – Hooking into the ArcGIS Server Response

image It’s at this point I realise that I’ve written another 1000+ words on something that started as a simple question and that having to read much more in one go might cause you to slowly lose the will to live. In order to save you at this point I’ll save the next part, taking the response and applying the watermark till another post. Probably after the ESRI Developers Summit where I know doubt will be shown a better way of doing this.

PS: What no code?

So you might be thinking where my sample is, how you can get access to it. Well, whilst I’ve provided all the tools to write this application, they haven’t been tested especially for use at scale. Modifying the pipeline of the ArcGIS is not to be taken lightly. The amount of work to actually do this isn’t very hard, it’s almost all provided with samples from MSDN and like I did, I would start by reading the custom HTTPModule section on that site and good luck!