Watermarking, WMS and maybe other things beginning with W.

I should caveat this post with the often used phrase, ‘don’t try this at home kids’. When tinkering with the guts of any system and modifying the information being sent to and from a service by ‘hacking’ into the request pipeline of a message your opening up a whole can of performance and stability worms that need a great deal of testing under load to understand the direct effect on the scalability of any site.

A Simple Questionimage

This post is based upon a question I had with a customer at our recent DeveloperHub conference in Birmingham. He asked how it would be possible to watermark an image that had been served from a request to a WMS service. Ed has given an excellent overview about why a customer might want to watermark their images here and some methods to do it. For this query though there was a need to do the watermarking at the server level and not the client, as you don’t want to restrict access to only those systems that have been modified to adopt whatever watermark solution you have adopted.

With WMS you also to need to make sure you don’t force a client process a response that is not compliant with the OGC WMS specification. So you end up with the need to do some invisible modification of the request or response in order to handle the addition of a watermark without any client realising anything has happened.

The WMS Request / Response Cycle

It’s worth taking a brief aside here to look at the WMS request / response cycle. WMS Services can be simply called from a URL as stated in the OGC specification (here) using a HTTP GET. Depending upon the type of operation you are trying to perform the parameters for the URL will vary. In our case we are most interested in the operation that requests maps, unsurprisingly enough the GetMap request. This uses parameters to control the location of the area to return a map of, the layers to be displayed and the format of the image to be returned.

Once the request has been processed the response is in the format of a image is returned as binary to the client. In a web browser that gets placed into an image for display. It’s this binary image that we are able to edit as it goes through to watermark.

Why not burn it in? image

One question that needs to be answered is that of why you don’t just create a tile cache with an image already burnt in. This would be the most ‘performant’ solution as it front loads any of the processing away from the actual request by the user, this increases the response time but leaves a cache that can possibly only be used for one task. Indeed with more than one client requiring more than one type of copyright notice, or image overlay, then each would possibly need their own set of tiles, or own service.

Alternatively, you could have a map service with a layer which contains any of the watermarking details. You can search the WMS request string for the inclusion of this layer, if it’s not there then you can always add it later. This is fine, but it means you are actually messing with the request that’s being made by the client, which could possibly cause for bugs to be introduced into any application making the requests.

A more flexible solution, albeit possibly less performing, would be to handle the addition of any information over the top of the image at a stage of the request where it can be applied post the actual creation of any map. In terms of ArcGIS this would be after the request has come in, been processed by the map service and then return the clear image, as a binary image object.

A Pipeline Solution

The last method was the answer I actually gave at the event. This was to intercept the response to the WMS service and stamp the returned image with the required watermark, either textural or image based. But how to achieve this, the serving of the image from ArcGIS is handled deep within the SOC process which was untouchable, what wasn’t untouchable was the request/response pipeline in the web server, in my case IIS. In the past this might have required the writing of some sort of ISAPI filter to hook into this pipeline, but since .NET came along it became possible to write a HTTPModule to do the same.

The HTTP Module allows you to hook into public events in the request / response pipeline. Specifically the BeginRequest and EndRequest events, which allow you to check the content of a request before it’s forwarded on to ArcGIS Server and process the returned image that is the result from ArcGIS server, before it’s returned to the client. This pipeline can be simply shown in the following diagram.


Bringing it all together

In order to get the application to run, and to be able to debug it (especially in IIS6 which can only work vsprojectwith files processed by the asp.net worker process) you need to create a handler that maps to the arcgisservices directory within your ArcGIS install (see why I say don’t do this at home!). The easy way of doing this is creating a visual studio project within that directory (as you an see in from the VS 2008 project to the right).

Once the solution is in the right place you can update the existing web.config within that directory. It will already contain the ESRI Handler and Module details that are needed for the operation of the ArcGIS server services, by placing an entry for a new module after the existing ones will allow you to hook into the pipeline before and after the ESRI modules (remember this could seriously damage your ArcGIS Server health, use with caution on a test machine before letting it anywhere near production). The entry would be similar to that given below.

wbcnfgOnce we have these elements in place we can add our class with the IHttpModule interface. You can see how to do this in the example at the MSDN site for the creation of a custom HTTP module.

Hooking into the ArcGIS Server Requestimage

In order to perform the watermarking task, its necessary to perform a number of steps, before and after th e request. Where we get involved is in the BeginRequest event handler, this gets fired once a request is made to ArcGIS. In any system it’s good to only do processing of requests when needed, therefore being able to test that a request to ArcGIS Server is for a WMS map is necessary. This can be done by converting the incoming request stream to a string and parsing that, code to be found here.

At this point our watermarking service could perform no end of housekeeping, checking the type of watermark to apply to a specific service or if indeed it is to be applied at all. At this point it also might be good to read any image to be applied to the response and add it into a cache layer (if we do this for lots of images we don’t want any disk access slowing us down more than once). We are now set to let the request filter down the stack to ArcGIS for processing and we can wait for the EndRequest event handler to fire and for use to get down and dirty with the WMS response.

In next week’s episode – Hooking into the ArcGIS Server Response

image It’s at this point I realise that I’ve written another 1000+ words on something that started as a simple question and that having to read much more in one go might cause you to slowly lose the will to live. In order to save you at this point I’ll save the next part, taking the response and applying the watermark till another post. Probably after the ESRI Developers Summit where I know doubt will be shown a better way of doing this.

PS: What no code?

So you might be thinking where my sample is, how you can get access to it. Well, whilst I’ve provided all the tools to write this application, they haven’t been tested especially for use at scale. Modifying the pipeline of the ArcGIS is not to be taken lightly. The amount of work to actually do this isn’t very hard, it’s almost all provided with samples from MSDN and like I did, I would start by reading the custom HTTPModule section on that site and good luck!

The Lure of Easy

imageThe other day I built a computer almost from scratch. I can admit it, I can nerd it with the best of them when pressed, ok I don’t even need to be pressed. I had a bunch of components lying around, a not too old processor, a bunch of fast RAM and a laptop hard drive all I needed was a case. That was easy to rectify as I’ve always fancied building a little PC and Shuttle do some excellent barebones machines. Now the premise of this post is not the coolness of my new computer (although it is quite nice) but the ease at which it took to build.

When I was Young.

the internet in the 1970's When I was young and the ‘internet was all fields’ I remember building many a machine, both in and out of work, I remember saving my cash for the components, carefully making sure I didn’t bend anything when I slotted processors into motherboards and affixed strange looking fans to the top. I remember screaming when one of the components didn’t work and whole machine failed to boot. I remember returning complete orders and vowing never to build another computer again. But, the lure is too much for something’s and time can heal all wounds, even those inflicted by bad memory modules.

I Haz PowrToolNow whilst I was away from the field of home brew machines a number of things have happened, component prices have reduced, hardware is much more modular and available, I have an electric screwdriver (my only power tool I might add) and I can buy ready small machines with integrated motherboards at every online store. Now what does this add up too? An ability to assemble a machine in under 30 minutes, from start to end. I was shocked, surely it must be harder than this and after a brief moment of screeching from the machine as I had forgotten to plug-in the graphics card power supply, I was up and running installing Ubuntu (it’s free damn you and until I know it’s stable I’m not putting Windows on it!) and hooking it up to the ‘interwebs’.

Now the question arises, why if it’s so easy, would I not recommend building all the machines I own, or use at work? I’d be able to save money and tinker with hardware, what’s not to like?

imageSo easy is good right?

If pushed I could probably build a wall, but would I want it to support my house, probably not until I’d had  a lot of time building walls, maybe not even until 10000 hours to become an expert has passed. It’s the same with my new PC, would I use it to store my families photos, no I use a RAID disk set for that and the cloud (hmm I do trust them right?) as I’m unsure that the machine I threw together would be able to stay working for a long time.  I find this to be the same in designing and developing applications.

Components and development tools and platforms have come a long way since the internet fields were paved over and with that have come rapid prototyping, development and easy deployment. It’s now possible with the use of wizards and samples to throw a demo together in a very short period of time, like the construction of one imageof these modern barebones PC’s. Lots of development is easy, but because you can throw something together it does not mean it will be robust and stable, because I was able to build one machine quickly it doesn’t mean I will have the same luck again, or that my machine, which it’s mismatch components will not let me down when I need it most, like watching Snog, Marry, Avoid on the iPlayer!.

It’s the same with code developed quickly, technical debit will often lead to decisions being made that could impact the delivery of a system down the line, be those due to difficulties in refactoring or failure to run performance tests on software during development. For demo purposes technical debit might not be important, the code might not need to ever see the light of day beyond the demo, although the consequences of showing functionality that might be hard to implement reliably might live to haunt any project in the future. Lobbing technology bombs between pre-sales and professional services is always something that should be avoided, for good profitability reasons.

The Cloud Lure.

The cloud is another case of easy, it sells itself as a way to remove yourself from the burden of machines, your application can scale so long as you have the money to pay for it. Again, like the 30 minute machine build or the quick copy and paste development job, nothing is as easy as it seems and even though the imagelure is there, careful planning still needs to be done in architecting any system especially for those cloud platforms server to emulate a real system. In a world where your application isn’t tied to a specific machine you need to be careful what you can trust, are you getting data from a machine that knows about your updates, or another machine that is just handling your request at that point in time? As your application scales to multiple worker or web processes in an environment like Azure or App Engine, how do make sure everything is tied together?

Understanding how applications run in the cloud will still be needed, in order to utilise existing or still eme
rging patterns of development, such as those in the O’Reilly Cloud Application Architectures book or being developed by Microsoft on their patterns and practices site for Azure. There is no magic going on here, fundamentally thread must be mapped to processors somewhere, hardware has to do some work and then notify other machines about what has gone on. How you handle this in any deployment and its efficiency will impact the performance of any system and solution.

image Deploying applications into the cloud will be as complex as deploying applications into any set of machines, the complexity might be more software focussed and rely less on the understanding of processor specs and more on the understanding of the best practices for writing scalable applications, such as these provided by Google for App Engine.

Easy come Easy Go.

imageWhen I heard David Chappell (the IT speaker and not the comedian) say the phrase ‘there is no lock in like cloud lock in’ I realised that whilst there is much promise of Cloud computing it still needs treated like any other system. Badly written and architected solutions will not magically perform in the cloud and will always cost you more in the end than those that are optimised for performance and tested for scalability.

The cloud allows us to abstract ourselves from some aspects of deployment, but at a cost of making the software we are to deploy possibly more complex. As tooling and patterns become set we will be able to benefit from the power offered to us by a service we can build and deploy within 30 minutes, just don’t bet your mortgage that it will be up in the morning just because it’s in the cloud.

Do you Cache?

Before you read on this isn’t a post devoted to image caching. This is a post about data caching in general imagewith image caching being an extreme form of data caching. It comes from a bit of work I did recently caching data from a tracking feed. It’s based around why you want to cache, what data you might need to cache and how you might cache (I used .NET but you can do it in all major web development languages). Caching has often been the premise of web sites that want to be, and I’m using a technical terms here, ‘screamingly fast’ and not ‘snail slow’.

Caching before caching was famous.

For many people developing with ArcGIS server caching means one thing, map caching. This has imagebecome the panacea for many scalability issues with applications. As if the maps are going to be the only slow part of your application. Whilst removing the bottlenecks of pretty maps away from your site is very important it often can form one part of your data caching strategy. There are often other areas that you might look at when deciding what to cache and when.

In web applications caching can occur in at a number of levels, specifically at the data level (caching data to avoid making costly requests) or the page output level (caching the output of web pages so they don’t have to be built in real time every time). Combining these areas can help an application scale but there are a number of considerations you should take into consideration before implementing any caching into any system.

Often with spatial systems, especially in the enterprise (within the firewall) the need for data to be current often outweighs the extra benefits that might be gained through getting a few more users onto the system. The difficulty is understanding with any application is how much it will scale and whether imagebuilding this stuff into the project is worth it, the problem is that usually when you realise that it might be useful it’s too late to refactor your application and you’re stuck in a world of application fail. Now it’s easy to crow at the big internet applications when you see they are struggling with the amount of users on the system, most of us though will never have the same architectural issues that plague Twitter, Flickr or Facebook, if you do I suggest having a read of the Art of Capacity Planning to understand how to monitor the load you’re going to be under and to mitigate appropriately!

For the rest of us it shouldn’t mean that the use of caching won’t help our applications, even under modest scale and if you’re a .NET developer there are easy ways to implement it within the system, either at the code level (for data caching) or the server level (for output caching).

So what does to cache mean?

The process of caching is basically the storage of the output of a request or set of processing; so that once it has been done it may be reused for a number of people. The basic premise being that the output imageof the work is valid for a number of users and any changes to the underlying data will not impact the usage of the information within the time period that the cache is valid. This means that you need to understand the nature of the data and how it is used within any workflow. Failure to do so can mean your users potentially seeing data that is temporally incorrect, something that wouldn’t possibly be out of place in a episode of Doctor Who. So the two main questions with caching are the concepts of what to cache and how long the data can be validly stored within the cache.

What to cache?

The question of what to cache is always going to be a tricky one that will be down to the application that imageis being developed, down to the need for currency of the data being used. If you think that data can be delayed 15, 30 or 60 minutes, or even longer, without affecting the users operation of the site then it’s probably ready for caching. This could save endless requests for the same data that in turn reduces the processing cycles on your servers, reducing the number of machines you need to use thereby reducing the cost of the implementation. Even in this world of unlimited cloud machines, there is still a cost associated with every processing cycle and every saved resource is saved money, again good in these times of thrift.

In terms of exactly what to cache these three areas given by Microsoft in their enterprise caching block information are a good start:

  • You must repeatedly access static data or data that rarely changes.
  • Data access is expensive in terms of creation, access, or transportation.
  • Data must always be available, even when the source, such as a server, is not available.
  • imageIn order to make your site as robust as possible you need to take a pessimistic view of software and servers (no comments please) and admit that at some time they will fail. If you want to protect yourself from such failure using a cache to store data and to make sure it never expires if you don’t get new information will allow your site to give the impression that it’s bullet-proof even though things are blowing up at the back end!

    How to cache?

    Microsoft make it amazingly straight forward to implement data caching, although as I said deciding what to cache and for
    how long is another matter. On the web there are ways of adding to the data cache of an application using the HttpRuntime.Cache property that can be used to get access to the applications cache. The report manager example given on this MSDN page gives a starting point for implementing a class that can control access to a piece of cached information.

    There are a number of choices that need to be made whilst using the cache object. Especially around the timing of objects, an absolute or sliding scale can be used to control how long an item stays in the cache. Long running static objects that don’t change can be kept alive using the sliding scale, as long as they are accessed within a time period they stay alive. Objects that need to be updated periodically can be made to expire at a certain time so as to make sure they are current to users but also to not require the server to process to much or request data from remote services too often. Further information about caching and ASP.NET can be found here.

    Timing the Updates

    Have a look at this picture. You can see the red line, which is the route the person is riding. You can see imagethe person, WTF? I hear you say… well maybe not. But it shows a good point about getting the timing right with any bit of cached data. Here we have two cached pieces of information. The position and the route. We get both in WGS 84 and want to project them into Web Mercator to lie over our map tiles.

    The point request can be performed very quickly but might not get updated very often; the route get’s updated even less but can take longer to project. We can see that if we update the position every 15 minutes, which is quiet timely enough when tracking someone on a bike, and the route every hour say, which again should be fine, then the application should be able to scale quiet well as the amount of requests we are actually making is quiet low. Why the problem?

    Well obviously there may be a point whereby the position is not updated in the cache between the time that the the route is updated and the position is updated. This might be because the feed you are using image(if it’s not your own) in turn is updated at different intervals. In the picture (at the start) we have the rider’s position slightly to the left of the end of the route as the caches are slightly out of sync. Five minutes later and the rider is back where they should be (and the corresponding rift in the space time continuum is closed, phew!), as shown in the picture to the right.

    This raises an interesting point about how to time your cache. Do you have one cache for everything; do you cache various pieces of data at different rates depending upon how often the information get’s updated?

    Do I need to care for my application?

    As with any architectural decision within your application the need to use caching or not should be down to the actual amount of users you are going to get and the complexity of service calls you’re going to make. imageOften though it’s the unknown unknowns that with affect the application over its lifecycle that will make the difference, will someone decide to roll your nice heavy weight intranet application out over the internet (it’s happened I’ve seen it, it wasn’t pretty)? Will your obscure application suddenly be tweeted about by Stephen Fry (he loves to watch websites die!)?

    There are many issues whereby having an understanding of the use of caching might be important, and in fact site saving! With the ease of implementation with today’s tools then there is little reason, apart from ignorance (usually mine…), whilst a certain level of caching can’t be built into even the most innocuous application.

    Cycling to the Ashes

    For me getting back into caching was all about developing a simple website for a charity bike ride for the ashes. Anyway whilst it might seem a different world from what we usually do at ESRI(UK) the actual scalability challenges that might be found in a consumer application that might be mentioned on a national radio or television show (or even by Mr Fry!) are the same in a large enterprise application and if designed correctly should be able to scale appropriately.

    If you’re interested the application can be seen here at CyclingToTheAshes or as a bigger version here MapsPerSecond. Whilst it’s a simple application, and for those who care it was built using ASP.NET and the ESRI JavaScript API using OpenStreetMap data, the challenges of scalability and robustness needed to be met due to the nature of how the site was being marketed. There were also long expensive calls to the REST services providing the location (from Sanoodi) and project the data (ArcGIS Server), both of which needed to be reduced where possible through caching!

    Anyway if you have read this far and are still awake (once again this post wasn’t meant to be so long!), please hop over to Oli’s site and go sponsor him, it’s a worthy cause.


    A rule of thumb.

    There has been a long standing rule of thumb when deciding how many instances to give a map service to give optimal performance. Finding this information has sometimes been hard although surprisingly when asked for this information the other day, and failing to find it, I decided to see if it was on the new resource centre. Fortunately the is a page on services performance.image


    Here it not only gives the ‘rule of thumb’ for the number of instances for a map service (2.5 * #CPUs) but also a whole series of information about the relative performance of each service type and the factors that will specifically effect the performance of any map service.

    With 9.3.1 it becomes a bit easier to automatically determine why a service might be slow either through using the new .MSD service type and the map services publishing toolbar or using the old school mxdperfstat script.

    The Perils of Synthetic Tools

    Of course any synthetic tool will only give you a level of guidance, any proof would have be got through actually performance testing any solution during development, preferably as early as possible. Such tests and examples are given in two recent ESRI whitepapers, the High-Capacity Map Services: A Use Case with CORINE Land-Cover Data and Best Practices for Creating an ArcGIS Server Web Mapping Application for Municipal/Local Government.

    Both documents cover the optimum use of data and their effect on how an application performs. The former in terms of a high scalability site but with information that can be applied to all sites, especially in terms of the recommendations about using file geodatabases for large performance gains. The latter document is important as it shows how a workflow can be mapped to making choices in implementation of an ArcGIS Server architecture, map and geoprocessing services for a medium sized authority.

    A Good Guide

    Guidance like that available in these two documents and on the Enterprise Resource Centre in general, whilst not indicative of how every site will perform, gives a good grounding in the pitfalls to avoid when translating user requirements to any specific solution architecture. With any performance and architecture though its important that you think of not only the performance now but also the performance implications of any site growing over time. Without any analysis of the capacity requirements of your site, you really don’t know how long your current performance will be applicable. It should be remembered though as is said so eloquently on this Ted Dziuba’s site ‘unless you know what you need to scale to, you can’t even begin to talk about scalability.’

    Understanding your current performance requirements and your short to medium term load requirements, and potential spike points, will mean that you can concentrate on worrying about the right parts of your application in terms of performance and stop worrying about those areas that might never become a problem. The book ‘The Art of Capacity Planning’ gives a good overview about how to tackle monitoring your sites performance over time, what to worry about, and when.