Home > GIS > How do we measure success?

How do we measure success?

2009/03/08

When we talk about our web mapping sites, how do we measure success? If we go by journal articles, success is simply getting something up that works (based on a quick search) and addresses some need. But how successful are the actual sites themselves? That’s when we have to define our metrics. Some of the options we have are

  1. Page hits, raw or unique.
  2. Number of registered users.
  3. Number of returning users (optional registration).
  4. Number of submissions (“add your …” sort of site).
  5. Number of times a registered user submits data.
  6. How often the site gets submissions.
  7. How often the data the site provides is used.

Obviously, we have to consider the type of site as well: 

  1. Basic data access site. No registration; no user-submitted data.
  2. Mapping site. No registration; user-submitted data.
  3. Mapping site. Registration; user-submitted data.

For simple data access sites, page hits or queries is more than enough. Your site is a success if it reaches the target you set out for it, say “I’d be satisfied if 100 people a day visited the site”. But when we start talking about sites with some public participation component, this measure isn’t enough. We’ll start with the sites without user registrations. For those sites, a reasonable measure may well be number of submissions a day. Although I do think it’s important to capture the number of submissions from returning visitors. You want to have a site that people want to return to, even if it’s only to see what else has been added. So success here would be a combination of page hits (you want people to look at the maps) and number of submissions. And to be successful in the long term, you have to keep the number of submissions up to keep the page views up.

The last type of site creates the most problems for defining success. You still want a lot of page hits, but you also want a lot of registered users who will continue to submit data. And, if you can submit data without registering, then you want a lot of registered users submitting data and a lot of unregistered users submitting data. But the measure of success ultimately depends on your project. If you’re working on a watershed-scale with a thousand stakeholders, then maybe getting 25% of those stakeholders to participate is a success. Which brings up another point – perhaps participation is only important during a limited time period. Then your have two distinct measures of success: the participation during that time period and the page counts after.

So we have something akin to this

Success = (weight * page counts) + (weight * number of registered users) + (weight * number of submissions) + (weight * some other metric)

If Success approaches 1 (weights are proportions), then we’re doing pretty well. And if you track this over time, then you can get an idea of what’s affecting your rates. For natural hazards, we’d expect traffic to rise right before or after a major event (I’m not going to really pay attention to a drought site when I’m not being affected by drought, for example). You could also track what types of outreach are bringing you more traffic – being a part of some geospatial portal or holding public meetings, etc.

It’s asking a lot of the developers, really. To build a site that looks good, works well, is easy to use and understand, generates traffic from new and repeat visitors, and serves some purpose (hopefully for the greater good). But if you get most of that, or at least meet your point of “good enough”, you still have a PR problem. Getting a paper published in an academic journal probably won’t reach your target audience; getting published in a newspaper is probably a pipe dream. 

I think I might try to do a more thorough review of the literature in the next week or two to come up with some better answers. Even if that answer is “We don’t know”. A negative result is still a result and will hopefully lead to some discussion of the topic.

Advertisements
Categories: GIS Tags: ,
%d bloggers like this: