UX Review

Dear all

One of the clear pieces of feedback from all the talk about improving the OSM UX was “show us these users who really find it difficult to use OSM”. So, we’re going to do that. We have a rough plan of action below

Nate Bolt of the splendiferous boltpeters.com has volunteered to help OSM with a user interaction review. Nate, as part of Bolt | Peters, did exactly this work with wikipedia:


Where they basically screen recorded people trying to edit in wikipedia with the following goals:

• identify obstacles that novice users encounter in editing a Wikipedia article—including, but not limited to—adding personal content, fixing a typo, adding a reference, and contributing to discussion pages
• identify obstacles in creating a new article
• evaluate the self-sufficiency and legibility of help materials and documents found on Wikipedia.org
• evaluate how novice users interact with templates
• discover user experience patterns and issues that have not been previously identified.

Together with Mike Migurski, famed geohead, creator of walking papers and graphical butterfly at Stamen.com, we’ve come up with a rough plan to get feedback from new users to OSM. It’s a very similar approach to that taken by Bolt | Peters with wikipedia. We want to find out from the real newbies what the issues are and lay them out clearly. Nate is independent of OSM and will be able to present a cold hard look at what’s good and what we need to work on to improve things so we get more newbies contributing.

* A small piece of javascript from ethnio.com is put on the new user page in openstreetmap
* Once in some very small sample size (perhaps between 1 in 1,000 and 1 in 10,000 signups) a popup appears
* The popup says something like “Hi! We’d really like to know why you came to OSM” and they say simply why. This is open ended on purpose so we catch as many things as we can, not just what we’re looking for, but things we won’t expect.
* They’re offered to record a short (10 minute max) screencast of them trying to achieve whatever it is (like look at a map, find OSMers, add a PoI and so on)
* That screencast is analyzed in aggregate with many others by Bolt | Peters with all their expertise in doing this stuff, and they come back with a set of findings.

We’re looking at both http://www.usertesting.com/ and http://www.openhallway.com/ to do the recording. Both Mike and I will pay for it, and might solicit donations and stuff if it looks beyond our budget.

I can’t say this enough, and I always get responses from people who think that I just set something out in stone – so I’ll be super super super clear: NONE OF THIS IS SET IN STONE. We need your feedback on everything before we go ahead. I mean _everything_. So, please tell us what you think about it all. Here are some questions:

What should our goals be? (General UX? How good/bad signup is? How good/bad editing is? How is it finding info?)
How often should we ask a signup for feedback? (the more the better but we can only look at so many)
How can we include more crowd source feedback? (I think of asking random signups for feedback as crowdsourcing it)
What else should we think about?

Yours &c.


8 thoughts on “UX Review

  1. Andrew Harris

    UX testing is essential – I definitely support it and would chip in a few $ to see it done well. Nothing beats observing users doing their thing so the screencast tool sounds good if it captures users in their ‘natural environment’. Questionnaires and interviews can help tease out issues, but will provide skewed results if used in isolation.

  2. Russ

    Comment to Posterous writes: <br/> <br/>Clarkson University has a usability testing lab. Camera on the <br/>keyboard, camera on the face, screen, mouse, and keyboard recording. <br/>I’m asking around to see who controls it.

  3. Erik Johansson

    Just reading the <a href="http://usability.wikimedia.org/wiki/Usability_and_Experience_Study">Wikipedia Usability and experience study</a> you linked give us a good indication of what the results will be, that is; Overwhelmed, "clean slate problem" and OSM editing not being visual enough.I can’t say anything about usertesting just try it and we will see what happens. The open ended testing scenario seems like a bad idea, I think you specifically want to test some parts of the editing. But sure it would be nice to know what people want to do, before deciding what to test. So I think you need both.

  4. chillly

    Start by asking why people are there. Are they looking for something in particular (where a place is , what does OSM look like, how to get to somewhere)? Do they want to change something (change a name, want to add a road, add a pub, add a whole town)? Asking what they want out of OSM is as valuable as asking how easy is it to use. If you know why they are there, maybe you would tailor they way the site looks / works, maybe dynamically based on a choice of options.

  5. Gerv

    Great idea, Steve – go for it! Although it would be good to develop some use cases for the front page, because I think a lot of the dispute in the community is about who the front page should be aimed at. Exactly what it looks like and what functions it has might well flow from a decision on that point.Gerv

  6. Rhea

    I am Rhea S. I have visited your website and I would like to congratulate you on building such a valuable online resource. I am sure your visitors find your site as useful as I did.Have a great day.Thanks and regards,Rhea S.

  7. Rhea

    I am Rhea S. I have visited your website and I would like to congratulate you on building such a valuable online resource. I am sure your visitors find your site as useful as I did.Have a great day.Thanks and regards,Rhea S.

Comments are closed.