Author Archives: RichardF

OpenStreetMap opens up to more contributors with easy ‘add a note’ feature

OpenStreetMap, the collaborative map of the world often called “the Wikipedia of mapping”, has unveiled a new feature designed to make it even easier to contribute.

For the first time, anyone viewing the map can suggest a correction – such as a renamed business, a missing footpath, or a changed road layout – without learning to use map editor software.

notes screenshot

The corrections will be picked up by OpenStreetMap’s million-strong army of volunteer mappers. A local mapper can visit the location to check the suggested information, and then update the map.

OpenStreetMap’s data is used by some of the biggest mapping names on the web, including Foursquare, Craigslist, Wikipedia, Mapquest, and Apple’s iOS Maps app. It is also extensively used in humanitarian operations, most famously after the Haiti earthquake of 2010.

The new feature is part of a drive to open up OpenStreetMap contributions to more people – so that everyone can add their local knowledge to the map. Later this spring, an all-new HTML5 map editor will be coming to openstreetmap.org; its intuitive controls and clear walk-throughs will encourage more map users to take the plunge and become map editors.

“It’s the detailed local knowledge of our contributors that makes OpenStreetMap so much in demand from web and app developers,” explained Simon Poole, Chairman of the non-profit OpenStreetMap Foundation and a Switzerland-based mapper himself.

“By making it even easier to add to the map, we’re increasing the amount of on-the-ground knowledge we can capture – further distancing OSM from the traditional map data companies and their lack of local expertise.”

To add a note or correction to OpenStreetMap, find your area on openstreetmap.org and click ‘Add a note’ in the bottom right corner.

Background information

Notes are freeform natural text, read by other people, making this is a very simple way to communicate any problems we notice about the map without needing to get to grips with OpenStreetMap and its tagging system. Although this functionality has been available for a while on the third party site OpenStreetBugs, adding it to the main OpenStreetMap site brings it to a much greater audience.

Although it is not necessary to have an OSM account to create a note or comment on it, registered users can see all the notes they have created or commented on, and will receive automatic email notifications if the status of a note changes. OpenStreetMap’s enthusiastic community is its greatest strength, and these notifications facilitate an exchange between the original reporter and experienced mappers.

A full API is available for the notes so that third party sites or apps can query existing notes, create and comment on them – opening up possibilities for mobile bug reporting and fixing. The system is expressly designed for high-quality reports from individual users, not automated error checks or ‘bots’.

OpenStreetMap takes copyright very seriously. Registered users agree not to copy from other maps when signing up, and are requested to verify each suggested correction from non-registered users using ‘on the ground’ information. OpenStreetMap is a map for the people, by the people, and made with personal local knowledge.

Further details are available on the OpenStreetMap community wiki.

Introducing OpenStreetMap’s JavaScript editor – alpha version

Screen_shot_2012-12-22_at_10

Back in July I wrote about building a new, friendly map editor for OpenStreetMap in JavaScript. Since then, and in particular in the last two months, the project has come on in leaps and bounds – and today marks the first alpha release.

Codenamed iD, it aims to provide an easy-to-use, comfortable editing environment for the OpenStreetMap newcomer that’s nonetheless fully featured – you’ll be able to edit any OSM data with it. Clear modes such as “Move map” and “Add point” make it easy to get started without having to read swathes of instructions.

So – try it out! You can play with a working instance at geowiki.com/iD. It’s connected to a test database, not the main OpenStreetMap database, so don’t worry – you won’t break anything.

If you find a bug, use github to check it’s not already been reported, and file one if not.

The fast progress over the past few months is entirely down to the work of Tom MacWright and Saman Bemel Benrud, John Firebaugh, and others. Some of them have been funded to work on this by a Knight Foundation grant from the Knight News Challenge programme.

As ever with OpenStreetMap, the code is fully open source, and we’re looking forward to the community getting involved with helping to build the project. For the first time since OSM was founded in 2004, this will give us a full suite of editing tools – iD and mobile tools for new users, Potlatch 2 and Merkaartor for intermediate editing, and JOSM for power users – so that anyone can bring their local knowledge to the map, whatever their expertise.

Read more about the alpha release in Tom’s blog post.

Building a friendly editor for OpenStreetMap in JavaScript

It’s been an amazing year for OpenStreetMap and it continues to get better. New users from Foursquare to Apple to Wikipedia to Esri, TV and press coverage around the world, innovative releases from MapBox, MySociety, Skobbler, Stamen, and a thousand others. More people are using OSM data than ever before.

More people are adding to it, too. Our data has grown by 43% in just one year. Most of this is added with two tools: JOSM, the offline editor (Java), and Potlatch, the online one (Flash). Launched in 2006 and 2007 respectively, they’re mature, stable, and enjoyed by thousands of users – the tools that have built the best map of the world.

But how do we harness the knowledge of millions of casual users who are now seeing a ‘Data by OpenStreetMap’ credit for the first time? How do we get people editing on the move, using phones and tablets that can’t run Flash or Java? Can we build something that doesn’t have the power of JOSM or Potlatch, yet is easier for the first-time user to grasp?

Hell yes!

Id_prototype

iD is a new project to build a simple, friendly editor for OpenStreetMap – an editor designed entirely for the first-timer to quickly add their street or their local cafe. It’s not a new Potlatch or a new JOSM: it’s ‘an editor for the rest of us’, a stepping stone into OSM.

It’s written in JavaScript: things that required a plugin just a few years ago, modern browsers can do natively, and fast. And with powerful frameworks that even out the differences between browsers, JavaScript can be a joy to program in.

Development is at its really early stages. It doesn’t do much yet: no tagging, no save. It’s not very pretty. That’s where you come in. OSM needs the best, most intuitive map editor there is, and you can help. Coder? Designer? UX specialist? Get involved.

You can read the project introduction, fork the code on github, and read the live docs. Check out the code, play to your heart’s content – tear it up and make it drastically different if you want – and let’s build something to take the OSM editing experience up another notch.

License change update: getting it right

With the new server successfully installed by our sysadmin team, we’re now onto the second part of our migration – the data ‘redaction’ work required to move to the Open Database License. We promised our first progress report next week, but lots of people have been asking, so here’s an update four days early.

The code changes to the OpenStreetMap API have been completed and successfully reviewed. openstreetmap.org is therefore ready to distribute the new data. (Thanks to Matt Amos for the code and Tom Hughes for the review work.)

The next part is the ‘redaction bot’. This is the piece of code that, for an area of OpenStreetMap data, goes through and redacts (removes/hides) any data that isn’t compatible with the new licence. This is the most crucial part of the whole process: we aim not to retain data whose creators haven’t given permission for it to be distributed under the Open Database License, and conversely, not to inadvertently delete anything from the vast majority which is compatible.

Since Wednesday we’ve been running tests against real-world data (thanks to Frederik Ramm for help with this). We’re not yet 100% happy with the results, so we are continuing to work on the code. As you would expect, we will not set the bot running until we are absolutely confident that it is producing accurate results. With the four-day Easter weekend just beginning, we currently expect that this will be next week. This puts us a few days behind schedule, but we owe it to our mappers to get this right.

If you’re a developer, you can help fix the currently failing tests: check out the code at https://github.com/zerebubuth/openstreetmap-license-change. If you’re a mapper, this gives you a few more days to get your area shipshape! And if you’re a data consumer, you can, of course, continue to use the data under our existing license, CC-BY-SA 2.0.

We’ll have a further update next week and, in any case, before the bot starts running.

A weekend of Potlatch

P2_hack_weekend_small

We usually talk about data on this blog, but OpenStreetMap wouldn’t happen without code, too. Running the world’s biggest user-editable map (and, increasingly, one of the world’s biggest maps, full stop) requires thousands upon thousands of lines of code… from the low-level stuff that keeps the servers flying along, via the API and editing software that enables you to contribute, to the programs and stylesheets that turn all this raw data into pretty maps.

Much of this work happens in isolation, co-ordinated by IRC conversations or mailing lists. But we also have ‘hack weekends’, where developers – experienced and newcomers alike – come together to share knowledge and bounce ideas off each other.

Last weekend saw a major hack weekend in Toronto, attracting developers from the US, Britain, and the Netherlands as well as Canada. This weekend saw a rather smaller gathering in Charlbury, a tiny town in the Cotswolds, England, which coincidentally is home to the lead developers of both Potlatch (our online editing software) and Mapnik (the ‘renderer’ which turns OSM data into map images).

The focus for this weekend was Potlatch, with a vast list of improvements undertaken over the two days. The theme was “little things that mean a lot”, so when the new version goes live soon, you’ll notice quicker loading, neater appearance, more reliable operation, and so on. (Those of a technical bent can see the long list of code changes.)

OpenStreetMap’s users are hungry for new features and our existing developers are run off their feet keeping up. So if you have technical skills, come and join in. Whether your knowledge is Ruby, JavaScript or design (the openstreetmap.org website), ActionScript (Potlatch), C++ (various site components) – or whatever it might be – we’d love to have you aboard. Check out the mailing lists (osm.org site, Potlatch, full list) and the #osm-dev IRC channel, and find out how you can get involved.

Introducing a new OSM editor… Potlatch 2

OpenStreetMap users will know all about Potlatch, the online editor that appears when you click the ‘Edit’ tab on the site. Well, there’s a whole new version coming soon!

Potlatch 2 is a complete rewrite still with the same principle in mind: an editor which hits the right balance between speed, ease-of-use, and flexibility. It’s under very active development at the moment and I’ll include a link at the end of this post where you can have a look.

But there are four big new features – and one behind-the-scenes change – to tell you about first.

New feature – friendly tagging system

Potlatch 2 has a friendly, intuitive tagging system. The mapper can use graphical menus, dedicated fields, and icons to get the tagging just right – without the need to remember tag names and values.

For example, you can choose highway types from a set of icons, then add a speed limit by selecting the appropriate restriction sign.

Potlatch 2 tag editor

All this is fully customisable using a straightforward presets file. Using this, you can create your own favourite tag combinations.

New feature – WYSIWYG rendering

Potlatch 2 has an all-new rendering engine far in advance of the current one.

With road names, patterned fills, rotated icons, and much more, the editing experience can be like working live on the familiar Mapnik rendering, the cyclemap, Osmarender, or anything you like -making it much more approachable for the beginner.

The Halcyon renderer used in Potlatch 2

Just like the tagging, the rendering is easy to customise. It uses a special form of CSS, called MapCSS, which lets you create wonderful-looking maps with just a few lines of text. The tagging and rendering together make Potlatch 2 ideal for ‘vertical’ mapping applications, such as a cycle-specific editor or a building/addressing editor. Stylesheets aren’t just about making the map look pretty: you can create stylesheets to help your mapping, such as one that highlights roads without names.

The rendering engine (Halcyon) is available as a compact (<100k) standalone component which you can embed in webpages, so your custom maps can be used outside Potlatch 2.

New feature – Beginners’ Guide

You couldn’t write instructions for Potlatch without writing instructions for OSM. The new Potlatch user needs to know about tagging, surveying, and copyright – but they’re certainly not Potlatch-specific.

So Potlatch 2 will have an accompanying ‘OSM Guide’, explaining the basics with friendly, illustrated text. It will be concise, focused and clear.

New feature – vector background layer

Mappers are working more and more with imports. But the approach until now has been to import data directly into the map – and many people have pointed out the problems this can lead to.

Potlatch 2 will support vector background layers. You can load OSM-formatted data from servers or files, and work on bringing it into the map the way you want, at your own pace.

Because this integrates fully with MapCSS stylesheets, you can choose to temporarily hide background data, or show (say) only footpaths… whatever you like.

Fully rewritten in ActionScript 3

Potlatch 2 is written in ActionScript 3, a Java-like language with an open source compiler and full docs available online. The Potlatch 2 source comes with instructions on getting started and is, of course, permissively licensed under the WTFPL.

Potlatch 2 thus far has been written by Dave Stubbs and Richard Fairhurst (me). But we would love to see more people hacking on the source. There’s a potlatch-dev mailing list especially for this.

Playing with Potlatch

So where are we up to right now, and how long do you have to wait?

The tagging system, rendering engine, geometry editing, and server communication are all up and running – the core of the editor, and the real hard work.

Some other features, like Yahoo and tiled backgrounds, are finished but not currently exposed through the editor: they’ll be along shortly. Others, such as GPS track support, the Beginners’ Guide and the vector background layer are not coded yet but are intended for the initial release.

Potlatch 1 has some three years of development behind it, of course, and much of this feature set has not yet been ported to Potlatch 2. There’ll be countless little UI tweaks (no keyboard shortcuts yet, for example!); and as you’d expect for an in-development version, performance can sometimes be sluggish and there’s a lot of optimisations we’d like to do.

But with work progressing so fast, this seemed a great time to talk about it. Both the tagging system and the renderer are enormously flexible and we’d like to see people hacking on them as soon as possible.

So how about some links? You can find a read-only running version at:
http://www.geowiki.com/potlatch2/

or play with the renderer alone at:
http://www.geowiki.com/halcyon/

Should you want to try a particular area, just put the lat and lon in the URL like this:
http://www.geowiki.com/potlatch2/?lat=52.2&lon=0.1
http://www.geowiki.com/halcyon/?lat=52.2&lon=0.1

and the source is at:
http://trac.openstreetmap.org/browser/applications/editors/potlatch2

and you can read MapCSS documentation at:
http://wiki.openstreetmap.org/wiki/MapCSS

Have a play, let us know what you think, and grab the source!

Google’s Data Liberation Front and aerial imagery

Data Liberation Front

Google have a couple of really enlightened guys called the “Data Liberation Front“. Their role is to make it easy for people to get their data out of Google – rather than it being locked in.

Usually, people are locked in by the lack of an export feature, or an obscure file format. In mapping, people are locked in by licences.

In Google Maps’ case, you can create your own work by tracing over aerial imagery. But you can’t use this work elsewhere, because of the licences and terms of use. (The phrase “derived work” usually crops up around now.)

At the Society of Cartographers’ Summer School last week, there was a great workshop on making a mashup from Google imagery – but when the question arose about taking the data out of Google and using it elsewhere… well, then things got a little hazy.

Google could fix this by saying that tracing from their imagery is ok – just like Yahoo have done. Or, alternatively, they could give us a clear “no”. Right now no-one really knows where they stand.

The Data Liberation Front ask people to vote for their favourite suggestions. We’ve added this as a suggestion and, at the time of writing, it has 585 votes – over four times the next most popular.

You can vote by going to http://url.ie/2ero. OpenStreetMap has shown the way in liberating geodata – maybe it’s time that some of these other guys caught up.

London to Brighton

A bike, yesterday

Everyone has their own reasons for enjoying OpenStreetMap, but for me, cycling is the “killer app” – in that OSM gives you the best cycling maps in the world (on the web and on your GPS), and mapping is also a great excuse to get out there and cycle.

This weekend, four intrepid OSMers are taking it a stage further by entering an OpenStreetMap team in the famous London to Brighton Bike Ride. Andy “Blackadder” Robinson, Etienne Cherdlu, Simon “Welshie” Hewison and Graham Seaman together form the OSM team. Another intrepid OSMer and keen cyclist, Gregory Williams, is also doing the ride though not officially as part of the team.

It’s a great fundraising event for the British Heart Foundation, and our riders are also seeking a little extra sponsorship to fund the printing of flyers explaining OSM – which can then be given out to other cyclists on the route. And with 27,000 other cyclists taking part, there’s plenty of opportunity for publicity.

Of course, the main route is already mapped… but rumour has it that our riders might be tempted to detour “off piste” to two unmapped villages, Crawley Down and Ditchling, if enough sponsorship comes in.

Find out more about the ride, and pledge your support, on the wiki page.

OSM Super-Strength Export

One of OpenStreetMap’s greatest advantages is that we don’t just give you a beautiful draggable map – we give you the data, so you can do what you like with it. Well, this weekend, that just got a whole lot easier.

OpenStreetMap now has an ‘Export’ tab, joining ‘View’ and ‘Edit’ at the top of the screen. It gives you an instant way to get the map data in a format you want.

Want a static map for your blog, without having to spend hours fiddling with JavaScript? No problem – just export in PNG or JPEG. Want a map for a book? PDF or SVG are the perfect formats – fully vectorised, so they look smooth on high-resolution printers at any scale, and are easy to restyle or edit. Want to play with the raw data? Get it in our easy-to-parse OpenStreetMap XML format. Here’s an example of a simple PNG streetmap generated in just two clicks.

And this is just the start – our mailing lists are already buzzing with possibilities for new formats, such as Adobe Illustrator for cartographers, or shapefiles for GIS professionals.

With this new feature, the difference between OpenStreetMap and the “corporate” mapping sites becomes a whole load clearer. Other mapping sites’ agreements with their data providers (such as Navteq, TeleAtlas or national mapping agencies) simply wouldn’t allow them to give the data out like this. With OSM, we actively encourage it!

The work behind this was done by Tom Hughes, winning OSM’s coveted Lolcat of Awesomeness developer award for the fifth time.

The licence: where we are, where we’re going

Ever since OpenStreetMap was founded three years ago, the licence has been one of the most debated aspects of the project.

At present, every OSM contributor agrees that their contributions can be used under the Creative Commons Attribution/ShareAlike licence, version 2 (CC-BY-SA 2.0, for short). This means:

  • Anyone can copy OSM data.
  • But if you incorporate it into something else, that “something else” also has to be copiable under the same terms and conditions (ShareAlike).
  • When you copy it, you have to give credit to the copyright owner (Attribution).

It seems uncontroversial on first glance. But, as ever with things legal, the devil is in the detail. Do you have to give credit to ten thousand different mappers named individually, or just to “OpenStreetMap”? Does “incorporating it into something else” include adding extra layers of info over the top, as in a map mashup?

These, and similar questions, have kept the subscribers to our legal-talk mailing list happily arguing for over a year now. Many of the problems arise because Creative Commons (the “CC” of our licence), as the name suggests, is largely concerned with “creative works” – music, literature, art, and so on. OpenStreetMap, on the other hand, produces data: a factual, uncontroversial recording of the world around us.

When OSM started, we were on our own. No other website had a significant corpus of user-generated, publicly-licensed data. Few had open data of any sort. So it’s not surprising that no obvious licence was available.

Today, the situation is a little brighter. Open data is recognised as a fast-growing movement – and fortunately, this means that those with talents in drafting licences are applying their knowledge to the same problems we’ve run up against.

The situation today

As the elected OpenStreetMap Foundation, we’ve been discussing this in our board meetings for the past few months.

We are aware there are serious risks in continuing as we are. Partly this is out of a desire to do better: if a substantial number of people are unhappy with the licence, we want to see if we can find a solution that also satisfies those who are happy with the status quo.

Most importantly, though, there is a strong body of legal opinion that our existing licence is not valid (for our purposes) in most of the world. Creative Commons bases its licences on copyright. In Europe, however, geographical data is principally protected by database right, and in the States, the only available protection is contractual. OSM data is potentially in a curious unlicensed limbo at the moment, which will not protect us if a major geodata company, for example, decides to take our data without respecting the intent of the licence. (See here for a EU angle, here for a US angle; for Creative Commons’ position, see the third paragraph here, and here at §5.2.)

As the Foundation, we do not control the project, but we look out for its best interests. So we would be negligent if we did not look for alternatives.

Finding a way forward

So here are the presumptions that we have brought to our work:

  1. We need to consider alternatives to CC-BY-SA 2.0 for future licensing of OSM data.
  2. At heart, to gain widespread community acceptance, the licence needs to give our database the same three basic licensing elements (freely copiable; share-alike; attribution required) as it has at present.
  3. It must acknowledge the reality that we are one database with many contributors.

Of course, it goes without saying that any recommendation we make must have the best interests of the project and the community at heart – however hard that might be for any of us who have our own strongly-held views on the matter.

The Creative Commons approach

Through its Science Commons initiative, Creative Commons has recently published a “protocol” on open access data. Its primary focus is scientific data.

Its approach is unusual. It mandates that all such data be placed in the public domain, with no share-alike or attribution clause. Instead, these requirements should be expressed as a “non-legally binding set of citation norms”. In other words – the data provider can ask that those using the data share-alike and give attribution, but cannot insist on it.

Why is this the case? In the protocol, Creative Commons restates the problem that copyright simply does not apply universally to factual information. “Many users choose to apply common-use licenses such as the GPL and CC in order to declare their intent [...] But a user would be able to extract the entire contents (to the extent those contents are uncopyrightable factual content) and republish those contents without observing the copyleft or share-alike terms. The data provider, based on our research, is likely to feel ‘tricked’ by this outcome. That is not a desired result.”

This is an accurate summary of the situation. However, their solution – putting all the data into the public domain – is unpalatable to some OSM contributors. Here’s why.

In the science world, for which this protocol is intended, citation is an everyday part of life. Learned articles will always cite those whose work they are building on, and will be disregarded by peers if they don’t. When Science Commons speak of a “non-legally binding set of citation norms”, they can do so in the expectation that these norms will be respected. SC’s John Willbanks says as much in this interesting post: “It seemed we had to think about taking all these social goals and moving them outside the legal world, and into the world that scientists controlled – norms.”

That isn’t necessarily the case with geographical data. If a big mapping company like TeleAtlas or Navteq were to use OSM contributors’ work in their data, there’s no commercial imperative for them to credit us – let alone to make the rest of their data available on our terms. Our work will be used outside the world that we control.

Other OSM contributors say “so what?”, reasoning that the low cost, high quality and regular updates of free data will always win out over the commercial product. But even those with such views (among whom I personally count myself) do acknowledge that they’re not held universally within the community, making a licence on those terms unlikely to be adopted by all.

The alternative

So if Creative Commons’ existing licence may not be valid, and their new licence is unlikely to be palatable to many OSM contributors, where do we go from here?

The most promising answer, so far, is provided by the Open Data Commons project. The ODC is principally the work of two lawyers, Jordan Hatcher and Charlotte Waelde, specialists in intellectual property law as applied to electronic content and with significant geographical expertise. They have worked with Creative Commons (indeed, the open access data protocol is largely their work), and have been generously sponsored by scientific data firm Talis.

As well as the public domain dedication that they have provided for Creative Commons, they also drafted a pair of licences (the Open Data Commons Database Licence and the Open Data Commons Factual Info Licence) which together provide a set of provisions particularly suitable for OSM data. Key points include:

  • The database is protected by share-alike (ODCD 4.4).
  • Attribution to the project is required (ODCD 4.2b).
  • The licence works through copyright, database right and contract, making it applicable across different jurisdictions (ODCD preamble).
  • The extent of share-alike is expressly stated (ODCD 4.5).

In other words, the licence retains the fundamentals of OSM licensing – freely copiable, share-alike required, and attribution required; is unambiguously valid for data; and provides definite answers to those questions of interpretation (such as who to attribute and how far sharealike extends).

Where now?

Work on the ODC Database Licence and Factual Info Licence has unavoidably been slowed down by the Creative Commons public domain project. But both Jordan Hatcher and the project’s sponsors Talis have assured us that work is going ahead, and that a new revision will be available in the near future. As Jordan writes on his blog, “the Open Data Commons Database Licence and the Factual Information Licence are on hold during development of the new legal tools (the PDDL and Community Norms) but not forgotten.”

Paul Miller at Talis has already kindly given us his time to discuss the licence and we hope to meet with Jordan as the project moves on.

If the Foundation then believes that the licence is settled enough to be considered a viable option for OSM, we will recommend to the contributors – you – that we adopt this as the new licence going forward, for both existing and new data. We can’t change anything without you. You, not the Foundation, own the rights to your mapping.

The ideal situation would be that the ODC licences are suitable, and that not just the Foundation, but everyone involved in OSM agrees. If that’s the case, then we can publish the new terms without significant loss of existing data. If there are major objections, of course, we will either withdraw the proposal entirely, or weigh up the dangers of retaining the existing licence against the potential withdrawal of data. We are hopeful that the ODC licences are sufficiently in tune with the “spirit of OpenStreetMap” that this will not arise, but need to be alert to the possibility.

OSM isn’t just “open” in name – we believe in openly discussing the matters affecting the future direction of the project. If you have concerns or, conversely, would like to express your support, then please do discuss it on the mailing lists or forum; or, as at any time, contact any of the Foundation board (Corey Burger, Etienne Cherdlu, Steve Coast, Michael Collinson, Richard Fairhurst, Mikel Maron, Andy Robinson), or e-mail us collectively at team@osmfoundation.org.

Fixing “the licence problem” might be something that many have thought impossible, but we are hopeful that if we really want to resolve it, we can find a solution that will benefit OpenStreetMap and stand us in good stead for the years to come.

– Richard Fairhurst