Over the past week the license change redaction bot has made automated redactions, sweeping across our entire worldwide dataset. The whole globe was covered yesterday. There has been substantial technical effort involved in developing and running the software to make those changes, and a fair degree of uncertainty about how long it would take, so this is a significant milestone. Congratulations and thanks are in order to all those who helped achieve it, and especially to Matt Amos, Andy Allan, Gnonthgol and MonkZ who carried out most of the coding work.
The data now in the live OpenStreetMap database is largely in a state where it can be declared ODbL licensed, however the license hasn’t changed yet. We will be posting a further update when this is imminent.
More than 99% of the data has been retained, and in most places, the difference is barely noticeable. There are, however, some areas of our map where the redaction was concentrated, in particular Poland and Australia. Though we would of course have preferred to retain this data, we do respect the original contributors’ decision, and we thank them for their past involvement in the project.
Fortunately the OSM community’s response in these areas has been magnificent and we believe we will be back to having a high-quality dataset in these areas in a short space of time. If, as an OSM mapper, you would like to help – by using aerial imagery and other sources, or ideally, from personal knowledge and survey – then please do get involved. Essentially we’re left with some new blank spots on the map, and can respond with the process we know and love. Use your favourite mapping techniques, go explore and fill in the blanks. (Note that you must not copy from CC-BY-SA datasets or map views based on the pre-redaction data; please treat this like you would any other incompatibly copyrighted map data.) We can begin this process in earnest now while final preparations for license changeover are made.
In the past week the redaction bot has progressed well. After the intial Ireland test, it has proccessed the UK and is now finishing off the ‘Western Europe’ area. Spain, and Italy are fully proccessed, France is very nearly complete, and the bot is (at time of writing) getting to work on some densely mapped regions such Germany and the Netherlands. You can see its progress on the redaction bot progress map
As you’ll see, the internal checks of the bot and the API occasionally throw up errors which cause a region (1 degree square) not to be fully processed. The developers working on the bot managed to track some of these failures down to specific bugs, meanwhile others are caused by temporary glitches in the API. The bot has been re-run in several areas for this reason.
You’ll also notice many yellow “current” regions being processed. These are parallel instances of the bot processing code. Although we’re not really in a hurry, we have a big dataset to get through. Running in parallel like this is proving to be a little faster.
There is still time to perform remapping ahead of the bot reaching your part of the world, though you may wish to refrain from editing in a region where the bot is actually runnning, to avoid any unnecessary complications. If you’re in a green area there is now a new kind of remapping to do. This is easier and clearer in many ways. Head out and remap those patches where the bot has redacted data.
Remember the license has not changed yet. Even in areas where processing is complete and redactions have been made, the license remains the same until we declare otherwise.
Follow the rebuild mailing list for more details and discussion.
I’m pleased to announce that the licence change bot is ready to get underway.
Starting this week, we will be ‘redacting’ the contributions (less than 1%) from the live database that are not compatible with the new Contributor Terms and Open Database Licence (ODbL) – in other words, they will no longer be accessible. We are expecting to begin on Wednesday (11th July) assuming a couple of final setup details are completed by then.
The bot will run in the following order:
- Western Europe
- North America
- rest of the world
Once it is complete, we will be ready to distribute data under the ODbL and we’ll advise of that with a separate announcement. The final pre-redaction dataset available under CC-BY-SA has now been generated at http://planet.openstreetmap.org/planet-120704.osm.bz2 . Where data has been redacted, any attempt to access it from the API or the site’s ‘browse’ pages will return a response to that effect.
Test runs have shown that the bot is functioning as we want it to, but we will of course be monitoring its progress. We are currently expecting it to take in the order of one month to complete; given the many variables I’m afraid we can’t give a more precise steer yet, but we’ll aim to keep everyone updated as it runs (via the announce@ and talk@ lists).
There will be no API outage and no other interruption to editing. When the bot is running in your area, please do save your edits frequently to minimise the likelihood of conflict.
(Separate messages are going to talk-ie@ and talk-gb@ as the first two areas to be affected. Please do forward and translate this for your local mailing lists.)
As you know we were expecting this to start just after 1st April and the complexity of the task incurred the delay. Thank you all very much for your patience in waiting for it to get underway. Thank you especially to those who have contributed to the code, whether by patches, suggestions or just helping to firm up the workings.
Posted to the mailing list by Richard for the OSMF board
Three weeks ago we mentioned we were still perfecting the ‘redaction bot’. The piece of code that goes through and redacts (removes/hides) any data that isn’t compatible with the new licence. Much to our dislike, it will take more time to get the bot working perfectly. Good news: the bot is now passing more tests than ever; bad news: still not all. Several people are working on this to make it work error-free.
Once these system tests are passing, live data testing will be conducted against a test server that is already configured and waiting. Subject to a successful test, a test of an isolated portion of the live database will be processed, most likely for the island of Ireland. If this goes successfully, the rest of the data will be processed.
On a positive note: in the last few weeks we’ve also managed to get agreement on several contributions to keep them in our database. We would like to thank all people who helped us make that happen.
We’ll give you another update next week.
Thank you all for your patience.
With the new server successfully installed by our sysadmin team, we’re now onto the second part of our migration – the data ‘redaction’ work required to move to the Open Database License. We promised our first progress report next week, but lots of people have been asking, so here’s an update four days early.
The code changes to the OpenStreetMap API have been completed and successfully reviewed. openstreetmap.org is therefore ready to distribute the new data. (Thanks to Matt Amos for the code and Tom Hughes for the review work.)
The next part is the ‘redaction bot’. This is the piece of code that, for an area of OpenStreetMap data, goes through and redacts (removes/hides) any data that isn’t compatible with the new licence. This is the most crucial part of the whole process: we aim not to retain data whose creators haven’t given permission for it to be distributed under the Open Database License, and conversely, not to inadvertently delete anything from the vast majority which is compatible.
Since Wednesday we’ve been running tests against real-world data (thanks to Frederik Ramm for help with this). We’re not yet 100% happy with the results, so we are continuing to work on the code. As you would expect, we will not set the bot running until we are absolutely confident that it is producing accurate results. With the four-day Easter weekend just beginning, we currently expect that this will be next week. This puts us a few days behind schedule, but we owe it to our mappers to get this right.
If you’re a developer, you can help fix the currently failing tests: check out the code at https://github.com/zerebubuth/openstreetmap-license-change. If you’re a mapper, this gives you a few more days to get your area shipshape! And if you’re a data consumer, you can, of course, continue to use the data under our existing license, CC-BY-SA 2.0.
We’ll have a further update next week and, in any case, before the bot starts running.
A message from Michael Collinson and the Licensing Working Group
We ask OSM mappers to check their local mapping areas, try and contact anyone who has not decided about re-licensing and then do as much remapping as you can. Please do this in good faith and not just copy things created by contributors who have declined or undecided! You can use information contributed by continuing mappers, resources like Bing imagery and your own knowledge. If you are mapping in Poland or the Czech Republic, please also note that we are aware of special issues that makes this difficult for you right now.
We would like to switch over licenses in two months time. In order to do that, un-relicensable data has to be removed from the active database, though it will remain in publicly available archives. The latest indications are that 1.9% of ways will likely not survive the license change and a further 0.6% may have to be reverted back to an earlier state.
We would like to decrease that still further.
Your mapping area may be much worse or much better. You can use the license change view on OSMinspector and zoom into your area (Information on this tool). You can then click on Potlatch or JOSM icons to find out more and to do remapping. More info
We ask that you look at your areas and contact undecided mappers via the OpenStreetMap messaging system or directly if you know them. So that they do not get bothered too many times, please log who you contacted on this wiki page
This page also has example messages in different languages. Ask undecided mappers to please log into their account and accept even if they no longer wish to continue mapping as their previous contribution is important to you. Contact from someone mapping in the same area is very useful.”
The text of this posting can also be found at Remapping/Contact And Remap Campaign. Please feel to simplify it to help non-native English speakers and to translate it into other languages.
Original mailing list post
The Licensing Working Group has obtained explicit special permission to incorporate geographic datasets from data.gov.au in the OpenStreetMap project database published under any free and open license, including ODbL, provided that…
a) we provide primary attribution in a reasonable manner (currently the Attribution wiki page), and…
b) that we explicitly list there each dataset used to give useful feedback within the Australian government on how folks are using open data.
We have been careful to point out that (under ODbL) we are not asking folks who make visual maps from OpenStreetMap to provide secondary attribution to each and every contributor, so would not be in compliance with the CC-BY Australia 2.5 and 3.0 license their data is normally provided under. They have raised no objection to this.
The LWG would like to publicly thank data.gov.au both for providing open geographic data and for providing this permission.
(Message on the talk-au mailing list from Grant Slater of the License Working Group)
Just announced on legal-talk is that Phase 4 of the license change process is scheduled for this Sunday, 19 June 2011. During Phase 4, the 166,000+ contributors who have accepted CT/ODbL will be able to edit. That is, the 406 contributors who have declined CT/ODbL will not be able to edit after phase 4 has begun.
Those who have declined with their existing account may reconsider or may open a new account to continue editing. Those who have already accepted should be unaffected by the move to Phase 4.
If you have an unexpected problem with editing, please see #osm-dev on IRC.
 use your favorite IRC client for #osm-dev on irc.oftc.net, or the web client at http://irc.openstreetmap.org
Matt Amos, of the OpenStreetMap Foundation Technical Working Group, announced the publication of the list of accounts accepting ODbL/CT.
as part of the voluntary relicensing phase of the move to ODbL,
existing contributors have had the ability to voluntarily accept the
contributor terms. to help the community assess the impact of the
relicensing it was planned to make the information about which
accounts have agreed available. this will help with the evaluation of
the process and analysis of any consequent data loss, should the
switch be made. at the last LWG meeting, having been put to the board
for approval, it was decided to make this available , and i’m
pleased to announce that this list is now up  and being regularly
refreshed from the database every hour.
i look forward to seeing the new analyses, visualisations and tools
that can be built using this data.
Matt’s announcement on email@example.com mailing list can be found in the archives,