Category Archives: Operations

Hardware and system administration related posts. Anything related to the Operations Working Group

New Australian caching server

Posted to the talk-au mailing list some news from Grant Slater and the sysadmin team:

Have you noticed faster tiles this week? Australia now has a map caching server located in Brisbane. The server is used to speed up the standard tile.osm.org “Mapnik” map style.

Browsing the map on http://www.openstreetmap.org/ should now be more responsive. This new server, named ‘bunyip’, first started providing tiles on Tuesday.

We thank Kris Amy for providing the server and hosting. Thank you, Kris!

If anyone experiences any issues or hiccups, please let me know.

Technical: 

Server specs are here

OpenStreetMap tile servers use GeoDNS (PowerDNS with Geo backend) to locate the closest tile server. The DNS regions / cache-server can be viewed here.

To check which server you are being directed to use the command:
nslookup tile.openstreetmap.org
or
dig tile.openstreetmap.org

Regards
Grant - Part of OSM sysadmin team


OpenStreetMap data license is ODbL

As of 9am (UK time), today, 12 September 2012, OpenStreetMap is now licensed under the Open Database Licence.

Thank you Open Data Commons for making legal tools available for the Open Data community.

[Update: Friday, 14 September 2012 approximately 18:30 UTC]
The XML-format Planet file took longer to generate than expected. It is now available in the new directory structure at http://planet.osm.org/planet/. You may experience slow downloads due to demand at this time.

Additional PBF-format files will follow as soon as they are completely generated.

Change to ODbL imminent

Hello OpenStreetMap-pers,

The change to ODbL is imminent. No, Really. We mean it.

At long last we are at the end of the license change process. After four years of consultation, debate, revision, improvement, revision, debate, improvement, implementation, coding and mapping, mapping, mapping, it comes down to this final step. And this final step is an easy one, because we have all pitched in to do the hard work in advance. The last step is so easy, it will be a picnic.

On Wednesday, 12 September 2012[1], generation of the next Planet file will begin. At that point, the API will switch over to ODbL and OpenStreetMap will be an ODbL-licensed Open Data project. API transactions and diffs consumed after that point will consist of ODbL-licensed OpenStreetMap data.

About thirty hours later, that newly-generated planet file will be available from planet.openstreetmap.org for you to consume with your renderers, routers, QA systems, convertors and re-imaginers.

You won’t want to mix ODbL diffs with old license planets or diffs. Purge and reload your systems with the ODbL planet. Then consume the ODbL diffs. Planet will have a new directory structure. We’re taking this opportunity to rationalize the layout of planet directories a bit. You should find it easier to understand afterwards. This also means that you won’t accidentally mix data of different licenses.

Mappers

Mappers shouldn’t see a difference and won’t have to change their mapping. Continue to improve OpenStreetMap by mapping from your own survey observations and using OSM-approved external sources. Never copy from other maps.

Data consumers

If you consume OpenStreetMap data and publish it, we have some guidance for you on the wiki. You’ll want to consider your obligations under the new license and then proceed to purge your old data and switch to the new. Many consumers, such as custom renderers, will only need to update their attribution of OpenStreetMap to the new simplified attribution.

Data consumers may time their upgrades to the new planet and diffs at their convenience.

Best regards and happy mapping,
The Communication Working Group

[1] in case of rain, we won’t cancel this picnic, just reschedule it for the subsequent Wednesday, 19 September 2012.

Image credit

Photo of cat in picnic basket is © Jacob Davies, licensed CC-By-SA.

Automated redactions complete

Over the past week the license change redaction bot has made automated redactions, sweeping across our entire worldwide dataset. The whole globe was covered yesterday. There has been substantial technical effort involved in developing and running the software to make those changes, and a fair degree of uncertainty about how long it would take, so this is a significant milestone. Congratulations and thanks are in order to all those who helped achieve it, and especially to Matt Amos, Andy Allan, Gnonthgol and MonkZ who carried out most of the coding work.

The data now in the live OpenStreetMap database is largely in a state where it can be declared ODbL licensed, however the license hasn’t changed yet. We will be posting a further update when this is imminent.

More than 99% of the data has been retained, and in most places, the difference is barely noticeable. There are, however, some areas of our map where the redaction was concentrated, in particular Poland and Australia. Though we would of course have preferred to retain this data, we do respect the original contributors’ decision, and we thank them for their past involvement in the project.

Fortunately the OSM community’s response in these areas has been magnificent and we believe we will be back to having a high-quality dataset in these areas in a short space of time. If, as an OSM mapper, you would like to help – by using aerial imagery and other sources, or ideally, from personal knowledge and survey – then please do get involved. Essentially we’re left with some new blank spots on the map, and can respond with the process we know and love. Use your favourite mapping techniques, go explore and fill in the blanks. (Note that you must not copy from CC-BY-SA datasets or map views based on the pre-redaction data; please treat this like you would any other incompatibly copyrighted map data.) We can begin this process in earnest now while final preparations for license changeover are made.

Redactions progressing well

In the past week the redaction bot has progressed well. After the intial Ireland test, it has proccessed the UK and is now finishing off the ‘Western Europe’ area. Spain, and Italy are fully proccessed, France is very nearly complete, and the bot is (at time of writing) getting to work on some densely mapped regions such Germany and the Netherlands. You can see its progress on the redaction bot progress map

As you’ll see, the internal checks of the bot and the API occasionally throw up errors which cause a region (1 degree square) not to be fully processed. The developers working on the bot managed to track some of these failures down to specific bugs, meanwhile others are caused by temporary glitches in the API. The bot has been re-run in several areas for this reason.

You’ll also notice many yellow “current” regions being processed. These are parallel instances of the bot processing code. Although we’re not really in a hurry, we have a big dataset to get through. Running in parallel like this is proving to be a little faster.

There is still time to perform remapping ahead of the bot reaching your part of the world, though you may wish to refrain from editing in a region where the bot is actually runnning, to avoid any unnecessary complications. If you’re in a green area there is now a new kind of remapping to do. This is easier and clearer in many ways. Head out and remap those patches where the bot has redacted data.

Remember the license has not changed yet. Even in areas where processing is complete and redactions have been made, the license remains the same until we declare otherwise.

Follow the rebuild mailing list for more details and discussion.

Licence redaction ready to begin

Hello all,

I’m pleased to announce that the licence change bot is ready to get underway.

Starting this week, we will be ‘redacting’ the contributions (less than 1%) from the live database that are not compatible with the new Contributor Terms and Open Database Licence (ODbL) – in other words, they will no longer be accessible. We are expecting to begin on Wednesday (11th July) assuming a couple of final setup details are completed by then.

The bot will run in the following order:

  1. Ireland
  2. UK
  3. Western Europe
  4. North America
  5. Australia
  6. rest of the world

Once it is complete, we will be ready to distribute data under the ODbL and we’ll advise of that with a separate announcement. The final pre-redaction dataset available under CC-BY-SA has now been generated at http://planet.openstreetmap.org/planet-120704.osm.bz2 . Where data has been redacted, any attempt to access it from the API or the site’s ‘browse’ pages will return a response to that effect.

Test runs have shown that the bot is functioning as we want it to, but we will of course be monitoring its progress. We are currently expecting it to take in the order of one month to complete; given the many variables I’m afraid we can’t give a more precise steer yet, but we’ll aim to keep everyone updated as it runs (via the announce@ and talk@ lists).

There will be no API outage and no other interruption to editing. When the bot is running in your area, please do save your edits frequently to minimise the likelihood of conflict.

(Separate messages are going to talk-ie@ and talk-gb@ as the first two areas to be affected. Please do forward and translate this for your local mailing lists.)

As you know we were expecting this to start just after 1st April and the complexity of the task incurred the delay. Thank you all very much for your patience in waiting for it to get underway. Thank you especially to those who have contributed to the code, whether by patches, suggestions or just helping to firm up the workings.

Posted to the mailing list by Richard for the OSMF board

License change still ongoing

Three weeks ago we mentioned we were still perfecting the ‘redaction bot’. The piece of code that goes through and redacts (removes/hides) any data that isn’t compatible with the new licence. Much to our dislike, it will take more time to get the bot working perfectly. Good news: the bot is now passing more tests than ever; bad news: still not all. Several people are working on this to make it work error-free.

Once these system tests are passing, live data testing will be conducted against a test server that is already configured and waiting. Subject to a successful test, a test of an isolated portion of the live database will be processed, most likely for the island of Ireland. If this goes successfully, the rest of the data will be processed.

On a positive note: in the last few weeks we’ve also managed to get agreement on several contributions to keep them in our database. We would like to thank all people who helped us make that happen.

We’ll give you another update next week.

Thank you all for your patience.

License change update: getting it right

With the new server successfully installed by our sysadmin team, we’re now onto the second part of our migration – the data ‘redaction’ work required to move to the Open Database License. We promised our first progress report next week, but lots of people have been asking, so here’s an update four days early.

The code changes to the OpenStreetMap API have been completed and successfully reviewed. openstreetmap.org is therefore ready to distribute the new data. (Thanks to Matt Amos for the code and Tom Hughes for the review work.)

The next part is the ‘redaction bot’. This is the piece of code that, for an area of OpenStreetMap data, goes through and redacts (removes/hides) any data that isn’t compatible with the new licence. This is the most crucial part of the whole process: we aim not to retain data whose creators haven’t given permission for it to be distributed under the Open Database License, and conversely, not to inadvertently delete anything from the vast majority which is compatible.

Since Wednesday we’ve been running tests against real-world data (thanks to Frederik Ramm for help with this). We’re not yet 100% happy with the results, so we are continuing to work on the code. As you would expect, we will not set the bot running until we are absolutely confident that it is producing accurate results. With the four-day Easter weekend just beginning, we currently expect that this will be next week. This puts us a few days behind schedule, but we owe it to our mappers to get this right.

If you’re a developer, you can help fix the currently failing tests: check out the code at https://github.com/zerebubuth/openstreetmap-license-change. If you’re a mapper, this gives you a few more days to get your area shipshape! And if you’re a data consumer, you can, of course, continue to use the data under our existing license, CC-BY-SA 2.0.

We’ll have a further update next week and, in any case, before the bot starts running.

API Read – Write returns

The sysadmin team completed the data base migration to the new DB server on schedule during the the morning of 04 April 2012. The API is now back to normal, Read – Write operation. Now the final steps of the license upgrade will proceed as outlined in the March – April service schedule announcement

Other items of possible interest as the license upgrade process proceeds:

osm.org map tile generation will recommence within the next few hours.

Replication diffs during the license upgrade period have started after community requests. These cc-by-sa data replication diffs are found in the redaction-period directory on planet. planet.openstreetmap.org/redaction-period. These diffs will only serve the period up until the switch to the new license. Mappers have requested these diffs for the redaction period. General consumers of OSM data may choose to consume these diffs or not at their discretion.

ODbL diffs will be located in another directory to be announced in future.

During the redaction period it is recommended that editors save their work early and often to reduce the chances of, and the complexity of conflicts with the back ground redaction process.

Bulk GPS point data


OpenStreetMap contributors have used track files from their GPSr devices for years while improving OSM data. They have shared those track files and the track points have been available to other mappers via editors and the web site. Now we are providing a way for you to get all of those points at once.

Announcing planet.osm.org/gps

This is the collected GPS point data from the first seven and a half years of OpenStreetMap. It is a very large collection of points and it is very raw data.

  • the compressed file is 7GBytes in size
  • uncompressed, the file is a 55GByte text file
  • the data consists of coordinate pairs only, with no track file or meta data
  • points were contributed by thousands of users
  • points were contributed as thousands of distinct track files
  • the data includes 2,770,233,904 points

Is this a big deal?

This might be the largest collection of Open Data GPS points published. Do you know of larger collections? Tell us in the comments.

Working with this file might not be your cup of tea. Over time, I expect that tools will emerge from the community to make this data easier to manage. For now, it is raw and it is extensive.

All of this data has been previously available to OpenStreetMap contributors in other forms, via editors and the web site. This file provides a new way to get the same data and to get all of it at once.

Example data

If you do decide to work with the file, this is the format that you can expect.

-900000000,1771882380
-900000000,1293757490
-891154290,1237501070
-877697750,1653442410
-871875000,1589069750
-871875000,1237507350
-843750000,1350007780
-843750000,1153132660
-843750000,1040632590
-843750000,1012507800
-843750000,1012507340
-824414060,1082922390
-815625000,1575007660
-805627440,1579614290
-805517580,1579284700
-805517580,1578845240
-804473020,1373773550
-787500000,1237507380
-787500000,1181257510
-787500000,1096882780
-787499970,1096882780
-778591613,1666901550
-778591613,1666898345
-778591384,1666911621

What format is that?

These are comma separated, raw lat / lon coordinates in a simple text format. To get the coordinates divide each number by 10**7. The points are sorted by location, starting in the far southeast of the globe (90 S,180 E) and moving northwest.

Thanks

Thanks as always to the hundreds of thousands of OpenStreetMap contributors over the seven-plus years of the project so far. Thanks to the syadmins for moving this data to a place where we can all access it.

This version of the GPS data file is CC-By-SA and published by OpenStreetMap and Contributors. The image in this article is a visualization of some of this point data in Europe. The image is licensed similarly and was created by Dave Stubbs.