Author Archives: Steve Coast

Google IP Vandalizing OpenStreetMap

Last week Mocality, a Kenyan business data startup, caught Google scraping their data and the post made it to boing boing. Mocality tracked this down with some analysis of their logs and a sting operation, even recording phone calls that Google staff made which contained false information. Google have apologized and the incident looked closed, at least from the outside.
 
Unfortunately we have to report something similar is still happening to us from the same source.
 
Preliminary results show users from Google IP address ranges in India deleting, moving and abusing OSM data including subtle edits like reversing one-way streets.

Two OpenStreetMap accounts have been vandalizing OSM in London, New York and elsewhere from Google’s IP address, the same address in India reported by Mocality.

The most obvious vandalism started around last Thursday last week from these particular users however it may take us some time to do a full analysis. In fact over the last year we have had over 102 thousand hits on OSM using at least 17 accounts from this Google IP.

These actions are somewhat baffling given our past good relationship with Google which has included donations and Summer of Code work. As a community we take the quality of our data extremely seriously and look forward to an explanation from Google and an undertaking to not allow this kind of thing to happen in the future.

Mikel Maron, OSMF board member, mikel@osmfoundation.org
Grant Slater, OSM Sysadmin, grant@osmfoundation.org
Steve Coast, OSM Founder, chairman@osmfoundation.org

—————–

Update 17th Jan 2012 5:30pm GMT
We’ve had many questions since this was posted and wanted to fill in some blanks.
 
Why was the post made? As an open community we respect the privacy of our members. We have to draw a line somewhere between open and closed communications not being available to community members. It was felt on balance that making the minimum facts public was the right thing to do.
 
What more details can we share? The source IP range: 74.125.63.* and two of the investigated accounts are: http://www.openstreetmap.org/user/kane123 (This user was blocked for a day on Friday… and they continued vandalising on Monday after being made aware of their bad edits Source: http://www.openstreetmap.org/user_blocks/79 ) and http://www.openstreetmap.org/user/sanganabongina 
 
Do you have an example of malicious data? Yes, here: http://www.openstreetmap.org/browse/changeset/10375538 (London), http://www.openstreetmap.org/browse/changeset/10375581 (New York City) where a user from that IP address modified one way streets [reversed and deleted]. Here: http://www.openstreetmap.org/browse/way/145893931/history where a user added a fake Olympic stadium.

Have we spoken to Google? We are talking to Google and sharing all the information we have.

Do we know if this was a coordinated activity? No. We simply know the IP address and the accounts and edit information, we’re not implying a grand conspiracy. The edits were made over many hours over multiple days – nothing that would happen as an accident by a new user.

When and how is vandalism escalated? – Our vandalism policy is here: http://wiki.openstreetmap.org/wiki/Vandalism

Was this all overblown? As a very diverse and large community of over 500,000 user accounts we have a wide set of opinions even amongst the key people running OSM. This is welcomed and every member of our community is free to speak about how they see things.

Hitting reset on talk-au

I’m speaking strictly personally here, posting to talk@ and opengeodata.

OSM often crosses bridges in it’s growth. Mostly they’re technical, like introducing color maps, rendering new things or speeding up the system. We have a much more ugly bridge to cross in front of us.

Would you want to be part of a community which includes people explicitly working to disrupt it, trolling it and breaking data? Would you want to be part of a community where people are literally scared for their jobs when thinking about helping run it?

Over the last few days there has been a bunch of discussion on talk-au which you can read in the archives, though for your own sanity you might want to skip it.

For the most part the posts revolve around the OSMF, the LWG and the license process. I considered my presence there over the last few days as both a last ditch attempt to salvage the data and more importantly the community that’s there. As RichardF pointed out, their license acceptance rate is about half what most EU communities have achieved. I would say that the people on that list feel disaffected with the process and their representation in it.

Despite multiple attempts at trying to have a reasonable dialog over both what happened and what we can do about it, mostly I’ve been met with extreme animosity.

Most of that comes from people either banned from the main lists, been deleted/blocked from OSM or been moderated or who have publicly stated they’re here to disrupt the project.

I’ve tried to get many people involved posting there in what I thought was a worthwhile effort, in effect to save that list. Almost everybody declined to do so. Only RichardF braved it and was met with a predictable response. Frederik has given up and from my reading of his email considers talk-au dead (I think you should make that email public). I find that understandable.

I’ve been trying to find someone to moderate the list along the Etiquette guidelines on the wiki. Mikel has given up, understandably, and he leads the main moderators. We found one native Australian to moderate but they backed out because they literally feared for their job safety, that the people who now inhabit the list would make life with their employer difficult. Thus, they declined to do so after initially accepting. I actually am convinced that was the right decision and the people on that list are capable of it.

I don’t think anyone I know in OSM would want to be part of a community like that. I think it’s a sad low point in what otherwise is a wonderful project to be involved in.

Let me be more clear, *I* don’t want to be part of a community that accepts this. Who in their right mind would want to be a part of a community run by people explicitly out to disrupt, fork and troll?

In the best traditions of open projects our ideas and code are Free. It’s not clear that our time and server resources should be. Unlike our ideas and code, they’re finite and open to abuse. Make no mistake that our time and resources are being used explicitly to destabilize the very project which provides them. Used by mostly anonymous or pseudonymous people who as I say have been kicked, banned or explicitly stated they want to destabilize OSM. This is not about censorship. If you read the lists, you’ll find we’ve made available repeatedly both the methods and the people to help resolve issues. These people are free to fork the project and the data, it’s all available for download. They have their own mailing lists. Are there genuine questions about license, it’s implementation and so on? Absolutely. But level-headed discussion is not welcome on talk-au for the most part. There are a few people who can discuss this stuff impersonally there but it’s a small part of the list.

Now – why are we at this point?

The OSMF and the working groups, the apparatus of how a chunk of this project is set up, are unable to deal with direct threats like this, even if it’s been going on for a year or more. One of the main forks of OSM (if you can call it main, it doesn’t yet display a map) is run by an ex-board member. When you have someone like that working together with those who’ve explicitly declared they want to disrupt OSM, it’s very hard for a young, open and democratic organization to deal with. For the most part we have no idea how many of these people are even real too, it’s been suggested that a few of the pseudonyms are in fact just one person creating them on the fly.

We simply don’t have the tools for it. Until last week we had no moderation at all, and that took many, many months (perhaps years) to set up. The board meets too infrequently to be able to respond to people explicitly working for its downfall, which perhaps is a little ironic. The working groups likewise I don’t think have the bandwidth as they currently operate. Generally in an otherwise do-ocracy there is a lack of people who feel they have the authority to take on a role like moderating. Even if they do, it’s an extremely thankless task that almost nobody will take on.

So – what do we do now?

Well to answer that I have to assume you agree with both the horrific tone on that list and that it should not be part of the community we represent. For that, you might be wanting references to some of the things I cite (like this http://lists.openstreetmap.org/pipermail/talk/2011-April/057947.html ) but I’ll allow others to do that exhaustively (Grant is usually good, hint hint).

I want to get back to mapping. I can only do that if we do something about these people on our lists.

I don’t want to contemplate ignoring the problem, which is one extreme end.

I don’t want to be a part of a community that accepts this, so leaving it as-is is not an option.

We’ve tried hard to find moderators and failed. If you want to volunteer and moderate under the Etiquette guidelines, this is the first option I would consider, but you will get a lot of flack. And a beer from me.
We can remove everyone from talk-au and start afresh. No pseudonyms, no license talk (would have to go to legal-talk) under the new list. This would hit reset but remove people who have legitimate concerns and those just trying to get on with mapping.

We can block the ‘main’ people. Then you have to draw the line somewhere between the good and the bad anonymous posters. I would suggest anyone who’s posted that they want to disrupt the project and anyone operating under a pseudonym.

We can place everyone under the emergency moderation flag and clear each post one by one, by moderator, by vote, I don’t care. I can log in and do that too.

Lots of people from talk@ could join talk-au@ and make it a nice place to be again, the way we took back legal-talk@ from the very same people.

Maybe you have a better option?

Either way, this is an ugly bridge to cross. We need to do something to make it clear this is not how things work in OSM. We need to make the message heard that this is not normal, this is not the reputation we want to be known by and we won’t let it be this way.

Steve

OSM mailing lists: time for a change?

OSM has had a good run with our mailing lists, especially if you grade that based on quantity. We have over 100 of them for Iranians to Australians, for importers to newbies.

The problem is that they’re an echo chamber. Every mail is sent to everyone. That means intelligent postings as well as complete garbage get the same precedence and everything is equally loud.

Unfortunately some of the main lists have become less than nice places to be. Low signal to noise and a scorched earth policy from the more flamey participants pushes people away.

Below: Newbies prepare to enter talk@openstreetmap.org

Alien-movie-spacesuits1

Let’s look at some solutions. One is to get rid of them. We spend too much time worrying about this crap, just kill talk@ and move to a new list discuss@ or something. Let that list have a clear and simple etiquette policy list. Put someone in charge of kick/banning people who don’t play nice. The problem here is there are very few people willing to take responsibility for being the moderator. It’s unpaid, nobody will be nice to you (except me, obviously) and it’s hard work.

Below: Mailing list infestation marines prepare to nuke the talk@ list from orbit, because it’s the only way to be sure.

Aliens-vs-predator-players-wanted-20090526000151286-0001

Another is that we let the talk@ list be. All the clueful people decamp to a new list with a nice policy as above. Stop linking to talk@ and leave the bozos there. Hopefully they don’t notice.

Below: a newbie after making a polite suggestion on legal-talk@

Aliens-half-synthetic1

We could all migrate (one imagines this follows a sea parting) to the forums. There we could be safe for awhile. Newbies are more at ease in forums. It’s easier to delete posts, they don’t have the same permanence that a mailing list does: Forever in the archive, and already sent to all participants.

None of these options are easy and we can just vote with our feet.

There is an interesting technical solution possible though. Take the forum mechanics to a mailing list. So instead of every post going to every participant just send it to participants with high karma. If they like the post (either explicitly by clicking/emailing something, or implicitly by not doing anything) then it goes to others on the list. That list can have override features so I can get sent everything in a thread or block certain people. I’ve actually started building something like this but it’s too big to fit in the margin.

Below: calm arrives at talk@ during a license discussion, unicorns descend from the skies and free healthcare for all

Seeking_utopia_11

Before utopia arrives, Steve’s top tips for the mailing lists are as follows

  • Stop asking permission on the lists, you will never get it
  • Don’t feed the trolls
  • Clueful people post more, and more cluefully to drown out said trolls
  • Someone should be appointed to moderate the lists and just warn, kick and then ban people who aren’t nice. Just use common sense to identify who they are

OpenStreetMap project sees serious decline, disbanding imminent

Negative-graph

The OSM Foundation today announced the imminent disbandment of the project due to a heavy loss of users, little uptake and not enough map data and community to sustain the project. Flocks of users abandoning OpenStreetMap for Google Map Maker and most formaly-supportive commercial entities now using ClosedStreetMap mean there are not enough donations to run the servers or hand out GPS units to the needy.

Mikel Maron, board member of the OSMF said today “We had no idea the project was in such a bad shape until it was discovered that one of our keenest users, RichardF, had in fact been paying hundreds of people on Mechanical Turk to enter reasonable looking data. We thought he liked living on a boat but that was just to keep costs down, so he could keep paying those turkers to map.”

MapQuest announced the closure of their well publicized ‘open’ project and renamed it the ‘MapQuest Closed’ project in response. Ant Pegg in an email stated “It took CM over 2 years to realise the project was worthless, it took us 1 year, hopefully the next sucker will only take 6 months.” In addition, MQ is closing it’s variety of open sites including open.mapquest.co.uk and instead replacing them under the new ‘Closed’ brand begining in the Cook Islands: closed.mapquest.co.ck.

A jubilent ReallyOpenStreetMap team have taken the loss in their stride and accepted new users to their ranks. A defiant member of the now-dorment legal-talk list known as “13gal 3ag13″ stated “HAHA WE WON!!!! NOW i CAN DeBATE TEH LICENSE OF FREEDZOMG” and asking all those leaving OSM to join ROSM, give up mapping and discuss the correct placement of commas in legal documents.

Steve Coast, founder of OSM, is to change roles at Microsoft and begin working on printer drivers for the upcoming Windows 8.

 

Nice simple animation of OSM growth

… and some basic stats ;

“A simple way of getting an idea of the coverage expansion trends is to look at the historic data. This article aims to provide visualizations for two elements: where was the coverage expansion happening and how did it break down on a weekly basis.

Beyonav has performed a statistical analysis on over 600GB compressed (10 TB uncompressed) weekly Planet-OSM database dump files since December 30th, 2009.”


http://www.beyonav.com/sites/beyonav/osmanalytics.aspx#