Placr News

History

Ten days ago I blogged on the strange case of the disappearing MDV MyTfL journey planner app, which appeared in the iTunes app store without fanfare (somewhat against TfL policy) and then disappeared again 24 hours later without explanation (pulled by TfL senior management). I described the affair as “a microcosm of political and economic conflicts to come around apps and data”, and so it has turned out with another tragic chapter in this story written in the last few days.

A week after the MyTfL app was withdrawn the open (but undocumented) TfL JourneyPlanner API was suddenly shut down, taking several apps with it, including two by Malcolm Barclay serving 100,000 users. The London DataStore was advised that this was a routine security procedure following attacks on TfL servers. It seems much more likely that this was revenge for the withdrawal of the app as I tweeted out last Thursday. The give-away was that not all the API calls were blocked: /lite, /imr and /bc still worked and only /user/XML_ was withdrawn, i.e. the part that gives back journey options in response to a query. If there had been a real security incident all of the service would have been taken offline. Wired have since picked up the story and have raised several interesting questions about the ownership of the API and data it serves.

Consequences

Although the consequences of this withdrawal were severe for current apps and development work e.g. OpenTfL, there were wider implications. By releasing transport data to the developer community and promoting the reuse of this data, TfL hope to benefit from the work of developers to produce value from the data. Hence the apps that have been released around the London Cycle Hire scheme e.g. London Cycle have meant that TfL have not needed to spend public money on writing their own. TfL simply cannot have this both ways… getting free apps and innovation around their services, but also issuing their own apps to achieve internal managerial agendas. If the climate for private sector innovation is clouded by TfL intervention with public funds then the developers will have to turn elsewhere as the risks will be too high. Ultimately, this could stop jobs being created in London if TfL’s strategy is not thought through.

Denouement

As it turns out, TfL have been forced to retreat (again) after concerted action by the London DataStore to take up the issue of the API withdrawal with TfL senior management. The API was restored yesterday and is working again normally. This episode has further raised the level of importance and visibility of the issue of data releases from TfL. Senior management have now seized back the initiative and are now promising to quickly produce a proper API for developers to use. I look forward to hearing more about this at tomorrow’s meeting of the Mayor’s Digital Advisory Board. Meanwhile, we at Placr will carry on with our investments in our transport API and hope to announce services for consumers and developers in the near future.

Jonathan Raper

The cover story

Yesterday morning a London travel planner app appeared in the iTunes app store called MyTfL. Nothing unusual there, there are already half a dozen available. However, MyTfL was different: it described itself as the ‘official’ Transport for London app, and it appeared to contain data feeds (e.g. live bus departures) that are not currently available to developers. This was evidently a significant development: so why was this it so quietly released with no fanfare?

On further examination the mystery deepened. The publisher was actually Mentz Datenverarbeitung Gmbh, who are a German system integrator working for TfL Group Marketing, who run the TfL Journey Planner. And the app used the full TfL branding, which is strictly controlled (see here for details). So what was this new app and how did it come to be released with hitherto unreleased data?

The implications

The short answer is that we don’t know what is going on here on the outside. Or even on the outside of the inside for those of us on the Mayor’s Digital Advisory Board. However, we can speculate on our slender knowledge. The key guess must be that all data that developers would like to access to make new apps is already polished and ready inside TfL. When we are told that it is not available for release then we must take this with a giant pinch of salt. Another guess must be that this is the equivalent of the ‘bureaucrats strike back’: the thinking presumably being that one way to stymie these pesky developers is to shoot their fox. With all the data in a pile, lots of public money and a few iPhone developers it must be easy to render independent attempts to create value from the datasets unnecessary and futile.

Finally, I would also guess that MDV were given this opportunity because they are the ones with most to lose from an open market in apps leveraging open transport data and so they were keen to take on the challenge. If developers were given access to the TfL Journey Planner API (search the web for the jailbreak access) then MDV would be enabling other commercial players to make money from their own work. However, the whole point of opendata access to government data is to break down stifling monopoly access to data to allow innovation around the data. If TfL release an all-singing app then it is hard for developers to justify the investments needed to do that innovation because the state has intervened in the market with taxation funding. With opendata access to all the live data feeds there is no market failure and no case for intervention by the state. Both this government and the last accepted that there is more value to UK PLC by allowing innovation (leading to increased tax receipts) than by allowing government departments to trade data to recover the cost. So this kind of app release is against all current policy.

Evidence, if we needed it, that this was a controversial release, came when equally mysteriously, the app disappeared from the iTunes app store today. From which we are forced to conclude that this was a freelance operation not cleared with TfL senior management. So there you have it: a brief glimpse within the Data Wars kimono, and a microcosm of political and economic conflicts to come around apps and data.

The backstory

The app itself is (was?) a conventional journey planner app in which you enter your origin and destination, the journey time and the modes to search:

The app also carried live departures for every mode of London Transport: tube, DLR, tramlink, bus and river. It also carries live departures from National Rail… yet there are no acknowledgments to National Rail Enquiries. It is the bus live departures that are of greatest interest as they are not available anywhere except central London bus stops on Countdown. But here they are in an app:

There are omissions… there is nothing about the Cycle hire scheme. But the app does do cycle routes, though not nearly as well as CycleStreets: they are drawn 1 pixel wide in yellow on a beige background and doesn’t avoid main roads very well. Here are the routing result page and the ‘interactive’ map page:

So this is not the app to set the world on fire, and there are alternative like Journey Pro from Navitime. It is more important for what it reveals about the battle to convert government bureaucracies to the cause of opendata, even if the political direction is clear.

Jonathan Raper

A couple of days ago TfL released timetable data via the LondonDataStore. We’ve been taking a look inside these XML files, and we’ve developed a little tool for browsing the data objects:

Browse the TfL Timetable TransXchange Objects

It’s a relatively raw object view (so not very recognisable as a timetable) We created this as an early experiment, and tool to help make sense of the data. Doubtless there are other developers investigating the format, so we thought we’d share it.

We have a ruby script to parse the XML from TfL and generate this HTML interface. This TransXchange static browser generator is also available for you to download and run on your local copy of the TfL XML, and to modify/extend/redistribute as you please. We’ll be aiming to make some improvements in the coming days.

TransXchange

The released XML data (in a zipped up set of 786 files, totalling 1.3 GB) is using the TransXchange format. Several object types can be seen. Here is a sketch of some of these types and relationships (elements nested or cross-referenced by id within the file). We also show some fields of particular interest (many simplifying omissions)

But if you’re looking for a more rigorous definition of TransXchange, the TransXchange website has all the documentation. It also has XSD files, and other tools for working with the data. The key document to have a look at is TransXChangeSchemaGuide-2.4b-v-50.pdf which includes “Figure 3-1 UML Overview of TransXChange Model for a StandardService“, on page 34, a diagram somewhat resembling the above. Page 33 gives a written summary of the object types.

Hopefully this basic information, and the browser tool will serve as a useful starting point for developers looking to understand the data format.

There’s been a whole conversation on Twitter about this proposal… here are some of the comments back and forth:

puntofisso Fri 20 Aug: I agree with this article> News on the tube API in @londondatastore and a proposal on ‘crowd-serving’ http://bit.ly/dnsUuF (via @MadProf)

paul_clarke Fri 20 Aug: RT @MadProf: Plan to crowd serve the tube API. Comments please on: http://bit.ly/dnsUuF .. fascinating approach!

londondatastore Mon 23 Aug: @MadProf interesting idea and look forward to response of the SME’s happy to point TfL in your direction if there is a head of steam

rollohome Mon 23 Aug: @MadProf sounds like an interesting approach to OD. I liked the SETI suggestion by 1 commentator: a ‘safe’ data set on which to experiment!

drewsonix Mon 23rd Aug: @MadProf Hi Jonathan – just read your posting re crowd serving. Wasn’t a big part of the problem the sheer size of data for all stations…

drewsonix Mon 23rd Aug: @MadProf ..and simply due to format? Couldn’t it be just “each train & its track locn” which we could crossref against cached tables?

drewsonix Mon 23rd Aug: @MadProf Something like Base64 representation of Set#, track loc, destcode, ismoving(y/n) for each train would mean a tiny fraction of data

MadProf Mon 23rd Aug: @drewsonix Data for all tube stations is not that large, nor too complex. But lots of users & some pulling whole feed at infeasible rate

MadProf Mon 23rd Aug: @drewsonix Challenge for tube API is finding a way to distribute data using open principles- hence crowdserving idea at http://bit.ly/dnsUuF

bensmithuk Mon 23rd Aug: @MadProf It’s an interesting idea, but given the loads it seems a shame TfL can’t just host and serve this properly from a CDN.

MadProf Mon 23rd Aug: @bensmithuk Absolutely agree, but meantime need a digital Dunkirk to rescue the feed! Rare opportunity for SMEs to get users & innovate

bensmithuk Mon 23rd Aug: @MadProf True, but isn’t complexity of proposed approach > than just proxying via Google AppEngine or Amazon?

MadProf Mon 23rd Aug: @bensmithuk Talking public sector procurement in a spending firestorm: not happening. If we take it over, its gets done fast, free & smart

bensmithuk Mon 23rd Aug: @MadProf I didn’t mean them, I meant you / SMEs by collaborating.

MadProf Mon 23rd Aug: @bensmithuk Delighted if people join the party & pay to proxy a few million hits; assumed most would want to offer API calls to generate biz

countculture Mon 23rd Aug: @MadProf Like idea, but wonder if enough SME’s willing to do it; conversely if there were lots, whether it would be economic for each

countculture Mon 23rd Aug: @MadProf …Thus mean that they would drop out, and meaning only small number again…

countculture Mon 23rd Aug: @MadProf However, SME with business idea could always apply to join service, I suppose….

MadProf Mon 23rd Aug: @CountCulture Thanx 4 points on tube API crowdserving plan: idea is that in return for distributor status SME wud hav a chance to advertise

iapainter Mon 23rd Aug: @MadProf Looks great and a great idea. Nothing better than the buzz of a new startup despite its all consuming nature Good luck #placr

daveaddey Mon 23rd Aug: @MadProf Problem is, SMEs (like ourselves) are hosting consumers, not hosting providers. We’re not set up to support high-load feeds.

daveaddey Mon 23rd Aug: @MadProf Also, demand for real-time transport data is inconsistent, eg v high demand during bad weather. Requires scalability we don’t have.

kemp_harper Mon 23rd Aug: Plan to crowd serve the tube API. Comments please on: http://bit.ly/dnsUuF

poggs Mon 23rd Aug: @MadProf Read that. In short, very fluffy and cute, ignores the fact we don’t know detail about what the issue is – bandwidth or API calls

poggs Tue 24th Aug: @MadProf I can’t see the business model in ‘reselling’ TfL’s data. Initial excitement will tail off; there’s only so much analysis we can do

MadProf Tue 24th Aug: @poggs what is biz model for any open data? No ‘white space’ coz no existing need. Need to make a market: transp apps sell by bucketload

poggs Tue 24th Aug: @MadProf I can only see two: a casual/simple “next trains” app on iOS/Android/web and a few people taking lots of data for detailed analysis

MadProf Tue 24th Aug: @poggs What bout live journey planners/ alerts/ oyster station footfall/ incident impact maps/ season ticket rebates/ station accessibility

poggs Tue 24th Aug: @MadProf Interesting uses, but I still can’t see more than a handful of people wanting to use the ‘big feed’ of raw data

MadProf Tue 24th Aug: @poggs Probably only a handful of SME tube API distributors needed. TfL will be able to it serve themselves eventually

The essence of these comments is that we shouldn’t have to do it ourselves but if we do do it then the business model for SMEs has to be clearer. I plan to post further on this… but the SME opportunity is to get users through their distribution of the TfL feed. Getting users is the hardest part of being a digital services SME, so if you get users but can’t monetise them then that is a more general challenge for open data. Persuading TfL to release data in this way… and then creating business for SMEs in London… would be a fantastic achievement for the open data movement.

Time is short… so let me have your offers to serve the tube feed if you can supply some bandwidth to join London’s great crowd-serving initiative!

Jonathan

Currently, TfL have an in-house live tube departures server based on the Trackernet system. This server couldn’t support the volume of external demand when opened through the London DataStore in July and the service had to be taken offline when the load overwhelmed the server. TfL need an interim solution that takes a feed from the internal server and redistributes it.

There are various conceivable solutions to this problem if it is not done in house. For example, a commercial API service like Mashery can do it. And TfL have discussed Google taking the feed and redistributing it for ‘free’. The problem with commercial services are that they cost money directly; and taking up Google’s offer brings Google’s terms and conditions plus data formats into all business models built on top of the service.

We would like to see a solution that creates London jobs and showcases London technology while following open data principles. Why not simply allow some London SME’s access to the raw feed and allow them to cache and redistribute it as real time open feeds? This would be ‘crowd-serving’ open data and could use public domain licences like the MIT licence. The SMEs would innovate around the raw feed, building additional API calls which they could monetise any way they could. Since this would be an intrinsically competitive space, developers wanting access to the live data would have access to the open feed and the choice of suppliers for additional services.

How could this be done in practice? Technically, we would foresee TfL authorising selected SMEs to access the raw feed and pull the tube data into a web, database or memory cache. In practice, we think that updates any more frequent than every 15 seconds would be pointless as system latency and limits to train movement reporting make it unlikely that you could see change in the service at any higher rate. With the tube running 20 hours per day this generates c. 4800 updates per day into the cache. Each SME would then offer developer access to their 15 second cache using whatever system they saw fit to provide e.g. via

• authenticated API key up to a limit of users to preserve service quality, or
• an open API service in which demand would be limited by the physical limits on the server bandwidth
• a Squid proxy to simply pass on the raw data.

If a developer found an open service or proxy to be too slow they would have to move to an authenticated service or moderate their usage.

The costs of providing one node of access to this kind of service are not huge. If a full ‘all stations’ read from the API is c. 200k uncompressed (we estimate), then getting this 4800 times per day requires nearly a gigabyte of data transfer per full user per day. A full user is equivalent in data transfer terms to about half a million users getting one station, once a day. Each SME in the scheme would have work out what sort of server package would provide processor, data transfer and bandwidth capacity sufficient to offer a distribution service. But we believe that the direct costs are affordable, in the order of £100 server costs per month to provide service to 50 full users… or 49 full users and half a million one hit end users. Of course each SME would also incur labour and overhead costs, but most would have the means to develop a freemium business model alongside their open feed.

The six million dollar question would be… who would qualifies to be a distributor? In our view the criteria should be that:

• you meet the EU definition of an SME
• you agree to offer the service for at least 6 months
• you redistribute the raw feed no slower than the ?15 second update guideline, and
• there are no access, charging or licencing constraints other than basic misuse terms and conditions.

TfL can probably handle 50 distributors pulling the raw feed on the 15 second interval base. If you met the basic criteria, participants could simply be the first 50 SMEs to apply. In time TfL could buy into the ‘best’ service or simply adopt the most efficient distribution method itself.

There are bound to be lots more questions, but the core principle of crowd-serving seems appropriate and suitable for one of the crown jewels of London’s open data. Let’s see if there are enough SME’s out there who can step up to the plate to offer an open feed and innovate around the data.

Jonathan Raper, http://www.placr.co.uk/