First, the good news. The UK government has made good on its promises to release open data across government in 2011, and this year has seen a dizzying sequence of open data announcements, most recently in the Open Data Measures in the Autumn Statement. Not only has the government opened the data, but it has put in place institutions (like the Transparency Board), portals (like data.gov.uk) and funding (through Technology Strategy Board). This is all profoundly good news and has enabled the growth of a cadre of open data companies like Cycle Streets, Open Corporates and my own company Placr. We are racing to build new companies built on the open data and we are already paying taxes that go back into the Exchequer, offering free services to the public and value-added offerings to businesses.
However, there is still a cloud on the horizon. Some of the most important reference data is still locked up like the detailed maps, addresses, land records, school databases, the national planning application register and court records (details in my blog post here). The government held a consultation in the summer over the formation of a Public Data Corporation (PDC), and we presented arguments as to why embedding a government trading monopoly at the heart of open data was a bad idea, and this seemed to resonate. However, when we read the government’s Open Data Measures in the Autumn Statement we were very disappointed to see that all they have done is change the acronym from Public Data Corporation (PDC) to Public Data Group (PDG), and kept the substance of the previous proposal. This leaves us with a problem, as we are not going to be the “world leader in open data” as George Osborne wants by taxing every digital service transaction in the UK for the core reference data that the government has already paid for!
To understand why this is happening and how to fix it, we need to see what the Autumn Statement is proposing to do. I have drawn the following diagram after a brainstorming session at the Open Rights Group to show the government’s plan:
The Public Data Group will be a merger of the Land Registry, Meteorological Office, Ordnance Survey and Companies House. Analysis of their 2010-11 Report and Accounts shows that the agencies collectively have revenues of £741m, costs of £649m and that they make profits of £92m. We are told that this trading is necessary to save the taxpayer the cost of these activities. However, when you realise that 83% of Met Office and 58% of Ordnance Survey revenues are from government itself, you can see that these costs are mainly being paid by taxpayers anyway. The non-government sales income from these two agencies (MO/OS) is only £84m. Companies House and Land Registry are different because they operate registries and manage transactions for business and house buyers, and so government usage is low, with users paying all the costs.
In the rest of the diagram you see how the government proposes to add two new agencies into the mix to moderate the operation of the PDG government trading monopoly. The DSB will take some dividends from the PDG trading operation and will be able to spend this on buying data to be freed, or on services. However the scale of its suggested funding is orders of magnitude lower than the costs freeing all the data outright and so its influence will be marginal unless it also directs the PDG business plan. In this system the taxpayer would be paying twice for the data: once for the core operations through taxes and then a second time to free the data through the DSB dividend income, which the Treasury would forgo. Meanwhile the Open Data Institute will spend its money on research over the long term… and cannot influence the trading activities.
Surely this is the wrong model for the future of government data as the paywall around the core reference data will continue to inhibit private sector investment and reduce the tax revenues from innovation. There are too many new players in the system, all incurring costs and adding friction to the movement of the data. I think the government should instead release all of the data freely to stimulate private sector growth, as I show in this diagram:
If government releases the data freely and encourages the agencies to do consultancy and packaging of the free data, these costs of these agencies will have to be (or are already being) restructured as follows:
- The Met Office has already adopted this plan by releasing most of its data in the Autumn statement… it will lose most of its data revenues on the one hand (83% of costs?) but government will not need to pay for the data on the other hand. If the Met Office makes 17% costs savings on operations OR raises 17% extra consultancy charges on the freely released data (=£33m), then its data releases are revenue neutral for government. The Met Office Business plan has clearly been revised to cover this change as is is not asking for more money (NET EXTRA COST: £0);
- The Ordnance Survey needs to release its data freely so government doesn’t need to pay its current £74m bill for map data and instead it should use this money to fund OS directly: this change would be revenue neutral. The OS would lose its £52m of consumer and business income from selling data, though it would be likely to earn some revenues through consultancy on packaging and delivering the released data, say £10m. So the extra cost to the taxpayer would be £42m if nothing else changes, though in reality the OS would be able to slim down and would not need all of its current 131 Sales and marketing staff. This could lead to savings of another £10m (NET EXTRA COST: ?£32m);
- Companies House is proposing to release its key data in the Autumn statement (more details needed!), so it looks likely to have to reduce its cost base and develop its services income as the private sector adds to, and replaces its services with new ones based on the open data. As it has already moved to open its data, this change is clearly now in its business plan (NET EXTRA COST: £0);
- The Land Registry is actually the biggest question… it can and should open access to its cadastral (land) records freely, but the government still need to supervise and assure land and property transactions. There is a straight choice here: pay its £281m costs out of government funds and give us all a tax cut (great stimulus when the economy is flatlining)… or carry on taxing land transactions (possibly in new and different ways?). This is not an open data question, as the data can be released at some limited cost while the government still charges for the transaction assurance (NET EXTRA COST: ?£5m).
This analysis (Excel spreadsheet here) implies that the only real shortfall on full release for these agencies would be the loss of OS non-government income of £32m net of savings on business-as-usual, plus ?£5m for the Land Registry to give open access to its data (=£37m). To keep the costs low the Land Registry could keep its registration income, but provide access to its land records as open data.
The moves by the Met Office and Companies House show that the government believes that agencies can release their data, cut their costs and still deliver core data. Therefore I don’t think free data release is “unaffordable” in any sense, and the Ordnance Survey also needs to be restructured to deliver this change in a broadly revenue neutral way. For example, it could close services that duplicate the private sector such as the ‘Get a Map’ service. Crucially, amongst the OS data releases would be AddressBase… the full UK national address database, which is a vital open dataset needed to power almost all digital services. The overall value to the economy of removing monopoly pricing from detailed maps and addresses would be a marginal increase in economic efficiency across a vast range of transactions.
I have shown here how the costs of these four agencies are either internal government trading, being cut voluntarily or being levied as transaction costs (which could remain while the data was opened). However, in theory losing the £92m profits of these agencies would also be a cost to government funds. On closer analysis… although MO, OS and CH returned £15m to the government, Land Registry had to pay a massive £87m for restructuring costs and returned no cash to the government. So if we add the “loss” of the £15m profit to the costs of releasing data, then the total bill for releasing all the data and losing profits could reach £52m on a business-as-usual scenario, though restructuring OS can probably eliminate most of this cost. If LR continued to trade and charge for transactions (whilst opening its data), then in a ‘normal’ year its gross profits of £69m could actually pay all the bills for data releases elsewhere!
So… given the “unaffordable” costs of data release are actually a mirage… the government should be acting boldly to release all the data and stimulate the economy. Therefore the only remaining job that the government has to do to get the final open data #WIN is to reform these agencies. As far as I can see the only people who still oppose this are the managements of these agencies… and you would expect that wouldn’t you?
Jonathan Raper
What it will cost to free the rest of UK government data (spoiler: £0)
First, the good news. The UK government has made good on its promises to release open data across government in 2011, and this year has seen a dizzying sequence of open data announcements, most recently in the Open Data Measures in the Autumn Statement. Not only has the government opened the data, but it has put in place institutions (like the Transparency Board), portals (like data.gov.uk) and funding (through Technology Strategy Board). This is all profoundly good news and has enabled the growth of a cadre of open data companies like Cycle Streets, Open Corporates and my own company Placr. We are racing to build new companies built on the open data and we are already paying taxes that go back into the Exchequer, offering free services to the public and value-added offerings to businesses.
However, there is still a cloud on the horizon. Some of the most important reference data is still locked up like the detailed maps, addresses, land records, school databases, the national planning application register and court records (details in my blog post here). The government held a consultation in the summer over the formation of a Public Data Corporation (PDC), and we presented arguments as to why embedding a government trading monopoly at the heart of open data was a bad idea, and this seemed to resonate. However, when we read the government’s Open Data Measures in the Autumn Statement we were very disappointed to see that all they have done is change the acronym from Public Data Corporation (PDC) to Public Data Group (PDG), and kept the substance of the previous proposal. This leaves us with a problem, as we are not going to be the “world leader in open data” as George Osborne wants by taxing every digital service transaction in the UK for the core reference data that the government has already paid for!
To understand why this is happening and how to fix it, we need to see what the Autumn Statement is proposing to do. I have drawn the following diagram after a brainstorming session at the Open Rights Group to show the government’s plan:
The Public Data Group will be a merger of the Land Registry, Meteorological Office, Ordnance Survey and Companies House. Analysis of their 2010-11 Report and Accounts shows that the agencies collectively have revenues of £741m, costs of £649m and that they make profits of £92m. We are told that this trading is necessary to save the taxpayer the cost of these activities. However, when you realise that 83% of Met Office and 58% of Ordnance Survey revenues are from government itself, you can see that these costs are mainly being paid by taxpayers anyway. The non-government sales income from these two agencies (MO/OS) is only £84m. Companies House and Land Registry are different because they operate registries and manage transactions for business and house buyers, and so government usage is low, with users paying all the costs.
In the rest of the diagram you see how the government proposes to add two new agencies into the mix to moderate the operation of the PDG government trading monopoly. The DSB will take some dividends from the PDG trading operation and will be able to spend this on buying data to be freed, or on services. However the scale of its suggested funding is orders of magnitude lower than the costs freeing all the data outright and so its influence will be marginal unless it also directs the PDG business plan. In this system the taxpayer would be paying twice for the data: once for the core operations through taxes and then a second time to free the data through the DSB dividend income, which the Treasury would forgo. Meanwhile the Open Data Institute will spend its money on research over the long term… and cannot influence the trading activities.
Surely this is the wrong model for the future of government data as the paywall around the core reference data will continue to inhibit private sector investment and reduce the tax revenues from innovation. There are too many new players in the system, all incurring costs and adding friction to the movement of the data. I think the government should instead release all of the data freely to stimulate private sector growth, as I show in this diagram:
If government releases the data freely and encourages the agencies to do consultancy and packaging of the free data, these costs of these agencies will have to be (or are already being) restructured as follows:
This analysis (Excel spreadsheet here) implies that the only real shortfall on full release for these agencies would be the loss of OS non-government income of £32m net of savings on business-as-usual, plus ?£5m for the Land Registry to give open access to its data (=£37m). To keep the costs low the Land Registry could keep its registration income, but provide access to its land records as open data.
The moves by the Met Office and Companies House show that the government believes that agencies can release their data, cut their costs and still deliver core data. Therefore I don’t think free data release is “unaffordable” in any sense, and the Ordnance Survey also needs to be restructured to deliver this change in a broadly revenue neutral way. For example, it could close services that duplicate the private sector such as the ‘Get a Map’ service. Crucially, amongst the OS data releases would be AddressBase… the full UK national address database, which is a vital open dataset needed to power almost all digital services. The overall value to the economy of removing monopoly pricing from detailed maps and addresses would be a marginal increase in economic efficiency across a vast range of transactions.
I have shown here how the costs of these four agencies are either internal government trading, being cut voluntarily or being levied as transaction costs (which could remain while the data was opened). However, in theory losing the £92m profits of these agencies would also be a cost to government funds. On closer analysis… although MO, OS and CH returned £15m to the government, Land Registry had to pay a massive £87m for restructuring costs and returned no cash to the government. So if we add the “loss” of the £15m profit to the costs of releasing data, then the total bill for releasing all the data and losing profits could reach £52m on a business-as-usual scenario, though restructuring OS can probably eliminate most of this cost. If LR continued to trade and charge for transactions (whilst opening its data), then in a ‘normal’ year its gross profits of £69m could actually pay all the bills for data releases elsewhere!
So… given the “unaffordable” costs of data release are actually a mirage… the government should be acting boldly to release all the data and stimulate the economy. Therefore the only remaining job that the government has to do to get the final open data #WIN is to reform these agencies. As far as I can see the only people who still oppose this are the managements of these agencies… and you would expect that wouldn’t you?
Jonathan Raper