Data was once the prerogative of the ruling classes. Those in power would collect it for the purpose of controlling the people; so they could raise their taxes and fund their armies.
The people - whom the data was about - never had access to it. Knowledge as they say is power - which governments even in the most liberal democracies are not so keen on sharing.
In this country enormous quantities of data are produced every year. But until lately this data was never shared - never exposed to public view.
Of course this lack of transparency meant that much of the data remained of poor quality - because there was little incentive for the quality to be improved.
What this government found out very quickly after we came into office two years ago was a total lack of information on how the Machinery of Government was working. My department was determined to hunt for efficiency savings and to get more value for taxpayer money.
But it soon became clear that it was going to be hard to make Whitehall run like a business when we totally lacked good quality management Information.
Sir Phillip Green’s early review into government efficiency exposed just this problem over the poor quality of data on where and how government spends its money.
Sir Philip concluded that because data was inconsistent and not available it was impossible for Government to buy efficiently and thus huge amounts of taxpayers’ money was being squandered unnecessarily. Of course that was something that the Labour government never particularly cared about or bothered to address.
These findings were echoed by the Non-Executives like Lord Browne that this government has appointed to bring business know-how to departmental boards. And the National Audit Office have echoed their concerns lamenting the lack of accurate, comparable, reliably-sourced data in Whitehall.
Something was clearly needed if we were serious about both improving the quality of the data we use to manage government - and so to improve the efficiency of the management of government. The answer was transparency.
Opening our data to public view would, of course, expose its inadequacies - but it would also drive us to be better.
Last year we introduced Quarterly Data Summaries. These are a snapshot of how each department is spending its budget, the results it has achieved and how it is deploying its workforce. In just a year the completeness of QDS responses has gone up from 59% to 86% over the course of a year.
This is a start. But there are further improvements to come to ensure we can properly compare how departments across Whitehall compare. What we found in 2010 was that Ministers and officials were all too often comparing apples and pears - and, in fact, as metrics were revised every year we couldn’t even compare this year’s apples with last year’s.
Because we want to ensure government functions like the best run businesses we are improving our management information and publishing it transparently to drive up its quality.
The better our data is - the more accurate and comparable and usable it is - the easier it will be for the public to demand that its accuracy improves. And as we get better data we can ensure government runs more efficiently, saving the taxpayer more and more.
Transparency and Open Data is central to this government’s agenda. Today we are seeing governments, societies and economies all over the world open up but I’m pleased that this government is leading the way.
New technology has meant for the first time people can access and exploit data in new ways. It is easier for the public to demand data. It is easier for governments to provide it.
It is easier for citizens to use data to inform their choices in public services and demand responses from the providers. And it is even easier for entrepreneurs and developers to drive innovation and economic and social growth based on the raw material of data.
The UK is leading the world in making more and more data freely available.
We’ve released almost 9,000 datasets on our flagship data portal data.gov.uk. These include key information on transport, health, education, transport, crime and justice and government spending.
Now everyone can now scrutinise local crime statistics, sentencing rates, school results and hospital infection rates. And we can all demand improvements.
Releasing data is giving businesses the opportunity to exploit the valuable data that previous governments left under-used. In their Big Data study McKinsey said that across Europe data could be worth a staggering 250billion Euros - the EU puts the prize at 140billion. Either way it’s big money.
The opportunities for enterprise won’t always be obvious.
For example when some years ago the US released its public weather service one surprise result was a boost to the insurance industry. The data helped farmers to protect their profits leading to dramatic improvements in agricultural productivity. Today the weather derivatives market in the US alone is worth $3.5billion - that’s all powered by Big Data.
Last autumn, this government announced that we would publish data from all 5,000 weather stations in the UK alongside real time transport information, NHS data to promote life sciences research, house price data and more valuable datasets.
And we will keep looking to put more and more data in the public domain.
Next month new data will be published on how well GP practices do in saving the lives of people with cancer - this data will allow patients to see the survival rates of every practice.
We are also encouraging businesses and voluntary groups and charities to be open and transparent. Care home owners, for example, are to publish outcomes data on their own initiative.
We have reached a pivotal moment - where we consider the rules and ways of working in a data-rich world. How we use this resource effectively, creatively and responsibly.
The Policy Exchange report published today is very timely and makes many valuable recommendations that I will be looking at closely.
And as you will be aware last week I launched a White Paper which details how in the future the UK will continue to unlock and seize the benefits of data in a responsible way.
It sets out how we will make it easier to access data; how we do more to build trust in data; and how across the public sector we will be smarter about using data.
As I said at the beginning, data, of course, isn’t new. What’s new is the technology that in the last twenty years has made it easier for everyone to access and exploit data.
What’s new is how we can aggregate data - make patterns and connections that can drive choice, efficiency and innovation.
But while ideally this will lead to a spread of knowledge that can empower citizens - not just here but across the globe -
There is also a danger that we could be left unequal data access to data.
Where only a select few have access to what’s called the Big Data - and the expertise to analyse it -
One of my key concerns is to make sure this data revolution benefits everyone -not just savvy entrepreneurs.
That’s why we believe that data - whether it’s big or small - has the most value when people can use it effectively.
Last week I announced that the whole of government is officially adopting Public Data Principles which set out that our Open Data must be published in machine readable form under the same open licence which enables free reuse, including commercial reuse.
We are also adopting Sir Tim Berners-Lee’s Five Star Scheme for assessing the degree to which open data sets are re-usable - to improve the consistency of the data landscape.
And because all our data has got to be easy to find, easy to use and in one place - we last week launched a completely overhauled data.gov.uk site that has better search facilities, much simpler ways to access information, an advanced GIS data search and better tools for developers.
As we reinforce the presumption to publish in the public sector - we are not losing sight of the fact that we must be vigilant about retaining trust in data. In fact we were determined to strengthen the protection of privacy - something that many responses to our Open Data Consultation agreed with.
Our White Paper enshrines our commitment to ensuring that privacy is not considered as an afterthought - but is at the beginning of all discussions concerning the release of new data.
I have announced the appointment of a privacy expert to the Public Sector Transparency Board to make sure we bring in the latest expertise on privacy measures.
We will also ensure that government keeps pace with the latest technology so that anonymised datasets remain anonymised and personal data remains personal.
This is enormously important because there is no doubt that the greater openness of datasets and the success of the information marketplace hinges on our ability to safeguard people’s data from fraud and misuse.
Moreover government is determined to see all our public services providing digitally by default - sharing data is a key enabler of this.
All this means users of digital public services must be able to assert their identities safely, securely and simply.
We’ve allocated £10 million to our Identity Assurance Programme, which will see the private sector take a lead role in producing identity solutions for public services - while government set standards and encourages the development of a market for them.
We’re not looking for a single solution for identity assurance - technology moves too quickly for that. And we won’t have a central database.
Instead we are creating a decentralised structure in which people can choose themselves their own identity providers - this system does not store data and puts the user in control of how it’s used.
We’ve said privacy must be protected at every step of this data revolution. But at the same time I think we are all agreed that the government and the public sector have got to get smarter about using their data assets.
In particular we need to overhaul the public sector’s outdated approaches to using data. Right now there are too many cultural and legal barriers that leave public sector workers from the police, to the Job centres advisers, to those in the NHS all working in isolation on the same problems. Data sharing is perceived to be too difficult - even when it is in the public interest.
This is particularly relevant in our approach to tackling fraud, error and debt - a key area the Policy Exchange have also identified.
The public sector loses an estimated £37billion a year across the public sector through debt, fraud and error - in a large part because of our failure to share data effectively.
Efforts to share information on known fraud and fraudsters have been consistently undermined by the complexity and general confusion over seemingly contradictory legal constraints.
The Cabinet Office is working with the National Fraud Authority to rationalise the 86 legal gateways that currently exist for data sharing on debt, between the 8 major debt departments and local councils. This will facilitate better sharing of information across the public and private sectors.
We are also aiming to create a single legislative framework that allows sharing of data on debtors.
This will not only reduce fraud, error and debt but help us take tough action on those who seek to defraud the public purse or deliberately avoid paying their debts. Equally we can better support those vulnerable people who genuinely cannot afford to pay and make sure that we are treating them fairly.
This is a key issue but we will be working with civil servants and public sector workers in all areas to identify and break down legal and cultural barriers to data sharing where it is in the public interest.
Data is the new raw material of the 21st Century.
Government has a duty to be the best steward of the data it holds, matching the best in the private sector. The Policy Exchange rightly make the point that data technologies alone are not the silver bullet.
To fully address issues like data standards and quality the government needs skilful people who can conduct and interpret data and analytics work intelligently.
Whitehall has traditionally focused on policy - not on running like a business. So data that you could reliably and objectively compare over time and across different departments has not been prioritised.
This is going to change. Last month we published our Civil Service Reform Plan - the first stage in an ongoing programme of reform. The plan made clear the importance of good quality Management in running more efficient government in the future. And our capabilities review will ensure we have the exceptional people skilled in accounting and quality assurance that government needs.
And over the summer we will also take forward the recommendations of Dr Martin Read, who has led a review of the management information we publish through the Quarterly Data Summaries.
There’s a long way to go before government - and the wider public sector -fully harnesses the potential of our data assets.
And we are not trying to claim that government has all the answers. The internet flourished precisely because it was shaped by its users not by governments - and the same approach should apply here - we want users to drive the release of data.
The new world of Open Data needs buy-in from everyone.
And transparency may be risky, difficult and uncomfortable at times - especially for those in government - but it does stick and once you start you can’t go back.
These are the first, formative years of the new age of open government. There are clear challenges for all of us ahead but I believe these will be overcome.
We will keep building towards effective personalised 21st century democracy, with transparency informing better choice over more modern and effective public services, and open data as a key driver of innovation and growth. The future is open and Britain is leading the way.