Towards a new Data for London Library
The Mayor of London’s Data for London programme wants to make it simpler for people to share and use data held across London to improve the city and benefit Londoners, the Data for London Library will be the service underpinning this…
For London government data is now regarded as essential city infrastructure supporting public service delivery, growth, and innovation. This is because achieving our goals — such as Growth, Net Zero, safety, opportunity and good health — depend heavily on data in various forms shared among partners. Through our new Data for London Library we aim for a smarter, more sustainable, and inclusive London as a result of better and more collaborative use of data.
Today it is crucial for government at all levels to create strong foundations for responsibly and effectively managing, sharing and using urban and other data — and to integrate these foundations into wider policies around infrastructure and innovation.
Why data libraries are important today
Data comes in many forms: spreadsheets, big data sources like sensors, research outputs, and qualitative insights about the city. Each organisation in London has the data that describes one piece of the puzzle: it becomes transformative when shared and connected. A data library seeks to improve the indexing, search and discovery of disparately held datasets like these.
Currently, poor collective data sharing between local and central government bodies hinders collaboration across departments, agencies and the formation of public-private and academic partnerships. Not only does inaccessible data hampers comprehensive analysis or insight, but these inefficiencies slow down the development and delivery of data-enabled public services. Finally, the adoption of advanced technologies such as AI, machine learning, and IoT is hindered by poor access to large, high-quality datasets, reducing their potential for innovative solutions. In the context of mission-led government, modern data architecture, governance and culture are critical.
The Mayor’s 2021 Manifesto for London, reiterated in 2024, commits to “harness data and emerging technologies for our city” by rebuilding the London Datastore to enable greater collaboration. Our proposed Data for London Library model work is sympathetic to the 2024 Labour manifesto pledge to create a National Data Library to “bring together existing research programmes and help deliver data-driven public services, whilst maintaining strong safeguards and ensuring all of the public benefit.”
This big picture focus is vital because better data sharing across UK local and central government is foundationally important to innovation, growth and better public services.
An effective urban data infrastructure and approach offers the following benefits for our city and country:
- For Citizens: Assure legal, ethical, and safe data sharing and usage; enhance transparency through open data, and improve public services.
- For Decision-Makers: Provide deeper insights and improve decision-making on complex place-based issues like Net Zero, homelessness, community safety.
- For Data Users: Enhance or develop better insights and services tailored for Londoners and their service providers.
- For Innovators: Encourage innovation through collaboration with the technology sector, universities and the wider community.
The Data for London programme
The existing London Datastore (established in 2010) is used by approximately 1 million times each month by specialist and non-specialist users in equal measure. It also supports a range of important and bespoke data services like the Planning Datahub (a live open data feed of planning applications from across the city); High Streets Data Service (neighbourhood spending and busy-ness); the Electric Vehicle Charging Point Dashboard (the city’s single source of information about charge point location and charge point utilisation) and the Infrastructure Mapping Service (co-ordinating utility digs across London). But we can go further by improving not just the platform but the wider culture and community around data. This can open the door to more data partnerships like the ones we’ve seen with Imperial and Breathe London air quality open data and the work with Faculty AI on Rough Sleeping Insights (the first pan-London view of homelessness across GLA, borough and VCS datasets).
The Mayor of London’s Data for London programme aims to further simplify data sharing and usage across London to improve the city and benefit its residents. The London Datastore platform addresses some, but not all, of these needs so our new Data for London Library will offer a new a user-friendly service for easy data discovery for urban datasets. To support this service, we will also create new standards to foster collaboration between councils, universities, government agencies, and others across the city.
Key Features of the new Data for London Library
The new Library, supported by the wider programme, will offer:
- Enhanced Discoverability: A single, searchable platform for urban data.
- Improved Data Sharing: Simplified and secure sharing of urban data.
- Collaboration- enablement: Curate and use urban data to tackle key challenges.
Over the first half of the year, we’ve focused our platform development process on refining the Library through user centred design by conducting user testing to deeply understand users’ goals, needs and the nuances of the task. We engaged on a number of different levels with the data community, software developers, data project owners and other users. This confirmed how urban data is hidden in silos across London, and innovation is stifled by having to identify the right partners and guessing at the datasets they might have.
To enable more benefits for London sooner, we’ve developed a catalogue to organise the diverse datasets held across London, aiding data users to discover the data they need for their projects. This will include open data sources relevant to London projects and metadata of private datasets across the public sector and partners. It will allow users to search by data description, publishing organisation, and license types to help users narrow down to the right dataset for their problem.
We expect the first release later in 2024 as we perfect the improved search function from what we have now. From the very beginning, it will have existing metadata from the London Datastore, the ONS, and two London boroughs: to show the benefits of being able to search across all of them at once. We will then reach out to London local government and functional bodies, universities and others via a roadmap to anchor organisations across London to onboard their data catalogues, broadening our coverage and maximising the variety of innovation we can support.
The next phase will focus on better sharing, access, and use of data, building on the current London Datastore. We will support complex data-sharing arrangements, personal data storage and security, and the Internet of Things and streaming data.
The Library can be characterised as B2B2C — by creating an easy way to join up data for users inside and outside the public sector, these users can create valuable products for their end-users. Contrary to traditional smart city models, this is not a central data lake but a federated system, and therefore goes with the grain of how London is governed. This model therefore requires data leadership from city government to onboard important datasets, develop standards for data quality, ethics, and public participation. The platform will be complemented by a ‘good data citizenship’ approach.
The Data for London Library is both an urban data platform and a digital service to connect datasets. As a digital service it departs from the traditional open data publishing platform approach by a relentless focus on involving data users in the design of the Library and its methods, as well as cultivating the data community so the platform can continuously evolve around their needs.
Additional programme initiatives
Technology build of the new Library core is only one part of our work: the Data for London Library is part of a wider move to address common points of friction with data sharing such as legal, ethical/trust and quality and innovation frameworks.
So we aim to create a collaborative environment for safe, responsible, and effective urban data use. This involves building strong collective data governance, a dynamic ecosystem, public engagement, and shared capabilities and resources for stronger collaborations, for example:
- Improving city-wide Information Governance: Establish a robust data user community to further develop the platform and ensure data sharing is legal, ethical, and purposeful. An important element of this is the Pan-London Information Governance service spun up by the London Office of Technology and Innovation at London Councils and their wider Data Foundations work.
- London Data Service Standard: Guide the development of new data services alongside standards for high-quality digital services aligned with Public Digital’s Data Service Standard (which has roots in their discovery for us in 2019).
- London Data Week: An annual event to bring together data experts and the public to discuss the ethics and opportunities of using data to improve the city. The second annual London Data Week was held 1–7 July 2024.
We’ll also undertake further work to reshape our Open Innovation challenges at City Hall around data and thinking has already started around how the Library can become a cornerstone for a future urban data exchange similar to how open banking standards transformed financial data exchange.
Our work is overseen by the Data for London Advisory Board, established in 2022 by the Mayor of London, and we work closely with the London Office of Technology and Innovation at London Councils, which provides the vital link with London’s boroughs and innovation networks. Our journey so far has been supported by discoveries and work by the Open Data Institute, Public Digital, Bloomberg Philanthropies and TPX Impact — as well as engagement with Business London.
Enabling Innovation Through the Data for London Library
The Data for London Library could serve as an example of how comprehensive data orchestration can drive innovation, enhance public services and (as I’ve argued before) give more agency in devolution. It goes beyond merely being a repository; it acts as an enabler for various stakeholders, including citizens, decision-makers, data users, and innovators by providing a fundamental service to support collaboration and encouraging the development of data-enabled solutions.
Furthermore, the emphasis on user-centred design, extensive user testing, and engagement with the data community ensures that the platform evolves according to the needs of its users from the light core described above to additional functionality in the future.
The change of government is a transformation opportunity: the previous government had no coherent policy in this area nor gave any practical support as we proceeded on our journey. So as London government progresses with the Data for London Library this year, we’re excited by the new optimism and focus on central and local government data (and our open door remains open as it embarks on its own discovery and policy development for a National Data Library).
By focusing on enhanced discoverability, improved data sharing, and collaboration enablement, Data for London could be a benchmark for similar initiatives and ensuring that data sharing becomes a catalyst for innovation and better public services across the UK.
