A new city data sharing platform for London

City Hall’s Christine Wingfield and Paul Hodgson from the City Intelligence Unit discuss aspects of progress towards London’s new city data platform, an important manifesto commitment

“How do big cities join-up data held by many different organisations to gain better insights or deliver better services to their residents?” This question became even more important during the pandemic as decision-makers sought to understand the spread and impact of the virus and will be vital for making decisions around the climate emergency and other pressing needs. Earlier this year we wrote about our emergent work to create a new city data platform and approach to city data sharing in London.

While the London Datastore was initially launched in 2010 by the Greater London Authority (GLA) as a tool to promote greater civic transparency and accountability, it has outgrown this scope in several ways. Today we look at sharing data from a wider variety of sources than open data. This includes sensor data and data which may be ‘non-open’: because of commercial or privacy characteristics. We are also keen to see what data we can share with the private sector, the research community and civil society.

Policy makers at City Hall are increasingly seeing the value in sharing data of all kinds to address city priorities through joint data projects. We’ve highlighted the benefits of this approach in a recent blog about our High Streets Data Service. This public and private sector collaboration uses data to understand the impact of the pandemic and lockdown on the health of London’s 600 High Streets and has provided a vital service for local authorities in London with the re-opening, recovery and renewal of these important social and economic centres.

The London Datastore can play a pivotal role in enabling the city to collaborate and join up the city’s data. Our early discovery with the Open Data Institute (ODI) confirmed the platform should pivot to provide a central registry of the city’s data, making discovery of data or analysis easier, rather than attempting to be a huge ‘data lake’ of all city data. Our public sector agencies need to be able to work together and to quickly and securely share private and open data — including live feed data — to support data driven policy making and services for London. This ‘federated’ approach reflects the how London government — the Greater London Authority, major agencies and 33 boroughs — works in practice.

To realise this vision, we will redevelop the platform so that it better meets the needs of the GLA, our partners and users. This summer, we undertook a second discovery exercise to translate these high-level needs and aspirations into a set of functional requirements that can be used to develop the platform. Importantly, we also wanted to develop a clear understanding of the technology options for the build.

The needs of our users underpins the approach we will take to technology and functionality. We conducted 13 user research workshops, 10 interviews and 1 user validation session. This provided us with a detailed understanding of the tasks, preferences, and frustrations according to the three main site user personas (consumers, publishers, and admin). We mapped user journeys and documented user stories for the future development roadmap and backlog.

There are several ways the new platform could be delivered, including open source with the option to customise and build, vs lease. We are planning to adopt a pragmatic approach, using off-the-shelf components where they meet our needs, and prioritising our development time on those parts of the new city data platform where full control over the architecture, technical design and operating model are key priorities for users and areas of future innovation and iteration. This includes the catalogue, search, and user management functionalities.

There are several ways to think about the site architecture, but this is our starting point.

When built, our new city data sharing platform for London will increase effectiveness by making our processes much more efficient, and our delivery of data services much more cost effective. It will give greater discoverability, ownership and control over the data; and greatly improve user experience and uptake.

We are going to consider these technology recommendations and the priorities for the roadmap with our partners and stakeholders across London and other UK cities, before tendering for a build partner early in the new year.

We will keep you updated on our progress and invite feedback soon.