Improving London-wide planning data: what we found…
By Molly Strauss, Senior Policy & Programme Officer, Growth and Infrastructure at City Hall
As part of the Mayor’s Smart London work, Chief Digital Officer Theo Blackwell has initiated a London-wide collaboration to streamline how data on coming development passes between London’s 33 council planning authorities and City Hall.
While results show the particular challenges of gathering London-wide planning data, they also give insights into the wider issue identified in the Smarter London Together Listening Tour, namely fragmented approaches and IT systems across London government.
The London Development Database
Currently, council planning authorities are required to submit data via the London Development Database (LDD).
The LDD contains details of all planning consents on:
any new build residential units
any loss or gain of residential units through change of use or conversion of existing dwellings
creation of seven or more new bedrooms for use as either a hotel, a hostel, student housing or for residential care through new build or change of use
1,000m2 or more of floor space changing from one use class to another or created through new build or extension for any other use
the loss or gain or change of use of open space.
The Database, established in 2004, remains an essential source of information so that the Mayor can monitor delivery of the London Plan. LDD data tells us which planning applications receive permission, when they start construction, and when they are completed.
How it works
Councils are expected to add permissions on a monthly basis, within three months of the permission being granted. Information on when work starts or is completed is updated annually. City Hall is not responsible for adding any information to the database and is not responsible for the quality or completeness of the data, only for ensuring conformity across London’s local planning authorities.
Improving the Database
Providing information is a requirement on London’s planning authorities, which approach it in different ways. This leads to a variation in quality of information gathered.
So at City Hall, we started to evaluate how ‘smart’ back-end changes to the process could improve efficiencies, lowering the administrative burden planning authorities face while also increasing both the quantity and quality of development data openly available to the public sector, businesses, and Londoners.
Method
To identify the best steps to achieve this outcome, we first set out to understand business-as-usual.
Planning authorities face constraints on all sides — including using legacy systems and processes. We wanted to know what is working and what isn’t, from their perspective.
We asked:
How do they manage LDD requirements and how long does it take?
Where does the data come from?
Are they satisfied with their data management software?
How does a planning application travel through systems and teams from submission to permission, and eventually, completion?
Is data collected that remains hidden in internal systems rather than being shared?
These questions were meant to uncover where manageable changes to technology and work practices could have big impacts.
We wrote to all 33 councils asking for information, and to date received replies or engagement from 26 of them. From February to May 2018, we met with the 26 and traveled to their offices in person — from the centre to the greenbelt — so that we could speak with wide-ranging groups including planners, development managers, monitoring officers, administrative support, and ICT.
This offered different teams within planning authorities the opportunity to sit down together, and often resulted in important information-sharing on the spot.
Findings
Of the 26 authorities we spoke with, many were frustrated with the systems they use to track planning applications.
Data extraction is a key issue, sometimes becoming so complex that it requires specialised staff. System fields frequently do not reflect planning authorities’ needs nor the database requirements. Proper integration between these systems and authorities’ other software — or even modules created by the same provider — is consistently a challenge.
Of those we met with, 15 authorities use development management systems created by IDOX, and 7 use Northgate products.
Many reported difficulties negotiating with these big providers to make changes reflecting authorities’ needs. Only 21 percent of authorities are satisfied with their systems, and half of those we spoke with are considering or actively pursuing procurement.
On average, 79 percent of planning applications enter authorities via Planning Portal, a webtool created as a joint venture between MHCLG and a private company. Planning Portal automatically populates most authorities’ systems with information on each application. However, because Planning Portal’s fields do not fully match authorities’ needs nor the LDD requirements — and because many applicants do not populate the existing fields correctly — only very basic information is pulled through into planning authorities’ systems.
LDD officers often face a difficult task: for at least half of authorities, information required for the LDD is not entered into development management systems. Therefore, officers must trawl through documents manually, hunting for the figures the need. Once the information is located, data entry into the LDD is still largely a manual process.
Time taken by councils to complete LDD requirements
For those we spoke with, completing the LDD took on average 60 hours monthly, including collecting and entering information on planning permissions, starts and completions.
Why? We were told starts and completions are particularly difficult to track, because building control records are often an unreliable source of information. Officers must therefore piece together starts and completions data through a variety of sources, from street naming and numbering to Google Streetview images to site visits.
To complicate matters, the details of a scheme can change between application submission and permission — so council LDD officers cannot rely only on original application documents for source data.
Our research identified some disagreement within authorities around who is responsible for ensuring data reflects the final, permitted scheme. Council LDD officers sometimes act as a double-check by referencing final officer reports to glean the most up-to-date information — but this isn’t always fed back into authorities’ systems or other reports.
Likewise, teams are sometimes unaware of useful data held by their colleagues. Those with the most knowledge of schemes — case officers — rarely contribute directly to the database.
Councils report different outcomes when contacting applicants to ask for additional pieces of information they require, but tend to agree that large, established companies are more reliable than small ones, and that requests are more successful if made before permission has been granted.
Because we are interested in whether useful data for understanding London’s growth sits within authorities without being shared more widely, we asked whether anyone across teams collects certain pieces of information not required by the LDD (for example, expected start and completion dates of projects, or their estimated construction cost). In many cases, authorities do have the additional information of interest to us, but it is distributed across systems, within disparate spreadsheets, or collected inconsistently.
Different approach
The purpose of completing the database differs between planning authorities. For some, the LDD is the primary (and sometimes, only) source of monitoring data for development, feeding directly into Annual Monitoring Reports and housing trajectories. For others that have their own separate monitoring systems, this can lead to potential duplication.
Finally, moving toward ‘open data’ practices is not consistently cascaded through authorities as a priority across London, for a variety of reasons including lack of resources and concerns about data privacy.
Conclusions
Across planning authorities, officers and managers know that database entry is not as efficient a process as it could be, and many are eager for change. However, the solution will not be as simple as creating automated data transfer between planning authority systems and the LDD (although this may play a role).
There are bigger issues at work: software that does not reflect the needs of its users, a Planning Portal that does not collect as much data in a useful form as it could, and teams who might benefit from tapping into another’s information and workflows. As we hoped, there do seem to be opportunities to lower workloads and yield more openly available development data — but this requires navigating complex systems, organisations, and companies.
Now that we have identified where the challenges and opportunities lie, we are exploring our next steps and determining the best role for City Hall to play. These issues are at the core of our Smart London work, and we are keen to see positive change.
