From PDFs to Machine-readable planning data: progress on changing London’s planning system
Since Peter and Molly’s last post on this project back in March, Simon Long has since joined the team to lead the delivery of the new London Development Database (LDD) Automation project. Here’s a big update on progress so far:
The Plan: change how planning data is gathered across London
The London Development Database was started in 2004 as a collaborative project between the Mayor and the London boroughs to monitor planning permissions, starts and completions across London.
The data on the LDD is supplied by the relevant planning authority. They are responsible for the quality and completeness of the data. In practical terms planning information is currently locked in PDFs rather than machine-readable fields. This stops information flowing smoothly and limits our real-time view of planning the city.
Officially launched in late 2018, and funded by MHCLG’s Local Digital Fund, the LDD Automation Project will revolutionise the way we do things — streamlining how planning application data travels from applicants, through London’s 35 local planning authorities, to City Hall, and out to the public. The initiative is in response to research showing the challenges Local Planning Authorities face when it comes to monitoring.
The ambition is to create a ‘live hub’ of planning and development information, accessible to all Londoners by reforming the information we collect and the way we collect it. We will request the data we need for monitoring up front, on the initial planning application. This data will pass seamlessly into authorities’ improved back-office systems, where planners will verify it, and then automatically out of those systems to City Hall and onto a public website.
The goal is to implement this by 2020.
Key Project Milestones
Defining a new approach
We have re-evaluated how we plan to collect data from borough back office systems, a key step that will allow us to consolidate all relevant information in one place. The initial plan was for all the back office suppliers to upgrade their software to add our new fields. However, this would require all London boroughs to upgrade to the newest versions of their back office system at the same time. It soon became obvious through conversations with boroughs and back office providers that this was quite an ambitious aim, and that boroughs would not be able to go through the upgrade process in time for our delivery date. So, we came up with an additional option for collecting the data that allows us to go live quickly and test the system as planned, rather than delaying until all boroughs have undertaken time-intensive upgrades:
Our two methods for collection are now:
1. The Full System Upgrade Option: our original approach would see the back office provider update their system(s) to create all new fields and export that data to us.
2. The Interim Option: back office systems will export and transfer whatever information they hold at the moment to us, without requiring an upgrade. Then, any information from the new fields we have added to the planning application will be passed to our database directly from the application submission portal. We’ll match up the data with the relevant application in our database.
This was a key pivot for us that arose from user research.
By providing this alternative interim option, we can go-live across London at the same time with less technical effort — rather than relying on all London boroughs to schedule in costly and time consuming major system upgrades at the same time.
The Prototype — Creation of our new Elastic Search Database and Kibana Reporting Tools
Our Supplier, WS Atkins, have finished the setup of a prototype of our new Elastic Search database including all the fields in our new data standard. They then created a standard method for loading the data into our database from any back office system. This has since been shared with all back office suppliers to help them to build their tools to provide the data to us in a standard way.
Kibana Dashboards — New Visualisation of Data in London
Atkins has also created some example user ‘Dashboards’ for how the data will be viewed in our new system. The GLA will undertake extensive user testing in order to set up a comprehensive suite of dashboards, allowing boroughs and the public to easily understand the data, as well as allowing boroughs to set up their own. This will compliment an updated version of the web map of London that currently exists on our website.
The images below are just some examples of how the data could be viewed in future. Please note these dashboards are based on the existing LDD data up to 2017, for demonstration purposes.
The next generation of planning application submissions — Planning Portal
We have also been working closely with all application submission portals in use across London — including the Planning Portal, which is the most heavily used. They are currently updating their application forms to include new questions that will provide us with fields from our data standard.
Planning Portal are currently on track to complete the final version of the new forms, ready for user testing, in December. These new forms will only be available for London Boroughs and any other authorities who wish to opt in to the new data standard. All other authorities in England and Wales will be unaffected by our changes.
Planning Portal are also introducing online forms for Prior Approvals in line with our project timescale, which will ensure many more applications are able to be submitted digitally.
Having the commitment from the Planning Portal is crucial to this project because it ensures we are able to capture the information we require up front, from developers — the key to the automation process we’re implementing.
Engagement with the Back Office Suppliers
We are now fully engaged with all the back office suppliers in London — Idox, Northgate, Agile, Ocella, Tascomi and Arcus to provide us with one of the following methods of supplying the data. These methods are based upon the two key options for collecting the data described above.
Any supplier that has chosen to do option 1 first will proceed to option 2 at a later date.
1. Build a connector to pass us the existing fields in their back office systems. This will not require a major system upgrade as no new fields will be added to the back office system in this phase. We will collect the additional GLA data directly from the submission portal once a valid application has been received.
2. Build all of the new GLA fields into the back office systems, update the 1App connector and build/update the API to pass us all the data.
Having the commitment from all of the back office suppliers is crucial to our plans because it allows us to progress with automation on a fast timeline, without boroughs needing to re-procure systems in order to participate.
Working with the Boroughs
We have been keen to work with all of the London boroughs as partners on this project and have completed a number of very useful workshops.
In one, we discussed how we will transition from the old LDD to the new one and the potential timescales involved. The session was well attended and extremely useful as it gave us a number of ideas to contribute to our transition plans.
Another workshop was aimed at discussing how the boroughs currently use the LDD for reporting, what they will need from the new LDD, and how we can ensure that anything required for go-live will be available to users. This session was also well attended, and many useful ideas were put forward that will guide our plans around reporting.
In future, once the back office systems and submission portal updates have completed, we will be working closely with the boroughs to carry out comprehensive user testing of the new Planning Portal forms and the flow of the data into our new database.
Developer Engagement
In September we held a key workshop geared entirely toward developers/agents, to present our project to this key group of users. We were really pleased by the response we got as it showed that despite the changes soon coming to planning application forms, participants were onboard with our vision for this project and the benefits it could bring to understanding what is really happening on the ground.
We are continuing to engage with developers and agents, with additional sessions upcoming (the next one on 12 November), and welcome all to join us and share their perspective. We will also be user-testing forms with developers when they are ready.
The challenges we’ve faced so far
Paper Application Submissions
One of the biggest challenges in our journey to create an automated, live feed of development data is how to deal with applications that originate from sources other than online submission portals, including paper and email submissions. We are keen for London to move away from these types of applications as they put the burden and cost of reporting on the local authority. We are working with boroughs on this matter currently.
Changes to Live Applications
Another one of our biggest challenges has been how changes to live planning applications feed through to the back office systems and our database — the current process is in some need of modernisation! Changing a planning application through the Planning Portal currently entails re-submitting supporting PDF documents, which is of no help to us. As a result, we are looking at how we can enable applicants to update fields within the portal, which would then automatically update not only our system but, in many cases, the back office providers’ also. This is still very much a work in progress but one which we intend to finalise over the coming months.
Our Next Steps
The immediate goal is for the back office suppliers, planning portal and Atkins to finish their product developments so we can start user-testing in earnest. Most of that development will be completing over the next few months and once ready we will be entering our next key stage of the project — testing! Between now and the end of 2019 we will be working hard to test a number of things — the new application forms, the flow of data from the back office systems and the submission portals into our database, and the Kibana Dashboards reporting capabilities. We will then implement the changes and improvements that flow out of the testing.
As well as all this, we will be working closely with the back office suppliers and the London boroughs to prepare our go-live plans for each authority, finalise our plans for transitioning from the current LDD to the new system, and the timescales for decommissioning the current LDD.
Check back here for updates as we move forward, and please get in touch with any questions or comments.
