Getting the data right: seamless data migration for the public sector
With the National Data Strategy setting out the Government’s desire to create a data-led public sector, as well as an appetite from local authorities to deliver data-driven services, many organisations now see that migrating from legacy systems to a new provider offers the ideal opportunity to improve data quality and use this investment to its fullest potential.
Data migration can be seen as a ‘necessary evil’ when implementing new tech, and may not be given the attention it deserves. The consequence of this is data decay – key data is excluded from transfer, or is clumsily aggregated in an attempt to retain information with little thought about its future use or accessibility.
Getting data migration right goes a long way to repressing fear and scepticism of change for stakeholders, which is vital in any digital transformation project. With careful planning and execution, data migration can be broken down into a series of manageable areas and councils can succeed in protecting valuable citizen data, with staff able to access it and use it to inform decisions and improve services.
Here are the most important elements for public sector organisations to consider when undertaking a data migration.
Data migrations require people with technical and non-technical knowledge. I’d recommend that teams include an expert on the area to which the system relates, who will take the lead on the initial data mapping exercise. This is a key individual who understands the current business process and legislative requirements for that data. They will also understand and consult with other stakeholders who rely on the data – maybe the local land charges department or an external Government department in need of solid statistics.
This expert should also understand the legal requirements for data retention and be empowered to make decisions about which data should and shouldn’t be migrated. These teams should also include an individual with technical knowledge of the source system. This person will have a good understanding of the source database and work closely with the above expert to convert business rules into technical requirements and can also read into the underlying data, helping to inform mapping decisions.
Identifying anomalies in the data up front lets the team correct them before the actual migration, cleansing records at source. They can also build rules into the migration to resolve issues during the extraction. Identifying data quality issues too late in the migration may result in unexpected issues, and ultimately a delay in the project’s timeline.
Data quality issues may originate from legacy data migrations with poor mappings or minimal data, invalid or duplicated records, accidental deletions, changes to the way staff used the system in the past or the poor administration of records. It’s important to identify any issues in the data before migrating it – side-stepping the potential for precious citizen data to be lost, damaged, or inaccessible.
Often public sector organisations, councils or otherwise, might believe that a successful transformation relates directly to how much data was carried over. Although this is a quick and convenient metric to use, it can lead to wasted effort, with migration teams focusing too much energy on data that won’t have much value in the future.
To ensure the effort pays off, take a value-based mapping approach. Each field of data you map out should have some value to the organisation. Ask these questions – is there a legislative requirement to hold this data? Do any external departments need this data? Will it aid future decision-making and improve services?
If the data does not meet any of these criteria, you should question whether it should be migrated. By taking this approach, effort will only be spent on high value data that is essential to the provision of local services.
It’s tempting to keep hold of every piece of data, ‘just in case’. But look at what data retention rules exist in your organisation. By excluding useless data from the migration (in line with the value based approach mentioned above), the new system will contain only compliant data and the time it takes to extract the data, and the storage requirement, will reduce.
When you reduce the storage requirement, data backups will run faster and the time taken to restore a full backup of data in the event of a disaster is reduced. It isn’t uncommon for councils to suffer data losses or even security breaches, so recovering that data should be as easy and painless as possible – something like Salesforce has almost 100% uptime, reducing the risk of this happening massively.
Data migration is a complex process that is prone to change between test cycles. Record all mapping decisions in a single table and make it easily searchable for when it is needed. Data extraction routines should also be re-runnable. All of this sets the team up for rapid migration and successful testing.
Testing is also crucial to the success of a full scale migration. Testing should include a range of record types at different stages of the business process, as well as manufactured test records. Test records do not need to ‘make sense’ or follow standard processes but are designed to touch as many mapping points as possible and hit every avenue. Manually migrating test records into the target system in line with the mapping document gives a preview of what to expect when records are migrated en masse, as well as highlight any areas that don’t line up with the new processes.
Finally, make sure to validate all data and make sure it is compliant with the new system.
Doing the above will ensure that the public’s data is kept safe and business can continue as usual following go-live. Local authorities hold a raft of sensitive data, from social housing waiting lists to benefits applications, planning applications and the like, and that’s excluding any internal data from staff working in the organisation. A digital transformation project is a huge undertaking. Getting the data part right will mean minimal disruption for staff or citizens, while valuable information is kept both safe and accessible.
Originally posted here
