Data retention — or, more critically, deletion — once again seized the spotlight following New Zealand's largest data breach earlier this year. The country's Deputy Privacy Commissioner Liz MacPherson aptly referred to data retention as "the sleeping giant of data security." The often overlooked data management challenge has grown so extensive and intricate that it can be daunting.

In accordance with New Zealand's Privacy Act, organizations are legally obligated under Information Privacy Principle 9 to not retain personal information longer than necessary. However, priority has often been given to compliance with IPP5 to prevent the loss, misuse or disclosure of personal information. New Zealand organizations will need strategies to purge the backlog of information to ensure compliance with IPP9. Plans should involve engaging stakeholders at all levels, conducting thorough record management reviews, implementing data-retention policies and considering technology solutions. To navigate this potentially overwhelming undertaking, we offer an overview of key considerations and practical pointers for the establishment and execution of data-retention policies, schedules and procedures.

Establish your goals

Drafting a policy is easy, but implementing an effective policy is challenging. While legal and information technology teams may outline essential tenets, investigations into standard operations may reveal the impracticality of perfect compliance. For this reason, it is helpful to understand your organization's systems and processes. 

Engage at the right (and all) levels 

Understanding the status quo is easier with endorsement from senior leadership and support from subject matter experts, data users and those who interact with data subjects.

• Senior leaders: Irrespective of financial implications, senior leaders' involvement is pivotal. Given time constraints on system and customer experts, endorsement from senior leaders ensures a synergistic approach.
• Subject matter experts: IT colleagues can help establish practical parameters regarding system capabilities, limitations and costs. Legal, finance, health and safety, and human resources professionals will have knowledge regarding the minimum retention spans for different types of information.
• Data stakeholders: Data owners, users and customer-service representatives contribute towards a pragmatic representation of typical business requirements, such as customer expectations.
• Risk management and compliance teams: Depending on organizational scale and structure, these teams offer crucial support, from aligning retention schedules with business needs and delivering training to executing relevant monitoring, reviews and assured activities per agreed-upon plans.

Locate and tag

Data-discovery work should be undertaken for structured and unstructured data across electronic and physical locations. This may include end-user devices, servers, clouds and physical storage. It can be a complex process that requires the use of a data-discovery tool. However, gaining a clear understanding of your data landscape makes it easier to classify this data and assign ownership to effectively ensure accountability. Determining a general classification approach for retained information allows different business areas to self-assess their risk. Changing content over a record's lifecycle and a potential tendency toward downplaying risk should be factored into any review. Simplicity, including examples, can guide participants to a harmonized approach, for instance:

• Low risk: Publicly available information, for instance on your website.
• Medium risk: Information intended for internal business use but which does not qualify as high risk.
• High risk: Personal information, commercially sensitive information, confidential documents. 

Identify big fish and low-hanging fruit

In reviewing data and systems, you will likely come across opportunities for simple, logical improvements, as well as challenges that require time and investment to resolve. The former may include implementing outlook tools, for instance, applying retention policies to folders or instituting digital clean-up days to align with New Zealand Privacy Week. The existing IT strategy may guide when and how resolution occurs for more challenging issues. For instance, times of system migration can be opportune for resetting rules and classifying data segments.

Defining success

As with any long-term project, a roadmap with reported achievements along the way will assist with momentum. Once implementation is underway, consider including it in regular reports to senior leadership. 

Tackling challenges

Setting up an effective data-retention program that is fit for purpose presents various challenges and demands ongoing efforts. The following key challenges should be addressed in parallel resolution to optimize data-retention practices.

• Assigning responsibility: Consider establishing a suitable oversight body encompassing cultural and technological facets to assure comprehensive oversight.
• Robust records management program: Develop a robust records-management program that includes consideration of Māori data sovereignty in its design. Te Ao Māori perspectives can be a useful reference to interpret the intentions of the Privacy Act. If personal information is taonga (a treasure), its governance should be proportionate to its value. Take legal advice on regulatory obligations, such as the Privacy Act 2020, Public Records Act 2005 and the Official Information Act 1982, to ensure compliance and protection of all data sources.
• Striking the right balance between automation and manual intervention: A successful data-retention program requires both manual processes, such as training and attestation, and automated processes via appropriate data management tools. Keep in mind automation does not necessarily mean a total "set and forget" mentality. These tools also require regular reviews and configuration updates to maintain efficiency and effectiveness.
• Embracing emerging tech such as artificial intelligence: AI-driven solutions can help automate data classification, identify high-risk data and predict retention needs to streamline data management. However, it is vital to consider ethical and lawful use in selecting and implementing these technologies. Large language models in AI often rely on processing vast data sets with unresolved questions over whether license to this data has been truly granted. It is essential to be mindful of data privacy and ethical considerations.

Moving beyond regulatory compliance and adopting industry best practice

Amid the dynamic landscape of data protection regulations, organizations should proactively anticipate future policies that make data retention and disposal easy. Organizations can uplift their data-management practices beyond minimum local regulatory requirements by considering concepts such as data minimization, anonymization and privacy-enhancing technologies. Other than potential adverse impacts on reputation, the risk of limiting compliance to local laws arises when dealing with foreign-owned suppliers or customers who are subject to more stringent obligations. Staying ahead of future policies allows organizations to align their data retention strategies with evolving privacy standards, ensure robust data protection and comply with industry best practices.

In pursuit of a secure future

As data continues to surge and the regulatory landscape shifts, it has become critical for organizations to evolve their data-retention practices. The bedrock of sustainable data management is formed by balancing compliance, innovation and ethical considerations. By embracing emerging technologies, engaging stakeholders and anticipating regulatory shifts, organizations can forge ahead with confidence, ensuring data integrity and safeguarding privacy in an interconnected world.