Healthcare organizations tackle AI governance in fragmented regulatory environment

Healthcare organizations are rapidly adopting artificial intelligence tools during a time of regulatory uncertainty.

Global entities are facing challenges with different regulatory requirements across larger jurisdictions like China, the EU and U.S. The operational obstacles exist on a national level as well with conflicting obligations across sectoral frameworks causing friction for organizations big and small.

During the Atlantic Council Cyber Statecraft Initiative's 8 Sept. webinar, stakeholders warned differing regulatory approaches to data governance, infrastructure and cross-border data transfers in the AI context could impact clinical trials, research efforts and opportunities to advance patient care.

Pfizer Enterprise Data Science and Advanced Analytics Vice President Ranjit Kumble noted AI tools are helping progress pharmaceutical innovation, including drug discovery, by helping researchers identify biological targets during the early stages of drug development. Despite the ability to potentially accelerate and innovate the sector, concerns over balancing growth and necessary safeguards remain top of mind as new healthcare AI deployment opportunities arise.

"When we develop an AI model, the goal is to drive a benefit for a population," Kumble said. "If a model is deployed without the right safeguards, the benefit to the population is much, much more uneven. Safeguards are what ensure that the benefit is going to be consistent."

Regulatory divides

Sensitive health data processing is at the forefront of sectoral concerns around AI development and use. There is no one-size-fits-all approach to varying regulatory requirements for sensitive data under U.S. comprehensive state privacy laws and global regimes like the EU General Data Protection Regulation. 

But skipping sensitive data processing altogether may impact healthcare entities' ability to access high-quality data for AI training to support their patient services.

Kumble said organizations are preparing data separately for individual AI use cases, using resources to ensure data is "AI ready."

"That's a challenge because the time and effort to prepare every use case, as you have more and more use cases across the company, starts to become unsustainable and starts to become unscalable," he said, adding the healthcare AI industry is moving toward a "precision advantaged world" where the "breadth and depth" of training data feeds patient confidence and guides competitiveness.

One potential path to streamlined compliance, according to Kumble, is the establishment of consensus privacy safeguards and data management standards among healthcare entities that policymakers can reference as they draft more concrete rules.

A range of risk requirements are also creating patchwork issues. High-risk system obligations for providers and deployers under the EU AI Act do not align with U.S. state-level regulations around AI chatbots, which focus more on transparency and disclosure.

Dreadnode Head of Policy Daria Bahrami indicated expedited business decisions are further clouding the risk calculus.

"We're at a really critical point where AI is accelerating all of the work we've already been doing, so now we're seeing results faster, and that means companies have to make decisions about what they're willing to tolerate versus not," she said. "I think those risk conversations have always been sort of on the back burner, but now we're seeing real-time consequences."

Governance challenges

Governance structures need to remain flexible to keep pace with the evolution of AI capabilities, Bahrami said. But that flexibility should not prevent pre-deployment assessments of AI tool data processing and retention standards or any other governance benchmarking.

Agentic AI's growing potential to go rogue puts a premium on governance steps, particularly in the healthcare context. Bahrami pointed to the recent OpenAI hacking incident where one of the developer's AI agents escaped its testing environment and breached AI startup Hugging Face.

Snowflake Global Public Sector Chief Technology Officer Stephen Moon said securing agents from the outset needs to become a priority for all entities.

"You've got to limit it at the agent governance level, but also at the data security level. Because that agent is only going to be able to do what you get access to. ... I spend a lot of time on that piece on building in semantic layers, so that it understands what it's looking at and then has the context and then applying those security controls at the data layer."

A similar agent hack in the healthcare sector could present more consumer consequences than those of the Hugging Face incident. Loose safeguards around patient records, diagnoses, prescription management and other sensitive information put patient safety at risk.

Targeted and measured communications around any potential breaches are equally important to proportionate governance practices to avoid an incident, Pfizer's Kumble said.

"I think the part that also worries me is that the incidents themselves, the coverage of those incidents, tends to sort of spark a lot of panic and fear, and maybe the conversation moves into a realm that's a little bit less fact-based," he said. "To me, that is equally concerning, and so, while the consequences are unintended, I think without the right guardrails and without the right sort of fact-based communications, it becomes very, very hard to reel things back in."