After postponement and a string of contentious hearings, a New York City law passed in 2021 finally took effect at the start of July. New York City's Local Law 144 governs employers and employment agencies that use automated employment decision tools in the hiring or promotion process. The requirement applies not just when candidates are selected for progression in the hiring process, but any time an automated tool "classifies them into groups."
Use of automated processes is widespread during hiring and promotion, and the law requires it to be subject to bias audits.
NYC's law specifies that an automated tool may not be used if more than one year has passed since its most recent bias audit, and clarifies that a bias audit is "an impartial evaluation by an independent auditor." The results of this bias audit must be made publicly available. The audit requirement generated significant debate and more than a few forceful comments during the bill's hearing process.
As the U.S. state and federal governments begin to build guardrails on AI usage, a targeted application like Local Law 144 could prove to be a critical testing ground.
In his recent Senate testimony, Anthropic AI CEO Dario Amodei suggested developing regulations for AI is an urgent but complex problem. Because of the unanticipated pace of AI development, the "science of testing and auditing AI systems is in its infancy."
AI system auditors, then, need a system of reliable and dynamic measures as the field develops. Amodei recommended the U.S. National Institute of Standards and Technology as the regulatory body naturally suited to develop such standards and urged for better funding of this existing agency.
Anthropic voiced further support of NIST in a subsequent blog, observing that though "the art of measurement may sound dry or possibly inconsequential," good regulation requires effective evaluation and measurement. "After all," the blog quips, "it's hard to manage what you can't measure."
Prior to launching its Risk Management Framework, NIST made an initial foray into guidelines on identifying and managing bias in AI. The special publication recognized that while "audits can be an effective accountability, bias, and general risk mitigation mechanism ... audits currently exist in a wide range of forms with varying levels of quality and consensus." Trustworthiness is the crux of the problem with auditing AI systems, and NIST has taken this challenge to heart.
Later, in its request for comments on AI accountability policy, NIST addressed the complexity involved in creating a trustworthy and accountable AI audit standard. It's a problem of layers.
The way AI models are implemented, and therefore assessed, depends on how these models are embedded in organizational systems. Because assessment is so context dependent, NIST suggests the most effective audits "should extend beyond the technical to broader questions about governance and purpose."
With AI systems being sociotechnical, a good audit must also involve trustworthy sociotechnical markers, acknowledging how the larger system is "influenced by societal dynamics and human behavior." Part of the sociotechnical solution could look like a set of recognized auditing standards from a regulator like NIST.
A critical and still unclear piece of this puzzle is who can and should conduct AI system audits.
For the NYC rule, there is no requirement for independent auditors to be approved by the Department of Consumer and Worker Protection. The only qualifying factor in the law is that auditors be "independent," which the DCWP rules define as "someone who exercises objective and impartial judgment in the performance of a bias audit."
In their comment on the NYC law, BSA The Software Alliance expressed concern about the potential windfalls of ambiguous requirements. Without industry agreement, companies may choose independent auditors according to individual preferences or price — which often correlates with quality — potentially undermining "efforts to establish common objective benchmarks."
Other prominent industry voices like Workday encouraged recognizing that internal audits can have similar value when they meet criteria for independence, especially in the absence of "a respected independent professional body to establish baseline auditing criteria or police unethical practices among auditors."
Fordham Journal for Financial and Corporate Law Lindsey Fuchs wrote in a recent law review article, "while Al regulation is new, auditing requirements in statutes are not." As several commenters suggested for the NYC law, system auditors need to conform to other similar industry practices and requirements. Typically, professional or educational bodies create a recognized baseline criteria for auditors. Professional bodies govern the professionalization, ethics and norms of fields from accounting to privacy. That form of binding standard does not yet exist for AI system auditors.
Further, embracing sociotechnical solutions for bias audits should prioritize the development of a standard of human trust in auditors. In a recent response to a National Telecommunications and Information Administration request for comments, 23 state attorneys general jointly urged for this action. They suggest NIST, NTIA or both should "work to spearhead consistent criteria and technical standards for testing, assessments, and audit policies and practices of emerging AI systems." These regulators could either create a certification system for trusted auditors, or "a system for establishing and overseeing" other certifying entities.
The IAPP response to the same request for comment recommends generally that professional credentials and other independent indicators of quality are an essential foundation for a more trustworthy and accountable AI governance marketplace.
As a wide variety of auditing firms are already offering services under the NYC requirement, their work may be building toward a new standard — if there is consistency among them.
Keith Sonderling, a commissioner on the Equal Employment Opportunity Commission, recently reflected on the significance of the first automated employment law. Not only is this effort prompting more questions on how to audit systems pre- and post-deployment, but he said "employers are now recognizing and investing in how to get AI compliant."
Local Law 144 is a step toward building not just familiarity with system audits, but a new set of standards and measures that will become part of organizations' everyday practice. Of course, applying these standards and measures outside of the auditing context will be another major challenge entirely.
Here's what else I'm thinking about:
- New regulator in town? Senators Elizabeth Warren, D-Mass., and Lindsey Graham, R-S.C., released the text of their new Digital Consumer Protection Commission Act, a bipartisan attempt to "rein in Big Tech." The lengthy bill would create a new independent regulatory body, empower the Department of Justice and the Federal Trade Commission as enforcers, and prioritize privacy standards like access to personal data and limitations on personalized advertising.
- From Mr. California to Mr. Worldwide? In a first-of-its-kind adequacy decision for a U.S. state, the Commissioner of Data Protection of the Dubai International Financial Centre found California's amended California Consumer Privacy Act equivalent with the DIFC's Data Protection Law. The recognition could signal the possibility for future recognition of California by other jurisdictions. Speaking of data transfers, the IAPP released a chart to help with understanding next steps in transatlantic data transfers.
Under scrutiny:
- Patience for Alphabet. In a letter, Senator Mark Warner, D-Va., raised questions about the potentially premature deployment of Alphabet's Med-PaLM 2 AI model, including concerns over the storage and usage of patient health data. In a later statement, a spokesperson affirmed the company's dedication to security and privacy in their roll-out of the model.
- Pondering the orb. Even before its official launch this week, Worldcoin's global iris-scanning mission has been under scrutiny for the sensitivity of the biometric data it collects. The company's "proof-of-personhood" methodology has already gathered a significant quantity of data, prompting serious privacy concerns around this new technology.
- About last rites. An antitrust lawsuit from the FTC has been looming over Amazon and its progression was marked by "last-rites" meetings with Chair Lina Khan and the other commissioners this week, signaling the last step of the process. This suit is separate from the consumer protection matters settled in June and the ongoing lawsuit about dark patterns in Amazon's Prime subscription service. Details of the complaint, including whether it includes arguments about anticompetitive uses of personal data, are still unknown.
Please send feedback, questions and human trust to cobun@iapp.org.
