iLIT & NYU Response to White House AI Bill of Rights Initiative Draws on Global Practice

Response to Request for Information from the White House Office of Science and Technology Policy on a Bill of Rights for an AI-Powered World

We are writing on behalf of the Digital Welfare State & Human Rights Project, Center for Human Rights and Global Justice (CHRGJ), NYU School of Law,¹ and the Institute for Law, Innovation & Technology (iLIT), Temple University, Beasley School of Law,² as well as a group of international legal experts and civil society representatives with extensive experience studying the impacts of biometric technologies.³

We welcome the focus on human and civil rights within the Bill of Rights for an AI-Powered World (“AI Bill of Rights”) initiative and the focus on the impacts of biometric technologies.⁴

2020_02_21 IDEAS Hub Opening and Ribbon Cutting

Where industry has long pushed for ethical principles, this is an opportunity to protect rights through binding regulations, an essential step given the existential threats such technologies pose to human rights, democracy, and rule of law. OSTP should reflect on both the substance of rights and potential barriers for enforcement. This includes striving to distinguish—as many new technology developers fail to do—between the need for innovation, new laws and new rights, and the need to fix what is broken in existing laws, rules, policies, practices, and institutions.

This response provides international and comparative information to inform OSTP’s understanding of the social, economic, and political impacts of biometric technologies,⁵ in research and regulation. Biometrics fuel automation globally,⁶ often at an accelerated, reckless pace, and these concerns transcend both political and geographic boundaries. Other powerful political actors—perceived as both peers and competitors—are attempting to understand and regulate in this area. This is an opportunity for the United States to be a world leader in ensuring that innovation is pursued in a way that safeguards human rights, both at home and abroad.

While we look forward to a consultative and transparent process for the AI Bill of Rights, we also note that the speed with which such technologies are being deployed requires urgent action. OSTP should work to establish immediate checks on the deployment of some of the most high-risk and contested tools, including an immediate moratorium on mandatory use in critical sectors such as health, education, and welfare, allowing time and space for democratic oversight before further intractable harms emerge. Our complete recommendations can be found in Section V.

I. The need for a comprehensive federal government response

There is already significant evidence that use of biometric identification in the United States can lead to harm, disproportionately impacting communities already discriminated against on the basis of, inter alia, race, sex, and national origin. For example, facial recognition technology disproportionately misidentifies people of color; use in law enforcement thus perpetuates racial bias, false arrests, and police brutality.⁷Moreover, the Department of Homeland Security’s (DHS) transnational network of biometric records, tracking, and automated profiling consistently evades scrutiny, but shows evidence of arbitrary, discriminatory, and harmful practices.⁸

Despite evidence of the harms of biometric technologies, regulation is woefully lacking,⁹ with the exception of some cities and states.¹⁰ A significant part of the population is not covered by this patchwork of prohibitions,¹¹ and while litigation and local regulation provide some oversight, the federal government and its contractors are not held accountable even to these inadequate standards.¹² The absence of, for instance, guidance for development and use of AI by the federal government and its agencies, as well as common binding standards for private actors, risks perpetuating fragmented and insufficient rights protection. Further, the federal government has a vital role to play in regulating all biometric technologies, including those which have been in place for decades, such as fingerprint scanning in the law enforcement and immigration contexts, as well as the extraterritorial application of technologies developed, produced, sold, and promoted by U.S. government agencies and corporations.

Two initial, fundamental concerns with a “Bill of Rights” approach must be highlighted, based on expert comparative legal analysis from several constitutional democracies. First, such an approach, if taken at face value as an effort to amend or modernize textually anchored rights, may exclude structural constitutional questions, such as separation of powers, the scope and quality of judicial review, and standing. Adoption of biometrics and predictive technologies increasingly concentrates power in executive agencies, inviting structural, slow-onset forms of injury.¹³ Yet, unlike most constitutional systems, U.S. judicial review of administrative actions is structurally divorced from constitutional law and rights protection. Much relevant technology is predicated on “improving” or “modernizing” the administrative state, but “administrative law in the USA is not concerned primarily with basic rights.”¹⁴Rights, however formulated, therefore risk being effectively unenforceable as executive discretion continues its extra-constitutional expansion.

Second, the absence of a cause of action for indirect discrimination (“disparate impact”) that applies generally across different sectors and to state, local, and federal departments, as well as private actors, is concerning in this context. In contrast to proportionality tests applied in the majority of constitutional frameworks,¹⁵ U.S. constitutional balancing tests are rigid and rules-driven, restricting serious scrutiny to the most obvious and intentional instances of racial discrimination.¹⁶ Though disparate impact exists as a theory of liability under some federal civil rights statutes,¹⁷ even here the availability of disparate impact claims is at executive agencies’ discretion in their enforcement of federal anti-discrimination laws.¹⁸ In most other jurisdictions, and under international treaties like the Convention on the Elimination of All Forms of Racial Discrimination, ratified by the United States in 1994, the term “indirect discrimination” is used to denote liability for discrimination based on the effect of laws and practices.¹⁹

The limited availability and lax enforcement of disparate impact leaves the equal protection clause gutted and insufficient to deal with AI-enabled biometric discrimination.²⁰Without providing for disparate impact claims, rights protections in the U.S. fall beneath international equality standards that the government has pledged to uphold and are not fit for purpose in an automated society which already exhibits structural bias and discrimination.

II. International evidence provides a critical resource

There is now a significant body of evidence that illuminates both the potential benefits and harms of biometric technologies in different contexts.²¹This response reflects input from leading experts who have worked in India, Jamaica, Kenya, and Ireland,²² where governments, international organizations, and private actors have used a combination of biometrics, data sets, and machine learning to mediate access to fundamental rights.²³ With cities and states in the United States poised to follow suit, ²⁴this research provides an invaluable resource, allowing for proactive actions that anticipate and mitigate known harms.

Most critically, evidence now extends beyond frequently raised concerns about surveillance and privacy in the context of law enforcement and national security, to encompass concerns about social rights such as health, social security, education²⁵housing, and employment²⁶

A recurring finding is that biometrics have potential to generate and exacerbate patterns of social exclusion, as well as direct and indirect discrimination. These technologies thus increasingly affect access, availability, affordability, and quality of fundamental public services.

A. How do AI and biometric technologies generate exclusion and discrimination?

Exclusion can be caused by innate problems with biometric technology. While much recent critique has focused on facial recognition and mass surveillance, the difficulties of mitigating the harmful effects of “lower-tech” solutions²⁷ such as fingerprinting, should be both a warning and opportunity for learning as “novel,” more advanced technologies emerge. As with most biometrics, specific notions of “normality” are built into fingerprinting systems; “hand scanners have particular sizes and shapes, with designated places to put the fingers,” and anyone falling outside of this “norm” will struggle to authenticate.²⁸Failure rates are significantly higher among people of color as systems are “infrastructurally calibrated to whiteness.”²⁹Further, as biometric systems are probabilistic and are often designed to tolerate significant exclusion errors, relying on them to definitively identify or verify will inevitably lead to exclusion.³⁰

Moreover, while laboratory-based testing of biometric technologies might show relatively high success rates, as was shown in a challenge to a nationwide digital ID system reliant on fingerprint authentication in Kenya, “the real-world data is very different.³¹Environmental conditions, including humidity, temperature, and light exposure, impact the quality of biometric data capture.³²Biometrics are not immutable, as they can alter over time and degrade with age. Capture and authentication often depend on fragile, expensive hardware, as well as quality internet and electricity. Thus, digital divides—which map onto other disadvantages—can be exacerbated through AI-enabled biometrics.³³

Consequently, when biometrics are yoked to essential services such as social security or health care, marginalization and exclusion may arise. This in turn results in decreased access to numerous fundamental entitlements, damaging physical and mental health, and impacting dignity. This has been extensively documented in India, home to the world’s largest biometric identification system, Aadhaar.³⁴

Persistent failures to authenticate fingerprints through Aadhaar at the point of service for welfare programs, including food rations depended on by four-fifths of Indian families, has resulted in numerous deaths by starvation, families cut off from rations for weeks, and a system that increasingly punishes the poor.³⁵In Uganda, card readers were unable to read older persons’ fingerprints or match their biometric profile to an accurate birth date in the national ID database. Although eligible to access certain social protection programs, such as cash transfers, older persons were consistently denied access to life-saving grants because of their inability to identify and authenticate biometrically.³⁶

Even where such technologies operate as intended, their use can facilitate other forms of indirect discrimination. They can sit atop existing barriers, while introducing further requirements, and access becomes contingent on digital literacy, specific forms of personal identification³⁷reliable access to basic ICT services, or fees related to travel, administration, and lost time spent navigating the system. For instance, in India, the use of Aadhaar in public services requires networks of data operators who continuously collect and verify biometric data. Without oversight, such operators become bureaucratic bottlenecks, sites of harassment and intimidation, and an insurmountable barrier to accessing services.³⁸

B. Civil death and other cumulative, systemic impacts of biometric systems

Taken individually, instances of exclusion may already constitute indirect discrimination. But the persistence of biometric information also means that the effects of exclusion replicate quickly, locking individuals out of multiple services. In Kenya, the United Nations High Commissioner for Refugees (UNHCR) collected biometric information to distribute food aid during a period of famine. Consequently, many Kenyans who were registered as children are victims of ‘double registration’: since their biometric data appears in a refugee database, the government denies them national ID cards, restricting access to services including employment, health care, and social security.³⁹In Ireland, the Public Services Card (PSC), which includes collection of biometric data, rapidly expanded beyond its original role in the welfare system, with other government agencies requiring it as the sole form of ID.⁴⁰This expansion was introduced without transparency, democratic debate, or adequate review of its necessity and proportionality. The use of biometrics can therefore quickly become de facto mandatory, even when not formally required.

Any failure to authenticate or ensure that data is consistent across different systems can therefore lead to “civil death,”⁴¹where an individual is cut off from all fundamental services. This is the case in Pakistan, where the government has unilaterally blocked certain individuals’ biometric digital IDs, forcing them into a vetting process to ‘prove’ aspects of their identity such as citizenship or gender.⁴²In Assam, India, the government recently conducted a mass citizenship verification process,⁴³placing approximately 2.7 million people on a ‘doubtful list’ of those whose citizenship is called into question. Many on this list have had their biometric profiles frozen; this means that they cannot use their Aadhaar record to receive health care, access food rations, get a drivers’ license, or register a SIM card.⁴⁴

This civil death phenomenon is especially concerning since use of biometric technologies can coincide with entrenchment of structural racism and discrimination. While the broad use of these technologies in public service delivery will ultimately affect everyone, at present harms disproportionately impact already marginalized communities; across many biometric systems, those unable to identify and verify are often those in poor, rural communities, ethnic and religious minorities, women, and older persons.⁴⁵

Widespread deployment may thus exacerbate and deepen structural and institutional patterns of harm.⁴⁶

Beyond exclusion, the extensive use of biometrics can also fundamentally affect democracy, the rule of law, accountability and transparency,⁴⁷while entrenching private sector control over public functions.⁴⁸ After alleged election rigging in the 2017 Kenyan presidential election, government officials were unable to comply with judicial orders to grant access to election results data tied to a biometric voter registration system, as the vendor’s servers were in France.⁴⁹In South Africa, the introduction of biometric technologies into welfare payment systems resulted in one company’s disastrous monopoly while weakening the government’s power to maintain any control over the welfare system.⁵⁰Use of biometrics can thus augment powerful market-based interests that do not reflect human rights and democratic principles.⁵¹

III. Comparative efforts to mitigate the exclusionary impact of biometric identification

While a data protection and privacy framework should be seen as a necessary condition to safeguard human rights in the context of biometrics, such measures are not sufficient to combat broader effects. Regulatory efforts that fail to include specific remedies for exclusion nor accessible accountability mechanisms render it extremely difficult to safeguard rights. For instance, India’s Aadhaar Act stipulates that children shall not be denied access to any subsidy, benefit, or service as a result of failed biometric authentication, but does not provide any specific cause of action or remedy.⁵²Most efforts to regulate the use of biometrics—and AI more broadly—have also failed to adequately engage affected communities in a meaningful, continuous way.⁵³ In Ireland, this was a core complaint about the expansion of the PSC’s scope.

Blocked by the lack of remedies, civil society organizations have resorted to litigation to challenge biometric identification systems. A series of such court cases highlights impacts on equality, dignity, autonomy, health, and social security, and demonstrates some ways in which legal frameworks and norms can be applied to biometric technologies.⁵⁴However, litigation is not an ideal mechanism, and challenges within litigation reflect broader difficulties regulating AI.⁵⁵For instance, biometric identification projects often involve proprietary technology and are implemented quickly and with little transparency; litigants therefore face significant barriers to accessing information necessary to challenge these systems. Judicial timelines also mean that harms may continue, and often replicate and deepen, while awaiting review.

Pushback from civil society and affected communities has also demonstrated the limitations of a purely individual rights framework that does not sufficiently recognize disparate impact; many of the impacts of biometric technology are structural, dispersed, and affect groups collectively. For instance, the legal challenge to the national ID system in Kenya required individual plaintiffs to show that they, as a member of a particular group, had been directly disadvantaged through the disparate impacts of biometric technologies.⁵⁶Similar issues have emerged in the United States,⁵⁷where victims of biased surveillance systems are left without constitutional protections.⁵⁸Thus, it is crucial to establish definitions of group harms and indirect discrimination, as well as evidentiary standards for demonstrating disparate impact.

Each application of biometric technology deserves its own legal assessment of harm, as well as of its legitimacy, necessity, and proportionality. However, some have concluded that, on the evidence, such technologies pose such serious risks to human rights and democracy that the potential benefits are outweighed, necessitating a ban on the sale and use of these technologies.⁵⁹Any steps taken by the U.S. government should seriously consider the gravity of these concerns.

IV. An international and comparative perspective is also necessary to reflect the global environment in which such technologies are being developed, used, and regulated

The United States plays a major role in the development and uptake of biometric technologies globally, through foreign investment, foreign policy, and development aid, as well as the activities of U.S. companies. The U.S. government has participated in mandating creation of biometric identification systems, such as through UN Security Council Resolution 2396, requiring states to “implement systems to collect biometric data” in order to “properly identify terrorists.”⁶⁰USAID provides active support for foreign governments’ collection of biometric data, while the World Bank finances the development of biometric systems in dozens of countries.⁶¹U.S. government actors and companies influence critical decisions in standard setting bodies about specifications for biometric data collection devices and biometric data analysis.⁶²Further, the Taliban’s seizure of U.S. military biometric devices and data in Afghanistan demonstrates the immense ramifications of U.S. actions abroad.⁶³The widespread use of biometric recognition at entry points at the Mexico border⁶⁴further influences other governments around the world to follow suit.⁶⁵

Meanwhile, the United States is one of the largest exporters of biometric surveillance technologies.⁶⁶ U.S. company L1 Identity Solutions was instrumental in the introduction of India’s Aadhaar system, for example;⁶⁷and U.S. companies such as Apple have also normalized the everyday use of biometric authentication.⁶⁸These companies have been largely unfettered by legal or regulatory constraints in their experimentation with biometrics.⁶⁹In large part, such initiatives have been paused only after public backlash and coordinated advocacy have forced companies to change course.⁷⁰Meta’s recent decision to shut down its facial recognition system and delete facial templates was explicitly driven by “societal concerns,”⁷¹but this came after Meta had been unconstrained in creating a database of over one billion faces; the company retains its DeepFace software and can resume use at any point.⁷²Further, existing models of self-regulation are insufficient and do not provide meaningful constraints on the development and deployment of biometric technologies.⁷³

Reticence in constraining U.S. technology companies’ advancements has been driven by a dominant narrative of an “AI arms race” with China.⁷⁴The National Security Commission on Artificial Intelligence (NSCAI) notes that China is setting a “chilling precedent.”⁷⁵Indeed, shocking reports detail the Chinese State’s use of biometrics to facilitate surveillance and persecution of Uyghurs in Xinjiang.⁷⁶

Yet U.S. government officials lament that technology companies in China can develop AI aided by unconstrained biometric data collection, claiming it is “not a level playing field.”⁷⁷This furthers the idea that “global AI leadership” requires low regulation, private sector access to troves of personal data, and expansive security use.⁷⁸The NSCAI urges that the United States “must win the AI competition”⁷⁹and identifies, somewhat uncritically, “surveillance,” “clearing of regulatory barriers,” and “enormous government stores of data” as factors enabling China “to leap ahead.”⁸⁰Viewing the development of AI enabled biometric technologies through this competitive, national security paradigm risks that law, regulation, and human rights are sacrificed in efforts to “win.”⁸¹The U.S. government must not allow a perceived AI arms race to dictate its approach to regulating biometric technologies.

Further, an arms race narrative simplifies complex realities around regulation in China itself.⁸² Growing public controversy around facial recognition, combined with tensions with Chinese Big Tech companies, have led the Chinese government to introduce regulations, including regarding the use of biometric technologies.⁸³The Supreme People’s Court of China has issued regulations requiring companies to obtain consent before collecting and processing facial biometric data.⁸⁴China’s recent Personal Information Protection Law mandates data minimization and user consent across the private sector when processing “sensitive personal information” including biometric data. China appears to be taking seriously the need to regulate biometric technologies.

Meanwhile, the European Union (EU) is claiming a leadership role in regulating biometric technologies and protecting human rights. For instance, the EU seeks to prohibit outright some uses of mass biometric surveillance by law enforcement.⁸⁵

The United States should view such attempts not as a ceiling, but rather a challenge to set standards even higher. Indeed, the EU’s proposed AI Act has been critiqued for its overly-broad exceptions; unnecessarily restricting prohibition of remote biometric identification to law enforcement; and applying prohibitions only to “real-time” uses rather than continuing or post-hoc uses.⁸⁶Further, the EU’s proposed Act gives providers significant discretion to assess the risks of their own technologies;⁸⁷it also fails to confer individual rights to those impacted by AI systems, or to provide for effective remedies where harms occur.⁸⁸We encourage OSTP to look to these parallel efforts and strive to go further still.

The U.S. government must also take account of the far-reaching impacts that its decisions and regulation of U.S. companies already have worldwide: the extraterritorial application of technologies developed, produced, sold, and promoted by U.S. government agencies and U.S. corporations must come into the remit of the AI Bill of Rights.

V. Recommendations

The outcome of this RFI and the AI Bill of Rights should be a comprehensive governance framework, including relevant laws, policies, and plans for implementation, which emphasizes human rights, regulatory oversight, and effective enforcement. In order to achieve this, OSTP should therefore work towards the following recommendations:

Impose an immediate moratorium for critical sectors: Define, classify, and enact a moratorium on the use of mandatory AI-enabled biometric identification technology.⁸⁹Such identification systems should never be mandatory in critical sectors such as education, welfare benefits programs, and health care, so as to preserve access to fundamental services.
Invoke legal action to address the indirect and disparate impact of biometrics: Propose and enact legislation that unequivocally applies the disparate impact doctrine, at a minimum in federal equal protection claims regarding the design and use of AI-enabled biometric identification technologies, encompassing their implementation in administering access to public and private services. Such legislation should be designated implementing legislation in line with the ratification of the CERD, affording a private right of action for racially discriminatory effects of the deployment of AI-enabled technologies.
Engage in further review of the human rights impact of biometrics and the components of different legal and regulatory approaches. This should include, inter alia:

a. Conduct and make public a comprehensive mapping of all federal systems currently or prospectively using biometric identification, including (1) the kinds of information collected, (2) the legal authority for collection and retention, (3) the purposes for which information is used, (4) how the information flows within public agencies, and (5) the impact of collection, retention, and sharing on rights.

b. Conduct a comprehensive analysis of other countries’ and regional bodies’ efforts to develop binding legal frameworks to regulate AI-enabled biometric technologies. Distilling key essons, the U.S. government should go beyond minimal standards to progress the field towards greater recognition and protection of human rights.
Build a comprehensive legal and regulatory approach that addresses the complex, systemic concerns raised by AI-enabled biometric identification technologies, including:

a. Commit to adoption of AI-enabled biometrics within administrative agency operations only to the extent that adoption demonstrably furthers the justification for delegated authority. Subject such adoption and use to regular oversight and review.

b. Establish clear safeguards for experimentation with these technologies, including but not limited to mandating rights-based impact assessments before a biometric technology can be piloted by the government or the private sector, and requiring a high level of justification, as well as suitable precautions, when such technologies are deployed first on marginalized groups such as migrants or welfare benefit recipients.

c. Address both (and distinguish between) public and private use, individual and group rights, and domestic and international use and data-sharing.

d. Place meaningful constraints on actions taken abroad. This includes U.S. companies’ operations abroad with regard to marketing, sale, or transfer of biometric data and technologies, as well as the U.S. government’s actions in spheres including, but not limited to, international development, counterterrorism, defense, and migration.
Ensure that any new laws, regulations, and policies are subject to a democratic, transparent, and open process. This should include, inter alia:

a. Hold further consultations, proactive outreach to affected communities, and engagement outside of the United States.

b. Ensure that public education materials and any laws, regulations and policies should be described and written in clear, non-technical, and easily accessible language.

Response to Request for Information from the White House Office of Science and Technology Policy on a Bill of Rights for an AI-Powered World

I. The need for a comprehensive federal government response

II. International evidence provides a critical resource

III. Comparative efforts to mitigate the exclusionary impact of biometric identification

IV. An international and comparative perspective is also necessary to reflect the global environment in which such technologies are being developed, used, and regulated

V. Recommendations

Share this:

Similar Posts

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

More Information