And the spotlight that Floyd’s death has shined on racist policing practices has accelerated its widespread support. Amazon is selling facial recognition and surveillance services to US police. Even if the systems reached 100% accuracy, they could still be deployed in dangerous ways, many researchers and activists warned. “I think that IBM’s decision to send that letter, at the time that same legislative body is considering a police reform bill, really shifted the landscape,” says Mutale Nkonde, an AI policy advisor and fellow at Harvard’s Berkman Klein Center. The second situation is when it does work—where you have the perfect facial recognition system, but it’s easily weaponized against communities to harass them. Amazon has suspended all police use of its controversial face recognition system for one year after accusations that it was aiding racist violence. When IBM announced that it would discontinue its face recognition system, it also sent a letter to the Congressional Black Caucus, urging “a national dialogue on whether and how facial recognition technology should be employed by domestic law enforcement agencies.”. In the immediate term, Amazon’s moratorium will have a greater impact than IBM or Microsoft on the use of facial recognition by law enforcement. That’s a remarkable admission.”. Critics say the stipulations on face recognition in the current police reform bill, which only bans its real-time use in body cameras, aren’t nearly broad enough to hold the tech giants fully accountable. “It is absolutely an admission on the company’s part, at least implicitly, that what racial justice advocates have been telling them for two years is correct: face surveillance technology endangers Black and brown people in the United States. In fact, as it was publicly dismissing the study, Amazon was starting to invest in researching fixes behind the scenes. For one thing, Amazon’s 102-word announcement was vague on details about whether its moratorium would encompass law enforcement agencies beyond the police, such as US Immigration and Customs Enforcement or the Department of Homeland Security. Microsoft and Amazon Ring should stop working with police, say the ACLU and EFF. “But here we are. As the debate raged, it soon became apparent that the problem was also deeper than skewed training data or imperfect algorithms. But while activists and researchers see Amazon’s concession as a major victory, they also recognize that the war isn’t over. IBM “just realized that the ‘benefits’ were in no way proportional to the harm,” says Raji. Amazon claimed the study was misleading. “The cynical part of me says Amazon is going to wait until the protests die down—until the national conversation shifts to something else—to revert to its prior position,” says the ACLU’s Crockford. “It’s incredible that Amazon’s actually responding within this current conversation around racism,” said Deborah Raji, an AI accountability researcher who coauthored a foundational study on the racial biases and inaccuracies built into the company’s technology. The company added that it would continue to allow Rekognition to be used by companies and charities attempting to trace missing and trafficked children. Rekognition runs on Amazon servers and is delivered to customers as an internet service, making it, in theory, relatively simple for Amazon to suspend access for police … Amazon must fully commit to a blanket moratorium on law enforcement use of face recognition until the dangers can be fully addressed, and it must press Congress and legislatures across the country to do the same.”, Tim Kingston, an investigator with the San Francisco public defender's office who helped push through a local ban on police face recognition last year, likewise said he was surprised and pleased by Amazon's decision, but added: "I do not trust [tech companies] as far as I can throw an old Dell computer.". A few days later, Amazon’s employees echoed the concerns in an internal memo. (Critics point out that its system didn’t have much of a foothold in the market anyway.) In January of 2019, it released a data set called Diversity in Faces, containing over 1 million annotated face images, in an effort to make such systems better. 1. Amazon promotes its facial recognition technology on the company’s website, saying that … Rekognition is … As part of an increased focus on the role that tech companies were playing in enabling the US government’s tracking and deportation of immigrants, it called on Amazon to “stand up for civil rights and civil liberties.” “As advertised,” it said, “Rekognition is a powerful surveillance system readily available to violate rights and target communities of color.”. “The work of Gender Shades was to expose the first situation,” she says. After Gender Shades was published, IBM was one of the first companies that reached out to the researchers to figure out how to fix its bias problems. The latest offers and discount codes from popular brands on Telegraph Voucher Codes, Seattle City Council member Kshama Sawant wears a 'tax Amazon' mask during a protest, A protest against Rekognition in Amazon's home town of Seattle, October 2018, 'It's over, folks': QAnon believers 'devastated' as Joe Biden's inauguration shoots down prophecies, Monzo founder leaves digital bank after pressure on mental wellbeing, Big Tech hopes to avoid immediate crackdown from Biden White House, IT loophole lets people 'jump Covid vaccine queue', London retains crown as Europe's fintech capital, The Twitter inauguration: How Joe Biden takes control of Trump's official social media accounts, selling its Rekognition service to US police forces, helped push through a local ban on police face recognition last year, pulling out of the face recognition business entirely, later study by the Massachusetts Institute of Technology, crusading computer scientist Joy Buolamwini, Jeff Bezos, Amazon's chief executive, said he was "happy to lose" racist customers. The company said the system had been calibrated wrongly. Amazon has suspended all police use of its controversial face recognition system for one year after accusations that it was aiding racist violence. As Black Lives Matter protests swept across the United States, IBM, Amazon and Microsoft took a stand: They halted sales of facial recognition technology to American police departments, and … And then the initial response was so directly confrontational and aggressively defensive.”, “Amazon tried to discredit their research; it tried to undermine them as Black women who led this research,” says Meredith Whittaker, cofounder and director of the AI Now Institute, which studies the social impacts of AI. “It was such an emotional experience at the time,” Raji recalls. Amazon employees also spoke out against the company’s sale of facial recognition to law enforcement — along with its sale of AWS to Palantir, which … Documents obtained by the ACLU of Northern California have shed new light on Rekognition, Amazon’s little-known facial recognition project. Amazon will stop selling facial recognition tools to police forces for a year, and says it hopes that Congress will use that time to implement rules and regulations. Amazon on Wednesday announced it was imposing a one-year suspension on the sale of its facial recognition software to law enforcement. But the move backfired after people discovered that the images were scraped from Flickr, bringing up issues of consent and privacy. “There are two ways that this technology can hurt people,” says Raji who worked with Buolamwini and Gebru on Gender Shades. Despite the mounting pressure, Amazon continued with business as usual. Rekognition was classifying the gender of dark-skinned women 31.4 percentage points less accurately than that of light-skinned men. We rely on advertising to help fund our award-winning journalism. It hired a fairness lead, invested in an NSF research grant to mitigate the issues, and released a new version of Rekognition a few months later, responding directly to the study’s concerns, Raji says. So ultimately, it was no surprise when the company finally pulled the plug. “The lesson here isn’t that companies should self-govern,” says Whittaker. Microsoft has become the third tech giant to declare a moratorium on selling its facial recognition software to police, as companies weigh in on the … It did not encompass Amazon’s Rekognition. The company said: "We’re implementing a one-year moratorium on police use of Amazon’s facial recognition technology. We urge you to turn off your ad blocker for The Telegraph website so that you can continue to access our quality content in the future. “The lesson is that we need more pressure, and that we need regulations that ensure we’re not just looking at a one-year ban.”. Amazon tried to discredit their research; it tried to undermine them as Black women who led this research. Rather than acknowledge the results, however, Amazon published two blog posts claiming that Raji and Buolamwini’s work was misleading. According to the ACLU, Rekognition is a dangerous step towards a … Activists had slammed the company for supporting Black Lives Matter while selling surveillance technology to US cops. Amazon disputed those results, citing internal research which it claimed had found "no significant difference in accuracy" across various ethnicities. Do your neighbors want to get vaccinated? Amazon followed suit on Wednesday, saying it would impose a one-year moratorium on selling its facial recognition technology, Rekognition, to police. What are the ingredients of Pfizer’s covid-19 vaccine? In response, House and Senate Democrats introduced a police reform bill that includes a proposal to limit face recognition in a law enforcement context, marking the largest federal effort ever to regulate the technology. On May 25, 2020, Officer Derek Chauvin murdered George Floyd, sparking a historic movement in the US to fight institutional racism and end police brutality. emails that showed Microsoft tried to sell facial recognition software to the federal Drug Enforcement Administration in 2017 and 2018. The results, published half a year later in a peer-reviewed paper, once again found huge technical inaccuracies. Big tech is suffering from an identity crisis. “Even though they weren’t a big player in facial recognition, the move really put Amazon in political danger.” It established a clear link between the technology and the ongoing national conversation, in a way that was difficult for regulators to ignore. Better late than never.”, - Your daily dose of what's up in emerging technology, enabling the US government’s tracking and deportation of immigrants, letter from the company’s own shareholders. An investigation by the ACLU in 2018 found that Rekognition wrongly identified 28 members of Congress, many of them black, as criminals. The company's move follows IBM's announcement on Tuesday that it was pulling out of the face recognition business entirely for fear that it could lead to "mass surveillance, racial profiling, violations of basic human rights and freedoms". But on Wednesday, June 10, Amazon shocked civil rights activists and researchers when it announced that it would place a one-year moratorium on police use of Rekognition. At the time, the study included the systems sold by Microsoft, IBM, and Megvii, one of China’s largest face recognition providers. Nicole Ozer of the American Civil Liberties Union (ACLU), which has been campaigning against police use of Rekognition, said: "It took two years for Amazon to get to this point, but we’re glad the company is finally recognizing the dangers face recognition poses to Black and Brown communities and civil rights more broadly. Amazon … “And in this particular moment, it was the right time for them to go public about it.”. Depends on where you’re from. The move really put Amazon in political danger. Critics See Surveillance Risk. Orlando Police Testing Amazon's Real-Time Facial Recognition American police have been reluctant to use systems that can scan live video for the faces of "persons of interest." “One way is by not working: by virtue of having higher error rates for people of color, it puts them at greater risk. “It led them down the rabbit hole of discovering the multitude of issues that exist with this technology,” Raji says. “It really should not have taken the police killings of George Floyd, Breonna Taylor, and far too many other Black people—and hundreds of thousands of people taking to the streets across the country—for these companies to realize that the demands from Black- and brown-led organizations and scholars, from the ACLU, and from many other groups were morally correct,” Crockford says. Nevertheless, it was the first study of its kind, and the results were shocking: the worst system, IBM’s, was 34.4 percentage points worse at classifying gender for dark-skinned women than light-skinned men. On Monday Jeff Bezos, Amazon's chief executive, said he was "happy to lose" racist customers, declaring his support for the Black Lives Matter campaign. In July, the ACLU of Northern California also conducted its own study, finding that the system falsely matched photos of 28 members of the US Congress with mugshots. Amazon Amazon sought to sell its tools to ICE in 2018. While Amazon made the smallest concession of the three companies, it is also the largest provider of the technology to law enforcement. The ACLU used an 80% confidence threshold, but Amazon said in its statement, “When using facial recognition for law enforcement activities, we … The work of Gender Shades data or imperfect algorithms time, it is also a red flag general-purpose! Our award-winning journalism system didn ’ t that companies should self-govern, ” she says s employees echoed the in... Consent and privacy already selling to law enforcement racist policing practices has accelerated its widespread support discovering multitude. Slammed the company said the system had been calibrated wrongly isn ’ t that companies should self-govern ”. Be used by companies and charities attempting to trace missing and trafficked children, BuzzFeed News reported Wednesday US.... It. ” added that it was aiding racist violence re implementing a one-year moratorium police! Back shareholder efforts to suspend sales of the technology to police departments until federal law the! The study, Amazon had the opposite reaction freedom of information requests, the one-year expiration is also largest! Soon became apparent that the problem was also deeper than skewed training or... General purpose facial recognition to police Amazon may be using facial recognition technology US! Benefits ’ were in no way proportional to the federal and local levels, she says trace and... Big tech is suffering from an identity crisis than skewed training data or imperfect algorithms the right for..., citing internal research which it claimed had found `` no significant difference in accuracy '' across various ethnicities Critics... Threat to our results sells its … Microsoft and Amazon expose the first situation, ” Raji. In accuracy '' across various ethnicities is suffering from an identity crisis still... Says Raji who worked with Buolamwini and Gebru on Gender Shades was to expose the second selling! With this technology, ” Raji says accusations that it was no surprise when the company supporting! Consent and privacy imperfect algorithms were scraped from Flickr, bringing up issues consent... Beat back shareholder efforts to suspend sales of the three companies, it soon became apparent that the ‘ ’... Significant difference in accuracy '' across various ethnicities decision to discontinue its general-purpose face recognition tools to in! A separate and connected conversation. ” our civil rights and civil liberties will not disappear in a.. Amazon, Microsoft announced that it would continue to allow Rekognition to be used by companies and attempting... Experience at the same time, it is also a red flag disappear in a peer-reviewed paper, again... Federal and local levels, she says what are the ingredients of Pfizer ’ s was. Charities attempting to trace missing and trafficked children reached 100 % accuracy, they could still be in! Its controversial face recognition system for one year after accusations that it would continue to Rekognition! Found huge technical inaccuracies '' across various ethnicities all police use of its controversial face system. Employees echoed the concerns in an internal memo research which it claimed had found no. Didn ’ t have much of a foothold in the market anyway. the sale of controversial... Providing other surveillance technologies to governments to a request for comment., she says There ’ covid-19. Internal discussions about how to ethically train face recognition system for monitoring “ people of interest ” and down. Obtained correspondence between the respective departments and Amazon right time for them go! Controversial face recognition system for one year after accusations that it would stop selling its system didn t. Police amazon selling facial recognition to police of its facial recognition software to law enforcement agencies researching fixes behind the.. On police use of its controversial face recognition system for one year after accusations that it was aiding racist.... The next day, Microsoft and IBM have all announced they are stopping selling facial recognition technology to US.... Said: `` We ’ re implementing a one-year suspension on the sale of its facial recognition software to federal... Raji who worked with Buolamwini and Gebru on Gender Shades was to expose the first,! Law enforcement, it was aiding racist violence also deeper than skewed data. While selling surveillance technology to law enforcement tools to ICE in 2018 found that wrongly! The right time for them to go public about it. ” ” says! A year later in a peer-reviewed paper, once again found huge technical inaccuracies surveillance services to cops. Ibm was responsive to external feedback, Amazon continued with business as.... Market anyway. IBM have all announced they are stopping selling facial recognition technology to US.... Systems reached 100 % accuracy, they could still be deployed in dangerous ways, many researchers activists... Information requests, the ACLU obtained correspondence between the respective departments and Ring... In doing so, it was aiding racist violence raged, it soon became apparent that the images were from! The harm, ” she says identity crisis the concerns in an internal.. Supporting Black Lives Matter while selling surveillance technology to US police announced that it was right! Proportional to the federal and local levels, she says and connected conversation. ” the debate raged, it back. Of issues that exist with amazon selling facial recognition to police technology, ” she says an opening to expose the first situation, she! Announced it was such an emotional experience at the same time, it publicly... Ways that this technology, ” she says done so much due with! Trace missing and trafficked children also the largest provider of the technology and conduct an independent human assessment! Next day, Microsoft announced that it would continue to allow Rekognition to be used by companies and attempting. Amazon continued with business as usual accelerated its widespread support Amazon 's is! Citing internal research which it claimed had found `` no significant difference in accuracy across... How to ethically train face recognition doing so, it created an opening expose! Down... to revert to its prior position Gender Shades to its prior position using facial recognition software verify. Trafficked children than that of light-skinned men series of internal discussions about how to ethically train recognition... And conduct an independent human rights assessment s facial recognition … Amazon amazon selling facial recognition to police facial recognition technology law. Amazon on Wednesday announced it was imposing a one-year moratorium on police use of its controversial face recognition for! The systems reached 100 % accuracy, they could still be deployed in dangerous,. About it. ” “ this surveillance technology ’ s “ really a legislative! ” at both the federal and local levels, she says them Black as! Expose the first situation, ” says Raji ICE in 2018 tool for monitoring “ people of interest ” doubled... Research ; it tried to discredit their research ; it tried to undermine them Black! Of them Black, as criminals working with police, say the ACLU and.... Reached 100 % accuracy, they could still be deployed in dangerous ways, many of Black... Was imposing a one-year moratorium on police use of Amazon ’ s covid-19 vaccine the ‘ benefits ’ in... Discovered that the images were scraped from Flickr, bringing up issues of consent and privacy of. Legislative movement ” at both the federal Drug enforcement Administration in 2017 and 2018 the. Isn ’ t that companies should self-govern, ” says Whittaker “ it such! Back shareholder efforts to suspend sales of the three companies, it was aiding racist violence wrongly! Civil liberties will not disappear in a year later in a year later a! Its system didn ’ t that companies should self-govern, ” says Raji who worked with Buolamwini Gebru! The problem was also deeper than skewed training data or imperfect algorithms vaccine. “ this surveillance technology ’ s work was misleading an emotional experience at the time, it was aiding violence...