Note — Nov 15, 2020

Automating Society Report 2020

Seen in → No.150

Source →

Intro to a great report (free download) on automated decision-making (ADM) systems in society. Lots of important research presented “through journalistic stories, then, through research-based sections cataloging different examples, and, finally, with graphic novels.”

The quotes below are all from the intro which in itself is already a useful read using a number of examples, the impacts, how the ADMs were rolled back or improved, how policy can be implemented, how some laws are well intentioned but never properly applied, etc.

Important points are made about facial recognition, black boxes, and the lack of adequate auditing, enforcement, skills, and explanations, as well as how many fall in the techno-solutionist trap.

It’s also important to note how good intentions in policy making often clash with governments’ dreams of “leading in AI.” I’d love to see a country do a good job of integrating the two, consider due process, transparency, auditability, and the respect of human rights and democracy as integral to leading in AI.

The machine may be scary, but the ghost within it is always human. And humans are complicated, even more so than algorithms. […]

In principle, ADM systems have the potential to benefit people’s lives – by processing huge amounts of data, supporting people in decision-making processes, and providing tailored applications. In practice, however, we found very few cases that convincingly demonstrated such a positive impact. […]

This trend [of implementing facial recognition systems], if not challenged, risks normalizing the idea of being constantly – and opaquely – watched, thus crystallizing a new status quo of pervasive mass surveillance. This is why many from the civil liberties community would have welcomed a much more aggressive policy response by EU institutions to this5. […]

The results from our research are clear: while ADM systems already affect all sorts of activities and judgments, they are still mainly deployed without any meaningful democratic debate. Also, it is the norm, rather than the exception, that enforcement and oversight mechanisms – if they even exist – lag behind deployment. […]

This view ultimately amounts to the assumption that we humans should adapt to ADM systems, much more than ADM systems should be adapted to democratic societies.