Lots of yes in this piece by Fabien Girardin at the Near Future Laboratory. “The business of dishonest automation and how the engineers, data scientists and designers behind it can fix it.” Missing context, forgetting (ignoring) edge cases, being dishonest, misleading users, stealing attention, lost trust. Notice item 4, The design for humane automation, with learning from the past, critiquing the present, and debating the future—including design fiction, which the lab has done very well for years.
Learning techniques becomes widespread, digital automation is becoming a commodity with systems that perform at Internet scale one task with no deep understanding of human context. […]
In the light of these examples of clumsy and dishonest automation, what concerns me is that many engineers, data scientists, designers and decision-makers bring these frictions into people’s everyday life because they do not employ approaches to foresee the limits and implications of their work. Apart from the engineering of efficient solutions, automation requires professionals to think about the foundations and consequences of their practice that transcend any Key Performance Indicator of their organization. […]
Many professionals in the tech industry (including me) embraced his description of Calm technology that “informs but doesn’t demand our focus or attention.” However, what Weiser and many others (including me) did not anticipate is an industry of dishonest automation or solutions that turn against their user’s intentions when things do not go as planned.”