It’s the first time I’ve seen an argument about the blackboxes of AI presented this way. Jonathan Zittrain makes a parallel with drug discovery. In many cases, we don’t now why they work, or we didn’t for a long while. He calls this intellectual debt and frames the AI blackboxes in the same way; what we don’t know about how they work is a debt, and the missing understanding opens the door to clashes with reality, abuses, and complex interactions and consequences.
This approach to discovery—answers first, explanations later—accrues what I call intellectual debt. It’s possible to discover what works without knowing why it works, and then to put that insight to use immediately, assuming that the underlying mechanism will be figured out later. […]
And yet, most machine-learning systems don’t uncover causal mechanisms. They are statistical-correlation engines. They can’t explain why they think some patients are more likely to die, because they don’t “think” in any colloquial sense of the word—they only answer. As we begin to integrate their insights into our lives, we will, collectively, begin to rack up more and more intellectual debt. […]
But intellectual debts don’t exist in isolation. Answers without theory, found and deployed in different areas, can complicate one another in unpredictable ways. […]
Intellectual debt can accumulate in the interstices where systems bump into each other, even when they don’t formally interconnect. Without anything resembling a balance sheet, there’s no way to determine—either in advance or retrospectively—whether any particular quantity of intellectual debt is worth taking on. […]
A world of knowledge without understanding becomes a world without discernible cause and effect, in which we grow dependent on our digital concierges to tell us what to do and when.
More: I’m definitely article around for future use and attaching it with this favourite from last year; Half Life: The Decay of Knowledge. Some of those changes in our knowledge are (or result from) payments of intellectual debt.