It's yet another preview of our future. While these are expert systems vs neural networks, but the issues are similar: the algorithm says that you shouldn't get health care and no one can explain why cause the math is a trade secret and no one understands it even when it's revealed. Then, when finally read out, it turns out that it was written by unqualified lowest bidders and contains 900+ errors in implementation.
Most importantly, when Idaho’s system went haywire, it was impossible for the average person to understand or challenge. A court wrote that “the participants receive no explanation for the denial, have no written standards to refer to for guidance, and often have no family member, guardian, or paid assistance to help them.” The appeals process was difficult to navigate, and Eppink says it was “really meaningless” anyway, as the people who received appeals couldn’t understand the formula, either. They would look at the system and say, “It’s beyond my authority and my expertise to question the quality of this result.”"
There was also no way to effectively challenge the system, as they couldn’t understand what information factored into the changes, De Liban argued. No one seemed able to answer basic questions about the process. “The nurses said, ‘It’s not me; it’s the computer,’” De Liban says.
They always said that learning programming was important, and I suppose they weren't kidding.