‹ Culture 3.14 · What we ask ourselves
Question · Physical Intelligence
What does robotics teach about silent software failures?
Robotics has spent decades assuming that a component can work and still produce no result at all; business software still discovers it case by case.
Why it matters
We write software that acts on ERPs, on processes and, lately, on machines. In all three the same failure appears: the execution was correct and the effect did not happen. A system does not only fail when a component breaks; it also fails when every component meets its contract and the whole produces a result nobody wanted.
Robotics has taken that for granted for decades and designed accordingly. Leveson frames it as a problem of control and of absent constraints; Thrun, Burgard and Fox start from the premise that the state of the world is estimated and never read. In business software we go on discovering it project by project.
What we know so far
We have stopped treating the record of an action as proof of its effect. We distinguish executed, confirmed and observed, and when there is no independent signal we declare it as a limit instead of taking the action as checked.
The transfer from robotics comes with a condition. A robot operates in an environment it can instrument with its own sensors; a process over an ERP depends on third-party systems that do not always offer a verification signal. Copying the method without that possibility produces a false sense of rigour.
What remains open
It remains open which part of the robotics method — state estimation, explicit constraints, independent verification — can be sustained when the signal has to be created on purpose and costs more than the action itself.
And how far the analogy reaches in processes that cross several applications, where the state of the world is the sum of records that no part of the system observes in full.