The Priesthood Model

In The You On AI Encyclopedia

Segal himself provides the evidence for the model's structural weakness with a candor that is both admirable and damning. He describes building a product early in his career that he knew was addictive by design. He understood the engagement loops, the dopamine mechanics, the variable reward schedules. He deployed them anyway, because the technology was elegant and the growth was intoxicating. The downstream effects — teenagers losing sleep, parents finding their children unreachable — took years to appear, and when they did, he was no longer in the room.

This is not an indictment of character. It is an illustration of what happens when incentive structures systematically reward failure of the stewardship ethic. Even well-intentioned builders, operating with genuine expertise and genuine concern, fail, because the market rewards the failure. Growth metrics reward engagement. Engagement rewards addiction. Addiction rewards the engineer who designs the most effective engagement loop. The structure selects for the behavior the ethic prohibits.

The priesthood model's deeper problem is that it resolves the knowledge asymmetry between AI builders and the public by concentrating institutional authority in the hands of those who understand the technology. If the authority derives from understanding that is by definition inaccessible to non-experts, the priesthood is accountable only to itself. The ethic cannot be evaluated by those who lack the knowledge to evaluate the priests' performance. The model becomes self-sealing.

The corrective is not better private judgment but democratic accountability — institutional mechanisms through which the people affected by private decisions can contest, constrain, and redirect those decisions. The institutions must be incrementally built: technical competence within regulatory bodies, structured channels for affected-community input, mechanisms for translating technical claims into language non-experts can evaluate. Each is a modest intervention. Together, over time, they create conditions under which democratic governance can function in the presence of concentrated expert authority.

Origin

Segal introduced the priesthood model in You On AI's chapter on attentional ecology, emphasizing the obligation that accompanies deep understanding. Lindblom's framework does not reject the ethic — it rejects the assumption that the ethic alone is sufficient.

Key Ideas

Sincere but structurally inadequate. The stewardship ethic is genuinely held but overwhelmed by incentive structures that reward its failure.

Self-sealing authority. Locating legitimacy in expertise makes the priesthood accountable only to itself, because non-experts lack the knowledge to evaluate priestly performance.

Structural correction. The alternative is not better ethics but better institutions — democratic accountability that does not require individual moral heroism.

Complement, not substitute. Well-functioning democratic institutions require technical expertise as input. The error is treating expertise as sufficient rather than necessary.