Moderation at scale is rarely a question of removing the worst content. The hard problem is everything in the middle: posts that are borderline, accounts that might be sincere, behavior that might be a misunderstanding. Treat that middle as a binary and you will be wrong half the time. Treat it as a ranking problem and you have a chance.
The teams that scale stop asking whether content is good or bad. They start asking how confidently a model can place it on a spectrum, and what to do at each point on that spectrum.
From verdicts to dispositions
Verdicts are brittle. A verdict says yes or no, and it is wrong whenever a model is uncertain. Dispositions are softer. A disposition says: surface this immediately, surface this after a delay, surface this to fewer people, route this to a reviewer, withhold this entirely.
Each disposition has a cost and a benefit, and the right disposition depends on the confidence of the underlying classifier and the cost of being wrong in either direction.
The architecture that follows
Once you adopt dispositions, the architecture inverts. Classifiers become signal sources, not decision-makers. A central orchestrator weighs the signals and chooses an action. Reviewers become a service consumed by the orchestrator, not a manual override.
The result is a system that gets quietly better as classifiers improve, without the architectural rewrite that a verdict-based system would need.