A developer building a chat application wants an extra safety layer that classifies both the text a user submits and the text their language model generates, flagging content across categories such as harassment, hate, self-harm, sexual, and violence before either is shown to end users. Referring specifically to OpenAI's Moderation API (the `omni-moderation-latest` model), which statement accurately describes how it is intended to be used?
- It replaces the need for the underlying language model to have any safety training of its own, since all content decisions are made entirely by the moderation classifier before generation even begins
- It can be called on both standalone user-submitted input and on text generated by a model, returning a flagged status plus per-category violation flags and confidence scores; per OpenAI's own documentation, the underlying model still generates its output normally, and the moderation results are meant to be reviewed before that output is shown to a user or acted on
- It can only be used on text a user submits before generation begins, and OpenAI's documentation states it cannot be applied to text a model has already generated
- It is a paid, metered endpoint priced per classified token, separate from OpenAI's chat completion pricing, that developers must budget for separately when adding a moderation layer
Why B? And why not the others?
Correct answer: B. It can be called on both standalone user-submitted input and on text generated by a model, returning a flagged status plus per-category violation flags and confidence scores; per OpenAI's own documentation, the underlying model still generates its output normally, and the moderation results are meant to be reviewed before that output is shown to a user or acted on
OpenAI's Moderation API can classify standalone text a user submits as well as text a model has already generated, returning a binary flagged status, per-category violation flags, and confidence scores for each of its content categories; the documentation is explicit that the underlying generative model still produces its output normally and that the moderation results should be reviewed before that output is shown to a user or used to trigger a downstream action, meaning it is a review layer rather than a generation-time filter. The option claiming it replaces the need for the underlying model's own safety training is wrong because it is described as an additional review step applied around generation, not a substitute for the model's own alignment. The option restricting it to pre-generation user input only is wrong because the documentation explicitly supports scoring model-generated text as well. The option describing it as a paid, metered endpoint is wrong because OpenAI's documentation states the moderation endpoint is free to use.
Source: OpenAI, 'Moderation' guide, platform.openai.com/docs/guides/moderation (redirects to developers.openai.com/api/docs/guides/moderation) -- describes the omni-moderation-latest model, its categories, its use on both input and generated output, and that it is free to use