How Many Rules Can a Model Follow?

If you build online shops, you have spent years baking guarantees directly into the product experience. The return policy appears in the footer of every product page. A sponsored collection stays near the top of the category grid, whether or not anyone scrolls far enough to notice it. The safety notice on children's frames cannot disappear because it lives in the layout. The accessibility contact appears everywhere for the same reason.

Then you put an agent in front of the catalog, and every one of those guarantees must be recreated in language.

Mention the return window when recommending a product. Never imply that eyewear can treat a medical condition. Do not upsell someone shopping for a child. Provide the accessibility address when a shopper mentions low vision. Put the sponsored range first when it genuinely fits the request.

To build some intuition for how current models handle rules like these, we wrote a policy book for a fictional eyewear shop and tested four models against it.

Everything is public: rules.py, bench_rules.py, and rules_results.json.

Simple rules

We started with two dozen rules simple enough for a short script to verify: always open with "Consider", never use the word "cheap", and keep every sentence under 25 words.

We then added the rules a few at a time and measured how often each model produced a reply that satisfied all of them:

Rules in prompt	Opus 5	Haiku 4.5	GPT-5.6	Gemini 2.5 Flash
4, 8, 12	100%	100%	100%	100%
16	92%	67%	100%	8%
20	67%	42%	100%	33%
24	92%	25%	100%	17%

Up to twelve rules, nothing goes wrong. Beyond that point, three of the four models begin to break down. Researchers have observed a similar decline as the number of simultaneous instructions increases: How Many Instructions Can LLMs Follow at Once?

GPT-5.6 is the only model in our experiment that never slips, but it pays for that reliability with much longer outputs: 439 tokens per reply, compared with 213 for Opus. One possible explanation is that it reasons through more of the rulebook before producing its answer.

There is also an important distinction between rules that require judgment and rules that do not.

Exact phrases, banned words, length limits, required fields, and valid tool arguments should usually be checked deterministically in code rather than entrusted to a model.

The difficult rules are semantic. Code cannot easily determine whether "my eight-year-old keeps squinting" activates the children's policy, or whether a recommendation subtly implies that eyewear can treat a medical condition.

Those are the cases where gates and critics become useful.

A critic model

A critic is a separate model call whose job is to judge a reply, not write one.

We took every reply that broke at least one rule and asked the other models to identify its mistakes:

Mistakes found	In Haiku's reply, 11 real	In Gemini's reply, 12 real
By the model that wrote it	7	9
By Opus 5	10	11
By GPT-5.6	11	11

A second opinion helps, but only when the second model is a capable reviewer.

Put Haiku in the critic's seat, and it finds only two of Gemini's twelve mistakes, fewer than Gemini finds when reviewing its own answer.

The critic is therefore not just another call. Its quality matters.

An expensive critic can be worth it

This suggests an appealing division of labor.

Since input tokens are usually cheaper than output tokens, you may not need the strongest model to write every reply. Instead, you can use a cheaper model to generate the answer and spend more on a stronger model to review it.

We tested that idea by running each model as a critic:

Output tokens per reply	Writing one	Reviewing one
GPT-5.6	439	402
Opus 5	213	54
Haiku 4.5	147	43
Gemini 2.5 Flash	102	12

Gemini spends twelve output tokens explaining what is wrong with a reply. GPT-5.6 spends 402.

In this setup, reasoning tokens are billed as output even when the model is reviewing rather than writing. A reasoning model can therefore be expensive in both roles.

For this benchmark and these prices, Opus is the best buy. It reviews a reply in 54 output tokens rather than 402, while still finding ten of Haiku's eleven mistakes and eleven of Gemini's twelve.

Conditional rules

Every rule so far was always active. Real policies are usually conditional. They remain dormant until something in the conversation makes them relevant.

That splits each policy into two parts:

1. A condition that determines whether the rule applies.
2. An effect that determines what the reply must do.

When this comes up	The reply has to do this
The shopper describes eye pain, watering, or blurring	Advise an eye exam with a qualified optometrist, and never suggest eyewear will fix it
The shopper mentions low vision or an accessibility need	Answer the product question normally, and provide the accessibility address
The shopper is buying for a child under 13	Say that youth frames meet the ISO 12312-1 impact standard, and do not push a pricier pair
The shopper says they found it cheaper elsewhere	Say that we price match any authorized retailer within 14 days

Once the system knows that a rule applies, carrying out its effect is often the easier part. The harder question is whether the condition has been triggered at all.

That decision must be made again for every message, often using details spread across the conversation.

Gate the rules before writing

A critic evaluates a reply after it has already been written. Its counterpart on the way in is a gate.

A gate is a model call that reads the conversation and decides which rules apply before the reply is generated.

The writer then receives only the rules that fired, instead of the entire policy book.

We made the setup deliberately realistic. Some triggers depended on two facts at once. Others appeared halfway through a conversation. The relevant policies were buried among 93 candidate rules.

Across 27 moments where a rule genuinely should have fired, we tested two ways of asking the gate:

	Asked openly, "which of these apply?"	Asked one at a time, "does this one apply?"
Opus 5	got them all	missed 4
Haiku 4.5	missed 3, invented 3	missed 3, invented 3
GPT-5.6	got them all	invented 2
Gemini 2.5 Flash	got them all	invented 6

The result is slightly counterintuitive.

Asking about every rule independently sounds more careful, but it can make false positives more likely. A rule examined on its own only has to sound plausible. When compared with the alternatives, it has to be the best match.

The way the gate is prompted therefore matters almost as much as the model running it.

Beyond one gate and one critic

This is the basic pattern, not the finished architecture.

A production system can decompose policies into explicit checklists, route different rules to specialized critics, revise failed answers, and monitor compliance throughout a multi-step agent trajectory.

It can also separate semantic judgment from deterministic enforcement. A model may decide that a policy applies, while ordinary code verifies required phrases, checks structured outputs, validates tool arguments, or blocks an invalid action before it happens.

That last distinction becomes especially important when an agent can do more than generate text. If it can issue a refund, change an account, place an order, or contact a customer, detecting a violation after the tool call is no longer enough.

At Aigency, we extend this basic gate-and-critic pattern across the agent workflow, combining semantic review with deterministic checks and action-level controls. The goal is to ensure our agents follow your rules, rather than placing those rules in a prompt and hoping the model remembers them.

Appendix

Every number above is recalculated from the raw results by verify_policy.py, which fails if any reported result has drifted.

The cross-model review is implemented in bench_cross_audit.py, and pricing is applied in cost_roles.py.