My LLM Pipeline Never Crashed. It Just Quietly Got Worse.

Normal software fails loudly. My article pipeline failed with HTTP 200 and valid JSON.

When a normal app breaks, it tells you.

A webhook is missing a field. A database constraint trips. An exception bubbles up, and the stack trace points at the exact line.

Annoying? Sure. But at least you know.

I've been building a pipeline that writes blog articles for real sites. A planner writes a brief, a writer drafts the article, an editor revises it. Each stage hands the next one its instructions.

This week I found this inside one of those briefs, in a section plan for an article about one of my own case studies:

Use note b51d42cf-cf42-4c28-ae

That's it. That's where the instruction ends.

Half a UUID. No error. No alert. Every check passed.

So how do you debug a bug that looks exactly like success?

Half a UUID, zero errors

Here's what happened.

The planner returns its brief as structured JSON. To keep briefs tight, the schema had length limits: 280 characters per outline point, 240 per note for the writer, 155 for the meta description, 1,600 for the brief itself.

Reasonable, right? Here's roughly what we sent the model (simplified):

outline: {
  type: "array",
  items: { type: "string", maxLength: 280 },
}

With OpenAI's strict structured outputs, the model doesn't fail when it hits a maxLength. In our runs, it just stopped the string. Mid-sentence. Sometimes mid-word.

Here's the outline point from the hook, exactly as the planner returned it (wrapped here so you can read it). It's 280 characters:

## Sniffo.ai separates collected inputs from
scoring decisions (220 words): Use note
4e7d7a7c-1c6e-422b-b865-4953727d9931 to
explain SourceItem rows, the separate
Opportunity mapper, business-profile
application, and re-scoring without network
work. Use note b51d42cf-cf42-4c28-ae

A few more from the same two weeks:

"…a product can assess separatel"

"…or validate the planned duration through a complete"

The response was valid JSON. Our Zod schema checked the length, and the length was fine (it was exactly the limit). So the repair step we had for bad briefs never ran.

Then the writer got its instructions with the ending missing. And the ending is usually the important part: which note to use, what the section should prove, how it should land.

The writer doesn't complain either. It works with whatever it got.

So I counted. Two weeks of briefs, 14 of them, across 6 sites:

• 23 of 120 writer notes stopped at exactly 240 characters.
• 7 of 96 outline points stopped at exactly 280.
• 11 of the 14 briefs had at least one field cut off.

Eleven out of fourteen.

Nothing crashed. Not once.

The guards were the bugs

Once I started looking for this shape, I kept finding it. Same week, same pipeline:

The link check removed every internal link. After a prompt update, the planner started writing links as bare slugs (/some-post) instead of real paths (/blog/some-post). The link check couldn't match them, so it did its job and unlinked them. All of them. Every target page actually existed. The drafts came out with orphan sentences like "read Anno 1800 vs Farthest Frontier." pointing at nothing.

A capital letter threw away a whole edit. The editor capitalized a link's anchor text. The check that makes sure the editor didn't break links compared links by their text, saw a "changed" link, and discarded the entire editorial pass. The article kept its unedited draft.

Thai text blew the token budget. The budget assumed English-like characters per token. A Thai draft of 12.4k characters is about 11k tokens, against a 9k budget. The editor's reply came back as invalid JSON, so the edit was skipped. Silently.

A word count halved the articles. The new prompts asked for a target word count in each brief. New drafts came in at 722–1,133 words. The old ones ran 1,128–2,193. The worked examples, the most useful part, were what got cut. No error. Just thinner articles. They lost a side-by-side comparison against the old prompts, 4–1.

Look at those again.

Every one of them is a safety check doing exactly what it was built to do. Too long? Cut it. Link doesn't match? Remove it. Edit looks wrong? Throw it away.

Each guard was built to keep the pipeline running.

So when something went wrong, it didn't stop. It deleted something and kept going.

That's not graceful degradation.

That's silent data loss with good manners.

More rules made it worse

The obvious move when an AI gets something wrong is to add a rule.

I tried that. It backfired twice.

First try: the platform prompts were full of "never claim X" and "don't add Y" lines. Those lists planted the exact words they were warning against. My own site's first generated article put a disclaimer on my own 30% result, because the prompts were full of warnings about numbers. Rewriting every rule as "do this" instead of "never do that" took the avoid-lines from 46–86 per site down to 0–3.

Second try: some articles had worked examples with numbers that went past real limits. So I added a "real limits" rule to the writer, the planner and the prompt generators. The planner turned it into "numerical examples must be supported by item data." One article dropped its example numbers completely. Others started hedging ("explicitly illustrative"). It lost the blind test to the version without the rule, 3–2.

Same lesson both times.

Tell the writer to be careful, and it gets vague.

What worked was moving the check out of the writer. The editor now walks through each worked example, checks it against the article's own rules, and fixes the result from the example's own inputs. Same drafts, blind comparison: 4 wins, 1 tie, 0 losses.

Let the generators write. Make a separate stage check.

What replaces the stack trace

If errors won't tell you, you need other signals. This is what I use now.

Look for outputs sitting exactly on a limit. One output at exactly 280 characters is a coincidence. A pile of them is truncation. It's one short query, and it's what showed me how big this was:

select
  count(*) filter (where length(note) = 240) as at_limit,
  count(*) as total
from content_briefs b,
  jsonb_array_elements_text(b.brief_json -> 'notesForGenerator') as note
where b.created_at > now() - interval '14 days';

If at_limit isn't close to zero, the model isn't finishing its sentences.

Don't put limits where the model can cut. The fix removes maxLength from the schema we send. Each field's description says "At most N characters." Our own code checks the length after the call and runs one repair pass if it's over:

function textSchema(limit: BriefTextLimit): JsonSchema {
  // No maxLength: strict outputs would stop mid-word to obey it.
  return { type: "string", description: `At most ${limit.max} characters.` };
}

// After the call: our own Zod check. Over the limit? One repair pass
// asks the model to shorten that field.

The limits also went up to what the writer actually needs (600 for an outline point, 800 for a note). Replayed on six sites first: nothing landed on a limit, and the one meta description that ran long (160) was repaired to 144. It shipped the same day. The first two real briefs after it had nothing on a limit, and notes ran up to 593 characters. That's how much the writer was missing.

Make links impossible to get wrong instead of checking them after. The planner now picks links by ID (L1, L2…) from the same list the link check uses, and code turns the IDs into paths. On replay, 14 of 14 links reached the page they were meant for.

Replay before you ship. Every prompt or schema change gets replayed offline against stored briefs and drafts first. It's cheaper than finding out on a customer's site, and it's the only way to see every case at once instead of one symptom per bug report.

Compare blind. Old output vs new, labels hidden. If you know which one is "the fix", you'll find a reason to like it.

Blind isn't enough on its own, either. In one model bake-off, I had three judges rank the same articles, labels hidden and order shuffled. Claude's judge put the two Claude-written articles first. GPT's judge put the GPT article first. Gemini, the only one with no horse in the race, sided with Claude.

Nobody told them who wrote what. They just liked the sound of their own voice. (Relatable, honestly.)

That's one topic, so it proves nothing. But if you use AI judges, use more than one company's.

Where this approach falls short

I don't want to oversell this.

Five-article blind tests are a reading, not a measurement. They caught real regressions. But 4–1 on five topics wouldn't survive a statistics class. They tell you which way to lean, not how big the effect is.

I still can't see how an article actually performs. Right now, a prompt change gets judged on how the drafts read, not on how they do in search. Keeping every version of every article (the first step toward that) is still being built.

Strict schemas are still worth it. Structured outputs mean I never parse broken JSON, and that's a big deal. The limit was the problem, not the schema.

Some failures are loud, and that's fine. Auth, billing and database errors still throw, still hit Sentry, still alert me. Normal engineering still applies. It just doesn't cover the part that matters most here: whether the article got worse.

What I'd do in your pipeline

If you're chaining LLM calls, check these this week:

• Remove maxLength from any schema a model fills in. State the limit in the description and check it in your own code.
• Count outputs that land exactly on a limit. That number should be close to zero.
• Find every guard that deletes or skips on failure. Make each one log what it dropped, with a counter you actually look at.
• Count token budgets in tokens, not characters, and test them with a non-Latin language.
• When the model gets something wrong, fix it in a checking stage, not with another rule for the generator.
• Replay stored inputs and compare blind before every prompt change.

So why is this harder?

Here's my honest answer.

It's not that LLMs are random. The truncation was perfectly deterministic. Same limit, same cut, every time.

It's that "correct" stopped being something code can check. A normal system is either right or it throws. A pipeline like this has a third state: valid, passing, and worse.

Loud failures vs quiet ones.

Every guard I added made the pipeline safer from crashing, and better at hiding its mistakes.

In an LLM pipeline, HTTP 200 doesn't mean it worked.

This pipeline is NichePressa, the content engine I built and run on my own sites.

Running LLM calls in a chain? I'd love to hear what's quietly breaking in yours, and how you caught it. Send me a note.