Chapter 8: Gotchas, Part 1

In Chapter 7, we saw how versatile a single LLM call can be. However, treating an LLM like a traditional deterministic backend function (f(x) = y) is a recipe for production outages.

LLMs are probabilistic token prediction engines. Even when you send a single, well-structured request, several subtle failure modes can occur.

Here are the four primary gotchas to defend against in single-shot architectures.

1. Hallucinations (Plausible Confabulation)

The Trap

An LLM has no concept of truth or verification. Its mathematical objective is to sample the most statistically plausible sequence of tokens that follow your prompt.

When an LLM does not know an answer or cannot find an entity in its training weights, it does not throw a 404 Not Found error. Instead, it generates a fluent, highly confident confabulation:

• Inventing plausible-sounding API methods (e.g., client.FetchUserWithMetadata()).
• Citing non-existent legal precedents, medical studies, or CVE identifiers.
• Fabricating package names or configuration flags.

Mitigation Strategies

1. Grounding in Provided Context: Provide the reference documents directly in the prompt and instruct the model to cite only facts from the provided text.
2. Explicit Refusal Contracts: Add an escape hatch in the system prompt:

   "If the provided documentation does not contain the answer, output 'NOT_FOUND'. Do not speculate."

3. Lower Temperature: Reduce sampling randomness (set temperature: 0.0) for factual or extraction tasks.

2. Output Format Inconsistency and Schema Drift

The Trap

When your backend application expects a strict JSON payload, human language models have a habit of decorating their output with conversational pleasantries or markdown formatting:

Sure! Here is the JSON response you requested:

```json
{
  "status": "active",
  "id": 1042
}

I hope this helps!

If your Go service attempts to run `json.Unmarshal()` on that raw response, it will immediately fail with a syntax error.

Other subtle format drift issues include:
- Emitting single quotes instead of valid double quotes (`{'status': 'ok'}`).
- Including trailing commas (`{"items": [1, 2, ],}`).
- Escaping internal quotes improperly.

### Mitigation Strategies
1. **System Prompt Enforcement**: State explicitly: *"Return raw JSON only. Do not include markdown code block formatting (e.g., ```json) or preamble."*
2. **Structured Outputs / JSON Mode**: Use provider-level schema enforcement (such as OpenAI Structured Outputs or Gemini JSON Schema) which constrains decoding tokens at the logits level to valid JSON grammar.
3. **Defensive Parsing in Host Code**: Strip markdown fences with a regex fallback before parsing JSON:

```go
func cleanJSON(raw string) string {
    // Strip ```json ... ``` fences if present
    re := regexp.MustCompile("(?s)```(?:json)?\n?(.*?)\n?```")
    matches := re.FindStringSubmatch(raw)
    if len(matches) > 1 {
        return strings.TrimSpace(matches[1])
    }
    return strings.TrimSpace(raw)
}

3. Token Truncation and the Abrupt Cutoff

The Trap

Every API request allows you to specify a max_tokens (or max_completion_tokens) limit. If the model generates a lengthy response that exceeds this parameter, generation is abruptly severed mid-sentence:

{
  "summary": "The migration completed successfully with 4 warnings",
  "affected_users": [101, 102, 103, 104, 10

Because the output is cut off, any downstream JSON parser will fail.

Mitigation Strategies

1. Inspect finish_reason: Always check the metadata returned by the provider. If finish_reason == "length", the response was truncated and should be treated as an error or retried with a higher limit.
2. Size Buffers Appropriately: Ensure max_tokens provides ample headroom for complex outputs.
3. Chunking: For large extraction tasks, break input text into smaller segments rather than requesting one massive response.

4. Temperature, Top-P, and Determinism

The Trap

Developers often assume setting temperature: 0.0 turns an LLM into a fully deterministic function.

While temperature 0 selects the highest-probability token at each step (greedy decoding), modern distributed LLM inference is not 100% bit-for-bit deterministic across runs.

Why does nondeterminism still occur at temperature 0?

• Floating-Point Non-Associativity: High-throughput GPU inference clusters split matrix multiplication across thousands of parallel CUDA threads. The exact order of floating-point addition operations varies slightly between GPU nodes and batches, which can occasionally flip token probabilities on closely tied logits.
• Model Quantization and Routing: Mixture-of-Experts (MoE) architectures and dynamic inference routing may route identical queries through slightly different hardware configurations.

Mitigation Strategies

• Treat LLM outputs as semi-deterministic.
• Never write tests that assert exact string equality (assert response == "Expected string").
• Assert against semantic properties, structured schemas, or valid JSON keys instead.

Now that we understand single-call capabilities and their failure modes, we are ready to take the next major leap in agent engineering: chaining multiple calls together.

In Part 2, we explore Multi-Turn Interactions & Loops.