Chapter 2

The same prompt can produce different answers

In traditional finance software, identical inputs always generate identical outputs.

If you refresh a spreadsheet containing an interest rate formula, the cell never displays 5.2% on Monday and 5.4% on Tuesday. If your accounting system runs an SQL query on last year's ledger, the total revenue figure remains identical to the penny across ten thousand consecutive runs.

When analysts move from financial software to large language models, this foundational certainty shatters.

You can paste the exact same earnings release into the exact same prompt window, click submit three times, and receive three noticeably different summaries. Most of the time, the wording merely shifts. But occasionally, a crucial financial digit drifts: an operating margin changes from 15.0% to 14.8%, or a bullet point disappears entirely.

This run-to-run variation is not a software crash. It is an observed mathematical consequence of how the machine selects its words.

Sampling draws candidate tokens from a probability bag

In Chapter 1, we learned that a language model computes the conditional probability of every possible next token. But computing probabilities is only half the job. The model must actually choose which token to emit.

How does the machine make that choice?

Think of the candidate tokens as poker chips inside a bag.

Close

When the model finishes reading the phrase Acme reported fourth-quarter revenue of, its neural network calculates a probability distribution across its entire vocabulary:

• The exact number from your text ($4,280) gets 82 chips.
• A rounded approximation ($4,200) gets 11 chips.
• A transposed number ($4,820) gets 5 chips.
• Other candidate tokens share the remaining 2 chips.

To select the next token, the harness reaches into the bag and draws a single chip at random.

Eighty-two times out of a hundred, the machine draws the correct number. But eighteen times out of a hundred, it draws an alternative chip. Once that alternative token lands on the page, the entire mathematical context shifts. Every subsequent token is now conditioned on that altered choice.

The dial that controls this selection process is called temperature:

When temperature is high, the neck of the bag widens. Unlikely tokens receive more chances to be drawn, making the writing colorful and varied. When temperature is low, the top-ranked token dominates.

Zero temperature narrows the draw but does not guarantee identical runs

When financial analysts first discover temperature, their immediate instinct is simple: Turn the dial to zero.

Setting temperature to zero instructs the model to use greedy decoding: always select the single candidate token with the highest mathematical probability. Never draw from the rest of the bag.

For simple qualitative drafting, greedy decoding reduces variation. But in enterprise environments, leaders make a dangerous assumption: they assume that zero temperature makes an LLM as deterministic as an SQL database.

It does not.

In production architectures, identical prompts sent with zero temperature still exhibit run-to-run drift for three technical reasons:

1. Floating-point non-determinism across parallel chips: Frontier models run across clusters of dozens of specialized graphics processors. Because floating-point addition in high-speed parallel silicon is non-associative, minor variations in thread execution order slightly alter the final logits. When two candidate tokens sit within 0.0001% probability of each other, microscopic rounding noise flips the winner.
2. Dynamic batching on server clusters: Enterprise API endpoints group concurrent queries from different users into dynamic computation batches to maximize throughput. Changing batch dimensions alters GPU matrix scheduling, shifting sub-word probability boundaries.
3. Silent model updates and weight quantization: Frontier model providers continually update internal routing weights, system prompts, and safety filters without incrementing version labels. A prompt run on Friday morning may run against different model artifacts than the identical prompt run on Monday afternoon.

In business and finance, you cannot eliminate variance by configuring a single software parameter. You must design workflows that verify outputs regardless of variation.

Document:
"Acme Industrial Technologies (NYSE: AIT) delivered a resilient finish to the fiscal year despite persistent supply chain friction in European markets. For the fourth quarter ended December 31, consolidated revenue reached $4,280 million, a 6% increase compared to $4,038 million in the prior-year period. Operating income expanded to $640 million from $595 million, reflecting price realization across heavy equipment lines. Full-year backlogs closed at record levels, though management noted cautious capital expenditure trends among regional industrial customers entering the first quarter."

Instructions:
Summarize the fourth-quarter performance for Acme Industrial Technologies.
State the total revenue and the year-over-year growth rate.
Use two concise sentences.