Working with LLMs

Limitations of Prompting

---

What prompting cannot do

Throughout this part, you’ve learned powerful prompting techniques: writing clear instructions, providing context, using examples, encouraging reasoning, and chaining prompts. These techniques enable impressive applications, but they have fundamental limits.

Understanding what prompting cannot accomplish is as important as knowing what it can do. This chapter explores the boundaries of prompting, helping you recognize when alternative approaches are needed.

---

Fixed knowledge cutoff

Providers may report a knowledge cutoff, indicating the time covered by a model’s training data. This does not guarantee complete or accurate knowledge of everything before that date.

During ordinary inference, the model’s parameters stay fixed. However, it can use new facts supplied in the prompt or retrieved by tools. Using those facts as context is different from updating the model through training.

For example, a model with a January 2025 cutoff cannot reliably report a March 2025 election result from its training alone. Supplying a trustworthy report can let it answer from that evidence. Rephrasing a question without adding new information does not provide the missing result.

Solutions: Use Retrieval-Augmented Generation to provide current information in prompts, choose models with recent cutoff dates, or accept the limitation for tasks involving historical or timeless information.

---

Hallucination

Language models sometimes generate content that sounds confident and plausible but is factually incorrect — called hallucination. Prompting can reduce this risk, but an instruction to be accurate is not a guarantee of correctness.

Models are trained to predict likely text continuations, not to verify truth. They learn patterns like “The capital of [country] is [city]” and fill these based on statistical regularities, without reliable mechanisms to distinguish facts they know from patterns they’re extrapolating.

Common hallucinations include fabricated citations with realistic-looking but nonexistent details, false specifics (numbers, dates) presented authoritatively, and confident errors in well-structured text showing no uncertainty.

What prompting can help with: Instructions like “If you’re not certain, say so” encourage expressing doubt. Requesting sources or asking the same question multiple times can reveal inconsistencies.

Limitations: A model’s own explanation or repeated answer does not independently verify its claims. Errors depend on training, available evidence, evaluation incentives, and the system around the model, not just its architecture. Research on hallucination distinguishes imperfect accuracy from inevitable false claims: abstaining can avoid an unsupported answer, although deciding when to abstain remains difficult in practice.

Solutions: Verify important information independently, use RAG for factual tasks to ground responses in retrieved documents, or accept that tasks requiring perfect factual accuracy may need different tools.

---

Reasoning limitations

While chain-of-thought prompting improves reasoning, models still struggle with certain logical and mathematical reasoning that humans handle easily.

Difficult reasoning types: Complex multi-step calculations, formal logic with many constraints, spatial reasoning about three-dimensional arrangements, causal reasoning distinguishing correlation from causation, and counterfactual “what if” scenarios.

Chain-of-thought prompting can change performance by giving intermediate calculations space in the context, without updating the model’s parameters. It still provides no guarantee of valid logic or correct arithmetic.

Solutions: Use specialized tools (calculators, code execution, formal verification), newer reasoning models designed for these tasks, or human verification for critical reasoning.

---

Limits of learned representations

Predicting tokens can produce useful internal representations. For example, Othello-GPT experiments identified representations of the board state in a model trained on game sequences. This does not establish reliable understanding of the wider world. Models can still apply learned relationships incorrectly or fail on situations outside their training experience.

Practical implications: Models produce subtle conceptual errors humans would catch, miss obviously problematic issues, and work well within training distribution but fail unpredictably on unusual inputs.

Solutions: Do not infer human-like understanding, intentions, or reliability from fluent language. Evaluate the model on the actual task, including unusual cases, and use appropriate human review for consequential decisions.

---

Context window constraints

Models can only process limited text at once — their context window. While modern models handle long contexts, limits exist.

Extremely long documents may exceed capacity. Prompt chains passing forward all outputs can hit context limits. Longer contexts cost more and process slower.

Solutions: Chunk large documents and process separately, store information externally and retrieve selectively, or choose models with larger context windows.

---

Consistency and reliability

When generation samples from a probability distribution, identical prompts can produce different outputs. Sampling randomness is optional: greedy decoding selects a highest-scoring token at each step and makes a deterministic choice for fixed scores and tie-breaking rules. Numerical differences in computation, serving infrastructure, or model-version changes can still affect reproducibility. Removing sampling does not guarantee identical responses from a hosted service.

Implications: Testing is harder, prompts that worked in testing might occasionally fail in production, and intermittent issues are difficult to reproduce and fix.

What helps: Record the model version and generation settings. Lower temperature concentrates sampling on higher-scoring tokens; where available, greedy decoding removes sampling randomness. Neither setting makes an answer more truthful by definition.

Solutions: Design systems handling variable outputs gracefully, validate outputs before use, or accept that applications requiring perfect consistency may need different approaches.

---

Training data biases

Models learn patterns from training data, including biases. Prompting can mitigate but not eliminate these.

Types of biases: Representation bias (some groups overrepresented or underrepresented), association bias (stereotypical connections learned from text), recency bias (recent texts potentially overweighted), and source bias (particular sources dominating training data).

Solutions: Use prompts to nudge outputs towards a more balanced perspective, but don’t rely solely on this. Human review for sensitive applications, diverse testing to identify bias patterns, transparency about limitations rather than claiming models are unbiased.

What prompting cannot fix: Deep patterns in training data persist despite instructions. You can only prompt against biases you’re aware of. Systematic gaps in knowledge cannot be conjured.