Improving role-play reliability, one failed hypothesis at a time
We tested prompt ideas as branches, closed the ones that failed, and kept only improvements that reduced factual mistakes without flattening character voice.
- Checkpoints
- 7 scored checkpoints
- Evaluation
- 20 fixed evaluation criteria
- Published
- August 26, 2026
The goal
Fewer story-breaking mistakes, without safer-but-flatter prose.
We focused on identity, current location, object continuity, character knowledge, player agency, response formatting, and whether the reply actually continued the latest turn.
Privacy and method
- An anonymized set of eight fictional role-play situations.
- The same model, runtime, cases, and scoring rules across branches.
- No conversation text, names, user identifiers, providers, source presets, or source locations are included here.
Fixed scoring rubric
Every branch faced the same checklist.
Higher is better for the five-point scores. Lower is better for severe errors and material misses.
- 1Completely missed or contradicted.
- 2Partial miss with no natural recovery.
- 3Mixed; recovery remains uncertain.
- 4Minor miss that can recover naturally.
- 5Satisfied with no meaningful error.
Test trajectory
Follow the branch, including the failures.
- Phase 01Baseline8 responses
Existing instruction stack
- Changed
- Captured the application runtime as the control condition.
- Observed
- Established the error profile and quality floor for every branch.
Correctness4.32 / 5Role-play quality3.67 / 51.25
Severe errors / response
2.63
Material misses / response
- Phase 02Branch closed8 responses
Consolidated behavioral contract
- Changed
- Replaced overlapping instructions with one behavioral contract.
- Observed
- Prose improved, but correctness regressed. This branch was closed.
Correctness4.25 / 5Role-play quality3.89 / 51.63
Severe errors / response
2.75
Material misses / response
- Phase 03Foundation retained24 responses
Split behavior from story data
- Changed
- Separated stable behavioral rules from mutable story evidence.
- Observed
- Improved correctness and role-play quality together; retained as the foundation.
Correctness4.48 / 5Role-play quality4.00 / 50.75
Severe errors / response
1.67
Material misses / response
- Phase 04ABranch closed8 responses
Minimal behavioral contract
- Changed
- Removed explanatory and style-shaping clauses.
- Observed
- The shorter contract removed useful guardrails without a reliable length benefit.
Correctness4.34 / 5Role-play quality3.88 / 51.50
Severe errors / response
2.25
Material misses / response
- Phase 04BBranch closed8 responses
Resolved state before recent history
- Changed
- Inserted conflict-resolved state before the newest raw messages.
- Observed
- Accuracy rose, but later raw text could reintroduce stale facts. Placement was wrong.
Correctness4.58 / 5Role-play quality4.33 / 50.88
Severe errors / response
1.50
Material misses / response
- Phase 05Best diagnostic8 responses
Resolved state after recent history
- Changed
- Placed conflict-resolved current state after all chronological evidence.
- Observed
- Produced the strongest correctness, quality, and error-rate result in the trajectory.
Correctness4.80 / 5Role-play quality4.50 / 50.13
Severe errors / response
0.75
Material misses / response
- Phase 06Branch closed8 responses
Minimal contract plus final state
- Changed
- Combined the shorter contract with the same final-state placement.
- Observed
- State helped, but missing behavioral guardrails still caused regressions. Branch closed.
Correctness4.62 / 5Role-play quality4.27 / 50.88
Severe errors / response
1.63
Material misses / response
What the trajectory taught us
Four lessons we are carrying forward.
Lesson 01
Prompt architecture mattered
Separating stable behavior from changing story evidence improved correctness and role-play quality together.
Lesson 02
Shorter was not automatically better
The minimal contract removed useful guardrails and did not reliably shorten responses.
Lesson 03
State placement mattered most
Resolved state worked best after recent messages, where it could reconcile every earlier source.
Lesson 04
State did not replace behavior rules
Knowledge limits, ownership, and output rules remained valuable even with accurate current state.
What happens next
Keep the stronger behavioral contract. Build better state tracking around it.
The best checkpoint used a curated final-state layer, so it is a research result—not something we can ship as-is. Our next task is a complete, validated state tracker that can preserve the same gains in normal conversations.