Managing Prompts Like Code: Versions, Tests and Rollback

A prompt is the instruction text an application sends to a large language model (LLM) along with the user's input, and it decides much of how an AI feature behaves. In many products it lives in a database row that anyone with admin access can edit, with no record of what changed or why. When the output suddenly goes wrong, nobody can say what was edited, when, or by whom. This post explains how to manage prompts the way teams already manage code, with a file, a history, a test, a reviewer and a way back.

Why a prompt change is a release

Picture a coaching institute with a feature that drafts replies to parent queries on WhatsApp. The prompt behind it sits in a database row, and the operations lead edits it from an admin screen whenever a reply sounds off. One Tuesday the drafts start promising a refund policy that does not exist, and there is no history to look at.

The cause is easy to understand once you see how much a prompt controls. An edit to one sentence in a system prompt (the standing instructions sent with every request) can change the tone of every reply, drop a field from every extraction, or make the model start answering a question it used to decline. The effect reaches the whole feature at once. Code with that reach goes through version control, review and tests. Prompts often skip all three because they are "just text", and the person editing one in production is only doing what the admin screen invited them to do.

Treating the prompt like the code it sits next to gives you three things you cannot get any other way: a diff that shows exactly what changed, a history that shows when, and a commit author who can explain why.

What one prompt version contains

The prompt text alone does not determine the behaviour. Here are the four parts that together decide what the feature does, and so the four parts that should be versioned together.

• Prompt textthe file, with placeholders
• Model and settingsmodel id, temperature, limits
• Output contractschema the app validates
• Evaluation setsaved inputs, expected results

Change any layer and the result is a new version.

1. The prompt text, with placeholders for the variable parts such as the parent's message or the institute's fee policy.
2. The model identifier and its settings. Temperature controls how varied the output is, and the output limit caps its length. If the model can call tools, their definitions belong here too.
3. The output contract, meaning the schema the application checks the response against, if the prompt asks for structured output such as JSON.
4. The evaluation set: saved real inputs, each with the result you expect.

A prompt tuned on one model and then run on the provider's next release is a different system, even when the text is identical. That is why the model identifier travels with the text.

Storing prompts as files in the repository

Store each prompt as its own file in the repository, next to the code that sends it. A directory per feature, a file per prompt, and a loader that reads them at build time is enough. Avoid prompts assembled from string fragments scattered across several modules, because the thing you need to compare between versions then never exists as one piece.

Here is what one such file, prompts/parent-replies/draft-reply.yaml, could look like for the coaching institute's reply feature. The model name is a placeholder.

model: example-model-2026-01-15   # exact id, never an alias like "latest"
settings:
  temperature: 0.2
  max_output_tokens: 400
output_schema: schemas/draft-reply.json
eval_set: evals/parent-replies.jsonl
template: |
  You draft WhatsApp replies to parents for {institute_name}.
  Only mention fees, batches and policies that appear in {policy_text}.
  If the answer is not in {policy_text}, say a counsellor will reply.

If you are still learning how to write the instructions themselves, Prompt Engineering 101 covers that side. This post is only about keeping track of them once they exist.

How a prompt change reaches production

A reviewer reading a prompt diff cannot tell whether the new wording is better. Only running it can. So the change follows the same path a code change takes, with one extra stage. Here is how it works.

1. Someone edits the prompt file and opens a pull request.
2. An automated job runs the evaluation set through both the old and the new prompt.
3. The reviewer reads the differences in outputs, rather than only the differences in wording.
4. If the outputs are acceptable, the change is merged and deployed.
5. When a wrong output turns up in production later, that input is added to the evaluation set.

1. 01Edit fileone change
2. 02Pull request
3. 03Run eval setold and new prompt
4. 04Review outputsnot wording
5. 05Merge, deploy

Failures join the eval set

The same path as a code change, with output comparison as the test stage.

The evaluation set does not need to be large to be useful. For the coaching institute, twenty real parent messages would be a good start, including the ones in Hinglish and the one asking about a batch that no longer runs, each with the reply a counsellor would have approved. Twenty real inputs are worth more than two hundred invented ones, because invented inputs tend to be polite and well formed, and real ones are not.

Results are judged against written criteria: required fields present, no invented details, length within bounds, tone as specified. Some of these checks can be automated with exact matching or a schema validator. Others need a person reading a sample. Record both. Once offline checks pass, running a new prompt alongside the old one on live traffic is the next level of safety, and shadow runs and canaries explains how that works.

Pinning the model and rolling back

Providers retire and replace models regularly. Pin the exact model identifier in the prompt file, not an alias such as "latest", so that a model change is a deliberate commit that runs the evaluation set, instead of a surprise on a Tuesday. When you do move to a new model, run the whole set first, fix the prompts that regress, and ship the model change and the prompt changes together. For an open-weight model you host yourself, the same rule applies to the weights and the serving configuration: record a checksum or tag alongside the prompt.

Ownership works like any other code. Prompt files have a named owning team, a code owners rule and a required review. The operations lead and the counsellors should still be able to propose changes, and a pull request is a good place for that, because the evaluation results appear next to the wording and the discussion about whether a reply "sounds right" happens with evidence in view.

Rollback is then a revert. Because the model identifier travels with the prompt, reverting restores the whole behaviour. Log the prompt version with each request in production, so that for any reply someone complains about, you can see exactly which version produced it.

Common mistakes with prompt changes

The first is the one this post started with: prompts edited in an admin interface in production, with no history, because it was quicker. The second is an evaluation set that exists but never runs, because someone has to trigger it by hand and nobody does. Versioning the prompt text but not the model identifier is the third, so behaviour drifts when the provider updates and the diff shows nothing. Reviewers sometimes read the wording of the prompt and never look at the change in outputs. And logs often store the output without the prompt version, so a bad reply cannot be traced back.

Where to start this week

1. Find every prompt in the product and move each into its own file in the repository.
2. Add the model identifier and settings to each file.
3. Collect twenty real inputs for the most used prompt and write down the expected result for each.
4. Make the evaluation run automatically on every pull request that touches a prompt file.
5. Log the prompt version with every request in production.

Step one comes first even when the admin screen is popular with the people who use it, because review and rollback both depend on the prompt existing as one thing with a history. Moving prompts into files is ordinary engineering work, the same kind described under generative AI development.