What an automatic model evaluation job actually tells you

You ran an automatic evaluation on a foundation model and got three numbers back. Should they decide your launch?

This article argues that every score an automatic evaluation job reports measures a proxy, never the thing you care about. Running the job is the easy part. Reading it well means knowing which proxy each number stands for and what it cannot see. The walkthrough below follows the decisions in the order you make them. It names one failure you can trigger deliberately, and it shows where a strong model score stops being evidence of anything.

An automatic job only works when a reference answer exists

Before you open any evaluation screen, answer one question: does a correct reference answer exist for this task? That question decides whether automation can help you at all.

An automatic evaluation runs a model against a dataset of prompts with known good answers. It then computes scores with no human in the path. A benchmark dataset is a standardised set of tasks of this kind, and its value is that it is repeatable. Two models measured the same way can be ranked against each other.

When no reference exists, nothing is available to compare against. Examples include judging whether a refusal is worded appropriately, or whether an answer has the right tone. That work needs human-in-the-loop evaluation, where people read outputs and score them. Where human review cannot keep up with the volume, it needs LLM-as-a-judge, where one foundation model scores another against stated criteria.

Whether a reference answer exists decides if an automatic job can score the task

Amazon Bedrock Model Evaluation, the managed service this article describes, runs both kinds of workflow. In automatic mode it scores outputs against built-in or custom datasets. In human mode it manages a review workflow staffed by your own work team or a managed one. Choosing the service therefore does not choose your method. You still have to make that choice yourself.

Setting up the job comes down to four choices

You can treat the setup of an automatic job as a short series of decisions. The exact screens and command syntax are not covered here, but the decisions are the same whichever route you take.

1. The model under test. This is the foundation model whose outputs will be scored. If you are comparing candidates, run each one against the same dataset so the scores can be compared.
2. The mode. Choose automatic, because your task has reference answers. If it does not, return to the previous section.
3. The dataset. This is either a built-in benchmark or your own prompts paired with reference answers.
4. The metric. Choose it by task shape, as the next section explains.

The third choice matters most, and it is the one most often made carelessly. A built-in benchmark is someone else’s data. It has never seen your documents, your customers’ phrasing or your edge cases. A strong result on it tells you how the model does on that benchmark, and nothing more.

A built-in dataset ranks models; your own dataset tells you about your workload

In our view, the most useful automatic job uses your own prompts and your own reference answers, even when there are only a few. It is better still if a built-in run sits beside it for comparison. When a model scores well on the benchmark and poorly on your set, that is not a contradiction. It is exactly the gap a benchmark cannot see.

The metric is chosen by task shape, not by sophistication

There are four metrics worth knowing. Three of them compare the model’s output, called the candidate, against a reference. The fourth uses a model as the judge.

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation) counts overlapping word sequences, called n-grams, with a recall orientation. It asks how much of the reference the candidate captured. Its home task is summarization, because a good summary keeps the important content.
• BLEU (Bilingual Evaluation Understudy) also counts overlap, but with a precision orientation. It asks how much of the candidate is supported by the reference. Its home task is translation, because a good translation adds nothing that the source did not say.
• BERTScore compares embeddings rather than words. An embedding is a list of numbers that places similar meanings close together. The metric therefore scores semantic similarity.
• LLM-as-a-judge applies stated criteria to open-ended output where no single right answer exists.

Route to a metric by the shape of the task the model performs

If you can hold onto only one rule, make it this one: summaries are scored with ROUGE, and translations with BLEU. The names carry the task. “Bilingual” points at translation, and “recall” points at keeping content.

Accuracy, robustness and toxicity each answer a narrower question than their names suggest

You may see scores with labels such as accuracy, robustness and toxicity. The service’s own documentation defines what each one computes, and those definitions are not repeated here. What can be stated is how to read any such score: ask what the number stands in for, and what it structurally cannot see.

Read an accuracy score on generated text as closeness to a reference. Classical accuracy, along with precision, recall and F1, needs a labelled test set with one right answer per item. A paragraph of generated text has no single right answer. A generation “accuracy” figure is therefore some form of overlap or similarity against the reference you supplied, and it is only as good as that reference. BERTScore will happily reward similarity to a reference that was itself wrong.

Read a robustness score as a statement about the inputs that were tested. This is our interpretation, not a definition. However robustness is measured, it can only cover the variations that appeared in the dataset. The rare failure you have not thought of lies outside it, just as it lies outside any human reviewer’s sample.

Read a toxicity score as a screen, not a verdict. Safety, tone and nuance are judgements, and human-in-the-loop review is the only genuine ground truth for them. An automatic score can flag outputs at scale. It cannot tell you whether a borderline refusal was worded well.

Each score label stands in for a narrower thing than its name implies

The scores share one blind spot: none of them knows whether an answer was true. A confident hallucination is fluent and well-formed, and it often sits close to the truth in embedding space. Those are exactly the properties that overlap and similarity metrics reward.

Break it on purpose: a correct paraphrase that scores badly

This is the most instructive thing you can do with an automatic job. Take one prompt from your dataset and set its reference answer to “The flight was cancelled.” Arrange for the candidate answer to be “They called off the flight.”

The meaning is identical, but the two sentences share almost no word sequences. An overlap metric such as ROUGE or BLEU will score the pair badly. This is not a fault in the metric. Overlap metrics punish paraphrase by construction.

Now score the same pair by semantic similarity. BERTScore, which compares embeddings, should treat the two sentences as close, because their meanings are close.

The same correct paraphrase fails an overlap metric and passes a meaning-based one

The lesson carries into real runs. Suppose your task allows many correct phrasings, such as answers to customer questions. A low overlap score may then mean only that the model worded things differently from your reference. Before you conclude the model is weak, check which proxy you are holding.

A strong model score is not a working application

Suppose the job comes back looking excellent. Two more layers remain before that result means much.

The first layer is the application. A foundation model inside a real system is one component among several. A retrieval-augmented system, which fetches documents before generating an answer, has a retrieval stage. An agent chooses tools, passes them arguments and sequences its calls. Any of those stages can fail while the model performs perfectly on the text it was handed.

Consider an assistant that confidently contradicts your policy documents. The first question to ask is whether the correct passage was ever retrieved. If it was not, the problem is a retrieval failure, and the fix lies in chunking, embeddings or the index. A better model changes nothing, because the model never saw the passage. If the passage was retrieved and then contradicted, the problem is a faithfulness failure, and the fix lies in the prompt or the model. The two fixes run in opposite directions, and an end-to-end score cannot tell you which failure you have.

The second layer is the business, the one that authorised the spend. Three metrics answer it:

• Task completion rate is the proportion of attempts that reached the intended outcome.
• User satisfaction is whether people find the system valuable.
• Cost per interaction is what one interaction costs to serve.

Model scores sit in the top layer; only the business layer says if the spend paid off

Cost deserves attention here, because it is the alignment metric most often left out. It is the only one that can turn an apparent success into a failure. An evaluation job has a cost of its own as well. An automatic run over an existing dataset sits at the cheap, automatable end of evaluation, while LLM-as-a-judge adds inference cost for every judgement it makes. When your experiment is finished, delete any evaluation resources and stored outputs you no longer need, following the service’s own guidance for doing so.

A final check concerns task engineering: whether the task was well framed for a model in the first place. Sometimes every technical score is fine and the outcome is still poor, because the task was the wrong shape for any model. Naming that possibility is part of reading the results honestly.

Key takeaways

• Decide first whether a reference answer exists. If one does, an automatic job can score the task. If not, use human review or LLM-as-a-judge.
• Choose the metric by task shape: ROUGE for summaries, BLEU for translation, BERTScore when correct answers are worded differently, and LLM-as-a-judge for open-ended output.
• Read every score label as a proxy. Accuracy measures closeness to your reference, robustness covers only the inputs you tested, and toxicity is a screen rather than a verdict.
• Run your own prompts beside a built-in benchmark. Benchmark strength is evidence about the benchmark.
• A strong model score proves nothing about the application or the business. Check retrieval separately, and always look at cost per interaction.

This article does not settle the exact console path or command syntax for creating a job. It does not give the precise definitions the service uses for its accuracy, robustness and toxicity scores, or what a run will cost. For those details, rely on the service’s current documentation rather than on this walkthrough.