An AI engineer take-home assessment mostly scores judgment, not cleverness. Reviewers usually check five things: does it run, did you measure whether it works, does it handle failure, did you think about cost and limits, and can you explain your choices in writing. Raw model knowledge and fancy techniques count for much less than most candidates expect.
Companies differ, and few publish their rubrics. What follows is the pattern that shows up most often, so treat it as a map and not a promise about any one employer.
What do reviewers look at first?
They open the repo and try to run it. If the setup takes ten minutes of guessing, you have lost goodwill before any code is read.
A reviewer’s first pass is short and mechanical:
- Is there a README with one command to install and one command to run?
- Does the program produce output on the sample input?
- Are secrets kept out of the code?
- Is the task that was asked for actually done, and not a bigger task you preferred?
Do these four well and you are ahead of a large share of submissions. Many fail at step one.
What are the five things that get scored?
Here is a typical rubric, written the way a hiring team might keep it in a spreadsheet.
| Area | What the reviewer looks for | Common way to lose points |
|---|---|---|
| Works end to end | Runs from a clean checkout, handles the sample input | Needs a hidden local setup |
| Evidence it works | A small test set, a score, a before and after | ”It looked good to me” |
| Failure handling | Timeouts, empty results, bad model output | Only the happy path is coded |
| Cost and limits | Token counts, model choice, caching reasoning | Biggest model everywhere |
| Communication | A short write-up of choices and trade-offs | No write-up, or a wall of text |
Notice that only one row is about the model itself. The rest is ordinary engineering applied to a system whose output is uncertain.
How do you show that your solution works?
This is the row that separates strong submissions from average ones. A language model can sound right and be wrong, so “I ran it and the answer looked fine” proves nothing.
Build a small evaluation set. Ten to thirty cases is enough for a take-home. Include easy cases, a few awkward ones, and at least two where the correct behavior is to refuse or say “I don’t know”. Then score your system against them and report the number.
A minimal version in Python looks like this:
cases = [
{"input": "Refund policy for damaged items?", "must_contain": "30 days"},
{"input": "What is the CEO's home address?", "must_refuse": True},
]
def score(run_fn):
passed = 0
for c in cases:
out = run_fn(c["input"])
if c.get("must_refuse"):
ok = "can't help" in out.lower() or "cannot" in out.lower()
else:
ok = c["must_contain"].lower() in out.lower()
passed += ok
return passed / len(cases)
This is crude, and that is fine. What matters is that you changed something, ran it again, and can say “the score went from 0.55 to 0.80 after I changed the retrieval step”. Our post on how to tell a prompt change actually helped walks through that loop in more detail, and the prompt evaluation project lets you practice it end to end.
What counts as good failure handling?
Reviewers know that model calls fail in boring ways. They look for evidence that you know it too.
- Timeouts and retries. A network call with no timeout is a red flag. Retry a small number of times, with a pause that grows each time.
- Bad output. If you asked for JSON and got prose, what happens? Validate the output against a schema and either repair it or return a clear error.
- Empty or irrelevant retrieval. If your search step finds nothing useful, the system should say so and not invent an answer.
- Unsafe input. Text from users or documents can contain instructions aimed at the model. Show that you treat it as data, not as commands.
You do not need all four in a few hours. Pick the two that matter most for the task and handle them properly. Then say in the write-up what you left out and why. Naming a gap honestly scores better than hiding it.
Do they care about cost?
More than candidates expect. A system that works but calls the largest model on every request, with the whole document pasted in each time, tells the reviewer you have not run anything in production.
Show three habits:
- Count tokens. Log the input and output token counts per request and mention the rough cost per hundred requests. Prices change, so state the date you checked and link the vendor’s pricing page, not a number from memory.
- Match the model to the step. A small, cheap model can often classify or route. Save the larger one for the step that needs reasoning.
- Avoid repeated work. If the same document is processed twice, cache the result.
Our guide on what LLM prompts actually cost shows how to do the counting.
What should the write-up contain?
The write-up is where reviewers decide whether they would want to work with you. Keep it to one page. Use this shape:
- What I built. Two sentences.
- What I measured. Your eval set size, your score, what changed it.
- What I chose and why. Model, retrieval method, chunk size, anything a reader might question.
- What fails. The cases your system still gets wrong. Show one.
- What I would do next. Two or three items, in order.
Section four is the one that builds trust. A candidate who points at their own weak spots looks like someone who will catch problems before customers do.
What mistakes cost the most?
- Scope creep. Adding a web interface nobody asked for while the core task is half done.
- No tests at all. Even three tests beat none.
- Hard-coded keys. Read secrets from the environment and add a sample env file with blank values.
- Overbuilding. Five frameworks where a short script would do. Reviewers read this as a lack of judgment.
- Ignoring the time limit. If they say four hours, spend about four hours and say what you would add with more time.
How do you practice before the real one?
Treat a practice run as a rehearsal of the habits above, not of any specific question. Pick a small task, such as answering questions from a set of documents, and give yourself a fixed time box.
Then check yourself against the list: clean run, eval set with a score, two failure cases handled, token counts logged, one-page write-up. If one is missing, that is what to train next.
If you are unsure where your gaps are, the AI and ML readiness quiz is a quick way to find them. The chapter on guardrails, evals and observability covers the measuring and safety side in depth, and the capstone build chapter gives you a larger project to practice a full submission on.
Next step
Once your project is built, the last skill is explaining it out loud. The chapter on mock interviews and your portfolio narrative is free to read on this site and covers how to turn a project like this into a clear story you can tell in an interview.