Foundation Models, Tokens, Embeddings, and the FM Lifecycle

Certification Blueprint

Field	Coverage
Exam	AWS Certified AI Practitioner (AIF-C01), exam guide v1.1
Domain	Content Domain 2 — Fundamentals of GenAI
Exam weight	24% of scored content
Task statement	2.1 Explain the basic concepts of generative AI (GenAI)
Objectives	2.1.1 foundational concepts · 2.1.2 use cases · 2.1.3 the FM lifecycle

What This Chapter Covers

Domain 1 is behind you — three chapters, 20% of the exam. This is the first chapter of Domain 2, which is 24% and the second-heaviest domain on the paper.

The change in character matters. Domain 1 rewarded classification: place the system, read the constraint, pick the technique. Domain 2 rewards precision. Several questions turn entirely on whether you know exactly what a word means — and these are words you almost certainly already use.

Domain 1 asked	Domain 2 asks
Is this AI, ML, deep learning or generative?	What is a foundation model, precisely?
Which learning type does the data permit?	What is the model actually reading? (tokens)
Which technique fits the output shape?	How does a machine represent meaning?
What does the ML pipeline produce at each stage?	What does the FM lifecycle produce at each stage?

This chapter takes the first three of Task 2.1's six objectives. Token-based pricing and context engineering are Chapter 05; agentic AI and MCP are Chapter 06.

What Is a Foundation Model?

A foundation model is a large model pre-trained on broad, general data that can be adapted to many downstream tasks without being trained specifically for any of them.

Two ways to hold it:

• A graduate hire rather than a specialist contractor. The graduate arrives knowing a great deal in general and is pointed at your particular job afterwards. The contractor arrives knowing your job and nothing else.
• A general-purpose kitchen rather than a bread machine. The bread machine makes excellent bread and nothing else. The kitchen makes anything, slightly less perfectly, and needs instruction.

The word doing the work is foundation. It is a base that other things are built on, not a finished product — which is why every later chapter in Domain 3 is a different way of building on top of it.

Note what is not the defining property: size. Large models existed before foundation models. What distinguishes a foundation model is adaptability without task-specific training.

Tokens: the Unit Everything Is Counted In

A token is the chunk of text the model actually processes. It is not a word and it is not a character — it is a learned sub-word unit.

Text	Roughly	Why
the	1 token	Common word, learned whole
unhappiness	~3 tokens	Rare; splits into sub-words
AIF-C01	several tokens	Digits and punctuation split

A rough guide for English: about four characters per token. Treat that as a rule of thumb rather than a formula — it does not hold for code, identifiers, or many other languages.

What to remember from this diagram: the model never sees your letters. It receives a list of integers, and it counts every one of them — in both directions. Output tokens are counted just as input tokens are.

Situation	What happens in tokens	Consequence
A prompt in fluent English	Efficient — most words are single tokens	Fewer tokens for the same meaning
A prompt full of product codes	Each code fragments	More tokens for the same meaning
A long conversation	Prior turns are re-sent each time	Token count grows as the conversation grows
A model asked for a long answer	Output tokens counted too	Output is not free

That last row is where most people's mental model is wrong. Nothing is remembered for free, and nothing is emitted for free.

Chunking

Chunking is splitting a long document into smaller pieces before it is processed or stored.

There are two reasons, and the exam uses both:

Reason	What it solves
A model processes a bounded amount at once	A 400-page manual cannot be handed over whole
Retrieval should return the relevant part	A whole manual answers no question usefully

Two ways to hold it:

• A book with chapters and an index — you fetch the chapter, not the library.
• Cutting a long interview into clips — each clip stands alone and can be found on its own.

Both failure directions exist. Chunk too large and you retrieve a great deal of irrelevant material alongside the answer. Chunk too small and you retrieve a sentence that no longer makes sense without the paragraph around it.

Embeddings and Vectors

An embedding is a representation of meaning as a vector — a fixed-length list of numbers.

Term	Precisely
Vector	An ordered list of numbers of fixed length. Implies no meaning by itself
Embedding	A vector produced so that position encodes meaning
Vector space	The space those vectors live in, where distance means relatedness

Every embedding is a vector. Not every vector is an embedding.

Two ways to hold it:

• Map coordinates. Two cities close in latitude and longitude are close on the ground, and the numbers themselves carry the relationship rather than describing it.
• A wedding seating plan. Guests with things in common are seated near each other, so the seat number encodes the relationship.

What to remember from this diagram: an embedding is not a summary and not a compressed copy. You cannot read the original text back out of it. It is a position, and the only operation a position enables is comparison.

Because meaning becomes distance, meaning becomes computable:

Capability	What it relies on
Search by meaning rather than keyword	"cheap flights" finds "budget airfare" — no shared word
Recommendation	Items near the ones a user liked
Grouping related documents	Positions that cluster together
Finding the relevant chunk to feed a model	The chunk nearest the question

Keyword search matches strings. Embedding search matches meaning. That is the exam-relevant difference.

The Transformer, and What Makes an LLM

A transformer is the model architecture behind current large language models. Its defining capability is relating every token to every other token in the input, so that meaning depends on context rather than on position alone.

A transformer-based large language model (LLM) is a transformer trained on a very large body of text.

The word	What it contributes
Large	Trained on a very large corpus, with very many parameters
Language	The domain it models is text
Model	It predicts — it does not look up

"Bank" in river bank and in savings bank is the same token with a different meaning. Relating tokens to each other is how that gets resolved. And note the third row: an LLM is not a database. It does not retrieve stored sentences; it predicts.

Prompt Engineering — the Term

Prompt engineering is the practice of structuring the input given to a model so that the output is more useful — supplying context, instruction and constraints rather than a bare question.

That definition is the whole of this chapter's requirement. Objective 2.1.1 lists prompt engineering among terms to define. The constructs, techniques, best practices, risks and versioning strategies belong to Objective 3.2 and are taught in Chapter 10. Being able to define it is examined here; being able to do it is examined later.

Three Model Families

Family	Handles	Typically produces
Transformer-based LLM	Text in, text out	Summaries, translations, code, replies
Multi-modal model	More than one modality in — commonly text and images	Usually text about the other modality
Diffusion model	Text in	Images, video, audio — generated by removing noise step by step

Diffusion works backwards from noise. It starts with random noise and repeatedly removes noise until an image consistent with the prompt remains. That mechanism makes the family unmistakable.

Be careful with multi-modal: the "multi" describes what the model can take in, commonly text and images together. It does not mean the model emits several kinds of output. If a question requires an image to be produced, that points at a diffusion model.

What to remember from this diagram: read the modalities, not the industry. What goes in and what must come out — that pair selects the family, and neither the sector nor the size of the company appears anywhere in the decision.

GenAI Use Cases

Objective 2.1.2 names these:

Use case	What the model is doing
Image, video and audio generation	Producing new media from a description
Summarization	Compressing text while preserving meaning
AI assistants	Multi-turn help grounded in context
Translation	Meaning preserved across languages
Code generation	Producing code from a description
Customer service agents	Handling enquiries conversationally
Search	Retrieving by meaning, not keyword
Recommendation engines	Suggesting items related to a preference

What to remember from this diagram: two of the eight named use cases are not generation at all. Search and recommendation are comparison problems and they run on embeddings. They appear in a generative-AI objective because embeddings come from the same family of models — and noticing this is what stops you reaching for a large language model on every Domain 2 question.

The FM Lifecycle

Objective 2.1.3 names seven stages:

#	Stage	The question it answers
1	Data selection	What will the model learn from?
2	Model selection	Which base model do we start from?
3	Pre-training	How does it gain general capability?
4	Fine-tuning	How does it fit our task or domain?
5	Evaluation	Is it good enough to ship?
6	Deployment	How is it made callable?
7	Feedback	What does real usage tell us?

This is not the AI/ML pipeline from Chapter 03. That pipeline describes a project — it starts at a business goal and ends at monitoring in production. This lifecycle describes the model itself. They overlap around evaluation and deployment, and conflating them is a reliable way to lose an ordering question in either domain.

What to remember from this diagram: the loop is the point. Feedback is a stage, not an afterthought — it is what sends a deployed model back for re-tuning. A lifecycle drawn as a straight line is wrong here for the same reason it was wrong in Chapter 03.

The objective says describe, so know what each stage produces, not just its name:

Stage	Produces
Data selection	A chosen, scoped corpus
Model selection	A named base model to start from
Pre-training	A general-capability model
Fine-tuning	A model adapted to a task or domain
Evaluation	A pass or fail against a defined bar
Deployment	A callable model
Feedback	Signals from real use that drive the next cycle

Ordering questions live here. Recall from Chapter 01 that ordering questions require the correct responses in the correct order, with no partial credit — so knowing seven names in the wrong sequence scores zero. Pre-training before fine-tuning is the pair most often inverted.

Decision Rules and Exam Signals

Rule 1 — define before you apply. In Domain 2, state what the term is before reaching for what it is used for. Several questions test only the definition.

Rule 2 — tokens are counted in both directions. Input and output. Any reasoning that counts only the prompt is incomplete.

Rule 3 — an embedding is a position, not a summary. Its only capability is comparison.

Rule 4 — modality selects the model family. What goes in, what must come out. Not the industry.

Rule 5 — "multi-modal" describes the input. If an image must be produced, that is diffusion.

Rule 6 — the FM lifecycle is about the model; Chapter 03's pipeline is about the project. Check which one a question is asking about before sequencing anything.

Distractor Patterns

Pattern	What it looks like	How to defuse it
Token as word	"1,000 words is 1,000 tokens"	Tokens are sub-word units; rare strings cost more
Output tokens forgotten	Counting only the prompt	Both directions are counted
Embedding as summary	Treating a vector as compressed text	It is a position; it enables comparison only
Diffusion for text	A diffusion model offered to summarise	Diffusion generates media, not prose
Multi-modal means multi-output	Assuming it must emit images	It takes more than one modality in
Fine-tuning before pre-training	Lifecycle stages reordered	Pre-training precedes fine-tuning, always
Search treated as generation	An LLM offered where embeddings fit	Search and recommendation are comparison problems

The first two share a root cause: thinking of tokens as a description of the prompt rather than as the unit of account for the whole exchange.

Scenario Walkthrough

A legal publisher holds 90,000 case documents, some over 300 pages. It wants lawyers to find relevant passages by describing a situation in their own words rather than guessing keywords, and it wants a short plain-English précis of each passage found. It also wants a cover illustration generated for each published digest.

Requirement	Reading	Decision
300-page documents must be processed	Too large to handle whole	Chunking first
Find by description, not keyword	Meaning compared, not strings matched	Embeddings and vector similarity
Short plain-English précis	Text in, text out	Transformer-based LLM — summarization
Cover illustration generated	Text in, image out	Diffusion model

One scenario, four different concepts — and only two of the four are generation.

The trap is answering the whole scenario with "an LLM". Retrieval here is a comparison problem, and chunking is a precondition rather than a technique choice.

Key Concepts

Term	Definition
Foundation model (FM)	A large model pre-trained on broad general data that can be adapted to many downstream tasks without being trained specifically for any of them
Token	The chunk of text a model actually processes — a learned sub-word unit, neither a word nor a character
Chunking	Splitting a long document into smaller pieces so it can be processed and so retrieval can return the relevant part
Vector	An ordered list of numbers of fixed length; implies no meaning by itself
Embedding	A vector produced so that its position encodes meaning, making relatedness measurable as distance
Vector space	The space embeddings occupy, in which distance between positions corresponds to relatedness in meaning
Transformer	The model architecture that relates every token to every other token, so meaning depends on context
Large language model (LLM)	A transformer trained on a very large body of text; it predicts rather than looks up
Multi-modal model	A model that accepts more than one modality as input, commonly text and images together
Diffusion model	A model that generates images, video or audio by starting from random noise and repeatedly removing it
Prompt engineering	The practice of structuring a model's input — context, instruction and constraints — so the output is more useful
Pre-training	The lifecycle stage in which a model gains general capability from broad data
Fine-tuning	The lifecycle stage in which a pre-trained model is adapted to a specific task or domain
FM lifecycle	Data selection, model selection, pre-training, fine-tuning, evaluation, deployment, feedback

Revision Flashcards

Say the answer aloud before revealing it.

1. What distinguishes a foundation model — and what does not? → Adaptability to many downstream tasks without task-specific training. Size is not the defining property; large models existed before foundation models did.

2. What is a token? → The chunk of text a model actually processes — a learned sub-word unit. Not a word and not a character. Roughly four characters per token in English, as a rule of thumb only.

3. Are output tokens counted? → Yes. Both input and output are counted. Nothing is emitted for free, and in a multi-turn conversation prior turns are re-sent, so the count grows with the conversation.

4. What two problems does chunking solve? → A model processes a bounded amount at once, so very long documents cannot be handed over whole; and retrieval should return the relevant part rather than the entire document.

5. What is the difference between a vector and an embedding? → A vector is any fixed-length ordered list of numbers. An embedding is a vector produced so that its position encodes meaning. Every embedding is a vector; not every vector is an embedding.

6. Can you recover the original text from an embedding? → No. It is a position in vector space, not a summary or a compressed copy. The only operation it enables is comparison.

7. What does keyword search do that embedding search does not, and vice versa? → Keyword search matches strings. Embedding search matches meaning, so "cheap flights" can find "budget airfare" despite sharing no word.

8. What does the transformer architecture contribute to an LLM? → It relates every token to every other token in the input, so meaning depends on context rather than position alone.

9. Does "multi-modal" mean the model produces several kinds of output? → No. It describes what the model accepts as input, commonly text and images together. Output is typically text about the other modality. If an image must be produced, that is a diffusion model.

10. How does a diffusion model generate an image? → It starts from random noise and repeatedly removes noise until an image consistent with the prompt remains — it works backwards from noise.

11. Which two of the eight Objective 2.1.2 use cases are not generation? → Search and recommendation engines. Both are comparison problems that run on embeddings.

12. Name the seven FM lifecycle stages in order. → Data selection, model selection, pre-training, fine-tuning, evaluation, deployment, feedback.

The Four-Beat Answer

The core question this chapter prepares you for: "Explain what a foundation model is and how one comes to be useful for a specific job."

Four beats, checked in this order. Missing a beat is a failure state — you will be probed on whichever one you skipped.

1. What it is — a large model pre-trained on broad general data, adaptable to many downstream tasks without task-specific training. Name adaptability rather than size as the distinguishing property.
2. What it operates on — tokens, which are learned sub-word units rather than words, counted in both directions. This is the unit that every later cost and capacity conversation is built on.
3. How it represents meaning — embeddings, which are vectors whose position encodes meaning, so relatedness becomes a measurable distance. Say what that makes possible: comparison, and therefore search and retrieval.
4. How it becomes useful — the lifecycle: data selection, model selection, pre-training, fine-tuning, evaluation, deployment, feedback. Name what the stages produce, and name the feedback loop, because a lifecycle without one is a description of a project that has already failed.

A strong answer names the unit and the loop. A weak answer describes capability.

Why This Helps You

On the job: the token is the unit that governs both cost and capacity in every foundation model system you will touch. Teams that treat tokens as words size their systems wrongly, and they discover it in production when a conversation grows.

In interviews: "what is an embedding?" is a standard screening question, and "it's a vector representation" is the answer everyone gives. Saying that it is a position whose only capability is comparison — and that you cannot read the text back out — signals that you have actually built with them.

On the exam: Domain 2 is 24% of scored content, and Task 2.1 questions frequently test the definition alone. The habit of stating what a term is before reaching for what it is used for is worth more marks here than anywhere else on the paper.

Chapter Checklist

• I can define a foundation model without using the word "big" as the distinguishing property
• I can say what a token actually is, and why it is neither a word nor a character
• I can state that both input and output are counted in tokens
• I can explain what chunking solves, and the cost of chunks that are too large or too small
• I can define an embedding as a vector whose position encodes meaning
• I can explain why an embedding cannot be read back as text
• I can say what embeddings make possible that keyword matching cannot
• I can explain what the transformer architecture contributes to an LLM
• I can define prompt engineering as a term, without straying into technique
• I can separate transformer-based LLMs, multi-modal models and diffusion models by modality
• I can name all eight Objective 2.1.2 use cases and say which two are comparison rather than generation
• I can sequence the seven FM lifecycle stages and say what each one produces

After the Chapter

1. Complete student/project.md — parts 15-18 of the AI/ML Decision Sheet you began in Chapter 01. Bring the same sheet; do not start a new one.
2. Take student/quiz.md closed-book, then review the reasoning for every question you guessed, including the ones you got right.
3. Open the official v1.1 exam guide's Domain 2 page and confirm you can attach a concept from this chapter to each of the first three bullets under Task Statement 2.1.
4. Next chapter: Chapter 05 — Token-Based Pricing and Context Engineering (Domain 2, Objectives 2.1.4 and 2.1.5). You now know what a token is; Chapter 05 turns that into money and latency, and introduces the role of context engineering in foundation model applications. It is short in objectives and heavy in consequence.