IESE AI Club

Role playbook

AI for AI product managers.

Every PM is becoming partly an AI PM. Model output is probabilistic, prompts are product surface, quality lives in evals and cost moves with every request. Learn those differences and write the playbook.

This assumes you have the base. Start there if not

The job

What this role actually does.

Six things in a loop. The model changes; customer expectations do not. Turn uncertain output into a product with a clear quality bar.

01Discovery 02Spec & prompts 03Model & stack 04Evals 05Ship & iterate 06Risk & trust

Most examples are vendor customer stories. Their self-reported, unaudited numbers show something was built and used—not a benchmark to expect.

01

Discovery at volume

Synthesis
The use case
Organising feedback, support volume and market signal into evidence about what an AI feature should actually do.
Job to be done
When I decide what to build with a model, I want the demand evidence sorted, so I choose a use case with a real job behind it rather than a demo.
Value to the business
AI investment aimed at problems users actually have.
How to evaluate it
Share of shipped AI features with measurable adoption, and the ratio of features retired to features shipped. Novelty is not demand.

What AI does here

AI examines the full signal—interviews, tickets, failed prompts, calls and usage notes—then clusters jobs, quotes evidence and exposes segment differences. Research gets broader, not wiser.

Workflow map

  1. InputCollect interviews, tickets, failed interactions, usage traces and segment labels around a job to be done.
  2. AI-assisted processCluster needs and failures, retrieve examples and highlight where segments experience the product differently.
  3. Human checkpointInspect the examples and decide which problem merits model complexity or a simpler solution.
  4. OutputAn evidence-backed opportunity brief with a target user, failure modes and next test.

What is still yours

Product taste: which pain matters, which user leads and whether AI adds an advantage or a complication. Volume does not remove the need to say no.

02

Specs and prompts

Drafting
The use case
Writing the behaviour spec: system prompt, instructions, refusal and escalation rules, worked examples.
Job to be done
When I define how the model should behave, I want intended and forbidden behaviour written down concretely, so the product does the same thing tomorrow as it did today.
Value to the business
Predictable product behaviour, and a written record of what was intended when it does not.
How to evaluate it
Behaviour consistency across releases, and how many production issues trace back to something never specified. Prompts are product surface: version and review them like code.

What AI does here

AI can draft prompts, generate hard inputs, compare variants and expose conflicting instructions. Treat prompts as code: version, test and explain changes.

Workflow map

  1. InputState the user job, permitted sources, desired behaviour, prohibited behaviour and representative requests.
  2. AI-assisted processDraft instructions, generate difficult variants and identify conflicting or underspecified directions.
  3. Human checkpointProduct and domain experts approve the behaviour and version the prompt alongside its tests.
  4. OutputA tested prompt/specification and a clearly defined set of known limitations.

What is still yours

Define behaviour plainly and cut what does not serve it. You own the flexibility-control trade-off and what stays out.

03

Model and stack choices

Analysis
The use case
Choosing which model does which job, and where inference and data sit.
Job to be done
When a feature has several viable models, I want the capability, cost, latency and data-residency trade-off made explicitly, so the choice still works at scale.
Value to the business
Margins that hold as usage grows, and deployments that pass procurement.
How to evaluate it
Cost per successful task, p95 latency, quality on your own evaluation set, and whether the architecture meets the customer's data requirements. Measure on your tasks, not on public benchmarks.

What AI does here

Benchmark models on the same task, estimate usage cost and prototype build-versus-buy. Replace leaderboard debate with evidence on capability, latency, reliability and cost.

Workflow map

  1. InputDefine representative tasks, data constraints, traffic expectations, cost ceiling and latency target.
  2. AI-assisted processRun comparable trials, summarise results and estimate the capability, latency and cost trade-offs.
  3. Human checkpointSecurity, engineering and product decide which failure modes, vendor dependency and operating cost are acceptable.
  4. OutputA documented stack decision with benchmarks, fallback and re-evaluation triggers.

What is still yours

The trade-off. The best model may be slow, the cheapest may need recovery logic and the easiest vendor may create an unacceptable dependency. Choose what the business can live with.

04

Evals and the quality bar

Analysis
The use case
Building the test set that decides whether a change ships.
Job to be done
When someone proposes a change, I want it measured against real cases and expert judgment, so shipping is a decision rather than a feeling.
Value to the business
The ability to move quickly without regressions reaching customers.
How to evaluate it
Eval coverage of real usage, regressions caught pre-release versus post, and how well eval scores track user outcomes. An eval set that never fails is measuring nothing.

What AI does here

AI expands known cases, runs them repeatedly and classifies failures. The result is a regression suite for behaviour no single expected string captures.

Workflow map

  1. InputTurn the product promise into labelled real cases, high-risk edge cases, a rubric and release threshold.
  2. AI-assisted processExpand test sets, execute repeated runs and classify failure patterns for review.
  3. Human checkpointDomain experts adjudicate ambiguous cases and approve the threshold or block release.
  4. OutputA versioned eval report, regression suite and prioritised improvement backlog.

What is still yours

Define a measurable bar: cases, grader, threshold and release-blocking failures. If the eval rewards the wrong behaviour, a better score means you are wrong more consistently.

05

Ship and iterate

Automation
The use case
Releasing changes under control: staged rollout, production quality signals, a way back.
Job to be done
When I change a model, a prompt or a tool, I want to see the effect on live quality before everyone gets it, so an improvement on paper cannot become an outage.
Value to the business
A faster release cadence that does not bet the product on each release.
How to evaluate it
Time from proposed change to safe rollout, incident rate per release, and time to roll back. Non-determinism means a change can regress a case no dashboard tracks; keep the human review loop.

What AI does here

Non-deterministic features need staged rollouts, visible feedback and a fast route from bad output to trace. AI groups failures and suggests changes; do not overreact to the latest example.

Workflow map

  1. InputSelect a pilot group, success metrics, escalation routes and the current production version.
  2. AI-assisted processSummarise feedback and traces, group recurring failures and propose candidate changes.
  3. Human checkpointReview evidence, test the change against evals and approve a controlled rollout or rollback.
  4. OutputA monitored release decision and an auditable iteration record.

What is still yours

Release judgment: who sees it first, the fallback and whether failure needs a prompt, product or no change. Gradual shipping earns trust.

06

Risk and trust

Knowledge
The use case
Designing the checkpoints, disclosures and controls: what the model may do alone, what needs approval, what the user is told.
Job to be done
When the product acts on someone's behalf, I want the boundaries and review points designed in, so a mistake is caught rather than compounded.
Value to the business
A product enterprises and regulators can accept, and one that survives its first public error.
How to evaluate it
Share of consequential actions with a review step, incident severity and time to detection, and adoption inside regulated accounts. Measure what happens after a failure, not only how often one occurs.

What AI does here

AI can map hallucination surfaces and data flows, generate adversarial cases, draft disclosures and explain answers. It cannot decide where the model must not act.

Workflow map

  1. InputInventory data flows, user harm scenarios, tools, permissions and actions that cannot be safely reversed.
  2. AI-assisted processGenerate adversarial cases, map policy gaps and draft user-facing explanations or escalation language.
  3. Human checkpointProduct, security, legal and domain owners set non-negotiable boundaries and human-review requirements.
  4. OutputA risk register, tested guardrails, disclosure plan and named accountable owners.

What is still yours

Draw the line the product cannot cross: privacy, irreversible actions, human review and the user promise. Trust is a product requirement.

Start this week

  1. Take one model-backed feature and write ten examples of good output, five bad outputs and three failures that must block release.
  2. Version its current prompt beside the spec, then change one instruction and run the same examples again.
  3. Compare two model options on your actual task for quality, latency and cost per successful result.

Where it fails in this role

  • Writing deterministic specs for probabilistic behaviour. The happy path can pass while the product still fails in ordinary use.
  • Optimising a public benchmark. Your users, prompts and failure costs are different, which makes your eval suite the only useful scoreboard.
  • Hiding uncertainty from users. Confidence theatre buys a clean launch and an expensive trust problem later.

Other playbooks

All nine playbooks Consulting Product management Sales Marketing Venture capital Operations Forward deployed engineer Founder / entrepreneur

Now do it somewhere real.

A playbook is a map. The Industry Fellowship is the terrain: a term spent talking to professionals who are implementing AI in one industry, working out where it actually creates value, and building a working prototype against what you find. Most fellows start as beginners.

Explore the Fellowship Apply for the Class of 2028