Define the improvement you actually need
A faster response is useful only if it leaves the user with less work. Start with a specific job your current model performs, such as extracting fields from a support request or drafting an answer from approved documents. Write down what makes an output acceptable before comparing models.
The November 13 briefing includes the rollout of Gemini 4 Hydrogen. A release is a reason to evaluate a candidate, not evidence that it will improve your particular workflow. This guide proposes a migration test; it does not report hands-on results for that model.
Save a baseline you can replay
Collect representative inputs you are allowed to use. Include routine requests, incomplete information, long documents and cases that previously failed. Save the prompt, model identifier, generation settings, tool definitions and retrieved evidence alongside each case. Remove unnecessary personal information before the cases enter a shared evaluation set.
Run the existing model and candidate against the same inputs and frozen tool responses. First keep the prompt unchanged to reveal compatibility problems. If you later tune a prompt for the candidate, report that as a separate configuration and check it against held-out cases. Otherwise, prompt improvements can be mistaken for a model improvement.
Score the result your application needs
For structured extraction, validate the schema and compare important fields with reviewed answers. For document questions, inspect whether the cited passages support the answer and whether missing evidence produces an honest uncertainty. For a tool-using assistant, inspect the chosen tool and arguments, not just the final message.
Google’s evaluation documentation describes rubric-based evaluation, deterministic metrics and custom functions. Use the method that matches the requirement: a parser can check a schema, while a knowledgeable reviewer may need to judge whether a summary preserves a qualification. If you use a model judge, review disagreements and critical cases yourself.
Measure the complete job
Record time until the result is usable, including tool calls, retries and correction. Keep first-response time separate from completed-task time. Review slow outliers as well as typical runs, because a quick average can conceal requests that leave users waiting.
Calculate cost per accepted result using the actual requests, including failed attempts and any fallback model calls. Track human correction time separately. A cheaper request can still make the workflow more expensive when it needs repeated repair. Repeat uncertain cases to see whether a result was dependable or just a lucky run.
Test integration failures before live actions
Check invalid tool arguments, unavailable tools, truncated outputs and refusals. Use a sandbox for writes and confirm the application handles an incomplete response without issuing a duplicate action. The model must also respect the same access boundaries and approval steps as the baseline.
Keep an error log with the input case, configuration and observed outcome. If the candidate loses an important requirement, narrow its scope to the tasks it handles well or keep the existing configuration. A migration does not need to move every task at once.
Roll out with a route back
Choose acceptance criteria before the pilot and assign someone to review failures. Start with a limited, reversible slice of traffic. If you run a shadow comparison, suppress its external writes so only the active workflow can act. Make the model configuration switchable, and test returning to the previous configuration before expanding.
Keep monitoring accepted-result rate, latency, fallback frequency and correction work after release. Re-run the saved evaluation when prompts, model versions or retrieval data change. For the action layer, see how to retry AI actions without duplicates.
Questions
- How do you decide whether a faster AI model can replace the one in your workflow?
- Replay representative tasks, compare correctness and end-to-end time, then try a limited rollout with a tested rollback. Keep the existing model where the candidate fails critical requirements.
- Should I change the prompt during the first comparison?
- Start with the same prompt and frozen inputs. Evaluate any candidate-specific prompt tuning separately on held-out cases so you can distinguish model changes from prompt changes.
- What should a shadow comparison do with write tools?
- Suppress external writes in the shadow path. Only the active workflow should act, so evaluating a second model cannot create duplicate messages, tickets or other changes.