Generated tests often mirror the implementation instead of the requirement, so a polished suite and high coverage can still provide weak protection.
Shared mistake: Code and assertions can encode the same wrong interpretation.
Weak signal: Tautologies, excessive mocks, and snapshots may never detect realistic faults.
Missing risk: Boundaries, invalid input, permissions, and failure paths are easy to omit.
False confidence: Tests may pass even after the behavior they claim to protect is broken.
Start expected outcomes from an independent specification. Review assertions, add integration coverage at important boundaries, and use mutation testing where its cost is justified.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
AI's own readingIt's the same reading behind the code.
Written specThe spec says orders of $50 or more ship free.
code ships free if
total > 50The AI read 'or more' as 'more than'.
total > 50It's the same code with the same bug.
test expects at $50
Pays shippingThe test repeats the same misreading.
Free shippingThis comes straight from the spec's wording.
suite result
Green, bug hiddenThe test and the code agree, so the bug ships.
Red, bug caughtA $50 order gets charged shipping, so the test fails.
Rule of thumbTake expected results from the spec, not from the code. Then review the assertions yourself, because a green run only proves the tests agree with the code.
1step
Expected result from: AI's own reading for Same AI reading and Written spec for Independent spec.
1 / 4← → keys
The AI's tests agree with its code because both come from the same misreading. Tests written from the spec expose the bug.
Saved in this browser - sign in to keep your review list.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Keep going - a few more words and AI can grade it.
Interview lens
How interviewers actually use this question
Likely follow-ups
When you review an AI-generated test in a pull request, what do you check first?Where the expected value came from, whether the test fails if the behavior breaks, and whether a mock replaces the very thing under test.
When does a mock in a generated test hide a real bug instead of isolating one?When it stubs the behavior under test or fakes a contract with another system. Add integration tests where that contract matters.
What you can say
The main pitfall is that AI-generated tests tend to mirror the implementation instead of the requirement, so they check that the code does what it already does.
If the model misread the requirement, the code and the assertions carry the same wrong interpretation, and the test passes anyway.
You also get weak tests, like tautological assertions, heavy mocking and snapshots, that rarely catch a realistic fault.
The risky cases often go missing too, like boundaries, invalid input, permissions and failure paths, so coverage looks high while broken behavior still passes.
So I take expected outcomes from an independent spec, review every assertion, and add integration tests at the important boundaries.
Mutation testing is the strongest check that the tests catch faults, but it costs run time and triage, so I use it only where the code justifies that cost.
Weak answers to avoid
Says 'just review the tests' and stopsToo vague. Say what the review checks: where expected values come from, whether each assertion can fail, and which risky paths are missing.
Treats high coverage as proof the suite worksCoverage shows the code ran, not that a wrong result would fail a test. Point to assertion review and mutation testing as the real checks.
Lists generic 'AI makes mistakes' complaintsMisses the mechanism. Code and tests built from the same reading share the same error, and that's why expected values need an independent spec.
Rejects AI-written tests outrightSounds dogmatic. A strong answer keeps the speed and adds checks: independent expected values, assertion review, integration and mutation tests.
Interview lens
Likely follow-ups, what you can say, and the weak answers to avoid.
