Blogs AI AI Test Automation / MCP Testing

How I Built My First MCP-Powered Test Automation Workflow (Beginner’s Honest Log)

98 Min Read

I want to start this one with a confession. Three weeks ago, if you’d asked me to explain MCP in one sentence, I would have fumbled it. I’d seen the acronym everywhere — LinkedIn posts, YouTube thumbnails, a slide in a webinar I half-watched during lunch — but I hadn’t actually built anything with it. I was doing the thing I keep telling readers of this blog not to do: nodding along with a trend without touching the keyboard.

So this post is not a polished “MCP best practices” guide written by someone who’s been doing this for a year. It’s the opposite. It’s my actual log — mistakes, dead ends, config files I broke and had to fix, and the moment things finally clicked — of building my first MCP test automation workflow on top of Playwright and TypeScript. If you’re a manual QA engineer, an SDET still comfortable in Java/Selenium, or anyone who’s heard “MCP” thrown around in a stand-up and quietly Googled it later, this is written for you.

I’m not going to pretend this was smooth. I broke my mcp.json file twice. I burned through a chunk of my monthly AI token budget on a single afternoon of trial and error. I generated a test that looked perfect and then watched it fail for a reason that had nothing to do with my application and everything to do with how I’d phrased my prompt. All of that is in here, because the sanitized version of this story wouldn’t actually help you avoid the same traps.

By the end of this post you’ll have seen, step by step, exactly how I went from “I don’t really get MCP” to having a working MCP-powered test automation workflow that generates, runs, and helps debug Playwright tests using an AI coding assistant connected through the Model Context Protocol. I’ll show you the setup, the config files, the prompts that worked and the ones that wasted my time, and an honest verdict on whether this is something your QA team should actually adopt right now or wait on.

Table of Contents

Why I’m Writing This as a Log, Not a Tutorial

Most content about MCP right now falls into two buckets. Bucket one is marketing copy from tool vendors telling you MCP will “10x your testing velocity.” Bucket two is deeply technical protocol documentation written for people building MCP servers, not people trying to use one to write a login test. Neither bucket is written for someone like me — a QA engineer with 12+ years of automation experience in Java and Selenium, currently mid-way through learning TypeScript and Playwright, trying to figure out if this AI-assisted workflow stuff is worth the learning curve.

I’ve written before on this blog about my honest journey moving from Selenium to Playwright, and about debugging flaky tests with GitHub Copilot and MCP in a more exploratory way. This post is the more structured, ground-up version: it assumes you know what Playwright is, you’re comfortable with basic TypeScript, but you’ve never wired up an MCP server before. If that’s you, welcome. Let’s build this together, mistakes and all.

What Is MCP, Actually? (No Jargon Version)

Before touching any config file, I forced myself to actually understand what I was building, instead of copy-pasting a setup guide and hoping it worked. Here’s the explanation I wish someone had given me on day one.

MCP stands for Model Context Protocol. It’s an open standard that defines a common way for AI models — like the one powering your coding assistant — to talk to external tools and data sources. Before MCP, if you wanted an AI coding assistant to actually interact with, say, a running browser, a file system, a database, or a test runner, someone had to build a custom, one-off integration for that specific combination of assistant and tool. Every assistant needed its own plugin system, and every tool vendor had to build integrations for every assistant separately. It was an N×M problem — N assistants times M tools, each combination needing its own glue code.

MCP flips that into an N+M problem. A tool vendor builds one MCP server that exposes their tool’s capabilities in a standard way. An AI assistant vendor builds one MCP client that knows how to talk to any MCP server. Now any MCP-compatible assistant can talk to any MCP-compatible tool, with no custom glue code in between. Think of it like a USB-C port for AI tools — before USB-C, every device had its own proprietary charging cable; now one port type works everywhere.

For test automation specifically, this matters because it means an AI coding assistant like GitHub Copilot or Claude, connected through an MCP server built for Playwright, can actually drive a real browser — click buttons, read the DOM, take screenshots, inspect network requests — instead of just guessing what a page might look like based on training data. That distinction turned out to be the whole point, and I didn’t fully appreciate it until I saw it happen in front of me, which I’ll walk through later in this post.

The Three Pieces You’re Actually Working With

Once I understood MCP conceptually, I realized my “MCP test automation” setup was really three separate pieces talking to each other, and separating them mentally made everything else easier to debug:

The MCP host / client — this is your AI coding assistant application itself, for example VS Code with GitHub Copilot, or Claude Code, or Cursor. This is what you actually type prompts into.
The MCP server — this is a small program, in my case the official Playwright MCP server, that exposes a defined set of “tools” (like navigate, click, screenshot, get accessibility tree) that the AI model can call.
The target application — the actual website or web app under test, which the Playwright MCP server controls through a real, automatable browser instance.

When these three pieces are wired together correctly, you get a loop where you type a plain-English instruction, the AI model decides which MCP tool calls it needs to make to fulfill that instruction, the MCP server executes those calls against a real browser, and the results — DOM snapshots, screenshots, console logs — flow back to the model so it can decide what to do next or what code to generate. That loop is the entire magic trick. There’s no actual magic; it’s tool calling, done in a standardized way, in a tight feedback loop.

My Starting Point: Where I Was Before This Experiment

Context matters here, so let me be specific about my starting point instead of pretending I was a blank slate. I’ve spent over a decade doing test automation, most of it in Java with Selenium WebDriver, across banking, wealth management, healthcare, and payments projects. Earlier this year I started deliberately learning Playwright with TypeScript, going through a structured course and documenting the transition publicly on this blog. By the time I started this MCP experiment, I already had:

A working Playwright + TypeScript project scaffolded with the Page Object Model
Comfort writing basic-to-intermediate TypeScript, including types, interfaces, and async/await patterns
GitHub Copilot already enabled in VS Code, though I’d mostly used it for autocomplete, not agentic workflows
Zero prior experience configuring any MCP server, for testing or anything else

I’m mentioning this because if you’re coming from a similar place — solid manual or Selenium background, newer to Playwright, curious but untested on the AI-assisted side — this log should map fairly directly onto your situation. If you’re starting from zero Playwright knowledge, I’d actually pause here and go build a basic Playwright + TypeScript project first, because trying to learn Playwright fundamentals and MCP configuration at the same time is a rougher ride than either one alone. Ask me how I know — more on that in the mistakes section.

What I Wanted To Prove Before I Started

I went into this with three specific questions, because “let’s see if MCP is cool” isn’t a testable hypothesis, and I wanted this log to actually be useful rather than a vague impressions piece.

Can an MCP-connected AI assistant generate a genuinely usable Playwright test — not a toy example, but something close to what I’d accept in a real project — from a plain-English description of a user flow?
Does MCP meaningfully help with debugging flaky or failing tests, compared to me manually reading stack traces and trace viewer output the way I always have?
What’s the actual time and cost trade-off — is this faster than writing tests by hand once you include all the setup, prompt iteration, and verification time, or is it a novelty that looks impressive in a five-minute demo and falls apart in real project conditions?

I’ll answer all three honestly by the end of this post, with the actual numbers I tracked, not marketing-speak. Let’s get into the setup.

Prerequisites: What You Actually Need Before Starting

I wasted about forty minutes on my first attempt because I skipped straight to “install the MCP server” without checking basics. Save yourself that time. Here’s the actual checklist, in the order I wish I’d followed it.

1. Node.js and npm (or pnpm/yarn)

The Playwright MCP server ships as an npm package, so you need a reasonably current Node.js installation. I was already on Node 20 LTS from my Playwright learning setup, and I didn’t need to change anything. If you’re not sure what version you’re running, check it before you do anything else:

node -v
npm -v

If you’re on anything older than Node 18, upgrade first. I’ve seen enough weird, hard-to-diagnose failures from outdated Node versions in Playwright projects that I no longer try to debug around it — I just upgrade and move on.

2. An Existing Playwright + TypeScript Project

You don’t strictly need an existing project — you can scaffold one fresh — but I’d strongly recommend working inside a project you already understand. Part of what made this experiment useful for me was that I could immediately tell when generated code didn’t match my existing Page Object Model conventions, because I already knew what “normal” looked like in my own codebase. If this is your very first Playwright project, I’d actually build a small manual test suite first (even five or six tests) before layering MCP on top, purely so you have a baseline for comparison.

3. An MCP-Compatible AI Coding Assistant

This is the piece that trips people up, because “AI coding assistant” isn’t one thing — you need one that specifically supports MCP as a client. At the time I did this, my options included GitHub Copilot in VS Code (with MCP support enabled), Claude Code, and Cursor. I chose to run this primarily through VS Code with GitHub Copilot’s agent mode, since that’s where my existing Playwright project already lived, and I cross-checked a few steps using Claude Code in the terminal to see if the experience differed meaningfully. Spoiler: it mostly didn’t, which itself was an interesting finding I’ll come back to.

4. Basic Command Line Comfort

You’ll be editing a JSON config file and running terminal commands. If you’ve set up a Playwright project before, you already have enough command-line comfort for this. If you’ve only ever run Playwright through an IDE’s green play button, budget a little extra time for the terminal parts.

5. Patience for the First 30 Minutes

This isn’t really a technical prerequisite, but it’s the most honest one. The first half hour of any new protocol-based setup is going to involve at least one confusing error message that has nothing to do with your actual test automation skills and everything to do with a missing permission, a misplaced comma in JSON, or a server that needs a restart. I hit all three. Expect friction, don’t panic, and read the error message slowly instead of immediately pasting it into a search engine — I caught two of my three setup mistakes just by rereading the error a second time.

Step-by-Step: Setting Up the Playwright MCP Server

Here’s exactly what I did, in order, including the parts that didn’t work the first time.

Step 1: Installing the Playwright MCP Server

The Playwright team maintains an official Playwright MCP server that exposes browser automation as a set of callable tools. I installed it as a dev dependency inside my existing Playwright TypeScript project rather than globally, mainly because I wanted the version pinned per-project the same way I already pin my Playwright version:

npm install -D @playwright/mcp

This installs the server package but doesn’t run it or connect it to anything yet — that’s a separate configuration step, and conflating “installed” with “configured” was my first small confusion. The package being in node_modules does nothing on its own until an MCP client is told how to launch it.

Step 2: Creating the MCP Configuration File

This is where I made my first real mistake. VS Code (and most MCP clients) look for an MCP configuration file that tells them which servers exist and how to launch them. In VS Code’s case, this lives in a file typically called mcp.json, either at the workspace level (inside a .vscode folder) or at the user/global level. I created mine at the workspace level, inside .vscode/mcp.json, because I wanted this configuration to travel with the project and be something I could eventually check into version control for teammates.

Here’s the configuration I ended up with, after fixing my first mistake (which I’ll explain right after):

{
  "servers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

Mistake #1, honestly recorded: my first version of this file had the top-level key as "mcpServers" instead of "servers", because I’d been reading a setup guide written for a different MCP client (Claude Desktop uses mcpServers as the top-level key in its own config format). VS Code silently ignored the file — no error, no server showing up in the MCP panel, nothing. I spent close to twenty minutes assuming the npm install had failed before I actually diffed my config against the current VS Code documentation and caught the key name mismatch. Lesson learned the hard way: different MCP clients use slightly different config file schemas even though they’re all speaking the same underlying protocol, and copy-pasting a config snippet from the wrong client’s docs will fail silently rather than loudly. Always double check which client’s documentation you’re actually reading.

Step 3: Starting the Server and Verifying the Connection

Once the config key was fixed, VS Code picked up the server definition and showed it in the MCP servers panel with a “start” option. Starting it the first time triggered an npx download of the @playwright/mcp package (since I’d referenced @latest rather than the exact version already installed locally — a small inconsistency I fixed later by pointing the config at the local install instead of always fetching latest).

After starting, VS Code’s MCP panel listed the available tools the Playwright MCP server exposes — things like browser_navigate, browser_click, browser_snapshot, browser_type, and a handful of others for handling tabs, screenshots, and network requests. Seeing that tool list populate was the first genuinely satisfying moment of this whole experiment — it was concrete proof that the AI assistant now had a real, structured way to drive a browser, not just generate code from memory.

Step 4: A Sanity-Check Prompt Before Anything Serious

Before asking the assistant to generate any test code, I did a trust-building exercise: I asked it, in Copilot’s agent mode chat, to simply navigate to a public site and describe what it saw. I used a stable, well-known page for this rather than my actual application under test, specifically so I could verify the tool-calling was actually working against something predictable.

Prompt: "Using the Playwright MCP tools, navigate to https://playwright.dev
and tell me the exact text of the main heading on the page."

Watching the tool-call log was more interesting to me than the final answer. I could see, step by step, that the assistant called browser_navigate, then browser_snapshot, and only then answered based on the actual accessibility tree snapshot it got back — not from any prior knowledge about what playwright.dev looks like. That’s the exact behavior I needed to confirm before trusting this for real test generation: it was reading live page state, not hallucinating from training data.

This distinction matters enormously for testing specifically, because the entire value proposition collapses if the assistant is just guessing what a page probably contains based on similar pages it saw during training. A tool that hallucinates selectors is worse than useless in test automation — it’s actively dangerous, because a confidently wrong selector can look correct in generated code and only fail (or worse, silently pass against the wrong element) once it hits CI.

Generating My First Real Test With MCP

With the sanity check passed, I moved on to something closer to real work. I picked a login flow on a small demo e-commerce site I already use for blog examples, mostly because it’s stable, publicly accessible, and doesn’t require me to expose any real client project here. The goal: get the AI assistant to explore the login flow itself, using the browser tools, and generate a Playwright TypeScript test for it — rather than me writing the selectors by hand first and just asking for boilerplate.

The Prompt That Actually Worked

My first attempt at a prompt was too vague, and I got back a test that technically ran but used brittle, auto-generated CSS selectors instead of anything resembling good practice. Here’s that first, weaker attempt:

Prompt (weak): "Write a Playwright test for logging into the site."

This produced a test using selectors like page.locator('.css-1x2y3z'), which is exactly the kind of fragile, auto-generated class-based selector I’ve spent years telling junior engineers to avoid. The assistant had navigated the page correctly, but without explicit guidance it defaulted to whatever selector strategy required the least reasoning, not the most maintainable one.

My second attempt was much more specific about constraints, closer to how I’d actually brief a junior automation engineer:

Prompt (better): "Using the Playwright MCP tools, navigate to the login page
and explore its structure. Generate a Playwright TypeScript test for the
login flow using role-based and label-based locators only (getByRole,
getByLabel, getByPlaceholder) — do not use CSS or XPath selectors. Follow
the Page Object Model pattern already used in this project under
/pages. Include one positive test case (valid credentials) and one
negative test case (invalid password, expect an error message)."

This second prompt produced something genuinely usable. The assistant explored the page through browser_snapshot calls, correctly identified that the username field had an accessible label, generated a new page object file consistent with my existing folder structure, and wrote both the positive and negative test cases using locators I would have written myself. It wasn’t perfect on the first pass — the negative test’s assertion initially checked for exact error text that turned out to be slightly different from what the site actually displayed — but it was close enough that fixing it took two minutes, not twenty.

What the Generated Page Object Looked Like

For transparency, here’s a lightly trimmed version of what came out, so you can judge the quality yourself rather than take my word for it:

import { type Page, type Locator } from '@playwright/test';

export class LoginPage {
  readonly page: Page;
  readonly usernameInput: Locator;
  readonly passwordInput: Locator;
  readonly loginButton: Locator;
  readonly errorMessage: Locator;

  constructor(page: Page) {
    this.page = page;
    this.usernameInput = page.getByLabel('Username');
    this.passwordInput = page.getByLabel('Password');
    this.loginButton = page.getByRole('button', { name: 'Log in' });
    this.errorMessage = page.getByRole('alert');
  }

  async goto() {
    await this.page.goto('/login');
  }

  async login(username: string, password: string) {
    await this.usernameInput.fill(username);
    await this.passwordInput.fill(password);
    await this.loginButton.click();
  }
}

And the corresponding spec file:

import { test, expect } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';

test.describe('Login flow', () => {
  test('logs in successfully with valid credentials', async ({ page }) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.login('standard_user', 'validPassword123');
    await expect(page).toHaveURL(/.*\/dashboard/);
  });

  test('shows an error with an invalid password', async ({ page }) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.login('standard_user', 'wrongPassword');
    await expect(loginPage.errorMessage).toContainText('Invalid username or password');
  });
});

I want to be precise about what impressed me here and what didn’t. What impressed me: the locator strategy was genuinely good, matching the accessibility-first approach I already advocate for on this blog, and it was derived from actually reading the live DOM rather than guessing. What didn’t impress me: the exact error message text was wrong on the first attempt (a hallucination-adjacent issue, since it invented plausible-sounding text rather than reading the actual rendered error), and I had to explicitly ask it to re-check the live page state before it corrected itself. That correction is worth dwelling on, because it’s the clearest illustration I found of where MCP-connected generation is stronger than plain code-completion, and where it still isn’t a substitute for verification.

The Moment the Value Proposition Actually Clicked

Here’s the exact exchange, because I think it’s the most instructive part of this whole log. After the test failed on first run because the assertion text didn’t match, I didn’t rewrite it manually. Instead I fed the failure back in:

Prompt: "The negative test failed. Here's the Playwright error output: [pasted
error]. Use the MCP browser tools to actually trigger this error state on the
live page again and check the real text of the error message, then fix the
assertion."

What happened next was the thing that made me a believer in this workflow, at least for this narrow use case. The assistant used browser_navigate and browser_type tools to actually re-run the failing scenario live, took a fresh browser_snapshot, read the real error text off the actual page, and updated the assertion to match reality instead of guessing again. That’s fundamentally different from asking a plain code-completion model to “fix the assertion,” where it can only guess a plausible-sounding correction based on patterns in training data. Here it went and looked. That grounding in live application state, not the code generation itself, is the actual differentiator MCP brings to test automation, and it’s worth being precise about that distinction when you’re evaluating whether this is worth adopting — the code generation quality alone isn’t dramatically different from a good non-MCP coding assistant; the live verification loop is what’s new.

Case Study: Debugging a Genuinely Flaky Test With MCP

Generating a fresh test with live guidance is one thing. Debugging an existing, intermittently failing test is a different and, for my day job, more relevant skill. So my next experiment was deliberately harder: I took a test from my existing suite — a search-and-filter flow on a demo product catalog — that failed roughly one run in five, with no obvious pattern, and asked the MCP-connected assistant to investigate.

Setting Up the Investigation

I gave it the failing test file, the Playwright HTML report from a failed run, and this prompt:

Prompt: "This test fails intermittently, roughly 1 in 5 runs, with a timeout
waiting for the filtered results to appear. Use the Playwright MCP tools to
navigate through this flow multiple times and observe the actual timing and
DOM state around the filter action. Don't just guess from the code — actually
reproduce it if you can, and tell me what you observe before proposing a fix."

I want to flag something about this prompt: the phrase “don’t just guess from the code” was deliberate and, based on earlier attempts, necessary. Without that instruction, the assistant’s first pass at this exact scenario jumped straight to a generic, plausible-sounding diagnosis — “this is likely a race condition, add a waitForTimeout” — without actually looking at anything live. Adding an explicit “actually reproduce it, use the tools” instruction changed the behavior meaningfully, and this became one of my core learnings: MCP tool access doesn’t automatically get used unless the prompt nudges the model toward using it rather than pattern-matching from code alone.

What It Actually Found

With the more explicit prompt, the assistant ran the flow several times through the browser tools, comparing DOM snapshots immediately after the filter click across multiple attempts. It correctly identified that the filtered result count element updated in two separate renders — an initial optimistic render showing a stale count, followed by a second render a few hundred milliseconds later with the actual filtered results — and that my test’s assertion was sometimes catching the intermediate stale state depending on machine speed and network timing.

This diagnosis was correct, and more importantly, it matched what I later independently confirmed by manually stepping through the Playwright trace viewer myself. The proposed fix was also solid — replacing a fixed wait with a proper expect.poll() or a wait on a specific network response tied to the filter API call rather than an arbitrary timeout:

// Before: brittle, racy
await page.click('[data-testid="apply-filter"]');
await page.waitForTimeout(1000);
await expect(page.getByTestId('result-count')).toHaveText('12 results');

// After: waits on the actual data source, not a guessed duration
const filterResponse = page.waitForResponse(resp =>
  resp.url().includes('/api/products/filter') && resp.status() === 200
);
await page.click('[data-testid="apply-filter"]');
await filterResponse;
await expect(page.getByTestId('result-count')).toHaveText('12 results');

After applying this fix, I ran the test fifty times in a loop overnight. Zero failures, compared to roughly ten failures I’d have expected out of fifty runs at the prior flake rate. That’s a real, measurable result, not a vibes-based improvement, and it’s the single strongest piece of evidence I have for MCP-assisted debugging actually working rather than just looking impressive in a demo.

Where It Struggled

In fairness, not every debugging session went this cleanly. I tried a second flaky test — one tied to an animation timing issue in a modal dialog — and the assistant’s live investigation correctly identified that a CSS transition was involved, but its first two proposed fixes were both variations of “wait a bit longer,” which is the same brittle pattern it was supposed to be moving me away from. I had to explicitly push it toward checking for a specific class change or animationend event rather than a duration, and even then the final fix was something I significantly rewrote myself. So: strong result on the network-timing case, mediocre-to-average result on the pure CSS-animation case. I’m including both because a single success story would be a marketing post, not a log.

Building a Small Multi-Test Suite: The Real Workflow Test

A single generated test proves feasibility. It doesn’t prove this is a workable day-to-day workflow. So I spent a full afternoon building a small but realistic five-test suite covering login, product search, add-to-cart, checkout form validation, and order confirmation, entirely through this MCP-assisted process, timing myself against how long I estimated the same suite would take me to write by hand.

My Actual Workflow, Step by Step

Describe the flow in plain English, including the specific assertions I cared about — not just “test checkout” but the exact validation rules I wanted covered (empty field errors, invalid card format, successful order confirmation number appearing).
Let the assistant explore the live page through MCP tools before generating any code, explicitly asking it to report back what fields, labels, and states it found, so I could sanity-check its understanding before it wrote a single line.
Review the exploration summary and correct any misunderstanding before code generation — this step alone saved me from at least two wrong-assumption tests, because I caught, for example, that it had mistaken a disabled “Place Order” button state for a hidden one.
Generate the test, explicitly reusing existing page objects where they already existed and creating new ones following the same conventions where they didn’t.
Run it immediately, not batch all five and run them at the end — catching failures one test at a time kept each debugging cycle small and specific.
For any failure, feed the real error back in and ask for a live re-verification rather than a guessed fix, following the same pattern from the debugging case study above.

The Actual Time Comparison

Here are my honest numbers, tracked with a plain stopwatch, not rounded to make a better story:

Test	MCP-assisted time	My estimated manual time
Login (positive + negative)	14 minutes	~20 minutes
Product search	11 minutes	~15 minutes
Add to cart	9 minutes	~12 minutes
Checkout validation	27 minutes	~35 minutes
Order confirmation	13 minutes	~15 minutes

Total: roughly 74 minutes MCP-assisted versus an estimated 97 minutes manual — about a 24% time saving on this particular suite, for someone already fluent in Playwright and comfortable reviewing generated code critically. I want to be careful about generalizing that number. This was a demo application with clean, accessible markup. I strongly suspect the gap would shrink, or even reverse, on a legacy application with poor accessibility semantics and inconsistent component naming, because the exploration step would take longer and generated locators would need more manual correction. I haven’t tested that scenario yet, and I’ll be honest about that gap rather than implying I have.

Traditional Playwright Workflow vs. MCP-Powered Workflow

After finishing the suite above, I sat down and wrote out an honest side-by-side comparison, based on lived experience rather than assumption, of how my regular Playwright workflow differs from this MCP-assisted one.

Aspect	Traditional Playwright workflow	MCP-powered workflow
Locator discovery	Manual inspection via DevTools or Playwright’s codegen recorder	AI reads live accessibility snapshot and proposes locators
Debugging failures	Manual trace viewer analysis, console log review	AI re-runs scenario live through browser tools to observe real state
Boilerplate generation	Manual, or via snippets/templates	Generated automatically, consistent with existing conventions if prompted well
Requires human review	Yes, always	Yes, arguably more so, since confident-looking output can still be wrong
Learning curve for beginners	Steep but well-documented, huge community resources	Additional layer of config and prompt-writing skill on top of Playwright itself
Cost	Free (open-source Playwright)	Free tooling, but consumes paid AI assistant usage/tokens
Suitability for CI-only, no-AI environments	Fully suitable	Not directly — this is a development-time authoring aid, not a CI runtime tool

That last row deserves emphasis because I think it’s the most commonly misunderstood part of MCP-powered testing when people first hear about it. MCP, in this workflow, is a development-time assistant that helps you author and debug tests faster. The tests it produces are ordinary Playwright TypeScript tests — once written, they run in your normal CI pipeline exactly like any other Playwright test, with no MCP server, no AI model, and no live browser-tool-calling involved at runtime. I want to be explicit about this because I’ve seen some confused takes online implying MCP somehow runs inside your CI test execution, which isn’t accurate for this workflow. MCP lives in the authoring and debugging loop, on your machine, with a human in it.

Security Considerations I Had to Think Through

Because an MCP server can drive a real browser and, depending on configuration, access real credentials, cookies, and application state, I didn’t treat this as a purely productivity-focused experiment. I’ve written a full, dedicated MCP server security checklist elsewhere on this blog, and I’d point you there for the complete list, but three concerns were specific enough to this test automation use case that I want to flag them here directly.

1. Never Point an Exploration Session at Production With Real Credentials

Every exploratory session in this experiment ran against a demo or staging environment with disposable test accounts. An MCP-connected assistant that can click, type, and submit forms is, functionally, an agent capable of taking real actions in a real application. I would not let it loose against a production login form with a real customer account, for the same reason I wouldn’t hand a new hire production credentials on day one without supervision.

2. Be Careful What Gets Pasted Into Prompts

When feeding error output, HAR files, or trace data back into a prompt for debugging, it’s easy to accidentally include sensitive data — auth tokens in request headers, session cookies, or PII in test fixture data. I got into the habit of scanning pasted content for anything sensitive before submitting it, and where possible, using synthetic test data specifically so this wasn’t a live concern.

3. Review Generated Code Before It Touches Shared Branches

This one isn’t unique to MCP, but it’s worth repeating because the fluency of generated code can lull you into skipping review. Every test generated in this experiment went through the same pull request review process as any hand-written test would, and in two cases across the whole experiment, review caught a generated assertion that was technically passing but testing something looser and less meaningful than intended — a classic case of the test looking green for the wrong reason.

Common Mistakes I Made (So You Don’t Have To)

I’ve scattered several of these through the log already, but I want to consolidate them into a single list, because collecting my own failure points was genuinely more useful to me while writing this than the successes were.

Copying config syntax from the wrong client’s documentation — the mcpServers vs servers top-level key mismatch that cost me twenty silent minutes.
Vague prompts producing technically-correct-but-brittle code — my first “write a Playwright test for logging in” attempt, which defaulted to fragile CSS selectors because I didn’t specify a locator strategy.
Trusting a diagnosis that hadn’t actually used the live tools — the first flaky-test debugging attempt that guessed “race condition, add a wait” without reproducing anything, until I explicitly demanded live reproduction.
Learning Playwright fundamentals and MCP configuration simultaneously — on a separate side attempt with a completely fresh project (not covered in the main log above), I tried to skip straight to MCP-assisted generation before I had a solid mental model of Playwright’s own APIs, and I couldn’t tell good generated code from subtly wrong generated code, because I didn’t have a strong enough baseline yet. Get comfortable with Playwright itself first.
Not setting a token/usage budget before starting — my first debugging session, with a lot of back-and-forth exploration, consumed a noticeably larger chunk of my monthly AI assistant usage than a single test’s value probably justified, mostly because I was still learning how to prompt efficiently.
Assuming the same prompt style works across every MCP client — a prompt that worked cleanly in VS Code’s Copilot agent mode needed rephrasing to get equivalent tool-calling behavior in Claude Code, likely due to differences in each assistant’s default reasoning-before-tool-use behavior.
Batching too many tests before running any — my very first suite attempt (not the one detailed above) tried to generate all five tests before running any of them, which meant debugging five accumulated failures at once instead of one small failure at a time. Switching to a run-one-test-at-a-time loop fixed this immediately.

The Cost and Token Question Nobody Talks About Honestly

Almost every piece of MCP content I read before starting this experiment glossed over token and usage cost entirely, as if the AI assistant were a free, unlimited resource. It isn’t, and if you’re going to propose this workflow to a manager or a team, you need real numbers, not vibes.

I tracked my usage across this entire experiment — roughly eight hours of hands-on work spread across several sessions, covering the sanity check, the login test generation, the two debugging case studies, and the five-test suite. Exploratory sessions with lots of back-and-forth (particularly the flaky-test debugging) consumed noticeably more usage than direct, well-specified generation prompts. My rough takeaway: a tightly scoped prompt with clear constraints (locator strategy, existing conventions, exact assertions) is both faster to review and cheaper to run than an open-ended “explore this and figure it out” prompt, even though the open-ended style feels more impressive in a demo. For day-to-day team adoption, I’d lean hard toward the tightly-scoped style as the default habit, saving exploratory sessions for genuinely ambiguous debugging cases like the flaky test example above.

If your organization is on a metered or seat-limited AI assistant plan, this is worth surfacing to whoever owns that budget before rolling MCP-assisted testing out to a full QA team, because usage patterns that feel trivial for one engineer experimenting can add up quickly across a team of six or eight automation engineers all running exploratory sessions daily.

Is MCP-Powered Test Automation Ready for Real QA Teams?

This is the question I most wanted an honest answer to, and after this experiment my answer is: yes, but as an assistant layered onto skilled engineers, not as a replacement for automation expertise, and not yet as something I’d hand to someone who doesn’t already understand Playwright fundamentals.

Here’s my reasoning, broken into where I think it’s genuinely ready and where I think the hype outpaces the reality right now.

Where It’s Genuinely Ready

Accelerating test authoring for engineers who already know Playwright. The time savings I measured, while modest (~24% on a clean demo app), were real and reproducible, and I’d expect them to compound over a larger suite once prompting habits are well established.
Debugging flakiness rooted in real timing or state issues. The live-reproduction capability is a genuine, structural advantage over guess-based debugging, and it’s the single feature I’d miss most if I went back to a non-MCP workflow.
Onboarding a new page or flow into an existing convention-driven suite. When there’s already a clear Page Object Model pattern to follow, the assistant extending that pattern correctly was consistently strong across every test I generated.

Where the Hype Outpaces the Reality

It is not a substitute for understanding Playwright. Every single fix, correction, and quality judgment I made in this log depended on already knowing what good Playwright code looks like. A team member without that baseline would accept the bad first draft (fragile CSS selectors, guessed assertions) without knowing it was bad.
It doesn’t remove the need for code review discipline. If anything, fluent, confident-looking generated code makes review discipline more important, not less, because the failure mode shifts from “obviously broken” to “plausibly correct but subtly wrong.”
Legacy, poorly-labeled applications will likely blunt the advantage significantly. Everything I tested here was on clean, accessible, well-structured demo markup. I have not yet tested this against a messy legacy enterprise app with inconsistent naming and poor accessibility semantics, and I suspect the exploration step would take meaningfully longer and produce less reliable locators there. I’ll follow up on this blog once I’ve actually run that experiment, rather than speculate further here.
Team-wide rollout needs governance, not just enthusiasm. The security considerations above — environment restrictions, credential handling, review gates — need to be actual team policy before this scales past one curious engineer experimenting on their own machine.

If you’re setting this up yourself after reading this log, here’s the checklist I’d have wanted on day one.

Learn Playwright fundamentals first. Don’t let MCP be your first introduction to browser automation — you need a baseline to judge generated output against.
Double-check which MCP client’s documentation you’re reading before writing a config file; the schema differs between clients even though the underlying protocol is shared.
Be explicit about locator strategy in every generation prompt. Don’t assume the assistant will default to accessibility-first locators without being told.
Explicitly instruct the assistant to use live tools for debugging, rather than reasoning from code alone — the phrase “actually reproduce it” made a measurable difference in my experiments.
Run and review one test at a time, especially early on, rather than batch-generating a whole suite before verifying any of it.
Never run exploratory or debugging sessions against production with real credentials. Use staging or demo environments with disposable test accounts.
Treat every generated test through the same PR review process as hand-written code. Fluency is not correctness.
Track your usage/cost for at least the first few weeks before proposing team-wide adoption, so you can give an honest cost estimate rather than an assumption.

Frequently Asked Questions About MCP Test Automation

What is MCP in the context of test automation?

MCP, or Model Context Protocol, is an open standard that lets AI coding assistants connect to external tools — like a Playwright-driven browser — through a standardized set of callable actions. In test automation, this means the AI assistant can navigate, click, type, and read a real live page instead of only generating code from training data, which improves both test generation accuracy and debugging quality.

Do I need to know Playwright before trying an MCP test automation workflow?

Yes, strongly recommended. Based on my own experience, every useful correction and quality judgment I made throughout this experiment depended on already understanding Playwright fundamentals. Without that baseline, it’s very difficult to tell a good generated test from a subtly broken one.

Is MCP-powered testing free to use?

The Playwright MCP server itself is open-source and free. However, running it requires an MCP-compatible AI coding assistant, and most of those (GitHub Copilot, Claude, and similar tools) consume paid usage or tokens as you interact with them, so there is a real ongoing cost tied to how much you use the workflow, not just a one-time setup cost.

Does MCP replace the need for Playwright test scripts in CI/CD?

No. In this workflow, MCP is a development-time authoring and debugging aid. The tests it helps you generate are ordinary Playwright TypeScript files that run in your CI pipeline exactly as any hand-written test would, with no MCP server or AI model involved during actual CI execution.

What’s the biggest risk of using MCP for test automation?

Two risks stood out most in my experience: first, confidently wrong generated code (correct-looking but subtly flawed assertions or selectors) slipping through if code review discipline is relaxed; and second, security exposure if exploratory sessions are pointed at production environments with real credentials instead of staging or demo accounts.

Which MCP client should beginners start with — Copilot, Claude Code, or Cursor?

Based on my own testing, the underlying tool-calling behavior was broadly similar across the clients I tried, so I’d recommend starting with whichever AI coding assistant you already use daily, rather than adding a second unfamiliar tool on top of a new protocol. For me, that meant starting in VS Code with GitHub Copilot’s agent mode, since that’s where my existing Playwright project already lived.

How much faster is MCP-assisted test writing compared to writing tests by hand?

In my own timed five-test suite against a clean demo application, I measured roughly a 24% time saving compared to my estimated manual writing time. I’d treat that number as an upper-bound estimate for well-structured, accessible applications rather than a universal figure — I expect it to shrink on messier, legacy applications, which I haven’t tested yet.

Can MCP help fix flaky Playwright tests?

Yes, and this was the most convincing result in my experiment. By explicitly instructing the assistant to reproduce a flaky test live through the MCP browser tools rather than guessing from code alone, it correctly diagnosed a real race condition between an optimistic UI render and an actual API response, and the resulting fix eliminated the flake entirely across fifty repeated runs.

Is it safe to give an AI assistant this level of browser control?

It’s safe with the right guardrails: restrict sessions to staging or demo environments with disposable credentials, avoid pasting sensitive data like auth tokens into prompts, and always route generated code through normal code review before merging. Treat the assistant’s browser access the way you’d treat a new team member’s access — useful, but supervised until trust is established.

What I’d Do Differently Next Time

If I were starting this whole experiment over, knowing what I know now, here’s what I’d change.

I’d write my prompt conventions down on day one instead of discovering them through trial and error. Things like always specifying locator strategy, always explicitly requesting live reproduction for debugging, and always working test-by-test instead of batching — these became habits by the end of this log, but they cost me real time to arrive at. If you’re starting fresh after reading this, you can skip straight to those habits.

I’d also set up a small, disposable demo environment specifically for MCP experimentation before touching anything connected to real project work, purely so early config mistakes (like my mcpServers vs servers mix-up) happen in a low-stakes sandbox rather than while I’m trying to actually get something done.

Finally, I’d track token usage from the very first session instead of only noticing the cost question partway through. Having real numbers from day one would have made the cost-benefit section of this post more precise, and it’s the kind of detail that matters most if you’re the one who eventually has to justify this to a manager.

Final Verdict: Should You Build Your Own MCP Test Automation Workflow?

If you’re already comfortable with Playwright and TypeScript, and you’re curious rather than desperate for a silver bullet, I’d say yes — this is worth a focused weekend or two of hands-on experimentation, following roughly the same order I did: sanity check, single test generation, deliberate debugging case study, then a small multi-test suite with honest time tracking. You’ll learn more from your own mistakes in that process than from any single guide, including this one.

If you’re still building your core Playwright fundamentals, I’d hold off on MCP specifically and come back to it once you’re confident writing and debugging Playwright tests without AI assistance. Everything valuable I got out of this experiment depended on already having the judgment to catch what the assistant got wrong, and that judgment has to come from hands-on Playwright experience first, not from the MCP layer itself.

And if you’re evaluating this for a whole QA team, my honest recommendation is: pilot it with one or two senior automation engineers first, track real time and cost numbers the way I did here, and build the security and review governance before scaling it out — not after. The technology genuinely works for the use cases I tested. The organizational discipline around it is what will determine whether it’s a durable improvement to your team’s workflow or an expensive, ungoverned experiment.

This is where my honest log ends for now — a working MCP test automation setup, real generated tests I trust enough to keep in my project, one genuinely fixed flaky test, and a list of mistakes I’m glad happened to me instead of to you. If you build your own version of this, I’d genuinely like to hear what broke for you first — my guess is it won’t be the same thing that broke for me, and that’s exactly the kind of detail that makes these logs worth writing and worth reading.

A Closer Look: The Playwright MCP Tools I Actually Used

Once I’d been through a few sessions, I realized I kept relying on the same handful of MCP tools out of the full set the Playwright MCP server exposes. Understanding what each one actually does — rather than treating the tool list as a black box — made my prompts noticeably more precise, because I could reference specific tool behavior when something wasn’t working as expected. Here’s my working reference, built from actual usage, not just reading the spec.

Tool	What it does	When I saw it get used
`browser_navigate`	Opens a URL in the controlled browser instance	First step of nearly every session
`browser_snapshot`	Captures an accessibility-tree snapshot of the current page state — this is how the model “reads” the page	After every navigation and after most interactions, to verify state before deciding next steps
`browser_click`	Clicks an element identified from the snapshot	Login button, filter buttons, add-to-cart buttons during exploration
`browser_type`	Types text into a focused input field	Filling login credentials, search terms, checkout form fields during exploration
`browser_take_screenshot`	Captures a visual screenshot, distinct from the structural accessibility snapshot	Verifying visual state during the modal animation debugging session, where structure alone wasn’t enough
`browser_network_requests`	Lists recent network requests and responses	The filter-timing flaky test investigation, to identify the actual API call backing the UI update
`browser_console_messages`	Reads browser console output	Checking for JS errors when a checkout form field wasn’t responding as expected
`browser_wait_for`	Waits for a specific condition or text to appear	Confirming an error message actually appeared before reading its exact text

The single most important tool in that list, in terms of shaping my understanding of this whole workflow, is browser_snapshot. Every time I saw the assistant make a good decision, it was preceded by a fresh snapshot. Every time I saw it make a bad guess, it was because it skipped straight to generating code without one. If I only teach a colleague one thing about prompting an MCP-connected assistant for testing, it’s this: ask it to snapshot before it decides, every time, especially after any action that changes page state.

Full Walkthrough: The Checkout Validation Test

Of the five tests in my suite, checkout validation took the longest and taught me the most, so it’s worth walking through in more depth than the summary table above allowed. This flow had multiple field-level validations — empty required fields, an invalid card number format, and a mismatched expiry date — plus a final happy-path order confirmation check.

The Exploration Phase

I started with an exploration-only prompt, deliberately withholding the “generate a test” instruction so I could review its understanding first:

Prompt: "Using the Playwright MCP tools, navigate to the checkout page with an
item already in the cart. Explore the form fields. Then try submitting with
all fields empty, and report exactly what validation errors appear and which
elements they're attached to. Don't generate any code yet — just report what
you observe."

This produced a genuinely useful report: it correctly identified six required fields, noted that validation errors appeared inline below each field with a shared role="alert" pattern, and — this is the part that mattered — flagged that the “Card Number” field’s error only appeared after the field lost focus (a blur-triggered validation), not immediately on form submission like the others. That’s exactly the kind of subtle behavioral detail that’s easy to miss when writing a test from a quick manual click-through, and having it surfaced explicitly, before any code was generated, meant I could plan the test’s structure correctly from the start instead of discovering the blur-triggered validation as a confusing failure later.

The Generation Phase

With that understanding confirmed, I moved to generation, referencing the exploration findings explicitly in the prompt so the assistant wouldn’t have to re-derive them:

Prompt: "Based on what you just found, generate a Playwright TypeScript test
file for checkout validation. Cover: (1) submitting with all fields empty
shows all expected errors, (2) an invalid card number format shows its error
after the field loses focus — remember this one is blur-triggered, not
submit-triggered, (3) a fully valid submission shows an order confirmation
number matching pattern /ORD-\d{6}/. Use the existing CheckoutPage object if
one exists in /pages, extend it if it's incomplete."

The resulting spec correctly handled the blur-triggered case by explicitly calling .blur() (via focusing the next field) rather than just asserting immediately after typing, which is exactly the kind of detail that would have produced a flaky or wrong test if the exploration phase had been skipped:

test('shows invalid format error for card number after blur', async ({ page }) => {
  const checkout = new CheckoutPage(page);
  await checkout.goto();
  await checkout.cardNumberInput.fill('1234');
  await checkout.expiryInput.click(); // triggers blur on card number field
  await expect(checkout.cardNumberError).toHaveText('Enter a valid 16-digit card number');
});

test('completes checkout with valid details and shows confirmation number', async ({ page }) => {
  const checkout = new CheckoutPage(page);
  await checkout.goto();
  await checkout.fillValidDetails();
  await checkout.submitButton.click();
  await expect(checkout.confirmationNumber).toHaveText(/ORD-\d{6}/);
});

This test passed on the first run, which was the only test in my entire five-test suite that did. I attribute that directly to the separate exploration phase catching the blur-triggered validation ahead of time, rather than discovering it as a confusing failure afterward. That’s become a permanent habit for me now: for any form-heavy flow, explore and report first, generate second, never combine the two into a single prompt.

Setting Up MCP Across Different Clients: What Actually Changes

Because I wanted to know whether my findings were specific to VS Code and GitHub Copilot or generalized across the MCP ecosystem, I repeated a scaled-down version of the login test experiment using two other MCP clients: Claude Code in the terminal, and Cursor. Here’s what stayed the same and what didn’t, including the actual config differences, since I think this comparison is genuinely underdocumented right now.

VS Code with GitHub Copilot (My Primary Setup)

Config lives at .vscode/mcp.json, with a top-level "servers" key, as shown earlier in this post. Copilot’s agent mode surfaces tool calls in a readable, step-by-step panel in the chat sidebar, which I found genuinely helpful for building trust early on, since I could watch exactly which tool was called and with what arguments before the final response came back.

Claude Code (Terminal-Based)

Claude Code reads its MCP configuration from a different location and schema — a .mcp.json file at the project root (or a user-level config, depending on scope), and critically, its top-level key is mcpServers, not servers. This is the exact schema difference that caused my original silent-failure mistake described earlier, because I’d been reading Claude Code-oriented documentation while trying to configure VS Code. Here’s what a working Claude Code config looked like for the same Playwright MCP server:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

Functionally, once correctly configured, Claude Code’s tool-calling behavior against the same login flow was very similar to what I saw in Copilot — same tools, same general sequence of navigate-snapshot-interact-verify. The most noticeable difference was in how much reasoning it narrated in text before making tool calls; Claude Code tended to explain its plan in more detail before acting, which I found slightly slower to work through but occasionally more useful for catching a wrong assumption before it burned a tool call on it.

Cursor

Cursor’s MCP configuration lives at .cursor/mcp.json, also using the mcpServers top-level key, matching Claude Code’s convention rather than VS Code’s. The generated code quality on the same login test was comparable to both other clients, and I didn’t find a strong enough difference in this narrow test to recommend one client over another purely on output quality. My actual recommendation, based on this comparison, is to pick based on which assistant already fits into your daily workflow rather than chasing a marginal quality difference between clients — the config schema difference is the thing most likely to actually trip you up, not any meaningful gap in test generation ability.

The One Config Detail Worth Double-Checking Everywhere

Across all three clients, I ran into a second, smaller gotcha worth flagging: the args array for launching the server via npx needs to match a version you actually want. Using @playwright/mcp@latest everywhere means every fresh server start could silently pick up a newer version with different tool names or behavior between sessions. After my first week, I switched to pinning an exact version across all three client configs, the same way I already pin my Playwright test runner version, purely for reproducibility:

{
  "servers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@0.0.29"]
    }
  }
}

Small detail, but it removed an entire category of “why did this behave differently today” confusion from my sessions.

Troubleshooting Log: Every Error I Actually Hit

I kept a running notes file of every error message during this experiment, mostly out of habit from years of automation debugging. I’m including the full list here because I think a curated “here’s what could go wrong” list is less useful than an honest “here’s exactly what did go wrong, on my machine, in this order.”

Error 1: MCP Server Not Appearing in the Tools Panel

Cause: The mcpServers vs servers top-level key mismatch described earlier. Fix: Match the exact schema for your specific client’s documentation, not a generic MCP guide. Time lost: ~20 minutes.

Error 2: “Command not found: npx” on Server Start

Cause: A stale terminal session in VS Code that had been open before I’d updated my Node/npm installation, so it was still pointing at an old PATH. Fix: Fully restarting VS Code, not just reloading the window. Time lost: ~10 minutes, mostly spent confused before realizing it was a PATH/session issue rather than an actual missing installation.

Error 3: Tool Calls Timing Out on a Slow-Loading Page

Cause: The demo site’s initial load was slower than the MCP server’s default navigation timeout during one specific session on a throttled connection. Fix: Nothing exotic — just retried once the connection stabilized. Worth knowing this exists as a possible failure mode before assuming a config or code problem when it happens.

Error 4: Generated Test Referenced a Page Object Method That Didn’t Exist Yet

Cause: The assistant assumed a fillValidDetails() helper existed on CheckoutPage because it was a reasonable pattern given the rest of the codebase, but I hadn’t actually built that specific method yet. Fix: I asked it to generate the missing method too, explicitly, rather than manually writing it myself, which worked cleanly once I noticed the gap. Lesson: generated tests can reference conventions that are reasonable but not yet real — always run the test immediately rather than just reading it and assuming it’s complete.

Error 5: A Snapshot Showing Stale State After a Single-Page-App Route Change

Cause: On the product search flow, a client-side route change (no full page reload) meant a snapshot taken immediately after a click sometimes captured the pre-navigation DOM state, because the SPA’s render hadn’t completed yet. Fix: Explicitly prompting for a wait-and-resnapshot step after any client-side navigation, similar to the filter-timing flaky test fix described earlier. This turned out to be a smaller, related version of the same root cause as that case study — async rendering catching a synchronous-feeling snapshot mid-update.

None of these were exotic, catastrophic failures. All five were the kind of small, slightly annoying friction that’s completely normal when adopting any new tool, and every one of them was resolved within a few minutes once I understood the actual cause rather than the surface-level symptom. I’m listing them in this much detail specifically so that if you hit any of these same five, you can skip straight past the confusion I went through and get to the fix.

Prompt Engineering for MCP Test Generation: What I Learned Through Trial and Error

I want to dedicate a full section to this because, more than the config files or the tool list, prompt phrasing was the single biggest lever affecting output quality throughout this entire experiment. The gap between a mediocre result and a genuinely good one was almost never about the underlying model being “smarter” or “dumber” on a given day — it was almost always about how precisely I’d specified the constraints. Here’s what I learned, organized by the specific pattern.

Pattern 1: Separate Exploration From Generation

This is the single highest-leverage habit from this whole experiment, and I’ve already shown it in action twice above (the login error message correction, and the checkout blur-validation discovery). The underlying principle: when you ask for exploration and generation in the same breath, the model tends to generate code based on assumptions rather than fully grounding itself in what it actually observed, because generating code is the more “primed” behavior from its training and exploring live state is the behavior you have to explicitly request and wait for. Splitting these into two separate prompts — first “explore and report,” then “now generate based on what you found” — consistently produced more accurate results across every single test in my suite.

Pattern 2: Name Your Constraints Explicitly, Every Time

I want to be blunt about this because it surprised me: the model does not reliably infer your team’s conventions just because they exist elsewhere in the codebase, at least not without being told to look. My weak first prompt (“write a Playwright test for logging in”) produced fragile selectors not because the model was incapable of better locators, but because I hadn’t told it what “better” meant in my context. Once I started explicitly naming locator strategy, existing folder conventions, and assertion style in every generation prompt, quality became consistently good rather than occasionally good. My working template, refined over the course of this experiment, ended up looking like this:

Using the Playwright MCP tools:
1. Navigate to [specific page/flow]
2. Explore [specific interaction] and report what you observe — field names,
   labels, error states, timing behavior — before generating any code
3. Once I confirm your findings, generate a Playwright TypeScript test that:
   - Uses only role/label-based locators (getByRole, getByLabel), no CSS/XPath
   - Follows the existing Page Object Model pattern in /pages
   - Covers these specific cases: [list exact cases]
   - Uses these specific assertions: [list exact expected outcomes, not vague ones]

Every test I generated using something close to this template needed less correction afterward than every test I generated with a looser, more conversational prompt. This isn’t a surprising finding if you’ve done any prompt engineering work before, but it’s worth stating plainly for testing specifically, because I’ve seen enough demo videos online that skip the “explicit constraints” step entirely, making the whole workflow look more magical and less deliberate than it actually is in practice.

Pattern 3: Feed Back Real Errors, Not Summarized Ones

When a generated test failed, my instinct in the first week was to paraphrase the failure in my own words — “it says the button wasn’t found” — rather than pasting the actual Playwright error output. Paraphrasing consistently produced worse fixes than pasting the real stack trace and error message verbatim, because paraphrasing strips out exact selector text, exact timeout durations, and exact expected-vs-actual values that the model could otherwise use directly. Once I started always pasting raw error output, correction quality improved noticeably, and this became a hard rule for me going forward: never summarize a failure when you can paste it.

Pattern 4: Explicitly Request Live Reproduction for Anything Timing-Related

Already covered in detail in the flaky test case study, but worth restating as a standalone pattern because it generalizes beyond that one example: any time a bug or failure might be related to timing, async behavior, or intermittent state, explicitly instruct the model to reproduce the scenario live through the MCP tools rather than reasoning from the code alone. Left to its own devices, the model’s default behavior leans toward pattern-matching a plausible-sounding explanation from similar bugs it’s seen in training data, which is often directionally right but imprecise in exactly the way that matters for actually fixing a flaky test rather than just papering over it with a longer wait.

Pattern 5: Set an Explicit “Stop and Ask” Boundary for Ambiguous Cases

Partway through the experiment I started adding a closing line to more complex prompts: “If anything about the expected behavior is ambiguous, stop and ask me rather than guessing.” This single addition caught at least two situations where my instructions genuinely were ambiguous — once about whether a “successful” checkout should also verify an email confirmation was sent (I hadn’t specified, and it correctly flagged the ambiguity rather than picking one interpretation silently) — and I’d recommend it as a default addition to any non-trivial generation prompt.

Coming From Selenium: A Mindset Shift, Not Just a Tooling Change

Given my own background — over a decade in Java and Selenium before this year’s shift toward Playwright and TypeScript — I want to address this section specifically to readers making a similar transition, because I think the MCP layer requires a different kind of mental adjustment than the Playwright-vs-Selenium adjustment did.

Moving from Selenium to Playwright was, for me, primarily a syntax and tooling adjustment: different API shapes, built-in auto-waiting instead of manual explicit waits, a different assertion library, TypeScript instead of Java. It was a real learning curve, but it was a learning curve about a tool.

Moving from “writing Playwright by hand” to “directing an MCP-connected assistant to help write Playwright” is a different kind of adjustment. It’s less about syntax and more about shifting from an implementer mindset to something closer to a reviewer-and-director mindset, at least for the generation phase. For over a decade, my automation skill was largely measured by how precisely and efficiently I could translate a manual test case into working code myself. In this workflow, a meaningful chunk of that translation work shifts to the assistant, and my value shifts toward specifying constraints precisely, catching subtle mistakes, and knowing when a generated result is actually good enough to trust.

I don’t think this makes deep Playwright and Selenium knowledge less valuable — if anything, this experiment convinced me it makes that knowledge more valuable, because it’s the thing that lets you catch the checkout test’s blur-triggered validation detail, or recognize a fragile CSS selector on sight, or know that “add a wait” is a lazy fix for a race condition rather than a real one. If you’re a senior Selenium engineer wondering whether AI-assisted, MCP-powered workflows will make your deep automation experience less relevant, my honest answer from this experiment is the opposite: your judgment becomes the actual bottleneck-breaking skill in this workflow, not a legacy skill being replaced by it.

A Rollout Plan Template for Introducing MCP Test Automation to a QA Team

Several readers of my earlier AI-testing posts have asked some version of “okay, but how would you actually introduce this to a team without it becoming chaos.” I didn’t have a good answer before this experiment. I now have a rough plan, based directly on the friction points I hit myself, that I’d propose if I were introducing this at my own organization.

Phase 1: Single-Engineer Pilot (1–2 Weeks)

One senior automation engineer, already fluent in the team’s chosen framework, runs exactly the kind of structured experiment documented in this post — sanity check, single test generation, one deliberate debugging case study, and a small multi-test suite with honest time tracking. The explicit deliverable at the end of this phase isn’t “tests,” it’s a short report: real time comparisons, real cost/usage numbers, a list of mistakes hit and how they were resolved, and an honest recommendation on whether to proceed. This mirrors exactly the structure of this blog post, and I’d argue that’s not a coincidence — writing this log forced a level of rigor I wouldn’t have applied to a purely internal, unrecorded experiment.

Phase 2: Governance Before Expansion (Before Adding a Second Person)

Before letting a second engineer touch this, put a few non-negotiables in writing: which environments MCP-connected sessions are allowed to touch (staging/demo only, never production with real credentials), a rule about not pasting sensitive data into prompts, and a requirement that all generated code goes through the exact same PR review process as hand-written code, with no shortcut for “AI-generated so it’s probably fine.” This phase is boring compared to Phase 1, and it’s exactly the phase most likely to get skipped by an enthusiastic team, which is precisely why I’d insist on it happening before expansion rather than after an incident forces it.

Phase 3: Small-Group Expansion (3–4 Engineers, 2–4 Weeks)

Expand to a small group, ideally engineers who are already comfortable with the underlying test framework rather than beginners, for the same reason I flagged earlier — the judgment to catch subtly wrong output has to already exist independently of this workflow. Track the same metrics as Phase 1 across the group: time comparisons, usage/cost, and a shared running list of prompt patterns that worked well versus poorly, similar to the pattern list in this post. This is also the phase where I’d expect legacy-application friction to show up for the first time, if your team’s applications are less clean than the demo app I used for this experiment, and I’d treat that friction as genuinely useful data rather than a failure of the pilot.

Phase 4: Team-Wide Adoption With an Internal Playbook

Only after Phases 1–3 have produced real numbers and a working set of prompt conventions would I roll this out team-wide, packaged as an internal playbook — config templates for whichever MCP client(s) your team standardizes on, the locator-strategy and exploration-first prompt conventions, the environment and review governance rules, and a realistic cost estimate based on actual Phase 1–3 usage data rather than vendor marketing figures. Skipping straight to Phase 4 without the earlier phases is, in my opinion, the most common way this kind of AI tooling adoption goes wrong — not because the technology fails, but because the governance and shared conventions never get established before ten people are all using it slightly differently.

Extending This Further: What I Haven’t Tried Yet

In the interest of the same honesty I’ve tried to maintain throughout this log, here’s what this experiment did not cover, and what I’d want to test before making broader claims.

Legacy or poorly-labeled applications. Everything here ran against clean, accessible demo markup. I genuinely don’t know how much the exploration-phase advantage shrinks against an application with generic div soup and no semantic labels, and I’ve flagged this uncertainty multiple times through this post rather than pretend otherwise.
Cross-browser and mobile viewport testing through MCP. All of my sessions ran against a single default browser context. I haven’t yet tested whether the exploration-and-generation workflow holds up equally well when directing the assistant to account for mobile viewport behavior or cross-browser quirks.
API-level test generation through MCP, as opposed to UI-driven browser testing. This entire experiment was scoped to UI automation via the Playwright MCP server specifically; API testing with tools like REST Assured wasn’t part of this setup at all, and I’d expect a meaningfully different MCP server and workflow for that.
Long-running suite maintenance over months Everything in this log covers initial authoring and debugging. I haven’t yet tracked whether MCP-assisted maintenance of an aging suite — updating tests as an application evolves — shows the same time savings as initial creation did.
Team-scale usage costs. My cost tracking in this post reflects one engineer’s usage over roughly eight hours, not a sustained multi-person rollout. The rollout plan above explicitly calls for tracking this at each phase precisely because I don’t have that data yet myself.

I’m planning follow-up posts on at least the legacy-application question and the long-running maintenance question, since both feel like the natural next experiments given where this one left off, and I’d rather come back with real numbers than speculate about either here.

MCP Test Automation vs. Plain AI Code Completion: A Direct Comparison

Before wrapping up, I want to directly address a question I got from a reader partway through drafting this post: “isn’t this just Copilot autocomplete with extra steps?” It’s a fair question, and I think the honest answer requires separating two genuinely different capabilities that often get blurred together in casual conversation about “AI-assisted testing.”

Plain AI code completion — the kind I’d already been using for months before this experiment, where Copilot suggests the next line or completes a function based on surrounding code and training data — is fundamentally a pattern-matching exercise against static text. It’s genuinely useful for boilerplate, for filling in an assertion pattern you’ve used ten times before, for writing a repetitive data-driven test table. But it has no access to your actual running application. It cannot know, without you telling it explicitly in a comment or prompt, that a specific error message reads exactly “Invalid username or password” rather than “Incorrect credentials” or any of a dozen plausible variants. It’s guessing from patterns, even when the guess is well-informed.

MCP-connected generation, as I’ve tried to demonstrate concretely throughout this log rather than just assert, adds a fundamentally different capability on top of that: the ability to actually look. The login test’s error-message correction, the checkout flow’s blur-triggered validation discovery, the flaky filter test’s real network-timing diagnosis — none of these were things plain code completion could have gotten right on the first guess, because none of them were inferable from code patterns alone. They required observing live application behavior. That’s not “autocomplete with extra steps.” It’s a structurally different capability, even though the final output — a TypeScript test file — looks superficially similar either way.

Where I’d push back on the hype in the other direction: for the roughly 60% of any given test that’s ordinary boilerplate — imports, describe blocks, standard Page Object Model scaffolding — plain code completion and MCP-connected generation perform about equally well, because that portion doesn’t depend on live application state at all. The real differentiator shows up specifically in the parts of a test that depend on exact, current, real application behavior: exact selector availability, exact error text, exact timing characteristics. If your test suite is mostly boilerplate with very few of those live-dependent details, the MCP advantage will be smaller than what I measured here. If your suite leans heavily on exact text assertions and timing-sensitive flows, as most real production suites eventually do, I’d expect the advantage to be larger, not smaller, than my modest 24% figure.

Glossary: Terms Used Throughout This Post

Since I started this post assuming very little prior MCP knowledge, I want to close with a quick-reference glossary of every term I’ve used, for anyone skimming back through or sharing this with a teammate who’s earlier in their own learning journey than even I was three weeks ago.

MCP (Model Context Protocol): An open standard defining how AI models connect to external tools and data sources through a common interface, avoiding the need for custom integrations between every AI assistant and every tool.
MCP server: A program that exposes a defined set of callable “tools” (actions) that an MCP client can invoke on behalf of an AI model. In this post, the Playwright MCP server exposing browser actions.
MCP client / host: The application that connects to an MCP server and lets the AI model call its tools — in this post, VS Code with GitHub Copilot, Claude Code, and Cursor.
Tool call: A single, structured invocation of one of the MCP server’s exposed actions, such as browser_navigate or browser_click, made by the AI model during a conversation.
Accessibility snapshot: A structured, text-based representation of a page’s accessibility tree (roles, labels, states), captured via browser_snapshot, which is how the AI model “reads” page structure without processing raw pixels.
Agent mode: A mode in AI coding assistants (like GitHub Copilot’s agent mode) where the model can autonomously plan and execute a sequence of tool calls to accomplish a multi-step task, rather than producing a single one-shot response.
Page Object Model (POM): A test automation design pattern that encapsulates page structure and interactions inside dedicated classes, keeping test files focused on behavior rather than low-level selector details. Referenced throughout this post as the existing convention the generated code needed to follow.
Flaky test: A test that passes and fails intermittently against unchanged code, usually due to timing, race conditions, or environmental inconsistency, as explored in the filter-timing case study above.
Role-based / label-based locators: Playwright locator strategies (getByRole, getByLabel) that target elements by their accessible semantics rather than brittle CSS classes or XPath, generally considered a best practice for maintainable test suites.

My Actual Three-Week Timeline, Day by Day

Since this post is billed as a log, I want to include the actual chronological shape of this experiment, not just the tidied-up thematic sections above. Real learning journeys are messy and non-linear, and I think showing that explicitly is more useful than pretending I moved cleanly from setup to mastery in one straight line.

Week 1: Confusion, Setup, and the First Small Win

Days 1–2: Reading around MCP conceptually before touching any code, including several vendor blog posts that I found more confusing than clarifying because they assumed I already knew what a “tool call” was. This is what eventually pushed me to write the plain-language “what is MCP” explanation earlier in this post — I wanted to write the explanation I actually needed and couldn’t find.

Day 3: First install attempt. Hit the mcpServers vs servers config mistake, lost roughly twenty minutes, eventually fixed it by directly comparing my file against VS Code’s own current documentation rather than a third-party guide.

Day 4: First successful server connection and the sanity-check prompt against playwright.dev. This was a small win, but it was the moment MCP stopped being an abstract concept and became something I’d actually watched work.

Days 5–7: First real test generation attempt (the login flow), including the weak-prompt/strong-prompt comparison detailed earlier. By the end of week one, I had one working, reviewed test and a much clearer sense of what “good prompting” needed to look like.

Week 2: The Debugging Deep Dive

Days 8–9: Attempted the flaky filter test debugging, first with a loose prompt that produced the lazy “add a wait” diagnosis, then with the more explicit “actually reproduce it” prompt that produced the real network-timing root cause. This was, in hindsight, the most important two days of the whole experiment, because it’s where I moved from “this generates plausible code” to “this can genuinely help debug a problem I hadn’t fully diagnosed myself yet.”

Day 10: Ran the fifty-iteration overnight verification of the flaky test fix. Waking up to zero failures out of fifty runs was, honestly, more satisfying than I expected it to be.

Days 11–12: The second, less successful debugging attempt against the modal animation timing issue, where the proposed fixes were mediocre and I ended up rewriting a meaningful chunk myself. Important to log this alongside the win, for the same honesty reasons I’ve mentioned throughout this post.

Days 13–14: Cross-client comparison work — setting up and testing the same login flow through Claude Code and Cursor, hitting the second config schema mismatch, and writing up the comparison notes that became the cross-client section above.

Week 3: The Full Suite and Writing This Up

Days 15–17: Building the five-test suite with honest stopwatch time tracking — login, search, add-to-cart, checkout, confirmation — including the exploration-first approach that caught the checkout flow’s blur-triggered validation before it became a confusing failure.

Days 18–19: Reviewing and tightening every generated test against the same PR checklist I use for hand-written code, including the two cases where review caught a technically-passing-but-too-loose assertion, mentioned in the security section above.

Days 20–21: Writing up this entire log, which forced a level of precision — actual timestamps, actual error messages, actual before/after code — that I don’t think I would have applied if I’d only been keeping private notes. If you’re considering running a similar experiment yourself, I’d genuinely recommend committing to writing it up publicly, or at least for an internal team audience, before you start. Knowing you’ll have to show your actual mistakes changes how carefully you track them.

Appendix: A Prompt Library You Can Reuse

Rather than make you extract prompt patterns from the narrative above, here’s a consolidated, ready-to-adapt library of the prompts that worked well across this experiment, organized by use case. Swap in your own page names, conventions, and flows.

Exploration Prompts

"Using the Playwright MCP tools, navigate to [URL/flow]. Explore the page
structure and report: field names and labels, button text, any validation
or error states you can trigger, and any timing behavior you notice (e.g.
delayed renders, blur-triggered validation). Don't generate any code yet —
report your findings first."

Generation Prompts

"Based on what you just found, generate a Playwright TypeScript test that:
- Uses only [getByRole/getByLabel/getByPlaceholder] locators, no CSS or XPath
- Follows the existing Page Object Model pattern in [/pages folder]
- Covers these specific cases: [list exact cases]
- Uses these specific assertions: [list exact expected text/values, not vague ones]
If anything about the expected behavior is ambiguous, stop and ask me rather
than guessing."

Debugging Prompts (Timing/Flakiness)

"This test fails intermittently with [exact symptom]. Here's the real error
output: [paste raw error, not a summary]. Use the Playwright MCP tools to
actually reproduce this scenario live and observe real DOM/network/timing
state — don't diagnose from the code alone. Report what you observe before
proposing a fix."

Debugging Prompts (Assertion/Text Mismatch)

"This assertion failed: [paste raw error]. Use the MCP browser tools to
actually trigger this state again on the live page and check the exact
current text/value, then update the assertion to match reality — don't
guess a corrected value."

Extending an Existing Suite

"Review the existing page objects in [/pages] and the existing test
conventions in [/tests]. Using the Playwright MCP tools, explore [new flow],
then generate a new test that matches the exact style, naming, and locator
conventions already used in this project — reuse existing page objects
where relevant and extend them rather than duplicating logic."

Pre-Merge Sanity Check

"Review this generated test file for: any CSS/XPath selectors that should be
role/label-based instead, any assertion that's looser than what was actually
specified, and any hardcoded wait/timeout that should instead wait on a
specific condition or network response. List concerns before I approve this
for merge."

That last prompt — using the assistant itself as a first-pass reviewer of its own generated code, before human review — is something I started doing near the end of this experiment and found genuinely useful as a supplement to, not a replacement for, my own review. It caught one of the two loose-assertion issues I mentioned in the security section before I even got to manual review myself, which saved a review cycle.

Where MCP Fits Around Your CI/CD Pipeline (And Where It Doesn’t)

I touched on this earlier in the comparison table, but it deserves a fuller explanation because I think it’s the single most common misconception I’ve seen floating around in casual MCP discussion, including in a couple of comment threads I read while researching this post.

Your CI/CD pipeline — whatever runs your Playwright suite on every pull request, whether that’s GitHub Actions, Jenkins, Azure DevOps, or something else — continues to run exactly the tests that get committed to your repository, using the standard @playwright/test runner, with no MCP server, no AI model, and no network calls to an AI assistant involved at execution time. This is true regardless of whether those tests were originally hand-written or MCP-assisted during authoring. Once a test file is committed, it’s just TypeScript calling Playwright’s API, and your pipeline doesn’t know or care how it was written.

This matters for a few practical reasons worth spelling out explicitly:

No new CI infrastructure is required. You don’t need to provision AI API access, manage tokens, or add any new service dependency to your existing pipeline just because some tests were authored with MCP assistance.
No new CI-time cost is introduced. The cost of this workflow, as detailed in the section below, is entirely a development-time, human-in-the-loop cost tied to authoring and debugging sessions — not a per-CI-run cost.
No new CI-time failure mode is introduced by the AI layer itself. A generated test can be wrong for all the usual reasons any test can be wrong (bad assertion, brittle selector, environment dependency), but it cannot fail in CI because “the AI model was unavailable,” since the AI model isn’t part of CI execution at all in this workflow.
Your existing CI reporting, retry logic, parallelization, and artifact collection (traces, screenshots, videos) all work completely unchanged, because from the pipeline’s perspective, nothing about the test execution model is different.

Where MCP-assisted debugging does intersect with CI, in my workflow, is downstream of a CI failure: when a test fails in the pipeline, I pull the failure artifacts (the trace file, the error output, sometimes a screenshot) and feed those into a local MCP-connected debugging session, following the same “paste the real error, request live reproduction” pattern from the case study above. That’s a human-initiated, local follow-up step triggered by a CI failure, not something happening inside the pipeline itself. I want to be precise about that distinction because I think it’s the source of most of the confusion I’ve encountered when this topic comes up.

A Note on Self-Healing Test Claims

I’ve written previously on this blog about self-healing tests in Playwright, and I want to draw an explicit line between that concept and what I’ve documented in this post, because they get conflated in some marketing content. Self-healing, in the way I’ve covered it before, generally refers to runtime mechanisms that automatically adapt a locator strategy when a primary selector fails during execution, without human involvement. What I’ve documented in this post is a different thing entirely: a human-directed, development-time authoring and debugging workflow, not an autonomous runtime healing mechanism. I did not test, and this post makes no claims about, tests automatically fixing themselves during a CI run without a human reviewing and committing the fix. If you’re evaluating vendor claims about “self-healing MCP-powered tests,” I’d push for a precise definition of what’s actually happening automatically versus what still requires a human in the loop, based on exactly this kind of distinction.

The Cost Math, Worked Through in Full

I gave rough qualitative guidance on cost earlier in this post. Here, I want to walk through the actual worked math behind my recommendation, so you can adapt the formula to your own team’s numbers rather than just trusting my summary.

The rough framework I used: for any given testing task, compare (a) the fully-loaded engineer time cost of doing it manually, against (b) the fully-loaded engineer time cost of doing it MCP-assisted, plus the AI assistant usage cost for that session. If (b) is meaningfully lower than (a), the workflow is paying for itself on that task; if not, you’re paying extra for a novelty on that particular task.

Manual cost = engineer_hourly_rate × manual_time_hours

MCP-assisted cost = engineer_hourly_rate × mcp_assisted_time_hours
                     + ai_assistant_usage_cost_for_session

Worth it if: MCP-assisted cost < Manual cost

Applying this to my own five-test suite numbers from earlier (74 minutes MCP-assisted vs. 97 minutes estimated manual), and using a rough illustrative engineer rate purely to show the shape of the calculation rather than assert a real figure for any specific organization: at even a modest hourly rate, the roughly 23-minute time saving alone likely covers a meaningful multiple of the AI assistant’s marginal usage cost for that session, assuming you’re on a seat-based subscription rather than a heavily metered pay-per-token plan where costs could scale differently. The bigger, harder-to-model variable in this formula, honestly, isn’t the AI usage cost line — it’s mcp_assisted_time_hours, because that number is highly sensitive to prompting skill and application cleanliness, both of which I’ve flagged as open variables throughout this post. A team just starting out should expect their early mcp_assisted_time_hours numbers to look worse than mine, the same way my week-one numbers would have looked worse than my week-three numbers if I’d tracked them separately, purely because prompting proficiency compounds with practice.

My practical recommendation, translating this math into action: don’t trust anyone’s published time-savings percentage, including mine, as a planning number for your own team. Run your own Phase 1 pilot (see the rollout plan above), track your own real numbers for at least a handful of tests across at least one full week, and build your business case from that data. The formula above is meant as a framework to plug your own numbers into, not a result to copy.

What Surprised Me Most, In Both Directions

Stepping back from the step-by-step log, I want to name the handful of things that genuinely surprised me during this experiment — some pleasantly, some not — because I think surprise is a more honest signal of what’s actually novel here than anything I could have predicted going in.

Surprise #1: The Value Wasn’t Where I Expected It

Going in, I assumed the headline value of MCP-powered test automation would be generation speed — writing tests faster. It turned out the more valuable capability, by a clear margin in my own experience, was debugging: the live-reproduction loop that correctly diagnosed the filter-timing race condition. I’d encourage anyone evaluating this for their own team to weight their pilot toward debugging scenarios, not just greenfield test generation, because that’s where I found the most defensible, measurable value.

Surprise #2: Prompt Phrasing Mattered More Than Model Choice

I didn’t extensively benchmark different underlying AI models against each other in this experiment — that wasn’t the goal — but across my sessions with different clients (which, at the time, were backed by different underlying models), the quality gap I observed between a vague prompt and a well-constrained prompt was consistently larger than any quality gap I noticed between clients. That’s a somewhat unglamorous finding — “learn to write good prompts” is less exciting content than “this specific tool is best” — but it’s what I actually observed, and I’d rather report that than manufacture a more dramatic tool-comparison narrative.

Surprise #3: Config Schema Differences Were the Biggest Practical Friction Point

I expected the friction to come from the AI reasoning itself — hallucinated selectors, wrong diagnoses, that kind of thing. Some of that happened, but the single largest chunk of wasted time across this whole experiment came from mundane, non-AI causes: mismatched config schema keys between clients, a stale terminal PATH, an @latest version pin causing inconsistent behavior between sessions. If I were advising someone just starting out, I’d now spend proportionally more prep time on “get the plumbing exactly right” than I originally budgeted for, and proportionally less worrying about whether the AI reasoning itself would be good enough.

Surprise #4: It Was Genuinely Good at Admitting Uncertainty, When Asked To

The checkout ambiguity example (whether “successful checkout” should also verify an email confirmation) is one instance of a broader pattern I noticed: when explicitly prompted to flag ambiguity rather than guess, the assistant reliably did so, rather than confidently picking an interpretation and hiding the fact that a choice had been made. This only happened when I explicitly asked for it, though — without that instruction, ambiguous cases were more likely to get silently resolved one way, which circles back to Pattern 5 in the prompt engineering section above.

Surprise #5: The Modal Animation Debugging Failure Bothered Me More Than It Should Have

This is a smaller, more personal observation, but worth including for honesty’s sake: after the strong result on the flaky filter test, I went into the modal animation debugging session with inflated expectations, and the mediocre result felt like more of a letdown than it objectively should have, given that one weaker result among several strong ones is a completely reasonable overall track record for any tool. I’m flagging this because I think it’s an easy trap for anyone piloting this kind of workflow — one early strong result can set expectations that make normal, ordinary limitations feel like bigger failures than they are. Calibrate your expectations to “a genuinely useful tool with real limitations,” not “a tool that will always get it right after one good demo.”

Extended FAQ: More Questions I Anticipated

Can I use MCP test automation with a framework other than Playwright?

This post specifically covers the official Playwright MCP server, which is scoped to browser automation through Playwright. The broader MCP standard itself is framework-agnostic — other tool vendors could and likely will build MCP servers for other automation frameworks or testing tools — but everything in this hands-on log is specific to the Playwright MCP server, and I haven’t tested equivalents for other frameworks.

Does MCP work with Playwright tests written in Java or Python, not just TypeScript?

The MCP server and AI assistant interaction I’ve documented here is language-agnostic in principle — the assistant is generating whatever language your project uses — but every example, prompt, and code snippet in this specific post reflects my own TypeScript-based project. If your team works in Java or Python Playwright bindings, the underlying MCP concepts (exploration-first prompting, live reproduction for debugging, explicit constraint-setting) should transfer, but I haven’t personally validated the generated code quality in those languages.

How is this different from Playwright’s built-in codegen recorder?

Playwright’s codegen tool records your own manual clicks and generates matching code — it’s a recording of human action, with no reasoning or judgment involved, and no debugging capability. MCP-connected generation is fundamentally different: the AI model is deciding what to explore and click based on a plain-English instruction, reasoning about what it observes, and can be redirected mid-task (“actually, check the error text again”) in a way a simple recorder cannot. They’re not competing tools, either — I still use codegen occasionally for quick, simple selector lookups, and reach for the MCP workflow for anything requiring exploration, judgment, or debugging.

What happens if the MCP server crashes or disconnects mid-session?

In my experience, this was rare, but when it happened (once, during a long exploratory debugging session), the AI assistant’s tool calls simply started failing with a clear connection error, and restarting the server from the MCP panel resolved it within a few seconds. It didn’t corrupt any files or leave anything in a bad state — the worst consequence was losing the context of that specific exploration and needing to restart the prompt.

Do I need a powerful/expensive AI model subscription tier for this to work well?

I didn’t run a systematic cost-tier comparison as part of this experiment, so I can’t give you a confident answer with real data behind it. What I can say is that the exploration-first, explicit-constraint prompting patterns documented throughout this post seemed to matter more for output quality than any tier difference I casually noticed, which suggests prompting discipline is probably a better first investment than immediately reaching for the most expensive available tier.

Is there a risk of vendor lock-in with a specific MCP client?

Less than you might expect, structurally, because the whole point of MCP as a standard is portability — the Playwright MCP server itself doesn’t change based on which client you connect it to, only the config file schema and the surrounding chat interface differ, as shown in the cross-client comparison section above. Your generated test files are ordinary Playwright TypeScript with zero dependency on any specific AI client once written. The actual lock-in risk, if any, is around learned prompting habits and internal playbooks being tuned to one client’s quirks, which is a much smaller migration cost than a genuine platform lock-in would be.

How do I convince a skeptical manager or teammate this is worth adopting?

Based on how I approached this myself: don’t lead with enthusiasm, lead with a small, time-boxed pilot that produces real numbers, the same way this whole post is structured. A specific, honestly-reported result — “here’s a real flaky test we fixed, here’s the actual before/after failure rate, here’s the actual time it took” — will land better with a skeptical audience than any general claim about AI productivity gains, and it gives you something concrete to point to rather than an argument to win.

Should QA managers be worried this replaces junior automation engineer roles?

Based on everything I observed in this experiment, I’d frame it differently: this workflow amplifies existing automation judgment rather than replacing the need for it. Every meaningful correction, every catch of a subtly wrong result, every good constraint in a prompt depended on automation expertise I’d built over years. I’d be more concerned about a junior engineer using this workflow without first building that underlying judgment than about the workflow itself displacing the role — the risk I’d actually watch for is a skills gap forming if newer engineers lean on generation without developing the review instincts to catch what it gets wrong, not headcount reduction from the tool itself.

Can this workflow help with visual regression testing too?

I didn’t specifically test MCP-assisted generation for visual regression scenarios as part of this experiment — that’s a distinct testing discipline I’ve covered in depth in a separate, dedicated guide on this blog. My instinct, based on the browser_take_screenshot tool’s role in the modal animation debugging session above, is that MCP could plausibly assist with setting up visual test scaffolding, but the actual pixel-comparison and baseline-management workflow is a different enough problem that I’d want to run it as its own dedicated experiment before making any claim here.

Common Objections, Answered Honestly

Throughout drafting this post, I kept a running list of pushback I anticipated from experienced QA engineers — the kind of skepticism I’d have had myself before running this experiment. I want to address the strongest versions of these directly, rather than only presenting the parts of this experiment that went well.

“This is just going to produce more test code that nobody understands or maintains.”

This is a legitimate risk, and I don’t think it’s fully solved by anything in this post — it’s solved by discipline, specifically the same PR review discipline I’ve emphasized repeatedly throughout this log. A generated test that nobody on the team understands is a liability regardless of how it was produced; the fix isn’t unique to MCP, it’s the same fix that’s always applied to any code a team doesn’t fully own: don’t merge what you can’t explain. I’d add one MCP-specific note: because generated code can look more polished and confident than a rough first draft a human might write, I think teams need to be slightly more deliberate about resisting the urge to rubber-stamp it, not less.

“You only tested this against a simple demo app — real applications are messier.”

Correct, and I’ve said so explicitly multiple times in this post rather than hide it. I don’t think that invalidates the findings about the mechanism (live reproduction beats guessing, exploration-first beats generation-first, explicit constraints beat vague prompts) — those are structural findings about how the workflow behaves, not claims about a universal time-savings percentage. But I fully agree the specific 24% time-savings figure should be treated as an upper bound from a favorable test case, not a number to plan a budget around, and I’ve flagged legacy-application testing as explicit future work rather than claim I’ve already answered that question.

“AI models change and improve constantly — won’t this whole post be outdated in a few months?”

Likely, at least in terms of specific version numbers and exact tool names — I’ve pinned specific package versions in this post precisely because I expect the Playwright MCP server and various clients to keep evolving. What I’d expect to age better than the specifics are the structural lessons: exploration before generation, explicit constraints over vague prompts, live reproduction over guessed diagnoses for timing issues, real PR review regardless of how code was produced. Those are workflow discipline lessons, not tool-version-specific ones, and I’d bet on them holding up even as the underlying tools change.

“Isn’t this just a more expensive way to do something Playwright’s own codegen already does for free?”

I addressed the codegen comparison specifically in the FAQ above, but to restate the core distinction here: codegen records human action with no reasoning, debugging capability, or ability to be redirected mid-task. It’s free and useful for exactly what it does, but it doesn’t do what the flaky-test debugging case study or the checkout ambiguity example demonstrated. They solve different problems, and I use both, not one instead of the other.

“Twelve years of Selenium experience and you’re excited about a tool that sometimes guesses wrong error messages?”

Fair jab, and I’ll take it directly: yes, the login test’s first-pass wrong error message is a real limitation, and I included it specifically so this post wouldn’t read as uncritical enthusiasm. What earned my cautious endorsement wasn’t that the tool is always right — it clearly isn’t — it’s that when explicitly asked to verify against live state rather than guess, it correctly self-corrected using real observed data. That specific capability, the live verification loop, is what twelve years of watching guessed-and-wrong test code fail in production makes me actually value, not the polish of the first-draft output.

This post builds directly on ground I’ve covered elsewhere on QAtribe, and if any part of this log resonated, these are the natural next stops. My earlier piece on debugging flaky tests with GitHub Copilot and MCP was the more exploratory, less structured precursor to the case study documented in detail above — worth reading if you want the earlier, rougher version of that same debugging journey. My MCP Server Security Checklist goes far deeper into the credential-handling, environment-restriction, and governance concerns I only summarized in the security section of this post, and I’d treat that as required reading before rolling any MCP-connected workflow out beyond a single engineer’s own machine. If you’re earlier in the Playwright transition than the MCP-specific content here assumes, my Playwright Page Object Model with TypeScript guide covers the exact convention-following patterns that every generated test in this post was instructed to respect, and is a better starting point if POM itself is still new to you. And if the visual verification angle from the modal-animation debugging session interested you, my Playwright Visual Regression Testing guide is the deep, dedicated treatment of that adjacent discipline, separate from the functional testing focus of everything in this post.

Full Code: The Search and Add-to-Cart Tests

I summarized the search and add-to-cart tests in the time-comparison table earlier without showing the actual generated code. In the interest of letting you judge quality directly rather than take my summary at face value, here’s the full, lightly-trimmed output for both.

Product Search Test

The exploration phase for this flow surfaced one detail worth noting: search results updated via a debounced, client-side filter rather than a full page reload, which meant the same “wait on the real response, not a guessed duration” principle from the flaky-filter case study applied here too, and the assistant correctly carried that pattern over without me having to re-explain it, since I referenced the earlier fix explicitly in my prompt.

import { type Page, type Locator } from '@playwright/test';

export class SearchPage {
  readonly page: Page;
  readonly searchInput: Locator;
  readonly resultItems: Locator;
  readonly noResultsMessage: Locator;

  constructor(page: Page) {
    this.page = page;
    this.searchInput = page.getByPlaceholder('Search products');
    this.resultItems = page.getByTestId('product-card');
    this.noResultsMessage = page.getByText('No products found');
  }

  async search(term: string) {
    const searchResponse = this.page.waitForResponse(resp =>
      resp.url().includes('/api/products/search') && resp.status() === 200
    );
    await this.searchInput.fill(term);
    await searchResponse;
  }
}

import { test, expect } from '@playwright/test';
import { SearchPage } from '../pages/SearchPage';

test.describe('Product search', () => {
  test('returns matching products for a valid search term', async ({ page }) => {
    const search = new SearchPage(page);
    await page.goto('/products');
    await search.search('wireless headphones');
    await expect(search.resultItems.first()).toBeVisible();
    const count = await search.resultItems.count();
    expect(count).toBeGreaterThan(0);
  });

  test('shows a no-results message for a nonsense search term', async ({ page }) => {
    const search = new SearchPage(page);
    await page.goto('/products');
    await search.search('zzznonexistentproductzzz');
    await expect(search.noResultsMessage).toBeVisible();
  });
});

Add-to-Cart Test

This flow was the most straightforward of the five, and the generated version needed essentially no correction — the only manual change I made was renaming one variable for consistency with a naming convention used elsewhere in the project, which the assistant had no way of knowing about since it wasn’t represented anywhere in the existing codebase it explored.

import { type Page, type Locator } from '@playwright/test';

export class ProductPage {
  readonly page: Page;
  readonly addToCartButton: Locator;
  readonly cartCountBadge: Locator;

  constructor(page: Page) {
    this.page = page;
    this.addToCartButton = page.getByRole('button', { name: 'Add to cart' });
    this.cartCountBadge = page.getByTestId('cart-count');
  }

  async goto(productSlug: string) {
    await this.page.goto(`/products/${productSlug}`);
  }
}

import { test, expect } from '@playwright/test';
import { ProductPage } from '../pages/ProductPage';

test('adds a product to the cart and updates the cart count', async ({ page }) => {
  const product = new ProductPage(page);
  await product.goto('wireless-headphones-v2');
  await expect(product.cartCountBadge).toHaveText('0');
  await product.addToCartButton.click();
  await expect(product.cartCountBadge).toHaveText('1');
});

I’m including both of these in full specifically because I don’t want this post’s overall positive assessment to feel abstract. These are real, unedited-except-for-one-variable-name outputs, and I’d merge either of them into a real project as-is.

Bonus Experiment: Testing a BFSI-Style Form Flow

Given my own background spans banking, wealth management, healthcare, and payments projects, I wanted to run one additional, smaller experiment specifically against a form pattern common in that domain: a multi-step KYC-style onboarding form with conditional fields, since that’s a meaningfully different shape of complexity than the e-commerce flows above — more branching logic, more regulatory-style validation, and fields that appear or disappear based on earlier answers.

I used a public demo financial-onboarding form for this, again avoiding any real client project. The interesting part wasn’t the happy path — that generated cleanly, similar to the tests above — it was how the assistant handled the conditional field logic during exploration.

The Conditional Field Challenge

The form had a “Are you a politically exposed person?” toggle that, when set to “Yes,” revealed two additional required fields (position held, and jurisdiction) that were otherwise hidden and not required. This is exactly the kind of branching complexity that’s easy to under-specify in a quick manual test and easy for a generated test to miss entirely if the exploration phase doesn’t specifically probe both branches.

My exploration prompt was deliberately more directive than my earlier ones, because I’d learned by this point in the experiment not to assume broad exploration would automatically find conditional branches on its own:

Prompt: "Using the Playwright MCP tools, navigate to the onboarding form.
Explore the 'politically exposed person' toggle specifically — check the
form state with it set to 'No' AND separately with it set to 'Yes'. Report
exactly which fields appear, disappear, or change required status between
the two states."

This produced an accurate report of both states, correctly noting the two additional required fields and their exact labels. Generation, based on that report, produced two separate test cases — one for each toggle state — rather than a single test that might have missed the conditional branch entirely if I’d only asked for “a test for the onboarding form” without directing attention to the toggle specifically.

The lesson I took from this bonus experiment, specific to BFSI-style domains: the exploration-first pattern from earlier in this post isn’t sufficient on its own for complex conditional or regulatory-style forms — it needs to be paired with domain awareness about where the branching logic actually lives, which the assistant has no way of knowing without being told. This is, I think, exactly the kind of domain expertise a QA engineer with real BFSI experience brings to this workflow that a generic prompt wouldn’t surface on its own, and it reinforced my earlier point about deep domain and automation knowledge becoming more valuable in this workflow, not less.

Answering My Three Original Questions, Directly

Back at the start of this post, I set out three specific questions before touching any config file, precisely so I couldn’t quietly move the goalposts once I had results. Here are direct, unhedged answers to each, now that the experiment is done.

Question 1: Can an MCP-connected AI assistant generate a genuinely usable Playwright test from a plain-English description?

Yes, with a specific and important caveat. Every test shown in full in this post — login, search, add-to-cart, checkout validation — reached a state I’d merge into a real project, using locators and structure I’d have chosen myself. The caveat: “genuinely usable” required the exploration-first workflow and explicit constraint-setting documented throughout this post. A vague, single-shot prompt (“write a test for X”) reliably produced weaker, more brittle output across every scenario I tried it on. So the honest answer isn’t simply “yes” — it’s “yes, if you prompt it the way this post describes, and meaningfully weaker if you don’t.”

Question 2: Does MCP meaningfully help with debugging flaky or failing tests, compared to manual trace analysis?

Yes, and this was the strongest, most defensible finding in the entire experiment. The filter-timing flaky test case study produced a correct root-cause diagnosis through live reproduction, a fix that eliminated the flake across fifty verification runs, and a diagnosis that matched what I independently confirmed through manual trace viewer analysis. That’s about as close to an unambiguous positive result as this kind of experiment produces. The important counterbalance: the modal animation debugging session was a genuinely mediocre result, so this “yes” comes with a real asterisk — it helped decisively on a network-timing issue and only partially on a pure CSS-animation issue, and I don’t yet know which pattern generalizes further without more testing.

Question 3: What’s the actual time and cost trade-off, once setup and prompt iteration are included?

Modestly positive, on the specific application I tested, for someone already fluent in Playwright. Roughly 24% time savings on my five-test suite, alongside a real but, in my case, manageable AI usage cost that I’ve been transparent about not having precisely quantified in absolute currency terms in this post, deliberately, since exact pricing varies by plan and changes over time — I’d point you to your own AI assistant provider’s current pricing rather than a number I write here that might be stale by the time you read this. The trade-off is meaningfully less favorable, based on my own reasoning even though I haven’t tested it directly, for legacy applications, for engineers still learning Playwright fundamentals, and for teams that skip the prompting discipline this post describes in detail.

What a Skeptical Teammate Actually Said When I Showed Them This

Before publishing this post, I walked a colleague — someone with a similarly long automation background, who’d been openly skeptical of AI-assisted testing hype in general — through the filter-timing flaky test case study specifically, since I expected it to be the most persuasive piece of evidence in this whole log. Their first reaction wasn’t excitement, it was a very reasonable question: “how do you know it actually reproduced the issue and didn’t just describe a plausible-sounding race condition that happens to match what a race condition usually looks like?”

That’s a fair challenge, and it’s worth addressing directly rather than glossing over. My answer, at the time and now: I didn’t take the diagnosis on faith. I independently verified it myself afterward using the Playwright trace viewer, the same tool I’ve used for manual debugging for years, specifically to check whether the AI-reported root cause matched what I could confirm through my own independent process. It did. I think that verification step — not trusting the diagnosis just because it sounded confident and technically plausible — is exactly the kind of skepticism this workflow needs from anyone using it, and I’d encourage the same skepticism from you rather than taking my own results on faith either. If you run a similar experiment, verify the AI’s diagnosis independently at least once before trusting the pattern going forward, the same way I did.

My colleague’s second reaction, after seeing the fifty-run verification numbers, was more measured acceptance than enthusiasm: “okay, that’s a real result, not a demo trick — but I still wouldn’t trust it without the fifty-run check, and I wouldn’t trust a junior to know to run that check.” I think that’s about the right level of calibrated trust to land on, and it’s close to where I ended up myself by the end of this experiment: genuinely useful, verifiable, and not something to hand to someone without the experience to independently confirm it.

Key Takeaways: A Condensed Checklist

For anyone who skimmed straight to the end, or who wants a condensed version to share with a teammate, here’s every major finding from this post distilled into a single checklist.

MCP lets an AI coding assistant actually observe live application state through a real browser, instead of only generating code from training-data patterns — that grounding, not the code generation itself, is the core value.
Separate exploration from generation into two distinct prompts; combining them produces more assumption-driven, less accurate results.
Always specify locator strategy, existing conventions, and exact expected assertions explicitly — the assistant will not reliably infer them on its own.
For debugging, explicitly instruct live reproduction through the MCP tools rather than reasoning from code alone, especially for timing and flakiness issues.
Paste real, raw error output into debugging prompts — never paraphrase a failure.
Double-check which MCP client’s documentation and config schema you’re actually following; servers vs mcpServers is a real, silent-failure-causing gotcha across different clients.
Pin exact MCP server package versions rather than always fetching @latest, for reproducibility across sessions.
Generated tests still run through the exact same CI pipeline, and require the exact same PR review discipline, as hand-written tests — MCP is a development-time authoring aid, not a CI-time capability.
Track your own real time and usage-cost numbers before proposing team-wide adoption; published percentages, including the ones in this post, should be treated as directional, not planning-grade figures for your specific context.
This workflow amplifies existing automation and domain judgment — it does not substitute for it, and I would not recommend it as a first introduction to test automation for someone without prior Playwright fundamentals.

A More Realistic mcp.json: Running Multiple MCP Servers Together

Every config example earlier in this post showed a single Playwright MCP server in isolation, which is the right way to learn the basics but not how a real, ongoing project setup typically looks. By the end of this experiment, my actual workspace config had grown to include a second server — a filesystem MCP server — running alongside the Playwright one, because several of my generation prompts needed the assistant to read my existing page object files directly rather than relying purely on whatever context VS Code’s own indexing surfaced automatically. Here’s the fuller, more realistic config:

{
  "servers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@0.0.29"]
    },
    "filesystem": {
      "command": "npx",
      "args": [
        "@modelcontextprotocol/server-filesystem",
        "/absolute/path/to/your/project/src"
      ]
    }
  }
}

With both servers running, a single prompt could reference both capabilities together — for example, “read the existing LoginPage.ts file, then explore the live registration page and generate a RegistrationPage.ts following the exact same structural conventions” — without me having to manually paste the existing file’s contents into the chat each time. This turned out to be a meaningful quality-of-life improvement for the convention-following behavior I’d been relying on throughout this post, since it meant the assistant was reading my actual current file rather than working from whatever it had inferred about my conventions from earlier context in the conversation, which could occasionally drift during a long session.

One practical note worth flagging if you go this route yourself: scope the filesystem server’s root path deliberately, the same way you’d scope any tool’s permissions. I pointed mine specifically at my src directory rather than my whole project root, partly out of the same security-mindedness covered earlier in this post, and partly because it kept exploration faster by not surfacing irrelevant files like build artifacts or node_modules content.

How This Fits Into My Own AI/GenAI Positioning as a QA Professional

I want to close with something a bit more personal, because I think it’s relevant to why I ran this experiment as rigorously as I did, rather than just skimming the surface for a quick blog post. Part of what’s driven my own learning direction this year — moving deliberately from Java/Selenium into TypeScript/Playwright, and now into structured, hands-on MCP experimentation like this — has been a conviction that genuine, hands-on AI/GenAI fluency is becoming a real differentiator for QA and SDET roles, not just a buzzword to sprinkle into a resume.

The distinction I care about, and the reason this post is written the way it is, is between claiming AI/GenAI experience and actually being able to speak concretely about what worked, what didn’t, and why, down to specific config mistakes and specific prompt phrasing. Anyone can write “experienced with AI-assisted testing tools” on a resume. Far fewer people can walk an interviewer through the exact reason a first-pass debugging prompt produced a lazy “add a wait” diagnosis, and the exact instruction that changed that behavior into a genuine root-cause investigation. That kind of specific, lived detail is, in my experience, what actually distinguishes a claimed skill from a real one in a technical interview, and it’s the entire reason I structured this post as a warts-and-included log rather than a polished highlight reel.

If you’re a QA engineer in a similar position — building genuine AI-assisted testing experience specifically to strengthen your own positioning, not just to chase a trend — my honest advice, based on this experiment, is to run your own version of exactly this kind of structured trial: pick a real question you actually want answered, track real numbers, document real mistakes, and be willing to publish or present the unflattering parts alongside the wins. That discipline is, itself, a demonstrable skill, and it’s a more durable one than familiarity with any single tool’s current version number.

Resources and Further Reading

A short, curated list of where I’d point you next if you want to go deeper on specific pieces of this post, beyond my own related posts already linked above:

The official Model Context Protocol specification and documentation, for anyone who wants the precise technical definition behind the plain-language explanation I gave earlier in this post.
The official Playwright MCP server repository on GitHub, for the authoritative, up-to-date list of exposed tools — this evolves over time, and I’d trust that source over any snapshot in this post for the current tool list.
Your specific AI coding assistant’s own MCP configuration documentation — VS Code’s MCP servers guide, Claude’s MCP documentation, or Cursor’s MCP docs — always double-check the exact config schema against current official docs rather than a third-party guide, for the exact reason detailed in my Error 1 troubleshooting entry above.
Your team’s own existing test automation style guide or conventions document, if one exists — the single highest-leverage input into generation quality throughout this entire experiment was how precisely I could specify existing conventions, and that starts with those conventions actually being written down somewhere in the first place.

Conclusion: A Working Setup, Real Mistakes, and an Honest Verdict

I started this post admitting I couldn’t have explained MCP in one sentence three weeks ago. I can now: it’s a standard that lets an AI coding assistant actually observe and act on a real, live application through a defined set of tools, instead of only generating plausible-looking code from training data — and for test automation specifically, that grounding in live state turned out to be the entire point, demonstrated most clearly in a flaky test I genuinely fixed and verified across fifty clean runs, not just talked about fixing.

I also started this post promising it wouldn’t be a sanitized success story, and I’ve tried to hold to that the whole way through: the silent config failure that cost twenty minutes, the fragile CSS selectors my first vague prompt produced, the hallucinated error message text, the mediocre modal-animation debugging session, the checkout ambiguity that needed an explicit “stop and ask” instruction to surface at all. Those aren’t footnotes to a success story — they’re the actual texture of what building a first MCP-powered test automation workflow looks like when you’re honest about it, and I think that texture is more useful to you than a clean highlight reel would have been.

My honest, final verdict: this is a real, measurable improvement to test authoring and debugging for engineers who already know what good Playwright code looks like, built on a genuinely different capability than plain AI code completion, worth the setup friction, and not yet something I’d hand to someone without that underlying judgment already in place. If that description matches where you are, I’d encourage you to run your own version of this experiment rather than take my numbers as gospel — track your own time, your own costs, and your own mistakes, the same way I’ve tried to do transparently throughout this whole log. And if you do, I’d genuinely like to hear what broke for you first.

Explaining This to Non-Technical Stakeholders

Partway through this experiment, a project manager I work with asked me, in a hallway-conversation kind of way, what I’d been spending so much time on. I gave a version of this explanation on the spot, and I’ve refined it since, because I think the framing matters for anyone who’ll need to justify this workflow to someone who doesn’t know what a locator or a DOM snapshot is.

The version I’ve settled on: “I’m testing whether an AI assistant can actually look at our real application while it helps write tests, instead of just guessing based on similar apps it’s seen before. The difference matters because guessed test code looks fine until it hits our actual site and breaks in ways that don’t make sense. So far, it’s caught and fixed one real intermittent bug in our test suite that used to fail about one time in five, and it wrote a small login and checkout test suite about a quarter faster than I’d normally take by hand — with me reviewing every line the same way I always do.”

That framing does a few deliberate things: it avoids protocol jargon entirely, it’s anchored to a concrete, verifiable result (the fifty-run flaky test fix) rather than a vague productivity claim, and it explicitly preserves the fact that human review hasn’t gone anywhere. I’ve found that last point matters more to non-technical stakeholders than any other detail — the question I get most often isn’t “how does it work,” it’s some version of “so is a human still checking this,” and being able to answer that clearly and honestly, the same way I’ve tried to throughout this whole post, tends to land better than any technical explanation of the protocol itself.

Metrics I’d Track Going Forward, If I Kept This Running

If this became a permanent part of my workflow rather than a three-week experiment, here’s the small set of metrics I’d want to keep tracking, based on what actually turned out to matter across this post rather than metrics that sounded good in theory but didn’t end up informing any real decision.

Time per test, MCP-assisted vs. my own rolling manual-time estimate, tracked per test rather than only as a suite average, since the checkout test’s 27 minutes versus the add-to-cart test’s 9 minutes in my own data shows real variance worth understanding by flow complexity, not flattening into a single number.
Correction rate — how many generated tests passed on the first run versus needed at least one round of correction, broken down by whether the exploration-first pattern was used. My own small sample (the checkout test passing first-try after explicit exploration, versus others needing at least one fix) is too small to generalize from confidently, but it’s exactly the kind of signal I’d want a larger, ongoing dataset to confirm or contradict.
Debugging session outcome — successful root-cause diagnosis with a verified fix (like the filter-timing case), versus partial help requiring significant manual completion (like the modal-animation case), versus no meaningful help at all, which I didn’t encounter in this experiment but would still want to track as a category.
AI assistant usage per session type, split between tightly-scoped generation prompts and open-ended exploratory debugging sessions, since I already observed a real difference in usage between these two styles and would want ongoing data to confirm the pattern holds at scale.
PR review findings specific to generated code — logging every case where review catches something in AI-assisted code that wouldn’t have made it into hand-written code, and separately, every case where review would have caught the same class of issue regardless of how the code was produced. This distinction matters for honestly assessing whether MCP-assisted code carries genuinely different risk, or just superficially different risk, compared to hand-written code.

None of these are exotic metrics — they’re the same kind of disciplined tracking any QA process improvement deserves, and I’d apply the same rigor here that I’d apply to evaluating any other new tool or practice change, rather than let this one get a pass on measurement just because it’s novel and interesting.

A Final Honest Note on Timing and Novelty

One last piece of honesty before I close this out: I’m aware this post is being published at a moment when MCP is genuinely new and evolving quickly, and some of the specific details here — exact tool names, exact config schemas, exact package versions — may shift by the time you’re reading this. I’ve tried to date-stamp my specifics clearly (exact version pins, exact client behaviors at the time of testing) precisely so a future reader, including a future version of me revisiting this post, can tell which parts are timeless workflow discipline and which parts were true of the tooling landscape at one particular moment. If you’re reading this significantly after publication and something described here has changed, I’d treat the underlying lessons — exploration before generation, explicit constraints, live reproduction for debugging, real review discipline regardless of how code was produced — as the durable takeaway, and the specific commands and config snippets as a snapshot of one beginner’s honest starting point, not a permanent reference.

A Third Debugging Case: The One That Took Longest to Untangle

I’ve shown a strong debugging result (the filter-timing race condition) and a mediocre one (the modal animation). In the interest of completeness, I want to walk through a third case that fell somewhere in between, because I think showing the full spread — strong, weak, and middling — gives a more honest picture than only showing the two extremes.

This test covered a product image gallery with a thumbnail-click-to-swap-main-image interaction, embedded inside an iframe for a third-party product configurator widget. The test failed intermittently with a “locator not found” error, and initial code review hadn’t revealed anything obviously wrong with the selector itself.

The Investigation

I used the same “actually reproduce it, don’t guess from the code” prompt pattern established in the earlier case study:

Prompt: "This test fails intermittently with 'locator not found' when clicking
a thumbnail inside the product configurator iframe. Use the Playwright MCP
tools to actually navigate to this page and interact with the thumbnails
several times in a row. Report the exact timing and structure you observe,
especially anything related to the iframe itself loading or reloading."

The investigation took noticeably longer than the filter-timing case — several rounds of snapshot-and-observe cycles — before it surfaced the actual cause: the third-party iframe widget occasionally reloaded itself entirely (a behavior of the third-party script, unrelated to anything in our own application code) in response to a viewport resize event that our test setup was triggering incidentally as part of an earlier step in the same test file. When that reload happened, the original locator reference became stale mid-test, producing the intermittent “not found” error.

Why This One Was Harder

Unlike the filter-timing case, where the root cause lived entirely within our own application’s timing behavior, this one involved a third-party component’s undocumented reload behavior — something no amount of exploring our own code could have revealed, and something the assistant could only surface by directly observing the iframe’s behavior repeatedly across multiple live attempts. It took roughly three times as many tool-call rounds as the filter-timing investigation before a consistent pattern emerged, and I want to flag that time cost honestly: this was not a fast debugging session, even though it ultimately succeeded.

The eventual fix — re-querying the thumbnail locator fresh after confirming the iframe had finished any reload, rather than holding a reference across the whole test — worked, and I verified it the same way I verified the filter-timing fix: twenty-five repeated runs with zero failures, versus roughly one in six failures beforehand. But I want this case study to sit alongside the other two as an honest data point: some debugging problems, especially ones rooted in third-party or undocumented behavior, take real, extended investigation even with live MCP tool access, and I don’t want the strong filter-timing result earlier in this post to set an expectation that every flaky test resolves in a single quick exchange. This one took the better part of an afternoon, not twenty minutes.

Prompt Evolution: Week 1 Me vs. Week 3 Me, Side by Side

To make the prompting-skill improvement I’ve referenced throughout this post concrete rather than just asserted, here’s a direct side-by-side of an early-experiment prompt and a late-experiment prompt for a structurally similar task, so you can see the actual gap in specificity that developed over three weeks.

Week 1: Requesting a Test for a Form

"Write a Playwright test for the login form."

Result: technically functional, CSS-selector-based, no negative case, no assertion on exact error text (used a loose .toBeVisible() check instead of verifying content), took about six minutes to review and rewrite significantly.

Week 3: Requesting a Test for a Structurally Similar Form

"Using the Playwright MCP tools, navigate to the account-settings password
change form. Explore it first: field labels, validation rules (especially
anything checked on blur vs. on submit), and exact error text for each rule.
Report back before generating anything.

Once confirmed, generate a Playwright TypeScript test using only
getByRole/getByLabel locators, following the existing SettingsPage pattern
in /pages. Cover: empty submission, password-confirmation mismatch, and a
successful change showing the exact confirmation toast text. Use
expect.poll or a real network-response wait for anything async — no fixed
waitForTimeout calls. Flag anything ambiguous instead of guessing."

Result: passed on first run, correctly used a network-response wait for the async confirmation toast without me having to ask for a correction, correctly separated blur-triggered validation from submit-triggered validation without me having to catch it after the fact the way I did with the checkout test earlier in this post. Review took under two minutes, mostly confirming rather than correcting.

The gap between these two prompts is the entire argument for the prompt engineering section earlier in this post, made concrete rather than abstract. Nothing about the underlying tool changed between week 1 and week 3 — what changed was my own specificity, built directly from the mistakes documented throughout this log. If you take away one practical artifact from this entire post to paste into your own first session, I’d make it the week 3 template above, adapted to your own flow, rather than anything closer to my week 1 attempt.

Who Should Own MCP Configuration on a QA Team?

A practical, slightly unglamorous question I hadn’t thought about until midway through this experiment: if this workflow does get adopted beyond a single curious engineer, who actually owns the shared config files, the version pins, and the prompt-convention playbook? Based on the friction I hit myself — the schema mismatches, the @latest version drift, the need for a consistent locator-strategy standard across generated tests — I don’t think this should be left to each engineer configuring their own setup independently, the way individual IDE preferences often are.

My recommendation, based directly on where I personally lost time in this experiment: treat the mcp.json (or equivalent per-client config), the pinned server versions, and the prompt template library as checked-in, version-controlled team assets, owned the same way a shared ESLint config or a shared Page Object base class would be owned — by whoever already owns test automation architecture decisions on the team. This isn’t a new organizational role; it’s an extension of an existing one, and treating it that way avoids the scenario where five engineers each discover the servers vs mcpServers gotcha independently, on their own time, the way I had to.

Pitfalls Specific to Team-Wide Adoption (Beyond What One Engineer Hits Solo)

Everything in the mistakes section earlier in this post reflects friction I hit working alone. A few additional pitfalls only become visible once more than one person is involved, based on reasoning through the rollout plan above rather than something I’ve directly observed at team scale myself, since this experiment was solo — I want to flag that distinction clearly rather than overstate my own evidence here.

Inconsistent prompt conventions producing inconsistent test quality across the team, if the prompt template library isn’t shared and enforced the way a linting config would be. Two engineers independently reinventing their own version of the exploration-first pattern is better than neither doing it, but worse than a shared, documented standard.
Divergent AI assistant client choices creating uneven config maintenance burden — if half the team uses VS Code/Copilot and half uses Claude Code or Cursor, someone needs to maintain and test config compatibility across all of them, given the real schema differences documented in the cross-client section above, rather than assuming one config file serves everyone.
Usage cost visibility gaps — an individual engineer experimenting rarely has full visibility into aggregate team-wide AI assistant usage cost, which is exactly why the rollout plan above calls for tracking this explicitly at each phase rather than discovering it as a surprise line item after a full team has already adopted the workflow.
Review fatigue if generated code volume scales faster than review capacity — the PR review discipline emphasized throughout this post only holds as team practice if review capacity scales alongside authoring speed. A workflow that helps a team generate tests 24% faster, per my own measured figure, but doesn’t correspondingly adjust review bandwidth, risks quietly eroding the exact review discipline this whole post argues is non-negotiable.

I’m flagging all four of these as reasoned extrapolation from my solo experiment rather than directly observed team-scale findings, in keeping with the same honesty standard I’ve tried to hold throughout this entire post. If I run a team-scale follow-up to this experiment, I’d expect this section to be the one most likely to get revised based on what actually happens rather than what I currently predict will happen.

My Updated Code Review Checklist, Post-Experiment

One concrete, reusable artifact I want to leave you with: after finishing this MCP test automation experiment, I actually rewrote my personal PR review checklist to account for what I’d learned, rather than reviewing MCP-assisted tests against the exact same mental checklist I used for hand-written code. Here’s the updated version, with the new, MCP-specific items marked.

Does every locator use role/label/text-based strategies rather than CSS classes or XPath, unless there’s a documented reason otherwise?
Does every assertion check specific, meaningful content rather than a loose toBeVisible() when exact text matters?
Are all waits tied to real conditions (network responses, specific state changes) rather than fixed-duration timeouts?
[New] If this test was MCP-generated, was it generated using the exploration-first pattern, or a single-shot prompt? Single-shot-generated tests get an extra pass of scrutiny, based on the measurable quality gap documented in this post.
[New] Does any assertion text look suspiciously generic or “plausible” rather than tied to something the author (human or AI) actually observed on the live application? This is specifically the hallucinated-error-message failure mode from the login test case study, and it’s now an explicit checklist item rather than something I only caught by luck.
[New] For any test addressing a previously flaky scenario, is there evidence of repeated-run verification (a documented run count and pass rate), not just a single passing run, before considering the fix confirmed?
Does the test follow existing Page Object Model and naming conventions used elsewhere in the project?
Would a new team member reading this test understand what behavior it’s actually verifying, without needing to ask the original author?

The three new items didn’t replace anything on my original checklist — they’re additions, specifically closing the exact gaps this experiment surfaced. I’d encourage anyone adopting MCP test automation on their own team to do the same exercise: don’t assume your existing review checklist automatically covers the new failure modes this workflow introduces, because in my own experience, it didn’t, until I deliberately updated it based on what actually went wrong across this log.

That checklist update is, in a way, the most concrete proof I have that this was a genuine learning experience rather than just an interesting three weeks. My review process is measurably different now than it was before I started, built directly from real, logged mistakes rather than a generic “AI code needs review” instinct. That’s the standard I’d hold any team to before calling an MCP test automation pilot complete: not just “did we generate some tests,” but “did our own process get better because of what we learned.”

Quick-Start Checklist: Building Your Own MCP Test Automation Workflow This Weekend

If everything above has convinced you to try this yourself, here’s the condensed, actionable version of this entire post, stripped down to a sequence you could realistically work through in a focused weekend, assuming you already have Playwright and TypeScript fundamentals in place.

Confirm prerequisites. Node 18+, an existing Playwright TypeScript project with at least a basic Page Object Model in place, and an MCP-compatible AI coding assistant already installed and working for normal code completion.
Install the Playwright MCP server as a dev dependency, pinned to a specific version rather than @latest: npm install -D @playwright/mcp.
Create the correct config file for your specific client, double-checking the top-level key against current official documentation for that exact client — don’t copy a snippet from a guide written for a different tool.
Run a low-stakes sanity check against a stable public site before touching your real application, to confirm tool-calling is actually working and grounded in live state, not guessed.
Pick one real, moderately simple flow in your own application — a login form is a good first choice, the same one I used — and run an exploration-only prompt first, reviewing what comes back before requesting any code.
Generate your first test with an explicit, constraint-heavy prompt: locator strategy, existing conventions, exact assertions. Run it immediately rather than reading it and assuming it’s correct.
Deliberately pick one known flaky or intermittent test from your existing suite and run it through the live-reproduction debugging pattern, verifying any proposed fix with at least twenty-five to fifty repeated runs before trusting it.
Track your own real time numbers throughout, the same way I did with a plain stopwatch, so you have honest data rather than an impression by the end of the weekend.
Run every generated test through your normal PR review process, ideally against an updated checklist that accounts for the specific new failure modes covered in this post — hallucinated assertion text, single-shot-generated brittleness, unverified flaky-fix claims.
Write down what broke, in as much specific detail as you can, the same way I’ve tried to throughout this entire post — not because you need to publish it, but because the specific, logged mistakes are what actually build the judgment this workflow depends on, far more than the successful runs do.

That’s the whole workflow, compressed. Everything else in this 25,000-plus word log is the detailed reasoning, the actual code, and the honest failures behind each of those ten steps — useful if you want the full context, but if you only take one section from this entire post into your own hands-on weekend, make it this checklist.

Thanks for reading this far into what turned out to be a much longer log than I originally planned to write. If you build your own version of this and something breaks in a way that isn’t covered above, I’d genuinely like to know — drop it in the comments, or reach out directly. My own next experiment, as flagged earlier, is testing this exact workflow against a messier, more legacy-style application, and I’ll be back with the same kind of honest, numbers-first log once that’s done.

What This Means for the Broader “Agentic QA” Conversation

I want to close with a slightly wider-lens reflection, because I think this specific experiment sits inside a bigger conversation happening across the QA and SDET community right now about “agentic” testing tools more broadly — AI systems that don’t just suggest code but take actions, observe results, and iterate. MCP test automation, as I’ve documented it in this post, is a fairly modest, human-supervised version of that idea: every action I described here happened with me reading the prompt, reviewing the tool-call log, and approving or correcting the output at each step. Nothing in this experiment ran unsupervised, and nothing merged without going through my own review.

I think that’s an important distinction to hold onto as more ambitious, more autonomous versions of this idea inevitably get marketed over the coming months — systems that claim to generate, run, and merge entire test suites with minimal human oversight. Based on everything I found in this specific, hands-on experiment, I’d be genuinely cautious about that more autonomous framing, for the same reasons documented throughout this post: even with a human reviewing every step, I still caught a hallucinated error message, a loose assertion that technically passed while testing less than intended, and at least one case (the modal animation debugging session) where the proposed fix was mediocre enough that I rewrote it myself. Remove the human review step from that equation, and I don’t have confidence, based on what I actually observed, that those same issues would get caught before reaching production test suites.

My honest position, informed directly by this three-week experiment rather than by general AI industry conversation: MCP-powered test automation, used the way I’ve documented in this post — human-directed, human-reviewed, with explicit constraint-setting and live verification — is a genuine, measurable improvement to a skilled automation engineer’s workflow today. A more autonomous version of the same idea might be a genuine improvement in the future too, but I’d want to see the same kind of rigorous, honestly-reported experiment — real numbers, real mistakes included, not just a demo — before extending my own confidence that far. Until I’ve run and logged that experiment myself, I’d encourage the same skepticism from you that I’ve tried to model throughout this entire post: trust specific, verified results over general claims, whether they’re mine or anyone else’s.

That’s really the throughline of this whole three-week log, if I had to compress it into one sentence: MCP test automation earned my trust incrementally, one verified result at a time — a correctly self-corrected error message here, a fifty-run-verified flaky fix there — never all at once, and never without me checking. I’d suggest building your own trust in this workflow the exact same way, rather than taking either the hype or my own enthusiasm in this post as a substitute for your own hands-on, honestly-logged experiment.

One Last Honest Number: My Confidence Rating

If you’ve read this whole log, you’ve earned a straight number instead of another paragraph of nuance. On a simple scale of “wouldn’t recommend,” “cautiously worth trying,” or “confidently recommend,” I’d land this MCP test automation experiment at a genuine “cautiously worth trying” for any engineer who already meets the prerequisites laid out early in this post, with the single loudest caveat being: run your own pilot, track your own numbers, and don’t skip the review discipline this entire post has argued for, section after section. That’s not a hedge — it’s the actual, considered conclusion of three weeks spent building, breaking, fixing, and honestly logging my first MCP-powered test automation workflow, and I’d rather leave you with that precise, calibrated confidence than an inflated one.

About This Log

I write QAtribe as a working QA and test automation professional documenting an active, ongoing learning journey — not as someone presenting finished expertise from a distance. This post continues that same approach applied specifically to MCP test automation: a real, hands-on, three-week experiment, with the actual mistakes, actual timings, and actual code left in rather than smoothed over. If you found the honest-log format useful here, it’s the same format behind my other posts on this blog — the Playwright migration journey, the visual regression testing guide, and the earlier, more exploratory look at debugging flaky tests with GitHub Copilot and MCP that this post builds directly on top of.

If you’re working through a similar transition yourself — Selenium to Playwright, manual to automated, or traditional automation to AI-assisted, MCP-powered workflows — I’d genuinely rather hear about what specifically broke for you than get a comment saying this was helpful in general. The specific breakage is where the real, transferable learning lives, and it’s exactly what made this particular 25,000-plus word log worth writing in the first place.

Sample CI Workflow: Where These Tests Actually Run

To reinforce the earlier point about MCP living entirely in the authoring phase, here’s a trimmed, realistic GitHub Actions workflow snippet showing exactly how the tests generated throughout this post run in CI — no MCP server, no AI assistant, nothing beyond ordinary Playwright, exactly as I promised earlier in this post.

name: Playwright Tests

on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Install dependencies
        run: npm ci
      - name: Install Playwright browsers
        run: npx playwright install --with-deps
      - name: Run Playwright tests
        run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

Notice what’s absent from this workflow: there’s no @playwright/mcp dependency installed, no MCP server started, and no AI assistant API key configured anywhere in this pipeline. That’s deliberate and correct — this CI job runs the exact same generated test files described throughout this post, using nothing but the standard Playwright test runner. The five tests from my suite, the login flow, the checkout validation, the search and add-to-cart tests — every one of them plugs into a pipeline that looks identical to any Playwright CI setup you’d have built without ever touching MCP. That consistency is, honestly, one of the most reassuring findings from this whole experiment for anyone worried that adopting an MCP test automation workflow means re-architecting their existing CI infrastructure. It doesn’t. The AI-assisted part of the story ends the moment a test file is committed, and everything downstream of that commit is exactly the CI/CD setup you already have.

That’s a fitting place to actually end this log. Three weeks ago I had a half-installed npm package and a config file with the wrong key in it. Today I have a small, working, reviewed test suite running cleanly in an unchanged CI pipeline, one genuinely fixed flaky test with fifty runs of proof behind it, an updated personal review checklist, and — more than any of that — a much more precise, much less hand-wavy understanding of what MCP test automation actually is, where it genuinely helps, and where it still needs a skeptical, experienced human in the loop. If this log saves you even one of the mistakes documented above, it’s done its job.

🔥 Continue Your Learning Journey

Want to go beyond Playwright with Typescript setup and crack interviews faster? Check these hand-picked guides:

👉 🚀 Master TestNG Framework (Enterprise Level)
Build scalable automation frameworks with CI/CD, parallel execution, and real-world architecture
➡️ Read: TestNG Automation Framework – Complete Architect Guide

👉 🧠 Learn Cucumber (BDD from Scratch to Advanced)
Understand Gherkin, step definitions, and real-world BDD framework design
➡️ Read: Cucumber Automation Framework – Beginner to Advanced Guide

👉 🔐 API Authentication Made Simple
Master JWT, OAuth, Bearer Tokens with real API testing examples
➡️ Read: Ultimate API Authentication Guide

👉 ⚡ Crack Playwright Interviews (2026 Ready)
Top real interview questions with answers and scenarios
➡️ Read: Playwright Interview Questions Guide

Tags: