# Midscene.js - AI UI Automation and E2E Testing

> Bring AI vision to E2E testing with a newly designed test framework for web, mobile, and desktop apps.

## Other

- [BDD-style scripts with Gherkin](/advanced/bdd-style-scripts-with-gherkin.md): This feature is available starting in Midscene 1.10.BDD-related capabilities are still in Beta. The API may change in future releases. Gherkin is a plain-text syntax for describing behavior examples. It is commonly used in BDD (Behavior-Driven Development) workflows. A BDD scenario usually contains three kinds of steps: Given describes the precondition, When describes the user action, and Then describes the expected result. For example, when you need to test adding a todo item, you can write the case as the following Gherkin script. Longer GUI cases can also be written in one Scenario. You can organize the steps by business phase, and each phase can use a group of Given, When, and Then. This style is useful when one business scenario naturally contains several UI phases, such as cart review, shipping selection, payment, and final confirmation. In AI-driven GUI automation, Gherkin is a natural fit for describing operation cases. It keeps the readability of natural language while providing a stable step structure. This structure also helps models generate more consistent cases. See the official Cucumber Gherkin reference for the full Gherkin syntax.
- [Midscene AndroidWorld Benchmark Report](/android-world-benchmark-report.md): This is Midscene's test report for the AndroidWorld benchmark. In this run, Midscene achieved Pass@1 93.10%, Pass@2 95.69%, and Pass@3 97.41%.
- [Midscene AppControlBench Benchmark Report](/app-control-bench-report.md): This is Midscene's benchmark report for AppControlBench. In this evaluation, each model ran the same set of 60 tasks, with the following results: Note: Doubao Seed 2.1 Turbo costs use Volcengine's official pricing: CNY ¥3 / M input tokens, ¥0.6 / M cached input tokens, and ¥15 / M output tokens. Other model costs use OpenRouter pricing. USD values use ¥6.8 per $1. The results show that the Midscene + GUI vision approach is highly competitive in both cost and pass rate.
- [Automate with scripts in YAML](/automate-with-scripts-in-yaml.md): In most cases, developers write automation scripts just to perform some smoke tests, like checking for the appearance of certain content or verifying that a key user path is accessible. In such situations, maintaining a large test project is unnecessary. Midscene offers a way to perform automation using .yaml files, which helps you focus on the script itself rather than the testing framework. This allows any team member to write automation scripts without needing to learn any API. Here is an example. By reading its content, you should be able to understand how it works.
- [Awesome Midscene](/awesome-midscene.md): A curated list of community projects that extend Midscene.js capabilities across different platforms and programming languages.
- [The Basics](/basics.md): This page explains Midscene's vision-based approach, usage options, and key APIs to help you understand and use the GUI Agent. To try it first, go to Try the GUI Agent.
- [Introducing Instant Actions and Deep Think](/blog-introducing-instant-actions-and-deep-think.md): From Midscene v0.14.0, we have introduced two new features: Instant Actions and Deep Think.
- [Chrome Bridge environment setup](/bridge-mode.md): Chrome Bridge enables Node.js scripts to control your desktop Chrome browser through the Midscene Chrome extension, allowing you to reuse existing login sessions, cookies, and browser extensions. The Node.js process runs the Bridge Server and handles AI model calls, while the Chrome extension connects to that server and executes operations on browser tabs.
- [Caching AI Planning and DOM Localization](/caching.md): Midscene supports caching two kinds of data: AI Planning steps and matched element localization information. The former can be used across automation workflows to reduce AI model calls and improve execution efficiency; for web automation, the DOM localization part (XPath) in the latter can significantly reduce repeated locating overhead, but it is currently web-only and has certain limitations. Effect With caching hit, time cost is significantly reduced. For example, in the following case, execution time was reduced from 51 seconds to 28 seconds. before after
- [Changelog](/changelog.md): {/* release-notes:start */}
- [Use a ChatGPT subscription](/chatgpt-subscription.md): Authorize Midscene through Sign in with ChatGPT (SIWC) to call GPT models using your ChatGPT subscription. This guide walks you through sign-in, configuration, and connection verification. A ChatGPT subscription and a model API key are two ways to connect Midscene. You can choose either based on your model payment plan.
- [](/common/get-cdp-url.md)
- [](/common/legacy-yaml-notice.md)
- [](/common/prepare-android.md)
- [](/common/prepare-ios.md)
- [](/common/sdk-common-methods.md)
- [](/common/setup-env.md)
- [](/common/test-collaboration.md)
- [](/common/test-philosophy.md)
- [](/common/test-workflow.md)
- [Consume Report Files](/consume-report-file.md): Midscene HTML report files capture the full execution history of a single Agent, making them useful for replay and debugging. Starting in v1.7.0, you can extract raw screenshots and JSON data from a report file, or convert the report into Markdown so other tools can consume it.
- [Data privacy](/data-privacy.md): ⁠Midscene.js is an open-source project (GitHub: Midscene) under the MIT license. You can see all the codes in the public repository. When using Midscene.js, your page data (including the screenshot) is sent directly to the AI model provider you choose. No third-party platform will have access to this data. All you need to be concerned about is the data privacy policy of the model provider. If you prefer building Midscene.js and its Chrome Extension in your own environment instead of using the published versions, you can refer to the Contributing Guide to find building instructions.
- [FAQ](/faq.md)
- [Integrate with any interface](/integrate-with-any-interface.md): You can use Midscene Agent to control any interface—such as IoT devices, in-house apps, and in-vehicle displays—by implementing a UI operation class that conforms to AbstractInterface. After implementing the UI operation class, you get the full capabilities of Midscene Agent: the TypeScript GUI Automation Agent SDK, supporting integration with any interfacethe playground for debuggingcontrolling the interface with YAML scriptsSkills support through CLI commands
- [Integrate with Playwright Test](/integrate-with-playwright.md): With PlaywrightAiFixture, you can use Midscene's visual interaction and assertion capabilities directly in an existing Playwright Test project. Playwright Test organizes and runs the tests, while Midscene interacts with the page and generates execution reports.
- [Web / Puppeteer SDK](/integrate-with-puppeteer.md): @midscene/web/puppeteer provides Agents bound to a Puppeteer page or Browser. Both inherit the base Agent’s AI methods and accept Web configuration options.
- [LLMs.txt documentation](/llm-txt.md): How to get tools like Cursor, Windstatic, GitHub Copilot, ChatGPT, and Claude to understand Midscene.js. We support LLMs.txt files for making the Midscene.js documentation available to large language models.
- [MCP integration has been retired](/mcp.md): Midscene no longer ships MCP servers. Use Skills to let AI coding agents drive Midscene through the platform CLIs. If you still need the MCP server packages, pin Midscene to 1.9.8. This is the final version that includes MCP support. Remove any agent configuration that references the retired MCP packages: @midscene/web-bridge-mcp@midscene/android-mcp@midscene/ios-mcp@midscene/harmony-mcp@midscene/computer-mcp@midscene/mcp If your previous MCP configuration set MIDSCENE_MCP_CHROME_PATH, move that value to MIDSCENE_CHROME_PATH for Skills and CLI usage. The old variable is still accepted as a temporary migration alias. For code-level automation, use the JavaScript SDK, YAML runner, or the platform CLIs listed in Skills.
- [Configure test projects](/midscene-test/configuration.md): midscene.config.ts is the project configuration file for Midscene Test. Use it to configure browser or device initialization, register Nodes for test cases to call, and set execution options such as concurrency and timeouts. This guide covers configuration structure, Agent and platform integration, multiple execution projects, and programmatic execution.
- [Develop custom Nodes](/midscene-test/extend.md): Custom Nodes turn business operations into steps that test authors can call from YAML. This guide covers defining and validating inputs, sharing data between Nodes, and handling execution and cleanup. Start with Create and use test projects if you do not have a test project yet. For platform setup, Agent integration, and execution settings, see Configure test projects. Register a Node: define an operation, call it from YAML, and verify registration through the generated Node Spec.Node execution arguments: distinguish business inputs from framework step settings.Share context across Nodes: create, read, and reset shared test data.Advanced usage: return results, handle errors and cancellation, and clean up resources.
- [Run in GitHub Actions](/midscene-test/github-actions.md): Midscene provides midscene-action to run Midscene Test in GitHub Actions. Automatically add test results to the workflow Summary.Save HTML reports and test data as artifacts for download.Optionally publish reports to Cloudflare Workers for online previews. If you do not have a test project yet, start with Create and use test projects. For setup instructions, configuration parameters, and complete examples, see midscene-action on GitHub Marketplace.
- [Midscene Test overview](/midscene-test/overview.md)
- [Create and use test projects](/midscene-test/use.md): This guide starts with project creation, then covers running an example, learning the available test operations, writing YAML cases, and inspecting results. If you already have a Midscene Test project, start with Understand available test operations. For the overall design, see Midscene Test overview. To implement business operations, see Develop custom Nodes. For runtime environments and execution settings, see Configure test projects.
- [Migrate to Midscene Test](/migrate-to-midscene-test.md): Legacy YAML and native Midscene Test share the same underlying execution kernel and report page, but they have separate user-facing entries. The midscene command owns legacy tasks/flow files, YAML batch configuration, and legacy CLI options. The midscene-test command owns native cases/steps files, TypeScript or JavaScript project configuration, lifecycle hooks, and Nodes. There are two migration paths: Upgrade in place: keep legacy cases, batch configuration, directory layout, and the midscene command unchanged. This adopts the shared execution kernel and new report without changing the input contract.Native migration: rewrite complete files as Test Documents, Cases, and Steps; replace batch YAML with project configuration; then switch those files to midscene-test. The complete legacy field reference remains available in YAML script runner and Workflow in YAML format for maintaining and migrating existing projects.
- [Midscene MobileWorld Benchmark Report](/mobile-world-benchmark-report.md): This report presents Midscene results on the MobileWorld benchmark. The evaluation covers 117 tasks, with a Pass@1 score of 78.63% (92/117).
- [Use a model API key](/model-api-setup.md): Connect Midscene using an API key from a model provider. This guide covers model configuration and connection verification. If you already have a ChatGPT subscription, you can also follow Use a ChatGPT subscription. The model configuration shown on this page consists of environment variables. Provide it according to how you use Midscene: In Playground, paste the configuration text from this page directly into the settings page.With the SDK or CLI, load the configuration as described in Set environment variables. For advanced parameter definitions, see Model advanced configuration and debugging.
- [Model advanced configuration and debugging](/model-config.md): This page covers advanced model settings, connection troubleshooting, logs, and tracing.
- [Model Selection Guide](/model-selection.md): This guide introduces the models supported by Midscene, two ways to connect them, and when to use one Default model or combine multiple models.
- [Android environment setup](/platforms/android.md): Midscene connects to and controls Android devices over adb, supporting UI automation across native apps, hybrid apps, and system interfaces. This guide walks you through setting up the Android environment required by Midscene Test, the JavaScript SDK, and Playground. You will first connect your device over adb, and then use Playground to verify screen capture, touch input, and model calls.
- [Desktop environment setup](/platforms/desktop.md): Midscene automates desktop applications across Windows, macOS, and Linux using native mouse and keyboard controls or Remote Desktop Protocol (RDP), with built-in multi-display support. It works with Electron, Qt, native desktop apps, and cross-application workflows. This guide walks you through setting up the desktop environment required by Midscene Test, the JavaScript SDK, and Playground. You will first prepare the desktop environment and permissions, and then use Playground to verify screen capture, mouse and keyboard input, and model calls.
- [HarmonyOS environment setup](/platforms/harmonyos.md): Midscene connects to and controls HarmonyOS NEXT devices through HarmonyOS Device Connector (HDC), supporting UI automation across native apps, atomic services, and system interfaces. This guide walks you through setting up the HarmonyOS environment required by Midscene Test, the JavaScript SDK, and Playground. You will first connect your device over HDC, and then use Playground to verify screen capture, touch input, and model calls.
- [Platform environment setup](/platforms/index.md): This section covers environment setup across all supported platforms. Before integrating with Midscene Test or the JavaScript SDK, select your target platform, connect your device or browser, and verify model and UI interactions in Playground. Midscene uses multimodal vision models to understand screenshots, operating directly on the rendered UI rather than relying on the underlying DOM or native view hierarchy. This allows the same interaction and assertion approach to work across native apps and cross-platform frameworks alike.
- [iOS environment setup](/platforms/ios.md): Midscene connects to and controls iOS devices through WebDriverAgent (WDA), supporting UI automation across native apps, hybrid apps, and system interfaces. This guide walks you through setting up the iOS environment required by Midscene Test, the JavaScript SDK, and Playground. You will first connect your device through WDA, and then use Playground to verify screen capture, touch input, and model calls.
- [Web environment setup](/platforms/web.md): For Web automation, you can either launch a controlled browser with Playwright / Puppeteer or connect to an existing desktop Chrome instance via Chrome Bridge to reuse active login sessions. Start by verifying your model settings and page interactions in the Chrome extension Playground, then prepare the browser environment for your chosen integration method.
- [Provide application knowledge to a GUI Agent](/provide-application-knowledge-to-gui-agent.md): When using a GUI Agent, developers often want to describe a goal and let the Agent handle the actions. But a clear goal does not mean the Agent already has the application knowledge needed to complete it. This article uses a practical example to explain how to supply relevant application knowledge and express your intent clearly, helping a GUI Agent execute tasks more reliably.
- [Try the GUI Agent](/quick-start.md): Use the Chrome extension to try the GUI Agent in a browser Playground without setting up a test project. This guide walks you through model configuration, running a natural-language instruction, checking its result, and reviewing execution. To write YAML tests directly, go to Midscene Test.
- [Shared Agent APIs](/reference/index.md): This page documents the shared APIs inherited by all platform Agents—covering configuration options, interaction methods, data extraction, element location, assertions and waiting, continuous observation, workflow execution, reporting and usage metrics, as well as lifecycle and runtime configuration. <a id="common"></a> <a id="constructors"></a>
- [Android JavaScript SDK](/sdk/android.md): @midscene/android controls Android devices over adb. AndroidAgent inherits all AI methods from the base Agent.
- [Chrome Bridge SDK](/sdk/chrome-bridge.md): AgentOverChromeBridge controls desktop Chrome through the Midscene Chrome extension and inherits all AI methods from the base Agent. Complete the Chrome Bridge environment setup before connecting.
- [Computer JavaScript SDK](/sdk/desktop.md): @midscene/computer provides Device implementations for local desktops (ComputerDevice) and remote RDP desktops (RDPDevice), along with ComputerAgent, which inherits all AI methods from the base Agent.
- [HarmonyOS JavaScript SDK](/sdk/harmonyos.md): @midscene/harmony controls HarmonyOS devices over HDC. HarmonyAgent inherits all AI methods from the base Agent.
- [iOS JavaScript SDK](/sdk/ios.md): @midscene/ios controls iOS devices through WebDriverAgent (WDA). IOSAgent inherits all AI methods from the base Agent.
- [Web / Playwright SDK](/sdk/playwright.md): @midscene/web/playwright provides Agents bound to a Playwright page or BrowserContext. Both inherit the base Agent’s AI methods and accept Web configuration options.
- [Write standalone automation scripts](/sdk/standalone.md): You can call the Midscene SDK directly from a Node.js script to operate apps, extract data, or automate everyday workflows without a test runner. Your script controls the execution order, branching logic, error handling, and resource cleanup.
- [Integrate with Vitest](/sdk/vitest.md): We recommend Vitest for writing JavaScript and TypeScript tests on Android, iOS, HarmonyOS, and desktop. Vitest organizes test cases, runs lifecycle hooks, and aggregates test results, while Midscene Agents connect to devices, interact with the UI, and perform assertions.
- [](/showcases-android.md)
- [](/showcases-computer.md)
- [](/showcases-harmony.md)
- [](/showcases-ios.md)
- [](/showcases-web.md)
- [Showcases](/showcases.md): This page introduces cross-platform GUI Agent, E2E test, and community showcases built with Midscene.
- [Control any platform with Skills](/skills.md): Agent Skills are a format for extending AI coding agents with specialized capabilities. Midscene provides Agent Skills that let AI coding tools (like Claude Code, Cline, etc.) drive UI automation through CLI commands. Skills work by running CLI commands directly in the terminal. The AI agent acts as the brain: it takes screenshots, analyzes the UI, and decides which actions to perform next.
- [YAML script runner](/yaml-script-runner.md): Midscene defines a YAML-based scripting format so you can quickly author automation scripts, then run them from the command line without extra setup. For more details on YAML scripts, see Automate with scripts in YAML. For example, you can write a YAML script like this: Run it with one command: The CLI prints execution progress and generates a visual report when it finishes, while keeping setup simple.