Validation
How a Syndai run checks its change: the checks it finds in your repo, the validation levels, phone app checks, and the evidence the API returns.
After the agent makes a change, Syndai checks it. It finds the checks in your repo and runs them on the changed code. You say how careful to be with level. Syndai picks the tools. You never send commands.
The rule before a pull request
A change can touch several parts of an app: web pages, a phone app, a backend. Each part the change touches needs at least one check that passed on the final code. If a part has none, the run does not open a pull request. It names the part, as in validation_evidence_missing_web.
A failed check goes back to the agent to fix. It gets up to budget.retry_budget tries. See Policy.
Levels
Set validation.level in policy_config. The default is standard.
| level | what changes |
|---|---|
smoke | Skips flutter analyze. Otherwise the same as standard. |
standard | The default. |
strict | Slow checks get more time. Phone app checks go from 1200 to 1800 seconds. Browser checks go from 900 to 1200 seconds. Slow first builds go from 1200 to 1800 seconds. |
Those are the only differences. Every other check runs at all three levels. Which checks run depends on your repo and on what the change touches.
What runs
Syndai looks at your repo's files to find its stacks. validation.profile can add a hint, but detection still runs.
| stack | checks |
|---|---|
Web (nextjs, vite) | package.json scripts named format, lint, typecheck, type-check, test and build. A test script that runs Playwright is left out. Also a build, tsc --noEmit, and unit tests near the changed files. |
| Web pages | When the change touches the web app, a Syndai browser check opens the pages and compares them to the base branch, with accessibility checks. Your own Playwright specs run when the change edits them. |
python | ruff, a type checker (ty or mypy), and pytest when your project uses it. Pytest skips integration and end to end tests, and runs only the tests near the change when it can. Makefile targets from a safe list also run. |
flutter | flutter pub get, flutter test when the repo has tests, flutter analyze (not at smoke), and a debug Android build. |
react_native | The same package.json scripts as web. With an Android module, also assembleRelease, testDebugUnitTest and lintDebug. |
native_android | assembleDebug, testDebugUnitTest and lintDebug. It needs an Android app module. |
| Swift package | swift test for a Package.swift outside ios/. |
Commands written in your docs never run. Syndai runs scripts and targets your repo defines, not text in a README.
To add a check, add it to your repo as a package.json script or a Makefile target. These policy_config keys are refused with INVALID_POLICY: commands, execution_environment, github_workflow_id, github_result_artifact_name and artifact_upload.
A repo with no checks
If Syndai finds no checks it can run, the run stops before the pull request. Set validation.bootstrap_when_toolless: true and the agent adds a small test setup for the language it sees, so the change can be checked.
Phone app checks
A phone app check runs a Maestro flow on a device. The flow is yours. Commit it to your repo as .maestro/*.yaml or .maestro/*.yml. With no flow in the repo, no device check runs.
| platform | device check |
|---|---|
| iOS (Flutter, native iOS) | on a simulator, with an unsigned build |
| Android | on an x86_64 emulator, with the debug APK |
| React Native | iOS and Android, with a Release build |
| Expo | iOS and Android, after expo prebuild |
Phone app checks run at every level. If the simulator fails on our side, the run says so with validation_evidence_unverified_ios_infra. That is our fault, not the change's.
Evidence
GET /runs/{id} returns a validations array. Each entry is one check:
| field | what it holds |
|---|---|
id | the check's id |
validator_name | the check that ran, such as flutter_analyze or playwright_tier0_smoke |
category | the kind of check, such as lint, test, build or browser |
status | pending, running, passed, failed, skipped, blocked or cancelled |
blocking | true when a failure stops the pull request |
iteration | which try of the run it belongs to |
tree_sha | the code it checked |
duration_ms | how long it took |
passed_count, failed_count, skipped_count | test counts, when the check reports them |
findings, errors | what the check found |
is_baseline_delta | true on a run of the check against the base branch. With status failed, it means the change added new failures. |
artifacts | evidence files: screenshots, logs and reports |
created_at, updated_at | timestamps |
GET /runs/{id}/live returns a shorter validation list while the run works. Each entry has name, cwd, state and p50_s.
The coding.validation.completed webhook fires once for each check as it ends. It carries that one check, not the full list. Treat it as progress, and read the run for the whole picture.
Evidence files
Each entry in artifacts has an artifact_id. Storage paths are never returned. To read a file, call GET /artifacts/{artifact_id}/open. It returns the text inline, or a signed_url that ends at signed_url_expires_at. How long files are kept is set by artifacts.retention_profile.