Test Playbooks

Declarative YAML/JSON test definitions — how to write, run, and manage playbooks in TINAA MSP.

Test Playbooks

A playbook is a declarative, version-controlled test definition that describes a user journey through your application. Playbooks are the fundamental unit of testing in TINAA MSP. They are YAML or JSON files that list the browser actions to perform, the assertions to verify, and the performance thresholds to enforce.

Playbooks page showing registered test playbooks and their status

Auto-Generated vs Manual Playbooks

TINAA supports two authoring modes:

Source How it works Best for
auto_generated TINAA scans your repository and infers journeys from routes, forms, and link structure Getting coverage quickly on an existing codebase
manual You write the YAML by hand or via Claude/MCP Critical flows requiring precise control
hybrid TINAA generates a starter playbook which you then refine Most teams — generated skeleton, hand-tuned details

Auto-generation is triggered automatically when you register a product with a repository_url. You can also trigger it on demand from the product settings page or via the explore_codebase MCP tool.

Playbook Schema

A playbook is defined by the following top-level fields:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

name: string               # unique name within the product, used as an identifier
description: string        # human-readable description (shown in reports)
priority: critical | high | medium | low
tags:
  - string                 # free-form labels (e.g. "smoke", "checkout", "auth")

# Trigger conditions — when this playbook runs automatically
trigger:
  on_deploy:               # run on deployment to these environments
    - production
    - staging
  on_pr: true              # run as a GitHub Check Run on every PR
  schedule_cron: "0 * * * *"  # also run on a cron schedule (hourly)
  on_change:               # only trigger when these paths change
    - src/checkout/**
    - src/cart/**

# Optional: variables available as ${VAR_NAME} in step params
variables:
  BASE_URL: "https://staging.myapp.com"
  TEST_EMAIL: "qa-robot@example.com"

# Optional: steps that run before the main steps (e.g. login)
setup_steps: []

# The main test steps
steps:
  - action: navigate
    url: "${BASE_URL}"

# Optional: steps that always run after main steps (e.g. logout, cleanup)
teardown_steps: []

# Optional: global assertions checked at end of run
assertions:
  no_console_errors: true
  no_network_failures: true
  max_accessibility_violations: 0

# Optional: performance thresholds that must be met for the run to pass
performance_gates:
  lcp_ms: 2500       # Largest Contentful Paint
  fcp_ms: 1800       # First Contentful Paint
  cls: 0.1           # Cumulative Layout Shift
  inp_ms: 200        # Interaction to Next Paint
  total_duration_ms: 60000  # entire playbook wall-clock time
  max_network_failures: 0

Supported Actions

Navigate the browser to a URL.

1
2
3
4

- action: navigate
  url: "https://staging.myapp.com/checkout"
  # Optional: wait for a specific network idle state before continuing
  wait_until: networkidle  # domcontentloaded | load | networkidle

click

Click an element on the page.

1
2
3
4
5
6

- action: click
  selector: "button[data-testid='submit-order']"
  # Optional: describe the click for reporting
  description: "Click the Submit Order button"
  # Optional: override the default 30s timeout
  timeout_ms: 10000

Selectors follow the Playwright selector syntax: CSS selectors, text=, role=, data-testid=, XPath, and more.

fill

Fill a form input with a value.

1
2
3
4
5
6
7

- action: fill
  selector: "input[name='email']"
  value: "${TEST_EMAIL}"

- action: fill
  selector: "input[name='password']"
  value: "SecurePassword123!"

Use variables (defined in the variables block) to avoid hard-coding credentials in playbook files.

type

Type text character-by-character, simulating keyboard input. Useful when fill bypasses input event listeners.

1
2
3

- action: type
  selector: "#search-box"
  value: "wireless headphones"

select

Select an option in a <select> dropdown by value or label.

1
2
3
4

- action: select
  selector: "select[name='country']"
  value: "US"           # select by option value
  # label: "United States"  # or select by visible text

press_key

Send a keyboard key press to a focused element or globally.

1
2
3
4
5
6

- action: press_key
  key: "Enter"

- action: press_key
  selector: "#promo-code-input"
  key: "Tab"

screenshot

Capture a screenshot. Screenshots are embedded in test reports and stored for trend comparison.

1
2
3

- action: screenshot
  name: "checkout-confirmation"   # used as the screenshot filename in reports
  full_page: false                # set true to capture the full scrollable page

wait

Pause execution for a fixed duration. Use sparingly — prefer wait_for_navigation or implicit waits.

1
2

- action: wait
  ms: 500

wait_for_navigation

Wait until navigation completes (page load, redirect, or SPA route change).

1
2
3

- action: wait_for_navigation
  wait_until: networkidle
  timeout_ms: 15000

hover

Move the mouse cursor over an element (useful for triggering tooltips or dropdown menus).

1
2

- action: hover
  selector: "nav .products-menu"

scroll

Scroll the page to reveal elements or trigger lazy loading.

1
2
3
4

- action: scroll
  selector: ".product-grid"   # scroll element into view
  # x: 0                      # or scroll by pixel delta
  # y: 500

clear

Clear the value of a text input.

1
2

- action: clear
  selector: "input[name='search']"

upload_file

Upload a file to a file input element.

1
2
3

- action: upload_file
  selector: "input[type='file']"
  file_path: "fixtures/test-document.pdf"

set_viewport

Resize the browser viewport. Useful for responsive design testing.

1
2
3
4

- action: set_viewport
  width: 375
  height: 812
  description: "Switch to iPhone 13 viewport"

evaluate

Execute arbitrary JavaScript in the browser context and optionally assert the return value.

1
2
3

- action: evaluate
  script: "document.querySelector('#cart-count').textContent"
  expected: "3"

group

Group related steps together for reporting clarity. Groups appear as collapsible sections in test reports.

1
2
3
4
5
6
7
8
9
10
11
12

- action: group
  description: "Complete payment details"
  steps:
    - action: fill
      selector: "input[name='card_number']"
      value: "4111 1111 1111 1111"
    - action: fill
      selector: "input[name='expiry']"
      value: "12/28"
    - action: fill
      selector: "input[name='cvv']"
      value: "123"

Assert Actions

Action Verifies
assert_visible Element exists and is visible
assert_hidden Element does not exist or is hidden
assert_text Element contains expected text
assert_url Current URL matches pattern
assert_title Page <title> matches
assert_no_console_errors No console.error() calls during the step
assert_no_network_failures No 4xx/5xx responses since last check
assert_accessibility No WCAG violations on the current page state 
1
2
3
4
5
6
7
8
9
10
11
12
13

- action: assert_visible
  selector: ".order-confirmation"
  description: "Order confirmation banner is shown"

- action: assert_text
  selector: "h1.order-title"
  text: "Thank you for your order"

- action: assert_url
  pattern: "/order/[0-9]+"

- action: assert_accessibility
  level: "AA"   # A | AA | AAA

Suite Types

Playbooks are tagged with one or more suite types that determine when they run:

Suite Purpose Typical duration
smoke Fast sanity check — does the app start and respond? < 2 minutes
regression Full user journey coverage before a release 5–20 minutes
accessibility WCAG compliance audit 3–10 minutes
performance Web Vitals, LCP, CLS, response times 5–15 minutes
security Headers, TLS, mixed content, form security 2–5 minutes

Tag a playbook with a suite type in the tags field:

1
2
3

tags:
  - smoke
  - regression

Running Playbooks

From the Dashboard (UI)

  1. Navigate to Playbooks in the left sidebar
  2. Find the playbook you want to run
  3. Click the Run button (or the three-dot menu for options)
  4. Select the target environment
  5. Click Execute

The run appears in Test Runs in real time as steps complete.

Via the REST API

1
2
3
4
5
6
7
8

POST /api/v1/playbooks/{playbook_id}/run
Content-Type: application/json
X-API-Key: <your-api-key>

{
  "environment": "staging",
  "target_url": "https://staging.myapp.com"  # optional URL override
}

Via MCP (Claude Code)

1
2
3
4

run_playbook(
  playbook_id_or_name="checkout-regression",
  environment="staging"
)

Or run the full test suite for a product:

1
2
3
4
5

run_suite(
  product_id_or_slug="checkout-service",
  environment="staging",
  suite_type="regression"
)

Via CI/CD

Use the TINAA CLI or API in your pipeline. Example for GitHub Actions:

1
2
3
4
5
6
7
8
9
10
11

- name: Run TINAA smoke tests
  run: |
    curl -X POST $TINAA_URL/api/v1/products/$PRODUCT_SLUG/run-suite \
      -H "X-API-Key: $TINAA_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"suite_type": "smoke", "environment": "staging"}' \
      --fail-with-body
  env:
    TINAA_URL: $
    TINAA_API_KEY: $
    PRODUCT_SLUG: checkout-service

Performance Gates in Playbooks

Performance gates define thresholds that a test run must meet to be considered passing. If any gate is breached, the run is marked failed even if all browser assertions pass.

1
2
3
4
5
6
7

performance_gates:
  lcp_ms: 2500        # Largest Contentful Paint must be <= 2.5 seconds
  fcp_ms: 1800        # First Contentful Paint must be <= 1.8 seconds
  cls: 0.1            # Cumulative Layout Shift score must be <= 0.1
  inp_ms: 200         # Interaction to Next Paint must be <= 200ms
  total_duration_ms: 120000  # Entire playbook must complete within 2 minutes
  max_network_failures: 0    # Zero 4xx/5xx responses allowed

When performance gates are defined, TINAA instruments the Playwright browser with the Web Performance APIs to capture metrics throughout the run. Results appear in the test run report alongside the step-level results.

Full Example: E-Commerce Checkout Flow

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126

name: checkout-full-flow
description: >
  End-to-end checkout journey: browse products, add to cart, enter
  payment details, and confirm the order.
priority: critical
tags:
  - regression
  - smoke

trigger:
  on_deploy:
    - production
    - staging
  on_pr: true
  on_change:
    - src/checkout/**
    - src/cart/**
    - src/payment/**

variables:
  BASE_URL: "https://staging.myapp.com"
  USER_EMAIL: "qa-robot@example.com"
  USER_PASSWORD: "QaPassword123!"

setup_steps:
  - action: navigate
    url: "${BASE_URL}/login"
  - action: fill
    selector: "input[name='email']"
    value: "${USER_EMAIL}"
  - action: fill
    selector: "input[name='password']"
    value: "${USER_PASSWORD}"
  - action: click
    selector: "button[type='submit']"
    description: "Log in"
  - action: assert_url
    pattern: "/dashboard"
  - action: wait_for_navigation
    wait_until: networkidle

steps:
  - action: navigate
    url: "${BASE_URL}/products"
  - action: screenshot
    name: "product-listing"
  - action: click
    selector: ".product-card:first-child .add-to-cart"
    description: "Add first product to cart"
  - action: assert_visible
    selector: ".cart-notification"
    description: "Cart notification appears"
  - action: navigate
    url: "${BASE_URL}/cart"
  - action: assert_text
    selector: ".cart-item-count"
    text: "1 item"
  - action: click
    selector: "a[href='/checkout']"
    description: "Proceed to checkout"
  - action: wait_for_navigation
    wait_until: networkidle
  - action: screenshot
    name: "checkout-page"

  - action: group
    description: "Fill shipping address"
    steps:
      - action: fill
        selector: "input[name='full_name']"
        value: "QA Robot"
      - action: fill
        selector: "input[name='address_line1']"
        value: "123 Test Street"
      - action: fill
        selector: "input[name='city']"
        value: "San Francisco"
      - action: select
        selector: "select[name='state']"
        value: "CA"
      - action: fill
        selector: "input[name='zip']"
        value: "94102"

  - action: group
    description: "Fill payment details"
    steps:
      - action: fill
        selector: "input[name='card_number']"
        value: "4111 1111 1111 1111"
      - action: fill
        selector: "input[name='expiry']"
        value: "12/28"
      - action: fill
        selector: "input[name='cvv']"
        value: "123"

  - action: screenshot
    name: "checkout-filled"
  - action: click
    selector: "button[data-testid='place-order']"
    description: "Place order"
  - action: wait_for_navigation
    wait_until: networkidle
  - action: assert_url
    pattern: "/order-confirmation/[0-9]+"
  - action: assert_visible
    selector: ".confirmation-banner"
  - action: assert_text
    selector: "h1"
    text: "Thank you for your order"
  - action: screenshot
    name: "order-confirmation"

teardown_steps:
  - action: navigate
    url: "${BASE_URL}/logout"

assertions:
  no_console_errors: true
  no_network_failures: true

performance_gates:
  lcp_ms: 2500
  cls: 0.1
  total_duration_ms: 120000

Full Example: Login and Authentication

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104

name: login-auth-flows
description: >
  Covers successful login, failed login, session persistence,
  and logout across the authentication system.
priority: critical
tags:
  - smoke
  - regression

trigger:
  on_deploy:
    - production
    - staging
  on_pr: true
  on_change:
    - src/auth/**
    - src/login/**

variables:
  BASE_URL: "https://staging.myapp.com"
  VALID_EMAIL: "qa-robot@example.com"
  VALID_PASSWORD: "QaPassword123!"

steps:
  # ---- Scenario 1: Successful login ----
  - action: navigate
    url: "${BASE_URL}/login"
  - action: assert_title
    title: "Sign In — My App"
  - action: assert_visible
    selector: "input[name='email']"
  - action: fill
    selector: "input[name='email']"
    value: "${VALID_EMAIL}"
  - action: fill
    selector: "input[name='password']"
    value: "${VALID_PASSWORD}"
  - action: screenshot
    name: "login-filled"
  - action: click
    selector: "button[type='submit']"
  - action: wait_for_navigation
    wait_until: networkidle
  - action: assert_url
    pattern: "/dashboard"
  - action: assert_visible
    selector: ".user-menu"
    description: "User menu visible after login"

  # ---- Scenario 2: Invalid credentials ----
  - action: navigate
    url: "${BASE_URL}/logout"
  - action: navigate
    url: "${BASE_URL}/login"
  - action: fill
    selector: "input[name='email']"
    value: "wrong@example.com"
  - action: fill
    selector: "input[name='password']"
    value: "WrongPassword!"
  - action: click
    selector: "button[type='submit']"
  - action: assert_visible
    selector: ".error-message"
    description: "Error message shown for bad credentials"
  - action: assert_text
    selector: ".error-message"
    text: "Invalid email or password"
  - action: assert_url
    pattern: "/login"
    description: "User stays on login page after failed attempt"

  # ---- Scenario 3: Logout ----
  - action: navigate
    url: "${BASE_URL}/login"
  - action: fill
    selector: "input[name='email']"
    value: "${VALID_EMAIL}"
  - action: fill
    selector: "input[name='password']"
    value: "${VALID_PASSWORD}"
  - action: click
    selector: "button[type='submit']"
  - action: wait_for_navigation
    wait_until: networkidle
  - action: click
    selector: ".user-menu"
  - action: click
    selector: "a[href='/logout']"
  - action: wait_for_navigation
    wait_until: networkidle
  - action: assert_url
    pattern: "/login|/"
    description: "User redirected after logout"
  - action: assert_hidden
    selector: ".user-menu"

assertions:
  no_console_errors: true
  max_accessibility_violations: 0

performance_gates:
  lcp_ms: 2000
  total_duration_ms: 60000

Next Steps