CI, reporting & safety

Reports

Every run writes report/report.html (step timeline, full request/response detail), report/junit.xml (for CI test-result ingestion), and report/results.json (the same redacted run report as JSON — read a run's outcome from a file instead of scraping stdout) — they all fall out of the same event stream tflw run already emits, nothing to wire up.

Both machine-readable files name their schema. results.json opens with "$schema": "https://deepak-tuteja.github.io/tflw/schema/results.schema.json", and the first line of events.ndjson names events.schema.json. The schemas are generated from the types tflw writes the files from, so an editor autocompletes a script that reads a run, and a validator can check one. Objects are open on purpose: a later tflw adds fields, and a reader built against the schema keeps working. Every run is also kept under report/runs/<id>/, with the same files (running a suite).

A run that evaluated a security, authorization or input-handling assertion writes a fourth file, report/findings.sarif, for GitHub code scanning. It is the one artifact that is not written unconditionally, and that is deliberate: an empty SARIF document tells GitHub every previously reported alert is fixed.

report.html is a single file unless the run captured a screenshot or trace large enough to be written out beside it, in which case the report is report.html plus report/assets/ — its footer tells you which, along with what the file actually contains. Read that footer before you attach a report anywhere: at the default evidence level it holds whole response bodies and page screenshots. See evidence levels to turn that down.

tflw check [files] runs the same parse + full checker pipeline run executes before it does anything, with no execution and no secrets required — a fast pre-commit/CI lint step. tflw run exits non-zero on any test failure. A GitHub Actions example:

- uses: actions/setup-node@v4
  with:
    node-version: 22
- run: npm ci
- run: npx tflw run
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: tflw-report
    path: report/

junit.xml — what your CI dashboard reads

report/junit.xml is the artifact GitHub Actions, Jenkins, GitLab and friends ingest to build a test-results view. Its shape mirrors your suite: a <testsuites> root, one <testsuite> per .tflw file, and a classname on every <testcase> naming the file it came from.

<testsuites name="tflw" tests="2" failures="1" errors="0" time="1.234" timestamp="…">
  <testsuite name="tests/smoke.tflw" tests="1" failures="0" errors="0" time="0.012" timestamp="…">
    <properties>
      <property name="env" value="local"/>
      <property name="seed" value="42"/>
      <property name="now" value="…"/>
    </properties>
    <testcase name="checkout works" classname="tests/smoke.tflw" time="0.012"/>
  </testsuite>
  <testsuite name="tests/regression.tflw" tests="1" failures="1" errors="0" time="0.020" timestamp="…">
    …
    <testcase name="checkout works" classname="tests/regression.tflw" time="0.020">
      <failure message="expected status to equal 200, but got 500">…</failure>
    </testcase>
  </testsuite>
</testsuites>

Dashboards identify a test by name + classname and track its history — flaky rate, "first failed in", "fixed in" — under that identity. Two files declaring a test with the same name (a smoke.tflw and a regression.tflw both having a "checkout works") therefore stay two distinct rows, which is why the file is on every testcase and not only on the suite.

Each suite repeats the run's <properties> — including the seed, so you can reproduce a failed CI run locally with tflw run --seed <n> straight from the dashboard. They're duplicated because JUnit only allows <properties> inside a <testsuite>, not at the root.

A workload-bearing test contributes one <testcase> per declared threshold (see Load testing); a run whose generator saturated marks them <skipped> rather than passed or failed.

Secrets are redacted automatically

Anything that ever flowed through env(NAME) — header, body, URL, a derived interpolation — prints as •••(NAME) in report.html, traces, and CLI output, automatically. See Config & environments for require env.

redact — name a secret by position, not by source

redact masks a named body field, header or query parameter regardless of where its value came from — useful for PII (email, address, ssn) and for credentials that were never read through env(...) in the first place:

env staging
  api "https://staging.example.com"
  redact body.email, body.*.address
  redact header "Authorization", header "X-Api-Key", query "token"

• body.<path> — .prop segments and a .* wildcard (matches every object key or array element).
• header "<name>" — one request or response header, matched case-insensitively. "*" matches every header.
• query "<name>" — one URL query parameter. Only the value is masked; the path, the other parameters and the parameter's own name survive, so the report still says which request this was. (There is no redact url on purpose — masking a whole URL makes the report unreadable.)

Header and query names are quoted strings, the same way every header name is written elsewhere in the language (header "Accept" is …) — and the only form that can express X-Api-Key or Set-Cookie.

Applied to the report-only trace — expect/capture always see the real, unmasked value.

redact means "this value is a secret." If you capture out of a position you named here, the captured value is tracked from then on and masked wherever it turns up later — a subsequent request's URL, a log line, another step's detail text:

env staging
  redact body.accessToken

test "reads a session"
  api POST /login
  capture body.accessToken as token
  api GET /session?token={token}    # the token is masked here too

Evidence levels — how much lands in the report

env staging
  evidence headers only

• full (default) — everything: method/url/status/headers/body, screenshots and traces included.
• headers only — drops the request/response body (replaced with an [omitted by evidence level] marker). Two words, not a hyphen — an identifier in this language cannot contain -, and every multi-word construct it has is space-separated.
• none — drops headers too; only method/url/status/duration remain.

The level governs three things, not just the trace:

1. The request/response trace, as above.
2. Step detail text — a step's own line never shows what the level already dropped. A failing assertion at evidence none reads expected body.token to equal "…", but got [omitted by evidence level]: you still see what was compared, just not the value.
3. Screenshots and traces — captured only at full. Nothing can redact a screenshot; it is pixels, and a page that renders a token on screen renders it into the image. So rather than promise to clean them, headers only and none don't capture them at all. A screenshot step still passes below full, reporting not captured (evidence level); a matches snapshot assertion still runs and still tells you how many pixels differed, it just doesn't attach the images. The cost is real: at evidence none a failing browser test has no trace to open.

--evidence <level> overrides the config value for one run — handy for a CI job that wants none-level reports by policy regardless of what any given tflw.config declares.

--forbid-insecure — a CI policy gate

Fails before any test runs if insecure true (TLS verification disabled — see Config & environments) is active for the env actually running. Use it in CI to make sure a self-signed-cert workaround never silently ships as the default for a shared pipeline.

--allow-public-target — the affirmation CI cannot commit

npx tflw run --allow-public-target https://stg.example.com

Required before an originating scan — expect response has no authorization violations, which re-issues your request under every other declared principal — may reach a host outside the private address ranges. Missing, it is TF065; naming an origin the run does not scan or does not declare, TF066.

The point of it being a flag is what it is not: there is no tflw.config key, and there will not be one. authorized target (the declaration, in config) says what may be scanned and why; this says this invocation is permitted to do it. A config is a file you can merge into main, so if it could grant both, a pull request could make your pipeline scan the internet on its own.

Two consequences for a pipeline:

• Put it in the CI job for the environment that needs it, not in a shared base config. A flag parked in every job is a flag that affirms nothing, and it will silently cover whatever host someone points that env at later — which is why it takes an origin and not a bare --yes.
• tflw check accepts it too, so a lint job for a suite that scans a public staging host can still come back clean. check still sends nothing; the flag only changes which diagnostics it reports.

A has no security violations assertion needs no flag at all — it inspects a response your suite already asked for, so there is no extra packet to authorize.

Skipping a test — skip "reason", everywhere or on some envs

test "refunds settle" skip "the payments sandbox is down until the 3rd"
  api POST /refunds body { orderId: 7 }
  expect status equals 202

A skipped test stays in the file and runs nothing — not its hooks, not its steps. Every report shows it as its own outcome: a - line with the reason on the console, N skipped on the summary line, a grey skipped row in report.html, and a <skipped message="…"/> testcase in junit.xml, so a CI dashboard counts it as skipped rather than passed. A run whose only non-passes are skips still passes. The reason is required — skip "" is TF084 — because it is the only record of why the test stopped and when it comes back.

A skip can hold in some envs only. on env names the env blocks from tflw.config where the test is skipped; everywhere else it runs as if the clause were not there:

test "refunds settle" skip "no payments sandbox in CI" on env ci, staging
  api POST /refunds body { orderId: 7 }
  expect status equals 202

Under --env ci the report reads no payments sandbox in CI (on env ci) and the summary says 1 skipped (1 by env); under --env local the test runs. A name the config does not declare is TF088, because a misspelt env would make the skip hold nowhere.

Leaving a test out of one CI job is a different thing, and it belongs on the command line: --tag !slow runs everything not tagged @slow. Inclusions OR together, exclusions AND together, and an exclusion beats an inclusion on the same test.

Replaying failures — --failed and --bail

npx tflw run --failed   # re-run only the tests that are failing
npx tflw run --bail     # stop at the first failing test

--failed re-runs the tests whose newest verdict in the kept runs (report/runs/) is a failure — a test stays failing until a run passes it, whatever else ran in between. Nothing failing, or no kept run yet: falls back to the full suite with a note, never a silent zero-test run. --bail stops after the first failing test's final verdict; under --parallel > 1 it stops starting new files, but files already in flight finish normally.

Splitting a suite across CI jobs — --shard

npx tflw run --shard 1/4   # in job 1 of 4
npx tflw run --shard 2/4   # in job 2 of 4, and so on

--shard i/n runs every nth file of the sorted suite, starting at the ith, so the n jobs of a matrix are disjoint and together run everything exactly once. Files are dealt round-robin rather than in blocks, so a directory of slow files is spread over the jobs. --tag, --only and --failed narrow within the shard, and every shard still checks the whole suite before running its part — no job passes a file another would refuse. The header names the shard: shard 2/4: 31 of 124 files.

Each job writes its own report/. The last job joins them into one report with tflw merge, and its exit code is the verdict of the whole suite:

# after downloading every shard's report/ into shard-1/, shard-2/, …
npx tflw merge shard-*/report --out report

The merged results.json holds every shard's tests in shard order, with counts re-derived from them, and mergedFrom naming the inputs. junit.xml and report.html are rendered from it, findings are deduplicated by fingerprint so findings.sarif uploads each once, and each shard's events.ndjson is joined in order. tflw merge exits 0 when the merged run passed and 1 when it did not, so a needs: job that runs it is the one check a branch protection rule has to name.

Sending a run to your tracing backend — tflw export otlp

npx tflw export otlp --endpoint http://localhost:4318/v1/traces
npx tflw export otlp report --endpoint https://otel.example.com/v1/traces --header authorization=Bearer…

A finished run becomes one OpenTelemetry trace — a span for the run, one per file, one per test, one per step — sent over OTLP/HTTP's JSON encoding to any collector that accepts it. A failed test or step is an ERROR span carrying its message; a skipped test is an OK span whose tflw.outcome is skipped. Exporting the same run twice sends the same ids, so a collector shows one trace.

The times are laid end to end, and every span says so. A report records the run's start and every duration, not when each test started, so the spans are placed one after another from the run's start. That is exact for a sequential run and a layout for a --parallel one; each span carries tflw.timing = "reconstructed" so nobody reads a waterfall the run never measured.

Structured logs — --format ndjson

npx tflw run --format ndjson

Replaces the human console output with one JSON object per line (RunEvents — run:start/ test:start/step:end/test:end/run:end, each tagged with its source file) — pure stdout, no human text mixed in, safe to pipe into a log aggregator or jq. Always full step-level detail, independent of --verbose. Also always written to report/events.ndjson, so the stream survives even when the invoking process didn't capture stdout. A before file / after file hook emits a test:start/test:end pair like any other unit of work, and both halves carry hook: "before file" or hook: "after file"; run:start.total counts the tests, a passing hook is work in flight and never one of them, and the field — not the name — is how a consumer tells the two apart. The file is written one line at a time, so a stream of any size lands whole.

The file is not a byte-for-byte copy of stdout: it is written after the run, so it gets the same final redaction pass as report.html/results.json (see secrets) with the fully-populated redactor. The live stream can't — a line has already left the process by the time a secret read later in the run reveals itself. If you archive one of the two, archive the file.

User-defined logging — the log statement

test "checkout narrates what it's doing"
  api POST /cart/checkout body { email: "shopper@example.com" }
  expect status equals 201
  capture body.orderId as orderId

  log "order {orderId} placed"
  log warn "stock for this order is running low"
  log error "payment gateway retried once before succeeding" to html

log [debug|info|warn|error] "message with {var}" [to console|html|both] narrates what a test is doing, in the author's own words — level defaults to info, destination defaults to tflw.config's log destination key (built-in default both). A log step always succeeds; it's author signal, never an assertion. An explicit to … clause always wins over both config and a --log-output override — only a bare log "…" (no to) resolves against config/CLI at all.

Two tflw.config keys (defaults/env, override semantics like evidence): log destination console|html|both and log level debug|info|warn|error — the minimum level a step must clear to render. Bare keywords, the same words log warn "…" already used bare in a test. results.json/--format ndjson always record every log step regardless of level or destination; only console text and report.html filter what's actually displayed. See CLI flags reference for --log-output/--log-level, which override both config keys for a single run.

Console ergonomics — timestamps, GitHub Actions grouping, --log-file

Every console line gets an HH:MM:SS.mmm prefix by default — --no-timestamps opts out. On GitHub Actions (auto-detected via the GITHUB_ACTIONS env var), --verbose's per-test step lines fold into a collapsible ::group::/::endgroup:: block — normal mode is already one line per test, so grouping only kicks in under --verbose. --log-file <path> duplicates console output to a file, always plain text regardless of whether stdout itself has color.

Keeping a suite current — tflw migrate

npx tflw migrate

When a piece of syntax is deprecated, the checker flags it as a warning (never silently — a deprecation always spends at least one release as a visible warning before removal) and carries its exact mechanical replacement. tflw migrate reads those warnings and rewrites every affected file in place, then tells you to re-run tflw check to confirm the result is clean. Run it from CI or a pre-upgrade script the same way you'd run check — it's a normal exit-0/non-zero command, not interactive.

The grammar has been additive-only since the very first internal milestone (no existing syntax has ever changed, only new syntax added), so today tflw migrate genuinely has nothing to do and reports exactly that — no deprecated syntax found — nothing to migrate. — rather than pretending there's work to review. It's real, tested machinery sitting ready for the day a deprecation actually ships, not a stub.

Full reference: SPEC.md §12, §13 (events/report).

Selecting what a pipeline runs — @tag

A test or crawl can carry tags on their own lines above it — @smoke @checkout — and tflw run --tag smoke runs only the tests that carry one. Tags are OR across a comma-separated list, so --tag smoke,checkout runs anything carrying either.

This is how one suite serves two pipelines: a pull request runs --tag smoke in a minute, and the nightly build runs everything. Tagging is a property of the test, so a test moved between files keeps the pipelines that select it.

What a run writes, and where — report

The report key in tflw.config says where a run's artifacts land — report "./report" is the default, and any path works. The directory it names is what a pipeline archives: report.html, junit.xml and results.json all appear there, written from the same event stream.

Whatever you choose, the artifacts fall out of the same event stream — there is no capture step to configure and no way for the report to disagree with the terminal, because neither is generated from the other.

The hosts a run may reach — allow hosts

allow hosts "api.example.com", "auth.example.com" in tflw.config is the safety half of this chapter's title. A request to any host not on the list is refused before it is issued — not asserted against afterwards, refused — so a suite that has been pointed at the wrong environment fails loudly instead of quietly talking to production.

It is worth setting even when it looks redundant. The failure it prevents is the one where a base URL is overridden by an environment variable in CI and nothing else notices.