Dashboard Setup

The dashboard reporter sends evaluation results to llmassert.com for trend analysis and team visibility. It is optional — matchers work without it.

1. Create an account

Sign up at llmassert.com/sign-up. The free tier includes 100 evaluations per month.

2. Create a project

After signing in, create a project from the dashboard. Note your project slug (e.g., my-chatbot).

3. Get your API key

Go to Settings > API Keys and create a new key. Store it securely — it's only shown once.

4. Configure the reporter

Add the reporter to your playwright.config.ts:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  reporter: [
    ["list"],
    [
      "@llmassert/playwright/reporter",
      {
        projectSlug: "my-chatbot",
        apiKey: process.env.LLMASSERT_API_KEY,
      },
    ],
  ],
});

Set the API key as an environment variable:

export LLMASSERT_API_KEY="lma_your-api-key-here"

5. Run your tests

pnpm exec playwright test

Results will appear in your dashboard automatically.

Local-only mode

Omit apiKey to run without dashboard integration. Evaluations are collected and assertions work normally — results just aren't sent to the dashboard.

["@llmassert/playwright/reporter", {
  projectSlug: "my-project",
  // No apiKey — local-only mode
}],

JSON file reporter

For local development or CI artifact storage, use the JSON reporter instead of (or alongside) the dashboard reporter:

reporter: [
  ["list"],
  ["@llmassert/playwright/json-reporter"],
],

Results are written to test-results/llmassert-results.json. See Reporters for full configuration options.

Next steps

• Configure the judge — timeouts, models, rate limiting
• Set up CI — GitHub Actions integration
• Understand scores — how scoring and thresholds work