Conformance suite

The conformance suite is a set of JSON test cases that every SDK must pass. It defines what "spec-compliant" means for a CrowdControl implementation.

Layout

The suite lives under conformance/suite/ in the repo. Each file is a single test case with a policy, an input document, and the expected results.

{
  "name": "basic-forbid",
  "description": "A forbid rule fires when all conditions match",
  "policy": "forbid \"no-interns\" {\n  user.role == \"intern\"\n  message \"no interns\"\n}",
  "input": { "user": { "role": "intern" } },
  "expected": [
    {
      "rule": "no-interns",
      "kind": "forbid",
      "passed": false,
      "message": "no interns"
    }
  ]
}

Running the suite

Each SDK has a conformance runner that iterates every case in conformance/suite/, evaluates the policy against the input, and compares the results to expected.

# Go
go test ./conformance/...

# Python
cd sdks/python && python conformance_runner.py

# TypeScript
cd sdks/typescript && npm run conformance

# Ruby
cd sdks/ruby && ruby conformance_runner.rb

# PHP
cd sdks/php && php conformance_runner.php

# Kotlin
cd sdks/kotlin && ./gradlew conformance

Running in Docker

Every SDK ships a Dockerfile so you can run its conformance suite without installing the language runtime locally. Build from the repo root — the build context needs to include conformance/ plus the SDK directory:

# single SDK
docker build -f sdks/python/Dockerfile -t crowdcontrol-python .
docker run --rm crowdcontrol-python

# all SDKs at once (parallel, via docker compose)
docker compose up --build --abort-on-container-exit

# or the helper script (serial, prints a pass/fail summary)
./sdks/run-conformance.sh

Each container exits 0 on full pass and non-zero on any failure, so they plug straight into CI.

What "spec-compliant" means

An SDK is considered spec-compliant when:

  1. It implements every construct in SPEC.md.
  2. It passes every case in conformance/suite/.
  3. Its result objects (rule, kind, passed, message) match the Go reference exactly for the same input.
  4. Its trace output is a structural superset of what the spec defines.

Adding new cases

When the spec changes, the conformance suite is updated first. SDKs must then be updated to match. The Go implementation is the reference for any behavior not explicitly nailed down by SPEC.md.