THOR-SEC

AI assurance case study · Case Studies

Model Identity Verifier

A CLI that flags model identity drift, provider or route mismatches, downgrade signals, and baseline changes without treating self-description as attestation.

prototype · Implemented prototype; live-provider release gate pending

Problem

A language model can state an identity in generated text, but that statement does not prove which provider route or model actually produced the response. Users still need a way to flag inconsistent identity behavior and metadata conflicts when stronger attestation is unavailable.

Why common approaches fall short

  • Trusting a model's self-description confuses generated text with evidence.
  • Behavioral consistency alone cannot prove cryptographic model authenticity.
  • Manual browser testing cannot verify provider route metadata when the provider does not expose it.

THOR-SEC approach

Model Identity Verifier runs repeatable identity probes, analyzes response consistency and available provider metadata, scores findings, and emits terminal, JSON, Markdown, or SARIF reports.

It supports API verification, manual prompt assessment, baselines, report comparison, and explicit statuses such as ROUTE_MISMATCH, DOWNGRADE_SUSPECTED, and INCONCLUSIVE.

Evidence

  • The repository contains the implemented Python CLI, mock/provider workflows, prompt-pack assessment, baseline comparison, structured reports, self-tests, local end-to-end scripts, and a documented release gate.
  • The documentation explicitly constrains interpretation: PASS means configured warning thresholds were not triggered; it does not prove the provider served the claimed model.
  • The release checklist requires live provider smoke tests before v0.1.3 may be tagged.

Limitations

  • The tool cannot prove model identity unless a provider exposes independently verifiable metadata.
  • Provider route metadata may be missing or opaque, and heuristic downgrade signals are not guaranteed detection.
  • Manual prompt mode is integrity testing only and cannot verify route metadata.
  • Version 0.1.3 remains held until the required live-provider smoke gates pass; it must not be presented as fully release-validated yet.

Current status

Implemented prototype; live-provider release gate pending

Implemented prototype with explicit interpretation limits. The v0.1.3 release gate remains pending required live-provider smoke validation.

Artifacts