InsightWorker Logo
  • contact@verticalserve.com

Schema Evolution & Compliance Audit

Weekly diff of your Confluent Schema Registry — every new version, every breaking-change candidate, every subject without compatibility set, cross-referenced against the consumer groups still on prior versions. Governance teams get the audit trail, platform teams get early warning.

← All use cases
Data Streaming — Confluent Kafka
Data governance, platform engineering, compliance — any org with a managed Schema Registry
Confluent CloudSchema Registrygovernancecompliancescheduling

The problem

  • Schema Registry accumulates changes silently. Producers register new versions; consumers don't always update; breaking changes ship to staging and surface in production a sprint later.
  • Subjects without compatibility levels set are landmines — anyone can register an incompatible version and the registry won't refuse it.
  • Data governance teams want a weekly digest of every schema change with owner, change type, and risk classification. Today that's manually compiled or skipped entirely.
  • When a downstream consumer breaks, the team scrambles to figure out which schema change caused it and when — no clean audit trail tying schema versions to deploy events.

How InsightWorker handles it

1
Schedule a weekly Monday 8 AM audit — cron '0 8 * * 1' — so leadership has the digest before the week's planning starts. scheduler · cron syntax
2
Pull every subject's version history for the past 7 days via Schema Registry REST API. Diff each new version against its predecessor. confluent_schema_registry
3
Run Confluent's compatibility check on every new version — flag BREAKING-class changes (field removal, type narrowing, required-field addition). confluent_schema_registry · compatibility check
4
Identify subjects with NO compatibility level set ('NONE' or unset) — these are governance landmines and the digest highlights them at the top. agent reasoning
5
Cross-reference: for each subject with a new schema, list the consumer groups subscribed to topics using that subject. Check if their deployed schema version predates the new one. confluent_consumer_groups · confluent_topics
6
Generate a governance Excel: one row per change, columns for subject, version range, compatibility-class, registered-by, downstream-consumers-on-old-version, breaking-risk-y/n. create_excel
7
Email the digest to data-governance@acme.com + the platform team with the breaking-change candidates highlighted at the top. send_email
8
Archive each weekly snapshot under .insightworker/schema-history/<YYYY-WW>.json so 'when did subject X change?' becomes a one-grep answer. write_file · memory

Sample prompt

"Run this week's schema audit — flag breaking changes and consumers stuck on old versions, email to data-governance@acme.com."
Deliverables: schema_audit_<YYYY-WW>.xlsx · breaking_changes.md · governance_digest.html · .insightworker/schema-history/ archive
Prefer the browser?
Run this in InsightStudio — no CLI install for the user.

Authors publish the app once with iw app publish; business users open it in the marketplace and click Run. Your worker box does the execution.

Visit InsightStudio →

Try this use case yourself

Free trial available — CLI, Desktop, VS Code, and the new --worker mode for InsightStudio. See download for details.

Download Free Trial