InsightWorker Logo
  • contact@verticalserve.com

Kafka Platform Daily Health Digest

A weekday 7 AM brief that tells the platform team what to look at — stale topics, lagging consumers, drifted connectors — pulled from Confluent Cloud + Datadog, synthesized by Bedrock, delivered as one email.

← All use cases
Data Streaming — Confluent Kafka
Platform engineering, data infrastructure, SRE — any team running Confluent Cloud at scale
Confluent CloudKafkaDatadogBedrockschedulingplatform ops

The problem

  • Platform team owns 200+ topics, dozens of consumer groups, and a fleet of Connect connectors. Nobody has time to manually walk the Confluent Cloud console every morning.
  • Stale topics (no produce in 30+ days) accumulate silently. Schema-less topics, undocumented owners, abandoned consumer groups — discovered only during quarterly cleanup or audit.
  • Datadog has the metrics but emails a wall of dashboards rather than a prioritized 'what changed yesterday' summary. Engineers tab through 5 dashboards before forming an opinion.
  • Senior platform engineers carry the institutional context (which consumer group's lag is normal vs. concerning, which connector pause is intentional). Junior engineers can't reproduce the morning sweep without that context.

How InsightWorker handles it

1
Schedule a weekday 7 AM digest with cron — '0 7 * * 1-5' fires before standup. scheduler · cron syntax
2
Pull topic inventory and recent produce/consume rates via Confluent Cloud REST API. Identify stale topics (no produce in 30+ days), schema-less topics, and topics without a documented owner tag. confluent_topics · web_fetch · gov-tags
3
Query consumer groups; pair with Datadog consumer-lag metrics to flag groups exceeding the per-group lag SLO. confluent_consumer_groups · datadog_query_metrics
4
Pull Connect connector status — failed tasks, paused connectors, configs drifted from declared (if connectors are managed via IaC, diff against the source of truth). confluent_connect · git_diff (for IaC drift)
5
Synthesize via Bedrock-hosted Claude — three sections, prioritized: stale topics (cleanup candidates), lagging consumers (with owner + Datadog trend link), connector drift (declared vs actual). agent reasoning · Bedrock provider
6
Compose an HTML email with deep links back to Confluent Cloud and Datadog dashboards for any flagged item — engineer one-click reaches the right pane. send_email · HTML body
7
Iterate the email format with --dry-run before going live to the team distribution. schedule run --dry-run

Sample prompt

"Build today's Kafka platform health digest — stale topics, lag SLO violations, connector drift — and email to platform-team@acme.com."
Deliverables: kafka_daily_digest.html · stale_topics.csv · lag_violations.csv · connector_drift.md (when IaC-managed)
Prefer the browser?
Run this in InsightStudio — no CLI install for the user.

Authors publish the app once with iw app publish; business users open it in the marketplace and click Run. Your worker box does the execution.

Visit InsightStudio →

Try this use case yourself

Free trial available — CLI, Desktop, VS Code, and the new --worker mode for InsightStudio. See download for details.

Download Free Trial