Skip to content

Incident Ops

Incident management and KTLO operations platform: incidents against severity-based SLAs, alerts turned into deduplicated incidents, escalation through an on-call rotation without polling, real-time console, and KTLO analytics with AI-drafted root cause analyses.

This site documents how it is built and how it is operated. The interface contract every module implements is contracts.md.

Live

What URL
Operations console https://incidents.marceloroman.com.br
Incidents API (Swagger) https://incidents-api.marceloroman.com.br/swagger
Insights API (OpenAPI) https://incidents-insights.marceloroman.com.br/docs
Source and local stack https://github.com/marcelo-roman/incident-ops

System at a glance

flowchart LR
    alerts["Prometheus / Alertmanager<br/>Azure Monitor"] -- webhooks --> api
    web["Console<br/>React + TS"] -- REST --> api["Incidents API<br/>.NET 8"]
    web <-- WebSocket --> sig["SignalR"]
    api --> sig
    api --> sql[("Azure SQL")]
    api -- CloudEvents --> bus{{"Service Bus"}}
    bus --> fn["Functions<br/>SLA timers, paging"]
    fn -- escalate --> api
    fn --> logic["Logic App<br/>Teams / Slack / email"]
    ins["Insights<br/>FastAPI + Pandas"] -- export --> api
    ins --> aoai["Azure OpenAI"]
    web -- REST --> ins

Sections

Section Start with
Architecture overview and C4 diagrams, SLA timers and escalation, deployment and delivery
Operations support model, alerting, incident response, runbooks
Engineering quality gates, definition of done
Leadership stakeholder updates, KTLO vs roadmap
Decisions architecture decision records

Modules

All modules live in one repository, marcelo-roman/incident-ops. Each bounded context is one module (context map).

Module Bounded context Role
services/api Incident Management system of record, SLA clock, alert ingestion, real-time hub
services/functions Escalation SLA timers, escalation, notifications
services/insights Operational Analytics KTLO analytics and RCA drafts
apps/web Operations Console operations console
infra Bicep, Azure DevOps samples, delivery scripts
docs this site
contracts the contract every module implements
local local stack configuration: Prometheus, Alertmanager, Service Bus emulator
.github/workflows one path-filtered workflow per module and a reusable container app deploy