VynOps Documentation

Overview

VynOps is an open-source, self-hostable SRE platform for teams running Kubernetes workloads. It provides a unified interface for incident management, multi-cluster Kubernetes operations, full-stack observability, automated remediation, and AI-assisted diagnosis.

The platform is built on Next.js 15, connects directly to the Kubernetes API, Prometheus, Loki, Jaeger, and AlertManager, and layers an AI reasoning engine on top to generate insights, remediation plans, and automated fixes.

Architecture

Browser (Next.js 15 / React 19)
  └─ API Routes (/api/*)
       ├─ Kubernetes API       — pods, nodes, workloads, events, RBAC
       ├─ Prometheus           — metrics, alerts, PromQL
       ├─ Loki                 — log aggregation
       ├─ Jaeger               — distributed tracing
       ├─ AlertManager         — active alerts, silences
       └─ Groq (LLM)           — AI Copilot, plan generation

Persistence : JSONL flat files — no external DB required
Auth        : NextAuth v5 (email/password, JWT sessions)
Real-time   : Server-Sent Events (/api/stream)

Requirements

• •Kubernetes cluster — any distribution (k3s, k8s, EKS, GKE, AKS, on-prem)
• •Prometheus — for metrics and alerting
• •Groq API key — for AI Copilot and autonomous planning (free tier available)
• •Loki, Jaeger, AlertManager — optional; enables logs / traces / alert features
• •Node.js 20+ for self-hosting

Dashboard

The main dashboard (/dashboard) provides a real-time platform health overview computed from multiple data sources.

Platform Health Score

A composite score (0–100) computed from SLA compliance rate, deployment velocity (DORA band), and active incident severity. Displayed with a trending indicator (improving / stable / degrading).

KPI Cards

Dashboard Sections

• •Cluster node & pod status — Nodes Ready/Not-Ready, Pods Running/Pending/Failed
• •Cost summary — total spend/month, projected cost, detected waste
• •DORA metrics — deployment frequency, success rate, change failure rate (7-day window)
• •AI insights panel — active predictions, cost hints, reliability recommendations
• •Recent incidents — last 5, with severity, state, SLA status, elapsed time
• •Resource utilization sparklines — CPU%, memory%, request rate, error rate

Incident Management

Tracks every failure event, correlates alerts, manages SLA deadlines, and drives resolution through a structured state machine.

State Machine

open → investigating → identified → monitoring → resolved → (re-opened)

List View

• •Filter by state, severity, service, environment, team, owner
• •Full-text search across title and labels; CSV export
• •Per-incident card: title, severity, state, owner, team, service, environment, SLA deadline & countdown, blast radius preview, source (auto or manual)
• •Metrics bar: open count, critical count, SLA breached, SLA breaching, MTTR, compliance %

Incident Detail

Live Timeline

• •Alert firings and resolutions
• •AI insight discoveries and RCA findings
• •Deployment activities correlated to the incident window
• •User actions — state changes, reassignments, notes
• •Escalation events — who was notified and when
• •Kubernetes events (pod restarts, node issues)

SLA Tracking

• •SLA deadline with live countdown; Breach status: OK / At Risk / Breached
• •SLA compliance % per service SLA target
• •Auto-escalation trigger as deadline approaches

Blast Radius Analysis

• •Affected services list with dependency depth
• •Estimated impacted user count
• •Affected geographic regions; downstream dependent services

AI Root Cause Analysis

• •AI-generated root cause with confidence score (0–100)
• •Evidence links: correlated metrics, logs, events
• •Affected components; contributing factors ranked by probability
• •Recommended next actions

Alert Correlation

All alerts auto-correlated to the incident. Per alert: source, severity, state, first/last fired time.

Linked Resources

• •Associated deployments and change descriptions
• •Runbook URL(s) — manually linked or auto-triggered
• •Slack channel link (auto-created on incident open)
• •Post-mortem link, Jira / GitHub ticket link (if integration configured)

On-Call & Escalation

On-Call Schedules

• •Multiple schedules (by team, service, region)
• •Rotation period in days with ordered member list
• •Shows current on-call person and next rotation date/time

Escalation Levels

Auto-Escalation Triggers

• •SLA deadline within configured threshold (minutes)
• •Active alert count exceeds configured threshold
• •Manual override: engineer can immediately escalate from incident detail
• •Slack notification sent to assigned person at each escalation level
• •Exhaustion logged when all levels have been notified

Deployments & Change Management

Tracks every deployment event and correlates it with reliability signals.

Deployment Event Fields

Post-Deployment Monitoring

• •Automatic CPU, error rate, and latency spike detection after deploy
• •Configurable automatic rollback trigger rules
• •Pre-deployment checks: config validation, image scan, RBAC requirements

Clusters

Register unlimited Kubernetes clusters across any cloud provider or on-prem distribution.

Cluster Configuration

Connectivity Probe

Each cluster is probed for: API version, node count, namespace count, uptime. Result shown with latency and last-checked timestamp. Test on-demand from Settings → Connections.

Workloads & Pods

Pod List

• •Status — Running / Pending / Failed / Succeeded / Unknown
• •Ready state (e.g. 2/3 containers ready)
• •Restart count — total and last 10 minutes; OOMKilled indicator
• •Pod age, assigned node, availability zone

Pod Detail Drawer

• •Containers — state, ready, restart count, image, CPU/memory requests & limits, ports, env vars, volume mounts, probe config (liveness/readiness/startup), init containers
• •Live Prometheus metrics — CPU and memory sparklines for current cycle
• •Pod conditions — Ready, Initialized, ContainersReady, PodScheduled with last transition time
• •QoS Class — Guaranteed / Burstable / BestEffort
• •Node selectors, tolerations, topology spread constraints

Container Operations

Workload Types

• •Deployments — replica state, image, strategy, rollout status, Helm release info, actual CPU/memory from Prometheus
• •StatefulSets — replica management, ordered rolling updates, volume claim templates
• •DaemonSets — desired/current/ready/available counts, update strategy
• •Jobs — status (Complete/Failed/Running), succeeded/failed/active pod counts, duration
• •CronJobs — cron schedule expression, last run, next run, active job count, suspension toggle

Nodes & Infrastructure

Node List

• •Status — Ready / NotReady / SchedulingDisabled / Cordoned
• •CPU capacity vs used (cores); memory capacity vs used (GiB)
• •Pod capacity and current pod count
• •Network bandwidth (in/out Mbps), packet loss %
• •OS image, kernel version, container runtime, uptime

Node Detail

• •Storage disks — device, mount path, capacity, used, IOPS, latency
• •Kubernetes conditions — Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable with last transition time and message
• •Pod list on this node with pod status
• •Taints, labels, zone, instance type, Kubernetes version

Auto-Detected Databases

• •Engines — PostgreSQL, MySQL, Redis, Kafka, Elasticsearch, MongoDB, Couchbase
• •Connection pool metrics — active, idle, waiting connections
• •Performance — QPS (read/write), avg query time, p99 latency, slow queries/min
• •Storage and memory utilization
• •Replication status — Primary / Replica / Standalone, lag
• •Backup status and last backup timestamp
• •Auto-linked to open incidents when health degrades

Network & Storage

Ingresses

• •Host-based routing rules with TLS configuration
• •Load balancer IP / hostname; backend service and port per rule path

Services

• •Type — ClusterIP / NodePort / LoadBalancer / ExternalName
• •Cluster IP, external IP, port mappings (port, targetPort, protocol, nodePort)
• •Selector labels, ready endpoint count, session affinity, external traffic policy

Persistent Volumes (PV)

• •Capacity, used space, storage class, access modes (RWO / ROX / RWX)
• •Reclaim policy — Retain / Delete / Recycle
• •Status — Available / Bound / Released / Failed

Persistent Volume Claims (PVC)

• •Bound volume, capacity, storage class, access modes, creation timestamp

Storage Classes

• •Provisioner, reclaim policy, volume binding mode
• •Volume expansion support flag

Policy & RBAC

Network Policies

• •Name, namespace, pod selector, ingress and egress rule counts
• •Pod/namespace/IP block selectors per rule

RBAC

• •Roles and ClusterRoles — verbs, API groups, resources, resource names
• •RoleBindings and ClusterRoleBindings — subjects (users/groups/service accounts) with namespace scope

Resource Quotas

• •CPU requests/limits, memory requests/limits, pod count limits, PVC/service limits

Limit Ranges

• •Default and maximum CPU/memory per container and per pod

Pod Security Standards (PSA)

• •Namespace-level security levels — Privileged / Baseline / Restricted

AI Copilot

A conversational AI assistant with direct programmatic access to live cluster state. Powered by Llama 4 Scout (17B, 16 experts) via Groq. Temperature: 0.2 (deterministic, tool-focused). Rate limit: 20 requests / 60 seconds per user. Model is runtime-switchable from Settings — no server restart required.

Operating Modes

Tool Reference (20+)

Conversation Persistence

• •All conversations stored per user, tagged by mode
• •History page (/ai-copilot/history) — filter by mode, delete, resume any past session
• •Messages include role (user/assistant), markdown-rendered responses, code blocks with syntax highlighting

Automation Runbooks

Structured multi-step remediation pipelines. Each step has a live status badge: ok / warn / error / info / pending / running. Steps are grouped into phases: Check → Remediate → Report.

Built-in Runbooks

Auto-Trigger Pattern Matching

Incident title / labels             matched runbook                  auto-execute?
─────────────────────────────────────────────────────────────────────────────────
"CrashLoopBackOff on payment-svc"  → diagnose-crash-loop             auto ✓
"OOMKilled: redis-cache"           → oom-patch-restart               approval required
"ImagePullBackOff: api-gateway"    → debug-imagepull                 auto ✓
"Deployment failed: checkout-v2"   → rollback-failed-deployment      auto ✓

Execution Controls

• •1-hour cooldown per runbook per incident — prevents remediation loops
• •Per-runbook auto-run permission toggle — operators control which runbooks fire autonomously
• •Dry-run mode — preview every step; no actual changes applied
• •Custom runbook builder — define org-specific runbooks inside the app
• •Role-gated — Operator role required; Viewer cannot execute
• •Execution history — 500 entries max, with timestamps, step outputs, duration, status; CSV export

Autonomous Healing Engine

A background agent running on a 5-minute cycle. It fetches AI insights, generates LLM-guided remediation plans, executes them with confidence gating, verifies outcomes, and updates per-pattern confidence thresholds based on real results.

Execution Cycle

Every 5 minutes:
  1. Fetch insights      — pull live AI predictions and RCA signals
  2. Filter              — critical/high severity, prediction and RCA types only
  3. Classify            — extract intent: restart_deployment | scale_deployment | delete_pod
  4. Confidence gate     — skip if confidence < adaptive threshold (default 0.85)
  5. Cooldown check      — skip if same action on same target within last 60 min
  6. Execute             — apply via Kubernetes API (or log if dry-run)
  7. Log                 — record action ID, timestamp, target, namespace, confidence, pattern key
  8. Verify (2h window)  — re-check health; classify: resolved | persisted | unknown
  9. Learn               — update per-pattern success rates; recalculate threshold multiplier

Allowed Actions

LLM Remediation Plans

For every open critical/high incident without an active plan, the engine calls Groq to generate a structured plan of up to 5 steps. The LLM is bounded to 11 pre-approved safe actions.

Safe Action Vocabulary

Read:      check_pod_status, check_pod_logs, check_rollout_history,
           check_events, check_resource_usage
Remediate: restart_deployment, rollback_deployment, scale_deployment,
           patch_memory_limit, delete_crashed_pod
Verify:    verify_health

Plan Lifecycle

pending → approved → executing → done
                ↘ dismissed
                ↘ failed

Learning Engine

Configuration Parameters

Metrics, Logs & Traces

Metrics (Prometheus)

• •Time windows — 15m / 1h / 6h / 24h (configurable)
• •Cluster-wide — CPU%, memory%, request rate (RPS), error rate%, p99 and p50 latency (ms)
• •Per-service breakdown — request rate, error rate, p95 latency, dependency health
• •Historical trend charts per metric; custom PromQL queries via AI Copilot

Logs (Loki)

• •Real-time log streaming with time window selection (15m → 24h)
• •Full-text search across log lines
• •Log level filtering — debug / info / warn / error
• •Previous container log support (crashed containers)

Traces (Jaeger)

• •Distributed trace waterfall visualization
• •Per-trace — service call chain, duration, status (ok / slow / error)
• •Depth visualization with color-coded spans; drill into individual spans
• •Service filtering

Kubernetes Events

• •Warning and Normal event types
• •Reason, message, involved object (kind/name/namespace)
• •Event count and last occurrence time; auto-linked to open incident when correlated

Analytics & DORA Metrics

Infrastructure Metrics

• •CPU and memory — current%, average%, peak%, 30-min history chart, forecast ETA to exhaustion (days)
• •Pod count — current with 24-hour history
• •Restart events — total in last 24 hours

DORA Metrics (7-day window)

SLA & Error Budget

• •Per-service SLA tracking — target, current availability, 1h/6h/30d breakdowns
• •Burn rate 1h and 6h — error budget consumption rate
• •Fast burn flag — alert when burn rate is unsustainably high
• •Budget used % and budget remaining in minutes
• •Visual status badges — OK / At Risk / Breaching; CSV export

Service Topology

• •Nodes — services, databases, queues, API gateways, external services (differentiated by icon)
• •Edges — request flows with real-time RPS, error rate%, and protocol label
• •Node colors — green (healthy) / amber (degraded) / red (critical)
• •Node detail drawer — metadata, live metrics, upstream/downstream service list
• •Diagnose button — launches AI Copilot Investigate mode pre-seeded with that service

Security & Compliance

Security Score

Letter grade (A–F) with numeric score (0–100), computed from weighted findings across all security dimensions.

KPI Indicators

• •Critical and high findings count
• •Privileged containers; containers using latest tag; containers without resource limits
• •Namespaces without Network Policies; namespaces without Pod Security Standards
• •Wildcard RBAC roles count
• •Falco runtime security — running status and pod count
• •Active threat event count

CIS Kubernetes Benchmarks

• •100+ CIS benchmark checks
• •Per check — name, category, pass/fail, severity, failure detail, affected resource count
• •Summary — Pass count / Fail count / Not applicable

RBAC Audit

• •Cluster admin bindings — name, role, subjects (kind/name/namespace)
• •Wildcard roles — flags dangerous * verbs or resources
• •Non-system ClusterRoleBindings — over-permissive role warnings

Workload Security Checks

• •Privileged containers; containers using latest tag
• •No resource limits (unbounded CPU/memory)
• •No read-only root filesystem; allow privilege escalation enabled
• •Host network access; host path mounts (sensitive directory access)

Threats (Falco)

• •Active threat events from Falco and Kubernetes audit events
• •Per event — reason, message, namespace, object kind/name, first/last seen, count, severity
• •Auto-linked to open incident when threat is active

Remediation

One-click fix modal — shows exact change before confirmation, applies patch, returns real-time result.

Settings

Profile

• •Display name, email, role (admin / operator / viewer)
• •Timezone preference — persisted locally, affects all time displays

User Management (Admin only)

• •View, add, edit, delete users; role and team assignment
• •All changes written to audit log

Data Sources

AI Provider

• •Provider selection — Groq (default) / OpenAI / Ollama
• •API key (masked), model, temperature, top-p, max tokens
• •Test connection button; usage tracking — tokens used, estimated cost this month
• •Autonomous AI toggles — enable plan generation and auto-execution

Notifications

• •Slack channel mapping per severity (critical / high / medium / low)
• •Do Not Disturb hours; digest mode for low-priority alerts; test button per channel

Audit Log

• •Complete user action history — who, what, resource, timestamp, success/failure
• •Filter by user, action type, date range; full-text search
• •Export all records — pagination disabled with ?export=1
• •Actions logged — user.create, user.update, user.delete, config.update, cluster.add, runbook.execute, incident.create, incident.resolve

Auth & RBAC

Authentication

• •Email/password login via NextAuth v5; JWT sessions with configurable expiry
• •Password change with current password verification

Role Permissions

Integrations

Real-Time Stream

• •/api/stream — SSE stream pushing CPU%, memory%, pod counts, alert counts, cluster health state
• •60-second heartbeat; auto-reconnects on disconnect
• •Active incidents auto-refresh every 30 s; pod/node metrics polled every 15 s

Data Persistence