--- title: "Monitoring & Logging" canonical: "https://kb.uconn.edu/space/IKB/28727574601/Monitoring%20%26%20Logging" format: markdown --- Monitoring and logging are essential for keeping your applications healthy on OpenShift. This guide covers how to view metrics, read logs, set up health checks, and configure alerts. --- ## Viewing Logs ### Via CLI ```shell # View logs for a running pod oc logs # Stream logs in real time oc logs -f # View logs from a crashed/restarted pod oc logs --previous # View logs from a specific container (multi-container pods) oc logs -c # View last 100 lines oc logs --tail=100 # View logs from the last hour oc logs --since=1h # View logs from all pods of a deployment oc logs deployment/my-app --all-containers=true ``` ### Via Web Console 1. Go to **Workloads → Pods** and click on a pod. 2. Click the **Logs** tab. 3. Use the dropdown to select a container (if multi-container). 4. Toggle **Wrap lines** and **Full screen** for readability. --- ## Application Logging Best Practices | Practice | Why | | --- | --- | | Log to **stdout/stderr** (not files) | OpenShift captures stdout/stderr automatically | | Use **structured logging** (JSON) | Easier to search and filter | | Include **timestamps** | Essential for correlating events | | Log **request IDs** | Trace requests across services | | Use appropriate **log levels** | DEBUG, INFO, WARN, ERROR | | Don't log **sensitive data** | No passwords, tokens, or PII in logs | Example structured log output: ```json {"timestamp":"2025-01-15T10:30:00Z","level":"INFO","message":"Request processed","requestId":"abc-123","duration":45,"status":200} ``` --- ## Monitoring Metrics (Web Console) OpenShift includes built-in monitoring powered by Prometheus. ### Dashboard View 1. Switch to the **Developer** perspective. 2. Click **Observe → Dashboard** in the sidebar. 3. Select your project to see CPU usage, memory usage, network bandwidth, and storage I/O. ### Metrics Explorer 1. Go to **Observe → Metrics**. 2. Enter a PromQL query to explore specific metrics: ```promql # CPU usage for your deployment (cores) sum(rate(container_cpu_usage_seconds_total{namespace="my-project", pod=~"my-app-.*"}[5m])) # Memory usage (bytes) sum(container_memory_working_set_bytes{namespace="my-project", pod=~"my-app-.*"}) # Network received (bytes/sec) sum(rate(container_network_receive_bytes_total{namespace="my-project", pod=~"my-app-.*"}[5m])) # HTTP request rate (if app exposes metrics) sum(rate(http_requests_total{namespace="my-project"}[5m])) # Pod restarts sum(kube_pod_container_status_restarts_total{namespace="my-project"}) ``` --- ## Monitoring via CLI ```shell # Real-time resource usage (CPU & memory) oc adm top pods # Resource usage for a specific pod oc adm top pods # Node-level resources (if permitted) oc adm top nodes # Check resource requests vs actual usage oc describe pod | grep -A5 "Requests\|Limits" # Events (useful for diagnosing scheduling/crash issues) oc get events --sort-by='.lastTimestamp' # Recent events for a specific deployment oc describe deployment/my-app | tail -20 ``` --- ## Health Checks (Probes) Health checks let OpenShift know when your app is ready for traffic and when it needs to be restarted. ### Types of Probes | Probe | Purpose | What Happens if It Fails | | --- | --- | --- | | **Readiness** | Is the app ready to serve requests? | Pod removed from service (no traffic sent) | | **Liveness** | Is the app still running correctly? | Pod restarted | | **Startup** | Has the app finished starting up? | Delays liveness/readiness checks | ### Configure via CLI ```shell # Add a readiness probe (HTTP) oc set probe deployment/my-app --readiness \ --get-url=http://:8080/health \ --initial-delay-seconds=5 \ --period-seconds=10 # Add a liveness probe (HTTP) oc set probe deployment/my-app --liveness \ --get-url=http://:8080/health \ --initial-delay-seconds=15 \ --period-seconds=20 \ --failure-threshold=3 # Add a startup probe (for slow-starting apps) oc set probe deployment/my-app --startup \ --get-url=http://:8080/health \ --failure-threshold=30 \ --period-seconds=10 # TCP check (for non-HTTP services like databases) oc set probe deployment/my-db --readiness \ --open-tcp=5432 # Remove a probe oc set probe deployment/my-app --readiness --remove ``` ### Configure via YAML ```yaml spec: containers: - name: my-app readinessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 15 periodSeconds: 20 failureThreshold: 3 ``` ### Health Check Endpoint Example Your app should expose a `/health` endpoint. A minimal example: ```python # Flask (Python) @app.route('/health') def health(): return {'status': 'ok'}, 200 ``` ```javascript // Express (Node.js) app.get('/health', (req, res) => { res.json({ status: 'ok' }); }); ``` --- ## Setting Up Alerts OpenShift allows you to create alerts that fire when metrics cross a threshold. ### Via Web Console 1. Go to **Observe → Alerting** in the Developer perspective. 2. Click **Create alert rule**. 3. Define the PromQL expression and threshold. 4. Set the severity and notification target. ### Example Alert Rules (YAML) ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: my-app-alerts namespace: my-project spec: groups: - name: my-app rules: - alert: HighPodRestartRate expr: increase(kube_pod_container_status_restarts_total{namespace="my-project", pod=~"my-app-.*"}[30m]) > 3 for: 5m labels: severity: warning annotations: summary: "Pod {{ $labels.pod }} is restarting frequently" - alert: HighCPUUsage expr: sum(rate(container_cpu_usage_seconds_total{namespace="my-project", pod=~"my-app-.*"}[5m])) > 0.8 for: 10m labels: severity: warning annotations: summary: "High CPU usage detected for my-app" - alert: HighMemoryUsage expr: sum(container_memory_working_set_bytes{namespace="my-project", pod=~"my-app-.*"}) / sum(kube_pod_container_resource_limits{namespace="my-project", resource="memory", pod=~"my-app-.*"}) > 0.9 for: 5m labels: severity: critical annotations: summary: "Memory usage exceeding 90% for my-app" ``` ```shell # Apply alert rules oc apply -f my-app-alerts.yaml # Check active alerts oc get prometheusrules ``` --- ## Centralized Logging (EFK Stack) If enabled on the cluster, OpenShift aggregates logs using the EFK (Elasticsearch, Fluentd, Kibana) stack. ### Accessing Kibana 1. From the web console, go to **Observe → Logging** (if available). 2. Or navigate directly to the Kibana URL provided by UConn ITS. 3. Filter by namespace, pod name, or container. > **Note:** Centralized logging availability depends on the cluster configuration. Contact UConn ITS if you need access. --- ## Debugging Checklist | Step | Command | What to Look For | | --- | --- | --- | | Pod status | `oc get pods` | CrashLoopBackOff, Error, Pending | | Pod events | `oc describe pod ` | Image pull errors, scheduling failures | | App logs | `oc logs ` | Application errors, stack traces | | Previous logs | `oc logs --previous` | Why the last instance crashed | | Resource usage | `oc adm top pods` | OOM or CPU throttling | | Project events | `oc get events --sort-by='.lastTimestamp'` | Cluster-level issues | --- ## Quick Reference ```shell # Logs oc logs # View logs oc logs -f # Stream logs oc logs --previous # Previous crash logs oc logs --tail=100 --since=1h # Last 100 lines, last hour # Metrics oc adm top pods # CPU & memory usage oc get events --sort-by='.lastTimestamp' # Recent events # Health checks oc set probe deployment/my-app --readiness --get-url=http://:8080/health oc set probe deployment/my-app --liveness --get-url=http://:8080/health oc set probe deployment/my-app --readiness --remove # Remove probe # Alerts oc get prometheusrules # List alert rules oc apply -f alerts.yaml # Apply new rules ``` --- *UConn ITS Infrastructure Services — OpenShift Knowledge Base*