---
title: "Monitoring & Logging"
canonical: "https://kb.uconn.edu/space/IKB/28727574601/Monitoring%20%26%20Logging"
format: markdown
---
Monitoring and logging are essential for keeping your applications healthy on OpenShift. This guide covers how to view metrics, read logs, set up health checks, and configure alerts.

---

## Viewing Logs

### Via CLI

```shell
# View logs for a running pod
oc logs <pod-name>

# Stream logs in real time
oc logs -f <pod-name>

# View logs from a crashed/restarted pod
oc logs <pod-name> --previous

# View logs from a specific container (multi-container pods)
oc logs <pod-name> -c <container-name>

# View last 100 lines
oc logs <pod-name> --tail=100

# View logs from the last hour
oc logs <pod-name> --since=1h

# View logs from all pods of a deployment
oc logs deployment/my-app --all-containers=true
```

### Via Web Console

1. Go to **Workloads → Pods** and click on a pod.
2. Click the **Logs** tab.
3. Use the dropdown to select a container (if multi-container).
4. Toggle **Wrap lines** and **Full screen** for readability.

---

## Application Logging Best Practices

| Practice | Why |
| --- | --- |
| Log to **stdout/stderr** (not files) | OpenShift captures stdout/stderr automatically |
| Use **structured logging** (JSON) | Easier to search and filter |
| Include **timestamps** | Essential for correlating events |
| Log **request IDs** | Trace requests across services |
| Use appropriate **log levels** | DEBUG, INFO, WARN, ERROR |
| Don't log **sensitive data** | No passwords, tokens, or PII in logs |

Example structured log output:

```json
{"timestamp":"2025-01-15T10:30:00Z","level":"INFO","message":"Request processed","requestId":"abc-123","duration":45,"status":200}
```

---

# OpenShift Logging: Find Application Logs in the Console or Splunk

Use this guide to find application logs from your OpenShift namespace.

| Environment | OpenShift console log search | Splunk search |
| --- | --- | --- |
| **OPP — on-premises OpenShift** | Available through **Observe → Logs** | Available with appropriate Splunk access |
| **ARO — Azure Red Hat OpenShift** | Centralized console log search is not currently configured | Available to authorized users |

## Before you begin

Have the following information ready:

- Environment: **OPP** or **ARO**.
- Your namespace, such as `its-ssg`.
- The approximate date and time of the issue.
- Your application, pod, or container name, if known.

Access to Splunk is managed separately from OpenShift. A namespace filter narrows results; it does not enforce access permissions.

For access or assistance, contact [techsupport@uconn.edu](mailto:techsupport@uconn.edu).

## Log retention

| Logging tool | Retention |
| --- | --- |
| OPP OpenShift console — Observe → Logs | **7 days** |
| Splunk — OPP and ARO | **Up to 5 years** |

Use the OpenShift console for recent troubleshooting. For logs older than seven days, search Splunk.

Logs must have been collected and remain within the retention period. Retention does not make logs available from before forwarding was enabled.

---

## Option 1: Search OPP logs in the OpenShift console

### 1. Open the Logs page

1. Go to [https://apps.uconn.edu](https://apps.uconn.edu/).
2. Sign in with your organizational account.
3. Expand **Observe** in the navigation menu.
4. Select **Logs**.

If the navigation menu is collapsed, click the menu button in the upper-left corner.

### 2. Select your namespace

1. Set the log-type dropdown to **application**.
2. Set the filter category to **Namespaces**, if necessary.
3. Open **Filter by Namespace**.
4. Select your namespace.
5. Remove previously selected namespaces that you do not want to search.

Start with one namespace. When searching multiple namespaces, you need application-log access to each one.

Leave **Severity** unfiltered initially because applications do not always supply a recognizable severity.

### 3. Choose a time range

Use the time-range selector in the upper-right corner:

- **Last 15 minutes:** A current issue.
- **Last 1 hour:** An earlier issue.
- **Custom range, if available:** A specific incident window within the seven-day retention period.

Click **Run Query**.

### 4. Read the results

Each result includes a timestamp and log message.

Expand a result using the arrow beside it. Inspect the namespace, pod, container, and message.

If a result displays a large JSON object, locate the `message` field.

### 5. Search for specific text

Click **Show Query** to open the query editor.

Replace `its-ssg` in these examples with your namespace.

**All logs in your namespace:**

```
{kubernetes_namespace_name="its-ssg"}
```

**Find a phrase:**

```
{kubernetes_namespace_name="its-ssg"} |= "connection refused"
```

Replace the phrase and click **Run Query**. The `|=` text filter is case-sensitive.

**Find common error terms regardless of capitalization:**

```
{kubernetes_namespace_name="its-ssg"} |~ "(?i)(error|exception|timeout)"
```

This searches text; it does not identify every possible application failure.

**Parse structured JSON log records:**

```
{kubernetes_namespace_name="its-ssg"} | json
```

**Combine a text search with JSON parsing:**

```
{kubernetes_namespace_name="its-ssg"} |= "connection refused" | json
```

### 6. Refresh or export results

- Click the **refresh icon** to rerun the search.
- Use the **Refresh** dropdown to select an automatic refresh interval.
- Turn automatic refresh off when you finish.
- Select **Export as CSV** if you need a copy of the results.

Review exported logs for sensitive information before sharing them.

### Troubleshoot console searches

| Message or symptom | What to do |
| --- | --- |
| **No datapoints found** | Expand the time range within the last seven days, remove text and severity filters, and confirm the application generated logs. |
| **Forbidden / Missing permissions** | Select only your authorized namespace. Contact support if access is still denied. |
| **Logs menu is missing** | Refresh the console. Contact support if it remains missing. |
| Results disappear after adding a filter | Remove the last filter and check spelling and capitalization. |
| Newly generated logs are missing | Allow a short interval, then refresh. |
| Logs are older than seven days | Search Splunk instead. |

**No datapoints found** means no records matched the query. It does not, by itself, indicate a permissions problem.

---

## Option 2: Search OPP or ARO logs in Splunk

ARO searches are intended for users authorized to access ARO logs.

### 1. Open Splunk

1. Go to [https://search.splunk.uconn.edu](https://search.splunk.uconn.edu/).
2. Sign in with your organizational account.
3. Open **Apps → Search & Reporting**.
4. Select **Search**.

Use the standard **SPL** interface for these examples.

### 2. Choose a time range

Use the time-range selector beside the search box:

- Start with **Last 15 minutes**.
- Expand to **Last 60 minutes** or **Last 24 hours** if necessary.
- Use a custom range for an older incident, subject to retention of up to five years.

Begin with a narrow time range to keep searches focused. For historical troubleshooting, search the specific incident window rather than the entire retention period.

### 3. Choose your environment and namespace

Both environments send logs to the index `redhat_oscp_its`.

Replace `YOUR-NAMESPACE` with your namespace, keeping the quotation marks.

**OPP application logs:**

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="YOUR-NAMESPACE"
host="*.opp.its.uconn.edu"
```

**ARO application logs:**

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="YOUR-NAMESPACE"
host="aro-cluster-01-*"
```

Click **Search**.

Always include the correct environment filter because the same namespace name can exist in both environments.

### 4. Read the results

Open the **Events** tab and expand an event.

| Field | Description |
| --- | --- |
| `message` | Application log message |
| `kubernetes.namespace_name` | Namespace |
| `kubernetes.pod_name` | Pod that produced the message |
| `kubernetes.container_name` | Container within the pod |
| `host` | OpenShift node associated with the event |
| `level` | Reported severity, when available |

A value of `level=default` does not mean the message is error-free.

### 5. Narrow your search

The following examples use the `its-ssg` namespace on **OPP**.

For **ARO**, replace:

```
host="*.opp.its.uconn.edu"
```

with:

```
host="aro-cluster-01-*"
```

Replace `its-ssg` with your namespace.

**Find a phrase in the message:**

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="its-ssg"
host="*.opp.its.uconn.edu"
message="*connection refused*"
```

The `*` characters allow other text before or after the phrase.

**Find common error terms:**

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="its-ssg"
host="*.opp.its.uconn.edu"
(message="*error*" OR message="*exception*" OR message="*timeout*")
```

**Search a specific pod:**

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="its-ssg"
host="*.opp.its.uconn.edu"
kubernetes.pod_name="YOUR-POD-NAME"
```

To match pods with a common prefix, replace the pod condition with:

```
kubernetes.pod_name="my-app-*"
```

**Search a specific container:**

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="its-ssg"
host="*.opp.its.uconn.edu"
kubernetes.container_name="YOUR-CONTAINER-NAME"
```

### 6. Display a readable table

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="its-ssg"
host="*.opp.its.uconn.edu"
| table _time kubernetes.pod_name kubernetes.container_name message
```

Select the **Statistics** tab to view the table.

### 7. Count events by pod and container

```
index=redhat_oscp_its log_type=application
kubernetes.namespace_name="its-ssg"
host="*.opp.its.uconn.edu"
| stats count AS events by kubernetes.pod_name kubernetes.container_name
| sort - events
```

Select the **Statistics** tab. Counts apply only to the selected time range.

### Troubleshoot Splunk searches

If no results appear:

1. Confirm the environment and namespace.
2. Expand the time range, keeping the retention period in mind.
3. Remove optional message, pod, and container filters.
4. Keep the namespace and environment filters.
5. Confirm the application generated logs during the selected period.
6. Allow a short interval for new logs to arrive, then rerun the search.
7. Contact support if results remain missing or access is denied.

---

## Important search considerations

- **Query languages differ:** the OpenShift Logs page uses LogQL; Splunk uses SPL. Queries are not interchangeable.
- **Check the timezone:** timestamps ending in `Z` are UTC. The interface may display local time.
- **Application logging matters:** logs written only to files inside a container might not be collected. Ask the application owner whether logs are written to standard output or standard error.
- **Retention differs:** OPP console logs are retained for seven days; Splunk logs are retained for up to five years.

## Request access or troubleshooting help

Contact [techsupport@uconn.edu](mailto:techsupport@uconn.edu).

Include:

- Tool: OpenShift console or Splunk.
- Environment: OPP or ARO.
- Namespace and application name.
- Pod or container name, if known.
- Incident date, time, and timezone.
- Exact search query and selected time range.
- Screenshot or error message.
- Whether the expected message appears in the pod’s own Logs tab in OpenShift.

Do not include passwords, tokens, or sensitive log content.

## Monitoring Metrics (Web Console)

OpenShift includes built-in monitoring powered by Prometheus.

### Dashboard View

1. Switch to the **Developer** perspective.
2. Click **Observe → Dashboard** in the sidebar.
3. Select your project to see CPU usage, memory usage, network bandwidth, and storage I/O.

### Metrics Explorer

1. Go to **Observe → Metrics**.
2. Enter a PromQL query to explore specific metrics:

```promql
# CPU usage for your deployment (cores)
sum(rate(container_cpu_usage_seconds_total{namespace="my-project", pod=~"my-app-.*"}[5m]))

# Memory usage (bytes)
sum(container_memory_working_set_bytes{namespace="my-project", pod=~"my-app-.*"})

# Network received (bytes/sec)
sum(rate(container_network_receive_bytes_total{namespace="my-project", pod=~"my-app-.*"}[5m]))

# HTTP request rate (if app exposes metrics)
sum(rate(http_requests_total{namespace="my-project"}[5m]))

# Pod restarts
sum(kube_pod_container_status_restarts_total{namespace="my-project"})
```

---

## Monitoring via CLI

```shell
# Real-time resource usage (CPU & memory)
oc adm top pods

# Resource usage for a specific pod
oc adm top pods <pod-name>

# Node-level resources (if permitted)
oc adm top nodes

# Check resource requests vs actual usage
oc describe pod <pod-name> | grep -A5 "Requests\|Limits"

# Events (useful for diagnosing scheduling/crash issues)
oc get events --sort-by='.lastTimestamp'

# Recent events for a specific deployment
oc describe deployment/my-app | tail -20
```

---

## Health Checks (Probes)

Health checks let OpenShift know when your app is ready for traffic and when it needs to be restarted.

### Types of Probes

| Probe | Purpose | What Happens if It Fails |
| --- | --- | --- |
| **Readiness** | Is the app ready to serve requests? | Pod removed from service (no traffic sent) |
| **Liveness** | Is the app still running correctly? | Pod restarted |
| **Startup** | Has the app finished starting up? | Delays liveness/readiness checks |

### Configure via CLI

```shell
# Add a readiness probe (HTTP)
oc set probe deployment/my-app --readiness \
  --get-url=http://:8080/health \
  --initial-delay-seconds=5 \
  --period-seconds=10

# Add a liveness probe (HTTP)
oc set probe deployment/my-app --liveness \
  --get-url=http://:8080/health \
  --initial-delay-seconds=15 \
  --period-seconds=20 \
  --failure-threshold=3

# Add a startup probe (for slow-starting apps)
oc set probe deployment/my-app --startup \
  --get-url=http://:8080/health \
  --failure-threshold=30 \
  --period-seconds=10

# TCP check (for non-HTTP services like databases)
oc set probe deployment/my-db --readiness \
  --open-tcp=5432

# Remove a probe
oc set probe deployment/my-app --readiness --remove
```

### Configure via YAML

```yaml
spec:
  containers:
  - name: my-app
    readinessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20
      failureThreshold: 3
```

### Health Check Endpoint Example

Your app should expose a `/health` endpoint. A minimal example:

```python
# Flask (Python)
@app.route('/health')
def health():
    return {'status': 'ok'}, 200
```

```javascript
// Express (Node.js)
app.get('/health', (req, res) => {
  res.json({ status: 'ok' });
});
```

---

## Setting Up Alerts

OpenShift allows you to create alerts that fire when metrics cross a threshold.

### Via Web Console

1. Go to **Observe → Alerting** in the Developer perspective.
2. Click **Create alert rule**.
3. Define the PromQL expression and threshold.
4. Set the severity and notification target.

### Example Alert Rules (YAML)

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
  namespace: my-project
spec:
  groups:
  - name: my-app
    rules:
    - alert: HighPodRestartRate
      expr: increase(kube_pod_container_status_restarts_total{namespace="my-project", pod=~"my-app-.*"}[30m]) > 3
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} is restarting frequently"

    - alert: HighCPUUsage
      expr: sum(rate(container_cpu_usage_seconds_total{namespace="my-project", pod=~"my-app-.*"}[5m])) > 0.8
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "High CPU usage detected for my-app"

    - alert: HighMemoryUsage
      expr: sum(container_memory_working_set_bytes{namespace="my-project", pod=~"my-app-.*"}) / sum(kube_pod_container_resource_limits{namespace="my-project", resource="memory", pod=~"my-app-.*"}) > 0.9
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Memory usage exceeding 90% for my-app"
```

```shell
# Apply alert rules
oc apply -f my-app-alerts.yaml

# Check active alerts
oc get prometheusrules
```

---

---

## Debugging Checklist

| Step | Command | What to Look For |
| --- | --- | --- |
| Pod status | `oc get pods` | CrashLoopBackOff, Error, Pending |
| Pod events | `oc describe pod <name>` | Image pull errors, scheduling failures |
| App logs | `oc logs <pod>` | Application errors, stack traces |
| Previous logs | `oc logs <pod> --previous` | Why the last instance crashed |
| Resource usage | `oc adm top pods` | OOM or CPU throttling |
| Project events | `oc get events --sort-by='.lastTimestamp'` | Cluster-level issues |

---

## Quick Reference

```shell
# Logs
oc logs <pod>                          # View logs
oc logs -f <pod>                       # Stream logs
oc logs <pod> --previous               # Previous crash logs
oc logs <pod> --tail=100 --since=1h    # Last 100 lines, last hour

# Metrics
oc adm top pods                        # CPU & memory usage
oc get events --sort-by='.lastTimestamp'  # Recent events

# Health checks
oc set probe deployment/my-app --readiness --get-url=http://:8080/health
oc set probe deployment/my-app --liveness --get-url=http://:8080/health
oc set probe deployment/my-app --readiness --remove  # Remove probe

# Alerts
oc get prometheusrules                 # List alert rules
oc apply -f alerts.yaml                # Apply new rules
```

---

*UConn ITS Infrastructure Services — OpenShift Knowledge Base*