/openhands-enterprise-troubleshooting

name: openhands-enterprise-troubleshooting
description: This skill should be used when a user reports an issue with OpenHands Enterprise (OHE) on a self-hosted (Replicated VM-based) installation. Use for diagnosing sandbox startup failures, auth issues, certificate errors, LLM connectivity problems, Keycloak login issues, Replicated Admin Console access, upgrade failures, or resource exhaustion. Helps triage symptoms, run diagnostic commands, guide through recovery steps, generate and analyze Replicated support bundles offline, and produce escalation handoffs.
triggers:
- openhands enterprise
- OHE troubleshooting
- openhands not working
- sandbox failed
- replicated admin console
- keycloak login
- certificate error
- LLM connectivity
- upgrade failed
- support bundle
- openhands install

OpenHands Enterprise Troubleshooting

This skill helps diagnose and resolve common issues on OpenHands Enterprise (OHE) self-hosted installations using Replicated. It covers triage, guided recovery, support bundle generation, and escalation handoffs.

Diagnostic Workflow

When a user reports an OHE issue:

1. **Collect symptoms** - Ask user to describe what they see, error messages, when it started 2. **Identify failure mode** - Match symptoms to one of the common issues below 3. **Run targeted diagnostics** - Use commands in `references/diagnostics.md` 4. **Guide recovery** - Follow resolution steps for the identified issue, one at a time 5. **Verify fix** - Confirm the original symptom is gone, not just that the last command succeeded 6. **Generate handoff** - If unresolved, produce a clear summary for the OpenHands team

Take recovery one step at a time. Before each step, state what it will change and what you expect to see afterwards; after it, run the check that confirms it before moving on. If the check fails or shows something unexpected, stop and re-diagnose — a step applied on top of a failed one buries the evidence, and several applied blind can leave the install worse than the fault you started with. Destructive steps (restarts, rollbacks, config changes) need the user's agreement first, and are worth recording as you go so the handoff can say exactly what was changed.

Work from whatever the user has. A described symptom starts at step 1; a support bundle goes to [Analyzing the Support Bundle](#analyzing-the-support-bundle). If they paste raw log output, the triage method in [`references/support-bundle-analysis.md`](references/support-bundle-analysis.md#triaging-a-log-file) applies to pasted text as well as to files, and notes what a hand-picked excerpt can hide.

Common Failure Modes

1. Sandbox Fails to Start / Timeout

**Symptoms:**

• Conversation hangs then times out
• "Sandbox failed to start" error
• A timeout in the logs. Read the actual value rather than assuming one: the timeouts in

`runtime-api` are configurable and differ between the Kubernetes client and the app, so quoting a fixed number back to a customer is how you end up chasing the wrong one.

**Diagnosis:** Check sandbox service status, the `sysbox-runc` RuntimeClass and its containerd runtime, resource availability

**Reference:** See `references/diagnostics.md` - Section "Sandbox Startup"

2. Git Provider Auth Broken

**Symptoms:**

• "Authentication failed" for the configured provider
• Can't clone or push repos
• The provider shows as disconnected

**Diagnosis:** Check the provider's secret in Kubernetes and that the provider is enabled. GitHub, GitLab, Bitbucket Data Center, and Azure DevOps are each configured separately — confirm which one the user is actually on before diagnosing

**Reference:** See `references/diagnostics.md` - Section "Git Provider Auth"

3. Certificate Errors

**Symptoms:**

• "certificate expired" or "self-signed certificate" errors
• TLS handshake failures
• Browser shows insecure connection warning

**Diagnosis:** Check cert expiry, certificate chain, ingress configuration

**Reference:** See `references/diagnostics.md` - Section "Certificate Issues"

4. LLM Connectivity Failures

**Symptoms:**

• "LLM endpoint unreachable"
• "Authentication failed" for LLM API
• Conversations fail to start

**Diagnosis:** Check LLM endpoint URL, API key secrets, network policies

**Reference:** See `references/diagnostics.md` - Section "LLM Connectivity"

5. Keycloak Login Issues

**Symptoms:**

• Can't access admin console
• Login loop or "invalid credentials"
• Keycloak pod showing errors

**Diagnosis:** Check Keycloak pod status, database connectivity, realm configuration

**Reference:** See `references/diagnostics.md` - Section "Keycloak"

6. Replicated Admin Console Unreachable

**Symptoms:**

• Can't access admin console URL
• Connection refused or timeout
• Browser shows "site cannot be reached"

**Diagnosis:** Check Replicated operator pod, ingress, service endpoints

**Reference:** See `references/diagnostics.md` - Section "Replicated Admin Console"

7. Upgrade Stuck or Failed

**Symptoms:**

• Replicated shows upgrade as "failed"
• Pods in crash loop after upgrade
• Migration jobs failing

**Diagnosis:** Check failed job logs, resource availability, pre-flight failures

**Reference:** See `references/diagnostics.md` - Section "Upgrade Issues"

8. OOM / Resource Exhaustion

**Symptoms:**

• Pods being OOMKilled
• "Too many open files" errors
• Services becoming unresponsive

**Diagnosis:** Check node resources (memory, disk, file descriptors)

**Reference:** See `references/diagnostics.md` - Section "Resource Exhaustion"

Diagnostic Commands Quick Reference

Access the VM and run these common commands.

**An empty result never means "healthy".** `kubectl get pods -l <selector>` prints `No resources found` and exits 0 both when a component is down and when the selector is wrong, so the two are indistinguishable. When you need to know whether something is running, ask its Deployment or StatefulSet for a READY count instead — that object exists either way