/workflow-debugging

name: workflow-debugging
description: Debug AEM Workflow issues on AEM 6.5 LTS and AMS including stuck workflows, failed steps, missing Inbox tasks, launcher failures, stale instances, thread pool exhaustion, queue backlogs, purge failures, and permissions errors. Use when the user reports workflow problems on AEM 6.5 LTS or AMS, asks why a workflow is stuck or failed, needs step-by-step troubleshooting, or provides thread dumps, configuration status dumps, or Sling Job console output for analysis.
license: Apache-2.0

AEM Workflow Debugging — 6.5 LTS / AMS

Production-grade debugging for AEM Granite Workflow engine, launcher, Inbox, Sling Jobs, thread pools, and purge on **AEM 6.5 LTS** and **Adobe Managed Services (AMS)**.

Variant Scope

• This skill is **6.5-lts-only** (includes AMS).
• Full JMX access via Felix Console or JMX client.
• Config changes via Felix Console or OSGi config in repository.

---

When to use this skill

• Workflow stuck, not progressing, failed, not starting, task not in Inbox, purge/repository bloat, permissions, queue backlog, thread pool exhaustion, auto-advancement not working.
• User provides thread dumps, configuration status ZIPs, Sling Job console output, or error.log excerpts.
• Environment: AEM 6.5 LTS / AMS (JMX available).

---

Step 1: Map symptom to first action

| Symptom | symptom_id | First action | |---------|------------|--------------| | Workflow stuck (not advancing) | workflow_stuck_not_progressing | Open instance; note current step type. No work item → stale. | | Task not in Inbox | task_not_in_inbox | Confirm Participant step; assignee = logged-in user; Inbox filters. | | Workflow not starting (launcher) | workflow_not_starting_launcher | Launcher enabled; path/event match payload. | | Workflow fails or shows error | workflow_fails_or_shows_error | Instance history; error.log for instance ID; payload and process. | | Step failed, retries exhausted | step_failed_retries_exhausted | Logs → process.label → JMX `retryFailedWorkItems` or Inbox retry. | | Stale (no current work item) | stale_workflow_no_work_item | JMX `countStaleWorkflows` → `restartStaleWorkflows(dryRun=true)`. | | Repository bloat / too many instances | repository_bloat_too_many_instances | JMX `purgeCompleted(dryRun=true)` or Purge Scheduler. | | User cannot see or complete item | user_cannot_see_or_complete_item | Assignee/initiator/superuser; enforce flags. | | Cannot delete model | cannot_delete_model | JMX `countRunningWorkflows` → terminate → delete. | | Slow throughput / queue backlog | slow_throughput_queue_backlog | JMX `returnSystemJobInfo`; `queue.maxparallel` on Granite Workflow Queue; Sling thread pool. | | Auto-advancement not working | workflow_auto_advance_failure | Check `default` thread pool saturation; Sling Scheduler; timeout jobs. | | New workflow not working | workflow_setup_validation | Model sync, launcher, process registration, permissions. |

---

Step 2: Decision tree (workflow stuck)

1. **No current work item?** → Stale. JMX: `countStaleWorkflows` → `restartStaleWorkflows(dryRun=true)`. 2. **Participant step** → Assignee exists? Inbox visible? Payload accessible? Dynamic participant resolver returning correct user? 3. **Process step** → Search error.log for instance ID. Check: `process.label` registered, payload path exists, bundle active, no exception in `execute()`. 4. **OR/AND Split** → Condition evaluates correctly? Routes exist? No dead-end branches? Model synced?

---

Step 3: Thread dump & thread pool analysis

Thread dumps on 6.5 / AMS are obtained via **jstack** or by requesting from AMS support. Configuration status ZIPs from **Felix Console → Status → Configuration Status**.

3a. Sling `default` thread pool (critical path)

The Sling Scheduler `ApacheSlingdefault` uses `ThreadPool: default`. This pool runs:

• Oak observation events
• All Quartz-scheduled jobs — including the workflow timeout-detection scheduler that emits `com/adobe/granite/workflow/timeout/job` events to the Sling Job system (the job itself then runs on the Granite Workflow Queue, see Step 3c)

**Check the Sling Thread Pools status page (`/system/console/status-slingthreadpools`):**

| Field | Healthy | Problem | |-------|---------|---------| | active count | < max pool size | **= max pool size** (saturated) | | block policy | RUN | **ABORT** (rejects tasks when full) | | max pool size | sized for workload | OOTB on AEM 6.5 LTS is **5/5** (Apache Sling default). Bump to 20+ in OSGi config for environments with many custom periodic schedulers, otherwise schedulers can starve. |

**If active count = max pool size AND block policy = ABORT:**

• New scheduled tasks (including workflow timeout/auto-advance jobs) are **silently rejected**
• This is the #1 cause of auto-advancement failure

**Check the Threads status page (`/system/console/status-Threads`) or the jstack thread dump (`/system/console/status-jstack-threaddump`):**

• Search for `sling-default-` threads
• If all threads show same stack (e.g. stuck on HTTP call, database, or external service), that's the blocking culprit
• Note `elapsed` time — threads stuck for hours indicate a hung external call without timeout

3b. Sling Job thread pool

**Check `Apache Sling Job Thread Pool`** in the Sling Thread Pools status page:

• active count vs max pool size
• If saturated, Sling Jobs cannot execute (workflow jobs stall)

3c. Granite Workflow Queue

**Check the Sling Jobs page (`/system/console/slingevent`):**

| Field | Healthy | Problem | |-------|---------|---------| | Queued Jobs (overall) | 0 | > 0 (jobs waiting) | | Failed Jobs | 0 | > 0 (step failures) | | Active Jobs | 0-N | 0 when Queued > 0 (jobs not picked up) |

**Check topic statistics for workflow model:**

• Topic: `com/adobe/granite/workflow/job/var/workflow/models/<modelName>`
• High `Failed Jobs` / low `Finished Jobs` ratio → process step throwing exceptions

**Check Granite Workflow Queue configuration:**

• Type: T