/concurrency-debugging

name: concurrency-debugging
description: Concurrency debugging skill for diagnosing data races and deadlocks. Use when reading TSan race reports, debugging deadlocks with GDB thread inspection, analyzing lock-order graphs with Helgrind, identifying std::atomic misuse patterns, or reasoning about happens-before in C++ and Rust. Activates on queries about data races, TSan reports, deadlocks, Helgrind, lock ordering, thread sanitizer output, or atomic ordering issues.

Concurrency Debugging

Purpose

Guide agents through diagnosing and fixing concurrency bugs: reading ThreadSanitizer race reports, using Helgrind for lock-order analysis, detecting deadlocks with GDB thread inspection, identifying common `std::atomic` misuse patterns, and applying happens-before reasoning in C++ and Rust.

Triggers

• "ThreadSanitizer reported a data race — how do I read the report?"
• "My program deadlocks — how do I debug it?"
• "How do I use Helgrind to find threading bugs?"
• "Am I using std::atomic correctly?"
• "How does happens-before work in C++ memory ordering?"
• "How do I find which threads are deadlocked in GDB?"

Workflow

1. ThreadSanitizer (TSan) — race detection

# Build with TSan
clang -fsanitize=thread -g -O1 -o prog main.c
# or GCC
gcc -fsanitize=thread -g -O1 -o prog main.c

# Run (TSan intercepts memory accesses at runtime)
./prog

# TSan-specific options
TSAN_OPTIONS="halt_on_error=1:second_deadlock_stack=1" ./prog

Reading a TSan report:

WARNING: ThreadSanitizer: data race (pid=12345)
  Write of size 4 at 0x7f1234 by thread T2:
    #0 increment /src/counter.c:8:5              ← access site in T2
    #1 worker_thread /src/counter.c:22:3

  Previous read of size 4 at 0x7f1234 by thread T1:
    #0 read_counter /src/counter.c:3:14          ← conflicting access in T1
    #1 main /src/counter.c:30:5

  Thread T2 created at:
    #0 pthread_create .../tsan_interceptors.cpp
    #1 main /src/counter.c:28:3

SUMMARY: ThreadSanitizer: data race /src/counter.c:8:5 in increment

How to read: 1. Line 1: type of access (write/read) and address 2. Stack under "Write of size": the thread that performed the write 3. Stack under "Previous read/write": the conflicting thread 4. "Thread T2 created at": where the thread was spawned 5. Fix: the `increment` and `read_counter` functions access the same address without synchronization

Common races and fixes:

| Race pattern | Fix | |-------------|-----| | Read/write on global without lock | Add mutex or use `std::atomic` | | Double-checked locking without `atomic` | Use `std::once_flag` + `std::call_once` | | `+=` on shared integer | Use `std::atomic<int>::fetch_add()` | | Container modified while iterated | Lock entire critical section | | `shared_ptr` ref count race | Already safe (ref count is atomic); but pointed-to object may not be |

2. Helgrind — lock-order and race detection

Helgrind uses Valgrind infrastructure to detect lock ordering violations (potential deadlocks) and data races:

# Run with Helgrind
valgrind --tool=helgrind --log-file=helgrind.log ./prog

# Lock order violation report
==1234== Thread #3: lock order "0x... M2" after "0x... M1"
==1234== observed (incorrect) order
==1234==    at pthread_mutex_lock (helgrind/...)
==1234==    by worker2 /src/worker.c:45           ← T3 takes M2 then M1
==1234==
==1234== required order established by acquisition of lock at address 0x... M1
==1234==    at pthread_mutex_lock
==1234==    by worker1 /src/worker.c:31            ← T1 takes M1 then M2

Lock-order violation = potential deadlock:

• Thread T1 acquires M1, then tries M2
• Thread T2 acquires M2, then tries M1
• Both can deadlock if they race

Fix: enforce a consistent global lock ordering. Always take M1 before M2 everywhere.

3. Deadlock detection with GDB

# Attach GDB to a deadlocked process
gdb -p $(pgrep prog)

# Or run under GDB then trigger deadlock

(gdb) info threads          # list all threads and current state
# * 1  Thread 0x... (LWP 1234) "prog" ... in __lll_lock_wait ()
#   2  Thread 0x... (LWP 1235) "prog" ... in __lll_lock_wait ()
# Threads blocked in __lll_lock_wait = waiting for mutex

(gdb) thread 1
(gdb) bt                    # show which mutex thread 1 is waiting for

(gdb) thread 2
(gdb) bt                    # show which mutex thread 2 holds/waits

# Find the mutex owner
(gdb) p ((pthread_mutex_t*)0x601090)->__data.__owner   # Linux glibc mutex
# prints TID of owning thread

# Python script to dump all mutex owners (GDB 7+)
python
import gdb
for t in gdb.selected_inferior().threads():
    t.switch()
    print(f"Thread {t.num}: {gdb.execute('bt 3', to_string=True)}")
end

4. std::atomic misuse patterns

// WRONG: atomic variable, but non-atomic compound operation
std::atomic<int> counter{0};
if (counter == 0) counter = 1;   // not atomic together! TOCTOU race

// CORRECT: use compare_exchange
int expected = 0;
counter.compare_exchange_strong(expected, 1);

// WRONG: relaxed ordering for sync flag
std::atomic<bool> ready{false};
// Producer:
data = 42;
ready.store(true, std::memory_order_relaxed);  // WRONG: no happens-before

// CORRECT: release-acquire for publishing data
// Producer:
data = 42;
ready.store(true, std::memory_order_release);   // syncs with acquire

// Consumer:
if (ready.load(std::memory_order_acquire)) {    // syncs with release
    use(data);  // safe to read data here
}

// WRONG: using data across threads without atomic/mutex
// int shared_data;  // non-atomic — UB on concurrent access

// CORRECT: protect with mutex or make atomic
std::mutex mtx;
std::unique_lock lock(mtx);
shared_data = 42;

5. Happens-before reasoning

In C++, happens-before is established by:

Sequenced-before (within a thread):
  Statement A comes before B in code → A happens-before B

Synchronizes-with (across threads):
  store(release) → load(acquire) on SAME atomic variable
    → store happens-before load
    → everything before s