/networking-debug

name: openstack-networking-debug
description: "OpenStack networking debug operations skill for SDN troubleshooting, packet tracing, and flow analysis. Covers OVS/OVN debugging (ovs-vsctl, ovs-ofctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-trace), security group analysis via OVS flow rules and conntrack, DHCP troubleshooting through namespace inspection and dnsmasq diagnostics, floating IP diagnosis with NAT rule and ARP verification, network namespace inspection (ip netns), MTU chain analysis for overlay networks, DNS resolution debugging, and east-west traffic diagnosis. Use when diagnosing network connectivity failures, tracing packets through the SDN stack, or analyzing flow tables in a running OpenStack cloud."
user-invocable: true
allowed-tools: Read Grep Glob
metadata:
  extensions:
    gsd-skill-creator:
      version: 1
      createdAt: "2026-02-23"
      triggers:
        intents:
          - "network debug"
          - "packet trace"
          - "flow analysis"
          - "connectivity"
          - "OVS"
          - "OVN"
          - "DHCP"
          - "floating IP"
          - "security group"
          - "namespace"
          - "MTU"
        contexts:
          - "troubleshooting network connectivity"
          - "debugging SDN"
          - "analyzing packet flows"
          - "diagnosing floating IP issues"

OpenStack Networking Debug -- SDN Troubleshooting Operations

Networking debug is the most hands-on troubleshooting domain in cloud operations. Virtual networks add multiple abstraction layers between user intent and physical packets -- an instance's traffic passes through a tap device, a Linux bridge or OVS port, integration bridge flows, tunnel encapsulation, and physical NIC before reaching the wire. When connectivity breaks, the operator must trace through every layer to find where packets stop flowing.

**The debugging mental model:** Start at the instance and trace outward. The packet path for a tenant instance is: instance vNIC -> tap device -> qbr bridge (if OVS with iptables) -> OVS br-int -> tunnel or VLAN tag -> OVS br-ex (for external traffic) -> physical NIC. For OVN, the path simplifies: instance vNIC -> OVS br-int (with OVN flows) -> tunnel or physical port. Every hop is inspectable. Every hop can be the failure point.

This skill is the primary reference for the CRAFT-network agent when diagnosing connectivity issues during Phase E operations.

Deploy

Debug Tooling Setup

Verify all diagnostic tools are available before beginning any debug session.

**OVS diagnostic commands** (available inside `openvswitch_vswitchd` container):

# Verify OVS tools are accessible
docker exec openvswitch_vswitchd ovs-vsctl --version
docker exec openvswitch_vswitchd ovs-ofctl --version
docker exec openvswitch_vswitchd ovs-appctl --version

# Show complete OVS configuration
docker exec openvswitch_vswitchd ovs-vsctl show

**OVN diagnostic commands** (available inside `ovn_northd` and `ovn_controller` containers):

# Verify OVN tools
docker exec ovn_northd ovn-nbctl --version
docker exec ovn_northd ovn-sbctl --version

# OVN trace (powerful logical packet tracing)
docker exec ovn_controller ovn-trace --version

**Network namespace tools** (on the host or inside Neutron containers):

# List all network namespaces
ip netns list
# Expected: qrouter-<id>, qdhcp-<id> (OVS backend)
# OVN uses fewer namespaces (metadata only)

**Packet capture** (tcpdump inside containers or namespaces):

# Capture on a tap interface (instance-facing)
tcpdump -i tap<port-id-prefix> -n -c 50

# Capture inside a network namespace
ip netns exec qrouter-<router-id> tcpdump -i qr-<port-prefix> -n -c 50

# Capture on physical NIC
tcpdump -i eth1 -n port 4789  # VXLAN traffic

Kolla-Ansible Debug Container Options

For persistent debug environments, Kolla-Ansible provides tooling containers:

# Enter the neutron_server container for API-level debugging
docker exec -it neutron_server /bin/bash

# Enter openvswitch_vswitchd for flow-level debugging
docker exec -it openvswitch_vswitchd /bin/bash

# Enter the relevant agent container for namespace access
docker exec -it neutron_l3_agent /bin/bash   # OVS backend
docker exec -it neutron_dhcp_agent /bin/bash # OVS backend

Configure

OVS Logging Levels

Adjust OVS logging to capture more detail during active debugging, then restore to production levels.

# Increase OVS daemon logging (temporary, resets on restart)
docker exec openvswitch_vswitchd ovs-appctl vlog/set vswitchd:dbg
docker exec openvswitch_vswitchd ovs-appctl vlog/set ofproto:dbg

# Restore production logging
docker exec openvswitch_vswitchd ovs-appctl vlog/set vswitchd:warn
docker exec openvswitch_vswitchd ovs-appctl vlog/set ofproto:warn

# Check current log levels
docker exec openvswitch_vswitchd ovs-appctl vlog/list

OVN Tracing Enablement

OVN trace simulates a packet through the logical pipeline without sending real traffic.

# Trace a packet from a logical port through OVN
docker exec ovn_controller ovn-trace <datapath> \
  'inport == "<logical-port>" && eth.src == <mac> && eth.dst == <mac> \
   && ip4.src == <src-ip> && ip4.dst == <dst-ip> && ip.ttl == 64'

Neutron Agent Debug Logging

Enable debug logging on individual agents for detailed event tracing.

# Check current log level
docker exec neutron_server grep -i "debug" /etc/neutron/neutron.conf

# Enable debug via Kolla-Ansible config override
# In /etc/kolla/config/neutron/neutron.conf:
# [DEFAULT]
# debug = True

# After config change, reconfigure the service
# kolla-ansible -i inventory reconfigure --tags neutron

Packet Capture Setup

# Identify the tap device for an instance port
openstack port show <port-id> -c id
# Tap device name: tap<first-11-chars-of-port-id>

# Identify the OVS port number for correlation with flow tables
docker exec openvswitch_vswitchd ovs-vsctl --columns=name,ofport