openstack-networking-debug

OpenStack Networking Debug -- SDN Troubleshooting Operations

Networking debug is the most hands-on troubleshooting domain in cloud operations. Virtual networks add multiple abstraction layers between user intent and physical packets -- an instance's traffic passes through a tap device, a Linux bridge or OVS port, integration bridge flows, tunnel encapsulation, and physical NIC before reaching the wire. When connectivity breaks, the operator must trace through every layer to find where packets stop flowing.

The debugging mental model: Start at the instance and trace outward. The packet path for a tenant instance is: instance vNIC -> tap device -> qbr bridge (if OVS with iptables) -> OVS br-int -> tunnel or VLAN tag -> OVS br-ex (for external traffic) -> physical NIC. For OVN, the path simplifies: instance vNIC -> OVS br-int (with OVN flows) -> tunnel or physical port. Every hop is inspectable. Every hop can be the failure point.

This skill is the primary reference for the CRAFT-network agent when diagnosing connectivity issues during Phase E operations.

Deploy

Debug Tooling Setup

Verify all diagnostic tools are available before beginning any debug session.

OVS diagnostic commands (available inside openvswitch_vswitchd container):

# Verify OVS tools are accessible
docker exec openvswitch_vswitchd ovs-vsctl --version
docker exec openvswitch_vswitchd ovs-ofctl --version
docker exec openvswitch_vswitchd ovs-appctl --version

# Show complete OVS configuration
docker exec openvswitch_vswitchd ovs-vsctl show

OVN diagnostic commands (available inside ovn_northd and ovn_controller containers):

# Verify OVN tools
docker exec ovn_northd ovn-nbctl --version
docker exec ovn_northd ovn-sbctl --version

# OVN trace (powerful logical packet tracing)
docker exec ovn_controller ovn-trace --version

Network namespace tools (on the host or inside Neutron containers):

# List all network namespaces
ip netns list
# Expected: qrouter-<id>, qdhcp-<id> (OVS backend)
# OVN uses fewer namespaces (metadata only)

Packet capture (tcpdump inside containers or namespaces):

# Capture on a tap interface (instance-facing)
tcpdump -i tap<port-id-prefix> -n -c 50

# Capture inside a network namespace
ip netns exec qrouter-<router-id> tcpdump -i qr-<port-prefix> -n -c 50

# Capture on physical NIC
tcpdump -i eth1 -n port 4789  # VXLAN traffic

Kolla-Ansible Debug Container Options

For persistent debug environments, Kolla-Ansible provides tooling containers:

# Enter the neutron_server container for API-level debugging
docker exec -it neutron_server /bin/bash

# Enter openvswitch_vswitchd for flow-level debugging
docker exec -it openvswitch_vswitchd /bin/bash

# Enter the relevant agent container for namespace access
docker exec -it neutron_l3_agent /bin/bash   # OVS backend
docker exec -it neutron_dhcp_agent /bin/bash # OVS backend

Configure

OVS Logging Levels

Adjust OVS logging to capture more detail during active debugging, then restore to production levels.

# Increase OVS daemon logging (temporary, resets on restart)
docker exec openvswitch_vswitchd ovs-appctl vlog/set vswitchd:dbg
docker exec openvswitch_vswitchd ovs-appctl vlog/set ofproto:dbg

# Restore production logging
docker exec openvswitch_vswitchd ovs-appctl vlog/set vswitchd:warn
docker exec openvswitch_vswitchd ovs-appctl vlog/set ofproto:warn

# Check current log levels
docker exec openvswitch_vswitchd ovs-appctl vlog/list

OVN Tracing Enablement

OVN trace simulates a packet through the logical pipeline without sending real traffic.

# Trace a packet from a logical port through OVN
docker exec ovn_controller ovn-trace <datapath> \
  'inport == "<logical-port>" && eth.src == <mac> && eth.dst == <mac> \
   && ip4.src == <src-ip> && ip4.dst == <dst-ip> && ip.ttl == 64'

Neutron Agent Debug Logging

Enable debug logging on individual agents for detailed event tracing.

# Check current log level
docker exec neutron_server grep -i "debug" /etc/neutron/neutron.conf

# Enable debug via Kolla-Ansible config override
# In /etc/kolla/config/neutron/neutron.conf:
# [DEFAULT]
# debug = True

# After config change, reconfigure the service
# kolla-ansible -i inventory reconfigure --tags neutron

Packet Capture Setup

# Identify the tap device for an instance port
openstack port show <port-id> -c id
# Tap device name: tap<first-11-chars-of-port-id>

# Identify the OVS port number for correlation with flow tables
docker exec openvswitch_vswitchd ovs-vsctl --columns=name,ofport list Interface | grep tap

# Set up continuous capture with rotation (for intermittent issues)
tcpdump -i tap<prefix> -n -w /tmp/capture-%H%M.pcap -G 300 -W 12

Flow Table Inspection Setup

# Dump all flow tables on br-int (primary integration bridge)
docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-int

# Dump flows for a specific table (table 0 = ingress classification)
docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-int table=0

# Watch flows in real-time (shows packet/byte counts)
docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-int --no-stats=false

Operate

Connectivity Diagnosis Workflow

Scenario: Instance cannot reach the external network.

Step-by-step trace from instance outward:

1. Verify the instance has an IP: openstack server show <instance> -c addresses
2. Check the port is bound: openstack port show <port-id> -c binding_vif_type -- must be ovs or ovn, not binding_failed
3. Check the tap device exists: ip link show tap<port-prefix> -- if missing, the port was not wired by the agent
4. Check OVS port attachment: docker exec openvswitch_vswitchd ovs-vsctl list-ports br-int | grep <port-prefix>
5. Trace through br-int flows: docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-int | grep <port-tag> -- look for matching ingress/egress rules
6. Check the router namespace (OVS): ip netns exec qrouter-<router-id> ip route -- verify the default route points to the external gateway
7. Check br-ex configuration: docker exec openvswitch_vswitchd ovs-vsctl list-ports br-ex -- the physical NIC must be attached
8. Check the physical NIC: ip link show <nic> -- verify it is UP, check for errors with ip -s link show <nic>

DHCP Troubleshooting

Scenario: Instance gets no IP address.

1. Check DHCP agent (OVS): openstack network agent list | grep dhcp -- must show alive and UP
2. Check DHCP namespace: ip netns exec qdhcp-<network-id> ps aux | grep dnsmasq -- dnsmasq must be running
3. Capture DHCP traffic: ip netns exec qdhcp-<network-id> tcpdump -i tap<dhcp-port-prefix> -n port 67 or port 68 -c 20
4. Run openstack server reboot <instance> to trigger a DHCP request
5. Look for: DHCP Discover (from instance), DHCP Offer (from dnsmasq), DHCP Request, DHCP Ack
6. Missing Discover: instance network stack or tap device issue
7. Discover but no Offer: dnsmasq config or port mismatch
8. Check lease file: ip netns exec qdhcp-<network-id> cat /var/lib/neutron/dhcp/<network-id>/leases
9. OVN DHCP: docker exec ovn_northd ovn-nbctl list DHCP_Options -- verify DHCP options are programmed for the subnet

Floating IP Diagnosis

Scenario: Cannot reach instance from external network via floating IP.

1. Verify floating IP assignment: openstack floating ip show <fip> -- check fixed_ip_address and floating_ip_address fields, confirm port_id is set
2. Check router namespace (OVS): ip netns exec qrouter-<router-id> ip addr show -- the floating IP must appear on the qg-<port> interface
3. Check NAT rules: ip netns exec qrouter-<router-id> iptables -t nat -L -n -v -- look for DNAT rule mapping floating IP to fixed IP
4. Check ARP on external network: arping -I <external-iface> <floating-ip> -- if no resp