Section 4: Debugging Tools

Introduction

Debugging distributed robotic systems is challenging:

10+ nodes running simultaneously - Which one is failing?
Async communication - Why isn't my subscriber receiving messages?
Complex state - What are the current parameter values?
Timing issues - Is the publisher sending at the right rate?

ROS 2 provides comprehensive tooling for introspection and debugging:

ros2 CLI - Command-line tools for inspecting every aspect of the system
rqt - Qt-based GUI tools for visualization and interaction
rviz2 - 3D visualization for robot state and sensor data
Logging - Built-in logging system with severity levels

ros2 CLI Tools

The ros2 command-line interface provides subcommands for every ROS 2 concept.

Node Introspection

# List all running nodes
ros2 node list
# Output:
# /talker
# /listener

# Show node information (publishers, subscribers, services, actions)
ros2 node info /talker
# Output:
# /talker
#   Subscribers:
#     /parameter_events: rcl_interfaces/msg/ParameterEvent
#   Publishers:
#     /chatter: std_msgs/msg/String
#     /rosout: rcl_interfaces/msg/Log
#   Services:
#     /talker/describe_parameters: rcl_interfaces/srv/DescribeParameters
#     /talker/get_parameters: rcl_interfaces/srv/GetParameters
#     ...

Topic Introspection

# List all active topics
ros2 topic list
# Output:
# /chatter
# /parameter_events
# /rosout

# Show topic details (type, publisher/subscriber count)
ros2 topic info /chatter
# Output:
# Type: std_msgs/msg/String
# Publisher count: 1
# Subscription count: 1

# Show topic details with endpoint info (-v for verbose)
ros2 topic info /chatter -v
# Output shows publisher and subscriber node names

# Echo messages in real-time
ros2 topic echo /chatter
# Output:
# data: 'Hello World: 42'
# ---
# data: 'Hello World: 43'
# ---

# Measure publishing rate
ros2 topic hz /chatter
# Output: average rate: 1.000

# Show bandwidth usage
ros2 topic bw /chatter
# Output: average: 45.2KB/s

# Publish a message from command line (one-shot)
ros2 topic pub /chatter std_msgs/msg/String "data: 'Test message'"

# Publish at a rate (10 Hz)
ros2 topic pub -r 10 /cmd_vel geometry_msgs/msg/Twist \
  "{linear: {x: 0.5}, angular: {z: 0.3}}"

# Find which nodes publish/subscribe to a topic
ros2 topic find std_msgs/msg/String
# Output lists all String topics

Service Introspection

# List all services
ros2 service list

# Show service type
ros2 service type /add_two_ints
# Output: example_interfaces/srv/AddTwoInts

# Find services of specific type
ros2 service find example_interfaces/srv/AddTwoInts

# Call a service
ros2 service call /add_two_ints example_interfaces/srv/AddTwoInts "{a: 5, b: 3}"
# Output:
# waiting for service to become available...
# requester: making request: example_interfaces.srv.AddTwoInts_Request(a=5, b=3)
# response:
# example_interfaces.srv.AddTwoInts_Response(sum=8)

Action Introspection

# List all actions
ros2 action list

# Show action type
ros2 action type /move_to_target
# Output: example_interfaces/action/Fibonacci

# Send action goal (with feedback)
ros2 action send_goal /move_to_target example_interfaces/action/Fibonacci \
  "{order: 5}" --feedback
# Output shows feedback updates and final result

# Show action server info
ros2 action info /move_to_target
# Output:
# Action: /move_to_target
# Action clients: 0
# Action servers: 1
#     /move_to_target_server

Parameter Introspection

# List all parameters for a node
ros2 param list /talker

# Get parameter value
ros2 param get /talker use_sim_time
# Output: Boolean value is: False

# Set parameter value
ros2 param set /talker publish_rate 10.0
# Output: Set parameter successful

# Dump all parameters to YAML file
ros2 param dump /talker
# Output:
# /talker:
#   ros__parameters:
#     publish_rate: 10.0
#     use_sim_time: false

# Load parameters from file
ros2 param load /talker params.yaml

Message Type Introspection

# List all message types
ros2 interface list

# Show message definition
ros2 interface show std_msgs/msg/String
# Output:
# string data

# Show detailed structure
ros2 interface show geometry_msgs/msg/Twist
# Output:
# Vector3  linear
#   float64 x
#   float64 y
#   float64 z
# Vector3  angular
#   float64 x
#   float64 y
#   float64 z

# Show message packages
ros2 interface package std_msgs
# Output lists all std_msgs types

Bag Files (Record and Replay)

# Record all topics to a bag file
ros2 bag record -a
# Creates rosbag2_YYYY_MM_DD-HH_MM_SS directory

# Record specific topics
ros2 bag record /chatter /cmd_vel /scan

# Record with custom name
ros2 bag record -o my_experiment /chatter

# Show bag info
ros2 bag info rosbag2_YYYY_MM_DD-HH_MM_SS
# Output:
# Files:             rosbag2.db3
# Bag size:          1.2 MB
# Storage id:        sqlite3
# Duration:          45.2s
# Start:             Jun 15 2024 10:30:22.123
# End:               Jun 15 2024 10:31:07.345
# Messages:          452
# Topic information: Topic: /chatter | Type: std_msgs/msg/String | Count: 452

# Replay bag file
ros2 bag play rosbag2_YYYY_MM_DD-HH_MM_SS

# Replay at 2x speed
ros2 bag play -r 2.0 rosbag2_YYYY_MM_DD-HH_MM_SS

# Replay only specific topics
ros2 bag play rosbag2_YYYY_MM_DD-HH_MM_SS --topics /chatter /cmd_vel

# Replay in loop
ros2 bag play -l rosbag2_YYYY_MM_DD-HH_MM_SS

System Health Check

# Check overall ROS 2 system health
ros2 doctor
# Output:
# UserWarning: Publisher and subscriber count mismatch on /chatter
# 1/3 checks failed

# Detailed health report
ros2 doctor --report
# Generates full system report

# Check specific issues
ros2 wtf
# Shows warnings and errors from all nodes

rqt - Qt-based GUI Tools

rqt is a framework for GUI tools built on Qt. Install the full suite:

sudo apt install ros-jazzy-rqt ros-jazzy-rqt-common-plugins

rqt_graph - Visualize Node/Topic Connections

# Launch graph visualization
rqt_graph

Features:

Nodes - Circles or boxes representing nodes
Topics - Arrows showing publish/subscribe relationships
Filters - Hide debug topics, group by namespace
Real-time updates - Graph updates as nodes start/stop

Use case: "Why isn't my subscriber receiving messages?"

Check if both nodes are connected to the same topic
Verify topic name matches exactly

rqt_console - View Log Messages

# Launch log console
rqt_console

Features:

Severity filtering - Show only WARN/ERROR/FATAL messages
Node filtering - Show logs from specific nodes
Message search - Filter by keyword
Timestamps - See when messages occurred
Real-time - Updates live as nodes log

Use case: "Find all error messages in the last 5 minutes"

Filter by severity: ERROR
Sort by timestamp
Search for specific keywords

rqt_plot - Real-Time Data Plotting

# Launch plotter
rqt_plot

Features:

Real-time graphs - Plot numeric message fields
Multiple topics - Compare different data sources
Custom time windows - Zoom to specific ranges
Export - Save plots as images

Example usage:

Topic: /cmd_vel/linear/x
Topic: /odom/pose/pose/position/x

Use case: "Is the robot actually moving when I send velocity commands?"

Plot /cmd_vel/linear/x (commanded velocity)
Plot /odom/twist/twist/linear/x (actual velocity)
Verify they correlate

rqt_topic - Topic Monitor

# Launch topic monitor
rqt_topic

Features:

Topic list - See all topics with types
Publishing rate - Real-time Hz measurement
Bandwidth - See data throughput
Message inspection - View latest message content

Use case: "Check if sensor is publishing at expected rate"

Find /scan topic
Verify it's publishing at 10 Hz (not 0.1 Hz due to bug)

rqt_service_caller - Call Services Interactively

# Launch service caller
rqt_service_caller

Features:

Service list - All available services
Request builder - GUI form for request parameters
Response viewer - See service response
History - Previous calls saved

Use case: "Test if my service works without writing client code"

Select /add_two_ints
Fill in a=5, b=3
Click "Call"
See response: sum=8

rqt_reconfigure - Dynamic Parameter Tuning

# Launch dynamic reconfigure
rqt_reconfigure

Features:

Slider controls - Adjust numeric parameters
Real-time updates - Changes apply immediately
Value ranges - Respects parameter constraints
Save/load configs - Export parameter sets

Use case: "Tune PID controller gains without restarting node"

Select /controller node
Adjust kp, ki, kd sliders
Observe robot behavior changes live

rqt - Master Plugin Loader

# Launch rqt with plugin selection menu
rqt

All plugins available in one window:

Plugins → Introspection → Node Graph, Topic Monitor
Plugins → Visualization → Plot, Image View
Plugins → Configuration → Dynamic Reconfigure
Plugins → Logging → Console
Plugins → Services → Service Caller

rviz2 - 3D Visualization

rviz2 is the primary tool for visualizing robot state, sensor data, and planning results.

# Launch rviz2
rviz2

# Load a config file
rviz2 -d my_config.rviz

Common Display Types

1. RobotModel - Visualize URDF

Setup:

Add → RobotModel
Set "Robot Description" topic to /robot_description
Robot appears in 3D view

Use case: Verify URDF loads correctly, check joint angles

2. TF - Coordinate Frames

Setup:

Add → TF
See all coordinate frames as axes
Enable "Show Names" to label frames

Use case: Debug transform tree, find missing frames

3. LaserScan - Lidar Visualization

Setup:

Add → LaserScan
Set Topic to /scan
Adjust Size and Color

Use case: Verify lidar is detecting obstacles correctly

4. Image - Camera Feeds

Setup:

Add → Image
Set Topic to /camera/image_raw
View live camera feed

Use case: Debug computer vision pipeline

5. Path - Planned Trajectories

Setup:

Add → Path
Set Topic to /plan
See planned robot path

Use case: Verify path planner is generating valid paths

6. Marker - Custom Visualizations

Setup:

Add → Marker
Set Topic to /visualization_marker
Display custom shapes (arrows, spheres, text)

Use case: Debug algorithm by visualizing intermediate results

Common rviz2 Operations

# Save current configuration
File → Save Config As → my_robot.rviz

# Load configuration on startup
rviz2 -d my_robot.rviz

# Common keybindings:
# - Mouse drag: Rotate view
# - Shift + Mouse drag: Pan view
# - Scroll: Zoom
# - F: Focus on selected object
# - R: Reset view

Logging System

ROS 2 uses a severity-based logging system with 5 levels:

Logging Levels

Level	Usage	Example
DEBUG	Detailed debug info	"Entering function X with param Y"
INFO	General information	"Node started successfully"
WARN	Warning, but system still works	"Sensor timeout, using last value"
ERROR	Error, feature doesn't work	"Failed to connect to database"
FATAL	Critical error, system shutting down	"Out of memory, aborting"

Using Logging in Python

import rclpy
from rclpy.node import Node

class MyNode(Node):
    def __init__(self):
        super().__init__('my_node')

        # Log at different levels
        self.get_logger().debug('Debug message (verbose details)')
        self.get_logger().info('Informational message')
        self.get_logger().warn('Warning message')
        self.get_logger().error('Error message')
        self.get_logger().fatal('Fatal error message')

        # Log with formatting
        value = 42
        self.get_logger().info(f'Sensor reading: {value}')

        # Log once (only first occurrence)
        self.get_logger().info('This only prints once', once=True)

        # Throttled logging (max every 5 seconds)
        self.get_logger().info('Throttled message', throttle_duration_sec=5.0)

Viewing Logs

# View all logs in terminal
ros2 run my_package my_node

# View logs with rqt_console
rqt_console

# View logs from specific node
ros2 node info /my_node

# Check /rosout topic (all logs published here)
ros2 topic echo /rosout

Configuring Log Levels

# Set log level for specific node at launch
ros2 run my_package my_node --ros-args --log-level debug

# Set different levels for different loggers
ros2 run my_package my_node --ros-args \
  --log-level my_node:=debug \
  --log-level rclcpp:=warn

Launch file configuration:

Node(
    package='my_package',
    executable='my_node',
    arguments=['--ros-args', '--log-level', 'debug']
)

Debugging Workflow Examples

Problem: Subscriber Not Receiving Messages

Symptoms: Publisher runs, but subscriber callback never fires

Debug steps:

Check if both nodes are running:

ros2 node list
# Verify both publisher and subscriber are listed

Check topic names:

ros2 topic info /my_topic -v
# Verify Publisher count: 1, Subscription count: 1
# Check if topic names match exactly (case-sensitive)

Visualize with rqt_graph:

rqt_graph
# Look for arrow connecting publisher to subscriber
# If no arrow, topic names don't match

Check message types:

ros2 topic type /my_topic
# Verify publisher and subscriber use same type

Check QoS compatibility:

ros2 topic info /my_topic -v
# Check QoS profiles match (Reliability, Durability)

Test with echo:

ros2 topic echo /my_topic
# If you see messages here, problem is in subscriber code
# If no messages, problem is in publisher code

Problem: Service Call Times Out

Symptoms: ros2 service call hangs, never returns

Debug steps:

Check if service exists:
```
ros2 service list | grep my_service
```
Check if server node is running:
```
ros2 node list | grep my_server
```
Check service type matches:
```
ros2 service type /my_service
```

Check server logs:

ros2 run my_package my_server
# Look for error messages when service is called

Test with simple request:

ros2 service call /my_service std_srvs/srv/Trigger

Problem: High CPU Usage

Symptoms: Node using 100% CPU

Debug steps:

Check publishing rates:

ros2 topic hz /my_topic
# Verify not publishing at excessive rate (e.g., 10000 Hz)

Check callback frequency:

# Add logging to callback with timestamps
self.get_logger().info('Callback fired')

Profile with top:

top -p $(pgrep my_node)
# See CPU% usage over time

Check for tight loops:
- Review code for while True: without sleep()
- Ensure timer callbacks aren't blocking

Use rqt_top:

ros2 run rqt_top rqt_top
# See CPU/memory usage per node

Problem: Transform Not Available

Symptoms: "Transform from 'base_link' to 'camera' not available"

Debug steps:

Check if frames exist:

ros2 run tf2_ros tf2_echo base_link camera
# Shows transform if available, error if not

View TF tree:

ros2 run tf2_tools view_frames
# Generates frames.pdf showing transform tree
evince frames.pdf

Visualize in rviz2:

rviz2
# Add → TF
# See all frames, identify missing ones

Check who publishes transforms:

ros2 topic info /tf
ros2 topic info /tf_static
# See which nodes publish transforms

Check URDF if using robot_state_publisher:
- Verify <joint> connects parent/child frames
- Ensure robot_state_publisher node is running

Key Takeaways

ros2 CLI provides commands for every ROS 2 concept (node, topic, service, action, param)
rqt_graph is essential for visualizing node/topic connections
rqt_console centralizes log viewing with filtering
rviz2 is the primary 3D visualization tool for robotics
Logging should use appropriate severity levels (DEBUG, INFO, WARN, ERROR, FATAL)
Systematic debugging follows a checklist: check node running → topic names → message types → QoS → actual data

Exercises

Try these debugging challenges:

Broken pub-sub: Intentionally use different topic names in publisher and subscriber
- Use ros2 topic list and rqt_graph to identify the issue
- Fix and verify with ros2 topic echo
QoS mismatch: Create publisher with RELIABLE QoS, subscriber with BEST_EFFORT
- Observe subscriber receives no messages
- Use ros2 topic info -v to see QoS settings
- Fix by matching QoS profiles
Parameter tuning: Create a node with adjustable speed parameter
- Use rqt_reconfigure to change speed while running
- Verify behavior changes without restart
Log analysis: Generate 1000 log messages with random severities
- Use rqt_console to filter only ERROR messages
- Find specific keywords in logs
Performance debugging: Create a node that publishes at 1000 Hz
- Use ros2 topic hz to measure actual rate
- Use ros2 topic bw to measure bandwidth
- Reduce rate if causing performance issues

Introduction​

ros2 CLI Tools​

Node Introspection​

Topic Introspection​

Service Introspection​

Action Introspection​

Parameter Introspection​

Message Type Introspection​

Bag Files (Record and Replay)​

System Health Check​

rqt - Qt-based GUI Tools​

rqt_graph - Visualize Node/Topic Connections​

rqt_console - View Log Messages​

rqt_plot - Real-Time Data Plotting​

rqt_topic - Topic Monitor​

rqt_service_caller - Call Services Interactively​

rqt_reconfigure - Dynamic Parameter Tuning​

rqt - Master Plugin Loader​

rviz2 - 3D Visualization​

Common Display Types​

1. RobotModel - Visualize URDF​

2. TF - Coordinate Frames​

3. LaserScan - Lidar Visualization​

4. Image - Camera Feeds​

5. Path - Planned Trajectories​

6. Marker - Custom Visualizations​

Common rviz2 Operations​

Logging System​

Logging Levels​

Using Logging in Python​

Viewing Logs​

Configuring Log Levels​

Debugging Workflow Examples​

Problem: Subscriber Not Receiving Messages​

Problem: Service Call Times Out​

Problem: High CPU Usage​

Problem: Transform Not Available​

Key Takeaways​

Exercises​

Further Reading​

Introduction

ros2 CLI Tools

Node Introspection

Topic Introspection

Service Introspection

Action Introspection

Parameter Introspection

Message Type Introspection

Bag Files (Record and Replay)

System Health Check

rqt - Qt-based GUI Tools

rqt_graph - Visualize Node/Topic Connections

rqt_console - View Log Messages

rqt_plot - Real-Time Data Plotting

rqt_topic - Topic Monitor

rqt_service_caller - Call Services Interactively

rqt_reconfigure - Dynamic Parameter Tuning

rqt - Master Plugin Loader

rviz2 - 3D Visualization

Common Display Types

1. RobotModel - Visualize URDF

2. TF - Coordinate Frames

3. LaserScan - Lidar Visualization

4. Image - Camera Feeds

5. Path - Planned Trajectories

6. Marker - Custom Visualizations

Common rviz2 Operations

Logging System

Logging Levels

Using Logging in Python

Viewing Logs

Configuring Log Levels

Debugging Workflow Examples

Problem: Subscriber Not Receiving Messages

Problem: Service Call Times Out

Problem: High CPU Usage

Problem: Transform Not Available

Key Takeaways

Exercises

Further Reading