Uncover the Hidden Truth of AI Performance
Probing is a production-grade performance profiler designed specifically for distributed AI workloads. Built on dynamic probe injection technology, it delivers zero-overhead runtime introspection with SQL-queryable performance metrics and cross-node correlation analysis.
- Debug Training Instabilities - Real-time insight into why training diverges or hangs
- Optimize Model Performance - Identify bottlenecks in forward/backward passes
- Memory Leak Detection - Track GPU/CPU memory usage across training steps
- Live Variable Inspection - Check tensor values, gradients, and model states without stopping training
- Runtime Framework Analysis - Understand how your framework performs in real-world usage
- Zero-Intrusion Profiling - Profile framework internals without code modifications
- Production Debugging - Debug issues reported by users in their actual environments
- Performance Benchmarking - Collect real performance data for optimization decisions
- Production Monitoring - Monitor AI services without service restarts
- Resource Optimization - Analyze resource usage patterns across the cluster
- Custom Metrics Collection - Gather any application-specific performance data
- Distributed Debugging - Correlate performance issues across multiple nodes
- Dynamic Probe Injection - Attach to running processes without code changes
- SQL-Powered Analytics - Use standard SQL to query performance data
- Live Code Execution - Run Python code directly in target processes
- Real-time Stack Analysis - Capture execution context with variable values
- Require Code Instrumentation - No need to add logging statements, insert timers, or modify your training scripts
- Force "Break-Then-Fix" Workflow - No waiting for issues to occur, then spending days trying to reproduce them
- Lock You Into Fixed Reports - No more deciphering pre-formatted tables; use SQL to create custom analysis reports that match your specific needs
- Disrupt Your Workflow - Attach to running processes without stopping your training jobs or services
- Force You to Learn New Tools - Use familiar SQL syntax and Python code for all your analysis needs
pip install probing# Enable instrumentation at startup
PROBING=1 python train.py
# Or inject into running process
probing -t <pid> inject
# Real-time stack trace analysis
probing -t <pid> backtrace- Dynamic Probe Injection - Runtime instrumentation without target application modification
- Distributed Performance Aggregation - Cross-node data collection with unified correlation analysis
- SQL Analytics Interface - Apache DataFusion-powered query engine with standard SQL syntax
- Interactive Python REPL - Live debugging and variable inspection in running processes
- Production-Grade Overhead - Efficient sampling strategies maintaining <1% performance impact
- Time-Series Storage - Columnar data storage with configurable compression and retention
- Real-Time Introspection - Live performance metrics and runtime stack trace analysis
- Advanced CLI - Comprehensive command-line interface with process monitoring and management
# Inject performance monitoring (Linux only)
probing -t <pid> inject
# Real-time stack trace analysis
probing -t <pid> backtrace
# Query performance data with SQL
probing -t <pid> query "SELECT * FROM python.torch_trace LIMIT 10"
# Evaluate Python code in target process
probing -t <pid> eval "import torch; print(torch.cuda.is_available())"
# Interactive Python REPL (connect to running process)
probing -t <pid> repl
# RDMA Flow Analysis
probing -t <pid> rdma
# List all processes with injected probes
probing list# Memory usage analysis
probing -t <pid> query "SELECT * FROM memory_usage WHERE timestamp > now() - interval '5 min'"
# Performance hotspot analysis
probing -t <pid> query "
SELECT operation_name, avg(duration_ms), count(*)
FROM profiling_data
WHERE timestamp > now() - interval '5 minutes'
GROUP BY operation_name
ORDER BY avg(duration_ms) DESC
"
# Training progress tracking
probing -t <pid> query "
SELECT epoch, avg(loss), min(loss), count(*) as steps
FROM training_logs
GROUP BY epoch
ORDER BY epoch
"Probing provides an interactive Python REPL that connects to running processes, allowing you to inspect variables, execute code, and debug in real-time:
# Connect to a process via REPL
probing -t <pid> repl
# For remote processes
probing -t <host|ip:port> replExample REPL session:
>>> import torch
>>> # Inspect torch models in the target process
>>> models = [m for m in gc.get_objects() if isinstance(m, torch.nn.Module)]The REPL provides:
- Live Variable Inspection: Access all variables in the target process context
- Code Execution: Run arbitrary Python code within the target process
- Real-time Debugging: Set breakpoints and inspect state without stopping the process
# Monitor all cluster nodes
probing cluster attach
# Inter-node communication latency
probing -t <pid> query "SELECT src_rank, dst_rank, avg(latency_ms) FROM comm_metrics"
# Cross-node stack trace comparison
probing -t <pid> query "SELECT * FROM python.backtrace"
# GPU utilization analysis
probing -t <pid> query "SELECT avg(gpu_util) FROM gpu_metrics WHERE timestamp > now() - 60"# Quick memory usage overview
probing -t <pid> memory
# Memory growth trend analysis
probing -t <pid> query "SELECT hour(timestamp), avg(memory_mb) FROM memory_usage GROUP BY hour(timestamp)"
# Memory leak detection
probing -t <pid> query "
SELECT function_name, sum(allocated_bytes) as total_alloc
FROM memory_allocations
WHERE timestamp > now() - interval '1 hour'
GROUP BY function_name
ORDER BY total_alloc DESC
"# Environment variable configuration
export PROBING_SAMPLE_RATE=0.1 # Set sampling rate
export PROBING_RETENTION_DAYS=7 # Data retention period
# View current configuration
probing -t <pid> config
# Dynamic configuration updates
probing -t <pid> config probing.sample_rate=0.05
probing -t <pid> config probing.max_memory=1GB
probing -t <pid> config "probing.rdma.hca.name='mlx5_cx6_0'"
probing -t <pid> config "probing.rdma.sample.rate='5'"Users: pip install probing — see Installation.
Contributors — new here? Read Contributing → Welcome (中文: 贡献指南), then:
python3 -m venv .venv && source .venv/bin/activate
pip install maturin
make develop
make check-dev
make test| Doc | Content |
|---|---|
| Contributing — Welcome | Pick a track (skills / Python / docs / Rust / web), first PR |
| Installation | PyPI, wheel, PROBING=1, platform support |
| Contributing — Dev setup | make develop, .pth hook, Makefile targets |
| examples/README.md | Optional torch/torchvision for demos |
Release wheel only: make frontend && make wheel && make install-wheel.