DOCS configure ipc bypass
Support# Configure IPC Bypass
## Overview
The IPC Bypass feature accelerates local ROS 2 DDS communication between processes on the same machine by intercepting loopback traffic through eBPF programs and redirecting it through a shared-memory pool, bypassing the standard TCP/IP loopback stack.
**Current Status**: Production-ready. The eBPF kernel programs, shared-memory pool, enterprise loader, and userspace consumers are all implemented and tested.
## Architecture
```text
┌─────────────┐ eBPF ┌──────────────┐ Userspace ┌──────────────┐
│ FastDDS │─── intercept ──│ sk_msg / │─── PerfEvent / │ SHM Pool │
│ CycloneDDS │ │ kprobe │──── SHM Poll ─────│ (/dev/shm) │
│ (local) │ │ cgroup_skb │ │ │
└─────────────┘ └──────────────┘ └──────┬───────┘
│
┌────────▼────────┐
│ IpcBypassService │
│ (userspace) │
└────────┬───────┘
│
┌────────▼────────┐
│ PacketBypassed │
│ Event (mpsc) │
└────────┬────────┘
│
┌────────▼────────┐
│ Scheduler / TUN │
│ → Mesh Network │
└─────────────────┘
```
### Dual Kernel Paths
| Path | Kernel Version | Hook | Behavior |
|---|---|---|---|
| **sk_msg** (preferred) | ≥ 5.15 | `sk_msg` on socket send | Drops packet from TX, writes metadata to SHM, optionally redirects via `bpf_msg_redirect_map` (~8 µs). Userspace discovers via SHM polling. |
| **kprobe** (fallback) | ≥ 4.14 | `kretprobe` on `tcp_sendmsg` / `udp_sendmsg` | Records metadata to SHM + `PerfEventArray` notification. Fires after TX enqueue (cannot prevent duplicate delivery). |
Both paths share the same SHM pool, BPF maps, and userspace `IpcBypassService`.
## What Is Implemented
- **eBPF programs**: sk_msg redirect (fast path + SHM fallback), TCP/UDP kprobes, sock_ops socket classification
- **SHM pool**: 64 MB shared memory at `/dev/shm/bauxite_ipc_pool` with small (4 KB) and large (64 KB) slots, lock-free free-list bitmap, 90% utilization guard, 250 ms watchdog
- **Enterprise loader**: `EbpfManager::initialize()` loads eBPF ELF via Aya `BpfLoader`, attaches programs, inserts intercept sockets, and populates `DDS_PORT_SET`
- **Userspace service**: `IpcBypassService` runs the watchdog loop, PerfEvent consumer (kprobe), and SHM polling fallback (sk_msg)
- **Slot processing**: `process_slot()` / `process_slot_from_shm()` read SHM data and emit `PacketBypassed` events for downstream TUN injection
- **Dynamic port sync**: `sync_priority_ports()` writes to the `DDS_PORT_SET` BPF map at runtime
- **Graceful degradation**: Falls back to SHM polling if PerfEvents are unavailable; falls back to standard loopback if SHM exceeds 90% utilization
## Enable the Feature
### Prerequisites
```bash
# Requires root for eBPF program loading
sudo apt install linux-headers-$(uname -r)
# Minimum kernel versions
# sk_msg path: 5.15+ (recommended)
# kprobe path: 4.14+ (functional)
```
### Cargo Feature Flags
In `bauxite-conduit/Cargo.toml`, enable one of:
```toml
[features]
# Preferred: sk_msg redirect path (kernel ≥ 5.15)
ipc-bypass-skmsg-only = []
# Fallback: kprobe path (kernel ≥ 4.14)
ipc-bypass-kprobe-only = []
# Base feature (enables infrastructure without eBPF hooks)
ipc-bypass = []
```
Build with the feature:
```bash
cargo build --release -p bauxite --features "ipc-bypass-skmsg-only telemetry"
```
Or for the kprobe fallback:
```bash
cargo build --release -p bauxite --features "ipc-bypass-kprobe-only telemetry"
```
### Runtime Configuration
Add to `config.toml`:
```toml
[ebpf]
# Enable eBPF IPC bypass (requires enterprise forge crate)
enabled = true
# SHM pool path
shm_path = "/dev/shm/bauxite_ipc_pool"
# Pool size in bytes (default: 64 MB)
pool_size = 67108864
[qos]
# DDS ports to intercept (default: 7400, 7410, 7412)
high_ports = [7400, 7410, 7403, 7412]
```
### Environment Variables
| Variable | Default | Description |
|---|---|---|
| `BAUXITE_EBPF_PATH` | `target/bpfel-unknown-none/release/bauxite-ebpf` | Path to the compiled eBPF ELF |
## SHM Pool Layout
```
/dev/shm/bauxite_ipc_pool (64 MB default)
│
├── Header (128 bytes): magic (0xBAUX), version, slot sizes, counters
│
├── Small Slots (512 × 4 KB): indices 0–511
│ └── Each: { header { sender_pid, receiver_pid, seq_num, flags }, data[4064] }
│
├── Large Slots (64 × 64 KB): indices 512–575
│ └── Same structure, data[65504]
│
└── Free-list Bitmap: atomic bitmask tracking allocated/free status
```
### Utilization Guard
When `active_count / total_slots > 0.9`, the SHM pool blocks new allocations and eBPF programs fall through to the standard loopback path. The watchdog reclaims expired slots every 250 ms (5-second TTL).
## DDS Port Filtering
The eBPF programs intercept traffic on ports configured in `config.toml` under `qos.high_ports`. Default RTPS ports:
| Port | Protocol | Purpose |
|---|---|---|
| 7400 | UDP | RTPS Discovery (PDP) |
| 7403 | UDP | RTPS Participant Discovery |
| 7410 | UDP | RTPS Data Distribution (DDP) |
| 7412 | UDP | RTPS Endpoint Discovery (EDP) |
Additional ports can be dynamically enrolled via `EbpfManager::sync_priority_ports()` at runtime.
## sk_msg Notification Model
**Important**: `sk_msg` programs cannot emit PerfEvents (missing BPF context pointer). The sk_msg path uses a **SHM polling loop** (10 ms interval) to discover new slots. This adds up to 10 ms of notification latency compared to the kprobe PerfEvent path.
The kprobe path uses `IPC_PERF_ARRAY` (`PerfEventArray`) for immediate per-CPU event delivery. If the pinned map is unavailable, it gracefully falls back to SHM polling.
## Performance Characteristics
| Metric | Standard Loopback | sk_msg Fast Path | sk_msg Fallback | kprobe |
|---|---|---|---|---|
| Typical Latency | 50–200 µs | ~8 µs | ~15–25 µs | ~30–50 µs |
| TX Suppression | N/A | Yes (`SK_DROP`) | Yes (`SK_DROP`) | No |
| Notification | N/A | SHM polling (10 ms) | SHM polling (10 ms) | PerfEvent (immediate) |
| Zero-copy | No | Yes (redirect) | Partial | Partial |
## Known Limitations
1. **kprobe duplicate delivery**: On kernels < 5.15, the kprobe fires after the packet is queued. The standard loopback path still processes the packet (potential duplicate delivery).
2. **IPv4 + IPv6 (kprobe only)**: kprobe handlers (`tcp_send`, `udp_send`) support both `AF_INET` and `AF_INET6`. IPv6 addresses are packed as first/last 32-bit chunks in the `KprobePerfEvent`/`ShmSlotHeader` to respect eBPF size constraints; full 128-bit resolution is deferred to userspace. The `sk_msg` redirect path currently supports only IPv4.
3. **TUN injection**: The `IpcBypassService` emits `PacketBypassed` events via an `mpsc::Sender`. Downstream consumers (e.g., the scheduler in `bauxite`) are responsible for TUN pipe injection.
4. **Payload copy**: Slot data is read from SHM by userspace via `memcpy`. This avoids kernel buffer allocations but is not truly zero-copy for the payload itself.
## Testing
Run the SHM pool unit tests:
```bash
cargo test -p bauxite-conduit --features "ipc-bypass" shm
```
Run service tests for a specific feature variant:
```bash
cargo test -p bauxite-conduit --features "ipc-bypass-skmsg-only"
cargo test -p bauxite-conduit --features "ipc-bypass-kprobe-only"
```
Verify eBPF program compilation:
```bash
cargo build --target bpfel-unknown-none --release -p bauxite-conduit-ebpf
```
Run workspace tests:
```bash
cargo test --workspace --exclude bauxite-conduit-ebpf --features "ipc-bypass"
```
> **Note**: The `bauxite-conduit-ebpf` crate must be excluded from `cargo test` due to `panic_impl` lang item conflicts when compiled with `std`.