# OmniNode

ROS2 node for Horizon Robotics S100P to run Qwen2.5-3B Omni multimodal LLM inference via xlm SDK.

## Directory Structure

```
OmniNode/
├── CMakeLists.txt
├── package.xml
├── config/
│   └── omni_config.json
├── include/
│   └── omni_node/
│       ├── omni_engine.hpp
│       └── omni_node.hpp
├── launch/
│   └── omni_launch.py
├── src/
│   ├── main.cpp
│   ├── omni_engine.cpp
│   └── omni_node.cpp
└── README.md
```

## Dependencies

- ROS2 Humble
- nlohmann/json (header-only, included via `single_include/nlohmann/json.hpp`)
- xlm SDK (pre-installed on S100P)

On S100P, download the single-header JSON library into the project:

```bash
cd ~/colcon_ws/src/OmniNode
mkdir -p include/nlohmann
wget https://raw.githubusercontent.com/nlohmann/json/develop/single_include/nlohmann/json.hpp -O include/nlohmann/json.hpp
```

## Configuration

Edit `config/omni_config.json` and fill in the paths to your HBM model files:

```json
{
  "visual_hbm_path": "/opt/models/visual.hbm",
  "audio_hbm_path": "/opt/models/audio.hbm",
  "text_hbm_path": "/opt/models/text.hbm",
  "embed_tokens": "/opt/models/embed_tokens.bin",
  "tokenizer_dir": "/opt/models/tokenizer/",
  "model_type": 5,
  "online_mode": true,
  "bpu_core": -1,
  "system_prompt": "You are Qwen, a virtual human..."
}
```

| Field | Description |
|---|---|
| `visual_hbm_path` | Vision model HBM file |
| `audio_hbm_path` | Audio model HBM file |
| `text_hbm_path` | Text model HBM file |
| `embed_tokens` | Embed tokens file |
| `tokenizer_dir` | Tokenizer directory |
| `model_type` | Model type (5 = Omni) |
| `online_mode` | Online inference mode |
| `bpu_core` | BPU core (-1 = any) |
| `system_prompt` | System prompt for the model |

## Build

On your development PC or directly on S100P:

```bash
cd ~/colcon_ws/src/OmniNode
colcon build --packages-select omni_node
source install/setup.bash
```

Or with ROS2 workspace:

```bash
cd ~/ros2_ws/src
# copy OmniNode folder here
cd ~/ros2_ws
colcon build --packages-select omni_node
source install/setup.bash
```

## Run

### Option 1: Direct launch

```bash
ros2 run omni_node omni_node --ros-args -p config_path:=config/omni_config.json
```

### Option 2: Launch file

```bash
ros2 launch omni_node omni_launch.py
```

### Option 3: With custom config path

```bash
ros2 launch omni_node omni_launch.py config_path:=/full/path/to/your/omni_config.json
```

## Topics

| Topic | Type | Direction | Description |
|---|---|---|---|
| `/asr` | `std_msgs/msg/String` | Subscribe | Input ASR text |
| `/llm_response` | `std_msgs/msg/String` | Publish | LLM response text |

## Usage

After the node is running, publish a text message to `/asr`:

```bash
ros2 topic pub /asr std_msgs/msg/String '{data: "请描述我在做什么"}' --once
```

The LLM response will be published to `/llm_response`:

```bash
ros2 topic echo /llm_response
```

## Architecture

```
omni_node (ROS2 Node)
  └── OmniEngine (xlm SDK Wrapper)
        └── xlm_init / xlm_omni_feed_text_online / xlm_omni
```

- **omni_node.hpp/cpp**: ROS2 interface — subscribes to `/asr`, publishes `/llm_response`
- **omni_engine.hpp/cpp**: xlm SDK wrapper — loads config, initializes model, handles callback
- **main.cpp**: entry point, creates rclcpp node and spins

## Notes

- BPU core `-1` lets the SDK select any available BPU core automatically
- Online mode must be `true` for streaming token output
- The node is single-threaded (ROS2 default executor), no extra threads are spawned
