README.md

---
license: cc-by-nc-sa-4.0
language: 
  - en
tags:
  - video-generation
  - vision-language-navigation
  - embodied-ai
  - pytorch
---

![SparseVideoNav Architecture](assets/caption.png)
# SparseVideoNav: Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation

## Model Details

### Model Description

SparseVideoNav introduces video generation models to real-world beyond-the-view vision-language navigation for the first time. It pioneers a paradigm shift from continuous to sparse video generation for longer prediction horizons. By guiding trajectory inference with a generated sparse future spanning a 20-second horizon, it achieves sub-second inference (a 27× speed-up). It also marks the first realization of beyond-the-view navigation in challenging night scenes.

- **Developed by:** Hai Zhang, Siqi Liang, Li Chen, Yuxian Li, Yukuan Xu, Yichao Zhong, Fu Zhang, Hongyang Li
- **Shared by:** The University of Hong Kong & OpenDriveLab
- **Model type:** Video Generation-based Model for Vision-Language Navigation
- **Language(s) (NLP):** English (Instruction prompts)
- **License:** CC BY-NC-SA 4.0
- **Finetuned from model:** Based on UMT5-XXL (text encoder) and Wan2.1 VAE.

### Model Sources

- **Repository:** [https://github.com/OpenDriveLab/SparseVideoNav](https://github.com/OpenDriveLab/SparseVideoNav)
- **Paper:** [arXiv:2602.05827](https://arxiv.org/abs/2602.05827)
- **Project Page:** [https://opendrivelab.com/SparseVideoNav](https://opendrivelab.com/SparseVideoNav)

## Uses

### Direct Use

The model is designed for generating sparse future video frames based on a current visual observation (video) and a natural language instruction (e.g., "turn right"). It is primarily intended for research in Embodied AI, specifically Vision-Language Navigation (VLN) in real-world environments.

### Out-of-Scope Use

The model is a research prototype and is not intended for deployment in safety-critical real-world autonomous driving or robotic navigation systems without further extensive testing, safety validation, and fallback mechanisms.

## How to Get Started with the Model

Use the code below to get started with the model using our custom pipeline. 

Ensure you have cloned the [GitHub repository](https://github.com/OpenDriveLab/SparseVideoNav) and installed the requirements.

```python
from omegaconf import OmegaConf
from inference import SVNPipeline

# Load configuration
cfg = OmegaConf.load("config/inference.yaml")
cfg.ckpt_path = "/path/to/models/SparseVideoNav-Models" # Path to your downloaded weights
cfg.inference.device = "cuda:0"

# Initialize pipeline
pipeline = SVNPipeline.from_pretrained(cfg)

# Run inference (Returns np.ndarray (T, H, W, C) uint8)
video = pipeline(video="/path/to/input.mp4", text="turn right") 
```

## BibTeX
```python
@article{zhang2026sparse,
  title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},
  author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},
  journal={arXiv preprint arXiv:2602.05827},
  year={2026}
}