Update README.md
Browse files
README.md
CHANGED
|
@@ -70,6 +70,32 @@ response = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
|
|
| 70 |
print(response)
|
| 71 |
```
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
## Citation
|
| 74 |
|
| 75 |
If you use CASA in your research, please cite our work:
|
|
|
|
| 70 |
print(response)
|
| 71 |
```
|
| 72 |
|
| 73 |
+
## Live video Captioning
|
| 74 |
+
In our [github repository], we provide a script to caption a video using our `CASA-Qwen2_5-VL-3B-LiveCC` model and generate the resulting video with subtitles embedded at the actual time they are generated.
|
| 75 |
+
|
| 76 |
+
Note that you will also need to install `ffmpeg` for this script to run. The Python dependencies are handled with `uv`
|
| 77 |
+
|
| 78 |
+
```bash
|
| 79 |
+
# Script options
|
| 80 |
+
uv run scripts/gen_livecc_subtitles.py --help
|
| 81 |
+
# Generation with Qwen2.5VL+CASA
|
| 82 |
+
uv run scripts/gen_livecc_subtitles.py --sample_path path_to_video.mp4 --srt True --temp 0.0
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
Additional qualitative samples are available in the associated [HuggingFace space](https://huggingface.co/spaces/kyutai/casa-samples).
|
| 86 |
+
|
| 87 |
+
<div align="center">
|
| 88 |
+
<p align="center" width="100%">
|
| 89 |
+
<video src="https://github.com/user-attachments/assets/cb205fe2-11fb-4e8d-98ac-e1a250e5573b" width="80%" controls></video>
|
| 90 |
+
</p>
|
| 91 |
+
<p>
|
| 92 |
+
The input video is taken from the Animal Kingdom dataset, and the subtitles displayed are generated with <code>CASA-Qwen2_5-VL-3B-LiveCC</code>.
|
| 93 |
+
|
| 94 |
+
Specifically, video frames are extracted at 2fps, and subtitles are displayed in real-time at the timestamp they are generated< </p>
|
| 95 |
+
|
| 96 |
+
<p><small> <i><b>Transcript:</b> "This video shows a fox in the Arctic. The Arctic is an area of Earth that's covered by ice and snow year -round, and it gets very cold there. Foxes are adapted to live in this cold environment because they have a thick layer of fur to keep them warm when they're out in the snow. This fox is walking through the snow and looking around for food or maybe just for safety from predators like wolves or bears that might be around. Foxes are also known for their ability to jump really high and"</i></small></p>
|
| 97 |
+
</div
|
| 98 |
+
|
| 99 |
## Citation
|
| 100 |
|
| 101 |
If you use CASA in your research, please cite our work:
|