ameroyer commited on
Commit
317e51f
·
verified ·
1 Parent(s): 44ca370

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +26 -0
README.md CHANGED
@@ -70,6 +70,32 @@ response = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
70
  print(response)
71
  ```
72
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
  ## Citation
74
 
75
  If you use CASA in your research, please cite our work:
 
70
  print(response)
71
  ```
72
 
73
+ ## Live video Captioning
74
+ In our [github repository], we provide a script to caption a video using our `CASA-Qwen2_5-VL-3B-LiveCC` model and generate the resulting video with subtitles embedded at the actual time they are generated.
75
+
76
+ Note that you will also need to install `ffmpeg` for this script to run. The Python dependencies are handled with `uv`
77
+
78
+ ```bash
79
+ # Script options
80
+ uv run scripts/gen_livecc_subtitles.py --help
81
+ # Generation with Qwen2.5VL+CASA
82
+ uv run scripts/gen_livecc_subtitles.py --sample_path path_to_video.mp4 --srt True --temp 0.0
83
+ ```
84
+
85
+ Additional qualitative samples are available in the associated [HuggingFace space](https://huggingface.co/spaces/kyutai/casa-samples).
86
+
87
+ <div align="center">
88
+ <p align="center" width="100%">
89
+ <video src="https://github.com/user-attachments/assets/cb205fe2-11fb-4e8d-98ac-e1a250e5573b" width="80%" controls></video>
90
+ </p>
91
+ <p>
92
+ The input video is taken from the Animal Kingdom dataset, and the subtitles displayed are generated with <code>CASA-Qwen2_5-VL-3B-LiveCC</code>.
93
+
94
+ Specifically, video frames are extracted at 2fps, and subtitles are displayed in real-time at the timestamp they are generated< </p>
95
+
96
+ <p><small> <i><b>Transcript:</b> "This video shows a fox in the Arctic. The Arctic is an area of Earth that's covered by ice and snow year -round, and it gets very cold there. Foxes are adapted to live in this cold environment because they have a thick layer of fur to keep them warm when they're out in the snow. This fox is walking through the snow and looking around for food or maybe just for safety from predators like wolves or bears that might be around. Foxes are also known for their ability to jump really high and"</i></small></p>
97
+ </div
98
+
99
  ## Citation
100
 
101
  If you use CASA in your research, please cite our work: