File size: 16,999 Bytes
51e0162 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766]
W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766] *****************************************
W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766] *****************************************
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
[rank1]:[W226 04:01:30.449847470 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
[rank3]:[W226 04:01:31.002032198 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
[rank2]:[W226 04:01:31.256592374 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
[rank0]:[W226 04:01:32.571792358 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
`torch_dtype` is deprecated! Use `dtype` instead!
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
`torch_dtype` is deprecated! Use `dtype` instead!
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:00<00:03, 1.22it/s]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 40%|████ | 2/5 [00:01<00:02, 1.27it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:00<00:03, 1.12it/s]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 60%|██████ | 3/5 [00:02<00:01, 1.28it/s]
Loading checkpoint shards: 40%|████ | 2/5 [00:01<00:02, 1.14it/s]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:00<00:03, 1.09it/s]
Loading checkpoint shards: 80%|████████ | 4/5 [00:03<00:00, 1.26it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.67it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.46it/s]
rank1: 0%| | 0/32 [00:00<?, ?it/s]
Loading checkpoint shards: 60%|██████ | 3/5 [00:02<00:01, 1.14it/s]
Loading checkpoint shards: 40%|████ | 2/5 [00:01<00:02, 1.13it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:00<00:03, 1.00it/s]
Loading checkpoint shards: 80%|████████ | 4/5 [00:03<00:00, 1.16it/s]
Loading checkpoint shards: 60%|██████ | 3/5 [00:02<00:01, 1.15it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.58it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.35it/s]
Loading checkpoint shards: 40%|████ | 2/5 [00:01<00:02, 1.06it/s]
rank2: 0%| | 0/32 [00:00<?, ?it/s]
Loading checkpoint shards: 80%|████████ | 4/5 [00:03<00:00, 1.20it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.62it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.37it/s]
Loading checkpoint shards: 60%|██████ | 3/5 [00:02<00:01, 1.11it/s]
rank0: 0%| | 0/32 [00:00<?, ?it/s]
Loading checkpoint shards: 80%|████████ | 4/5 [00:03<00:00, 1.16it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.58it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00, 1.32it/s]
rank3: 0%| | 0/32 [00:00<?, ?it/s]
rank1: 3%|▎ | 1/32 [00:11<05:51, 11.34s/it]
rank2: 3%|▎ | 1/32 [00:10<05:28, 10.61s/it]
rank3: 3%|▎ | 1/32 [00:08<04:33, 8.83s/it]
rank0: 3%|▎ | 1/32 [00:11<05:44, 11.11s/it]
rank1: 6%|▋ | 2/32 [00:20<05:03, 10.13s/it]
rank2: 6%|▋ | 2/32 [00:20<05:08, 10.28s/it]
rank3: 6%|▋ | 2/32 [00:19<05:03, 10.11s/it]
rank0: 6%|▋ | 2/32 [00:21<05:20, 10.69s/it]
rank1: 9%|▉ | 3/32 [00:29<04:33, 9.43s/it]
rank3: 9%|▉ | 3/32 [00:28<04:33, 9.44s/it]
rank2: 9%|▉ | 3/32 [00:34<05:46, 11.95s/it]
rank0: 9%|▉ | 3/32 [00:35<05:47, 11.97s/it]
rank3: 12%|█▎ | 4/32 [00:36<04:13, 9.05s/it]
rank2: 12%|█▎ | 4/32 [00:43<05:01, 10.76s/it]
rank1: 12%|█▎ | 4/32 [00:45<05:35, 12.00s/it]
rank0: 12%|█▎ | 4/32 [00:44<05:11, 11.11s/it]
rank3: 16%|█▌ | 5/32 [00:47<04:23, 9.76s/it]
rank1: 16%|█▌ | 5/32 [00:53<04:51, 10.80s/it]
rank0: 16%|█▌ | 5/32 [00:51<04:21, 9.67s/it]
rank2: 16%|█▌ | 5/32 [00:54<04:48, 10.70s/it]
rank3: 19%|█▉ | 6/32 [00:57<04:13, 9.76s/it]
rank0: 19%|█▉ | 6/32 [01:01<04:14, 9.78s/it]
rank2: 19%|█▉ | 6/32 [01:04<04:35, 10.60s/it]
rank1: 19%|█▉ | 6/32 [01:09<05:27, 12.61s/it]
rank0: 22%|██▏ | 7/32 [01:13<04:17, 10.29s/it]
rank2: 22%|██▏ | 7/32 [01:14<04:18, 10.33s/it]
rank3: 22%|██▏ | 7/32 [01:12<04:48, 11.55s/it]
rank1: 22%|██▏ | 7/32 [01:20<04:58, 11.94s/it]
rank2: 25%|██▌ | 8/32 [01:27<04:26, 11.10s/it]
rank1: 25%|██▌ | 8/32 [01:28<04:16, 10.70s/it]
rank0: 25%|██▌ | 8/32 [01:26<04:31, 11.31s/it]
rank3: 25%|██▌ | 8/32 [01:30<05:19, 13.33s/it]
rank0: 28%|██▊ | 9/32 [01:36<04:06, 10.74s/it]
rank2: 28%|██▊ | 9/32 [01:38<04:20, 11.31s/it]
rank1: 28%|██▊ | 9/32 [01:40<04:13, 11.01s/it]
rank3: 28%|██▊ | 9/32 [01:39<04:38, 12.12s/it]
rank0: 31%|███▏ | 10/32 [01:43<03:33, 9.69s/it]
rank1: 31%|███▏ | 10/32 [01:49<03:47, 10.35s/it]
rank2: 31%|███▏ | 10/32 [01:49<04:05, 11.18s/it]
rank3: 31%|███▏ | 10/32 [01:48<04:05, 11.15s/it]
rank0: 34%|███▍ | 11/32 [01:55<03:39, 10.46s/it]
rank1: 34%|███▍ | 11/32 [01:59<03:35, 10.26s/it]
rank2: 34%|███▍ | 11/32 [01:58<03:38, 10.39s/it]
rank3: 34%|███▍ | 11/32 [01:56<03:33, 10.18s/it]
rank0: 38%|███▊ | 12/32 [02:05<03:21, 10.10s/it]
rank1: 38%|███▊ | 12/32 [02:07<03:13, 9.68s/it]
rank2: 38%|███▊ | 12/32 [02:08<03:28, 10.43s/it]
rank0: 41%|████ | 13/32 [02:14<03:05, 9.79s/it]
rank1: 41%|████ | 13/32 [02:16<03:02, 9.59s/it]
rank3: 38%|███▊ | 12/32 [02:13<04:05, 12.30s/it]
rank1: 44%|████▍ | 14/32 [02:25<02:49, 9.40s/it]
rank3: 41%|████ | 13/32 [02:22<03:35, 11.35s/it]
rank2: 41%|████ | 13/32 [02:25<03:56, 12.47s/it]
rank0: 44%|████▍ | 14/32 [02:26<03:11, 10.62s/it]
rank1: 47%|████▋ | 15/32 [02:34<02:36, 9.21s/it]
rank3: 44%|████▍ | 14/32 [02:33<03:18, 11.02s/it]
rank0: 47%|████▋ | 15/32 [02:35<02:51, 10.11s/it]
rank2: 44%|████▍ | 14/32 [02:43<04:09, 13.86s/it]
rank1: 50%|█████ | 16/32 [02:45<02:36, 9.79s/it]
rank0: 50%|█████ | 16/32 [02:43<02:33, 9.59s/it]
rank1: 53%|█████▎ | 17/32 [02:53<02:16, 9.09s/it]
rank3: 47%|████▋ | 15/32 [02:50<03:37, 12.81s/it]
rank2: 47%|████▋ | 15/32 [02:53<03:35, 12.70s/it]
rank0: 53%|█████▎ | 17/32 [02:53<02:24, 9.65s/it]
rank1: 56%|█████▋ | 18/32 [03:01<02:02, 8.71s/it]
rank3: 50%|█████ | 16/32 [02:59<03:06, 11.68s/it]
rank2: 50%|█████ | 16/32 [03:02<03:06, 11.64s/it]
rank0: 56%|█████▋ | 18/32 [03:02<02:11, 9.36s/it]
rank3: 53%|█████▎ | 17/32 [03:08<02:43, 10.93s/it]
rank1: 59%|█████▉ | 19/32 [03:11<02:01, 9.31s/it]
rank0: 59%|█████▉ | 19/32 [03:11<01:58, 9.14s/it]
rank2: 53%|█████▎ | 17/32 [03:15<03:03, 12.24s/it]
rank1: 62%|██████▎ | 20/32 [03:19<01:46, 8.91s/it]
rank3: 56%|█████▋ | 18/32 [03:18<02:29, 10.71s/it]
rank0: 62%|██████▎ | 20/32 [03:19<01:48, 9.05s/it]
rank3: 59%|█████▉ | 19/32 [03:26<02:10, 10.02s/it]
rank2: 56%|█████▋ | 18/32 [03:29<02:55, 12.55s/it]
rank0: 66%|██████▌ | 21/32 [03:30<01:44, 9.54s/it]
rank1: 66%|██████▌ | 21/32 [03:35<02:01, 11.02s/it]
rank3: 62%|██████▎ | 20/32 [03:36<02:00, 10.02s/it]
rank0: 69%|██████▉ | 22/32 [03:38<01:29, 8.92s/it]
rank2: 59%|█████▉ | 19/32 [03:40<02:40, 12.33s/it]
rank1: 69%|██████▉ | 22/32 [03:43<01:41, 10.18s/it]
rank0: 72%|███████▏ | 23/32 [03:46<01:17, 8.65s/it]
rank3: 66%|██████▌ | 21/32 [03:46<01:50, 10.03s/it]
rank1: 72%|███████▏ | 23/32 [03:51<01:25, 9.51s/it]
rank2: 62%|██████▎ | 20/32 [03:51<02:20, 11.68s/it]
rank0: 75%|███████▌ | 24/32 [03:56<01:12, 9.08s/it]
rank3: 69%|██████▉ | 22/32 [03:58<01:44, 10.42s/it]
rank1: 75%|███████▌ | 24/32 [04:04<01:22, 10.32s/it]
rank2: 66%|██████▌ | 21/32 [04:03<02:09, 11.77s/it]
rank2: 69%|██████▉ | 22/32 [04:10<01:44, 10.43s/it]
rank3: 72%|███████▏ | 23/32 [04:08<01:33, 10.40s/it]
rank0: 78%|███████▊ | 25/32 [04:12<01:18, 11.26s/it]
rank1: 78%|███████▊ | 25/32 [04:16<01:16, 10.90s/it]
rank2: 72%|███████▏ | 23/32 [04:19<01:30, 10.11s/it]
rank3: 75%|███████▌ | 24/32 [04:17<01:20, 10.08s/it]
rank0: 81%|████████▏ | 26/32 [04:21<01:03, 10.52s/it]
rank1: 81%|████████▏ | 26/32 [04:25<01:02, 10.44s/it]
rank2: 75%|███████▌ | 24/32 [04:29<01:20, 10.02s/it]
rank3: 78%|███████▊ | 25/32 [04:29<01:13, 10.52s/it]
rank0: 84%|████████▍ | 27/32 [04:30<00:50, 10.19s/it]
rank1: 84%|████████▍ | 27/32 [04:42<01:01, 12.28s/it]
rank0: 88%|████████▊ | 28/32 [04:40<00:39, 9.98s/it]
rank2: 78%|███████▊ | 25/32 [04:42<01:16, 10.88s/it]
rank3: 81%|████████▏ | 26/32 [04:43<01:08, 11.44s/it]
rank1: 88%|████████▊ | 28/32 [04:51<00:45, 11.48s/it]
rank2: 81%|████████▏ | 26/32 [04:51<01:02, 10.36s/it]
rank0: 91%|█████████ | 29/32 [04:51<00:30, 10.26s/it]
rank3: 84%|████████▍ | 27/32 [04:51<00:52, 10.47s/it]
rank2: 84%|████████▍ | 27/32 [04:59<00:48, 9.66s/it]
rank3: 88%|████████▊ | 28/32 [05:00<00:40, 10.21s/it]
rank0: 94%|█████████▍| 30/32 [05:03<00:21, 10.78s/it]
rank1: 91%|█████████ | 29/32 [05:07<00:38, 12.79s/it]
rank2: 88%|████████▊ | 28/32 [05:09<00:38, 9.67s/it]
rank0: 97%|█████████▋| 31/32 [05:10<00:09, 9.86s/it]
rank1: 94%|█████████▍| 30/32 [05:16<00:23, 11.58s/it]
rank0: 100%|██████████| 32/32 [05:19<00:00, 9.40s/it]
rank0: 100%|██████████| 32/32 [05:19<00:00, 9.97s/it]
rank3: 91%|█████████ | 29/32 [05:18<00:37, 12.34s/it]/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
rank2: 91%|█████████ | 29/32 [05:25<00:35, 11.73s/it]
rank3: 94%|█████████▍| 30/32 [05:29<00:23, 11.98s/it]
rank1: 97%|█████████▋| 31/32 [05:33<00:13, 13.07s/it]
rank2: 94%|█████████▍| 30/32 [05:35<00:21, 10.95s/it]
rank1: 100%|██████████| 32/32 [05:40<00:00, 11.44s/it]
rank1: 100%|██████████| 32/32 [05:40<00:00, 10.65s/it]
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
rank2: 97%|█████████▋| 31/32 [05:42<00:09, 9.82s/it]
rank3: 97%|█████████▋| 31/32 [05:46<00:13, 13.64s/it]
rank2: 100%|██████████| 32/32 [05:52<00:00, 9.89s/it]
rank2: 100%|██████████| 32/32 [05:52<00:00, 11.01s/it]
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
rank3: 100%|██████████| 32/32 [05:55<00:00, 12.01s/it]
rank3: 100%|██████████| 32/32 [05:55<00:00, 11.10s/it]
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
|