W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766] 
W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766] *****************************************
W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0226 04:01:15.642000 1093345 site-packages/torch/distributed/run.py:766] *****************************************
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once
[rank1]:[W226 04:01:30.449847470 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once
[rank3]:[W226 04:01:31.002032198 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 3]  using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once
[rank2]:[W226 04:01:31.256592374 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 2]  using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once
[rank0]:[W226 04:01:32.571792358 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
`torch_dtype` is deprecated! Use `dtype` instead!
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
`torch_dtype` is deprecated! Use `dtype` instead!
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:03,  1.22it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.27it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:03,  1.12it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.28it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.14it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:03,  1.09it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:03<00:00,  1.26it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.67it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.46it/s]

rank1:   0%|          | 0/32 [00:00<?, ?it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.14it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.13it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:03,  1.00it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:03<00:00,  1.16it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.15it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.58it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.35it/s]

Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.06it/s]
rank2:   0%|          | 0/32 [00:00<?, ?it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:03<00:00,  1.20it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.62it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.37it/s]

Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.11it/s]
rank0:   0%|          | 0/32 [00:00<?, ?it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:03<00:00,  1.16it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.58it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.32it/s]

rank3:   0%|          | 0/32 [00:00<?, ?it/s]
rank1:   3%|▎         | 1/32 [00:11<05:51, 11.34s/it]
rank2:   3%|▎         | 1/32 [00:10<05:28, 10.61s/it]
rank3:   3%|▎         | 1/32 [00:08<04:33,  8.83s/it]
rank0:   3%|▎         | 1/32 [00:11<05:44, 11.11s/it]
rank1:   6%|▋         | 2/32 [00:20<05:03, 10.13s/it]
rank2:   6%|▋         | 2/32 [00:20<05:08, 10.28s/it]
rank3:   6%|▋         | 2/32 [00:19<05:03, 10.11s/it]
rank0:   6%|▋         | 2/32 [00:21<05:20, 10.69s/it]
rank1:   9%|▉         | 3/32 [00:29<04:33,  9.43s/it]
rank3:   9%|▉         | 3/32 [00:28<04:33,  9.44s/it]
rank2:   9%|▉         | 3/32 [00:34<05:46, 11.95s/it]
rank0:   9%|▉         | 3/32 [00:35<05:47, 11.97s/it]
rank3:  12%|█▎        | 4/32 [00:36<04:13,  9.05s/it]
rank2:  12%|█▎        | 4/32 [00:43<05:01, 10.76s/it]
rank1:  12%|█▎        | 4/32 [00:45<05:35, 12.00s/it]
rank0:  12%|█▎        | 4/32 [00:44<05:11, 11.11s/it]
rank3:  16%|█▌        | 5/32 [00:47<04:23,  9.76s/it]
rank1:  16%|█▌        | 5/32 [00:53<04:51, 10.80s/it]
rank0:  16%|█▌        | 5/32 [00:51<04:21,  9.67s/it]
rank2:  16%|█▌        | 5/32 [00:54<04:48, 10.70s/it]
rank3:  19%|█▉        | 6/32 [00:57<04:13,  9.76s/it]
rank0:  19%|█▉        | 6/32 [01:01<04:14,  9.78s/it]
rank2:  19%|█▉        | 6/32 [01:04<04:35, 10.60s/it]
rank1:  19%|█▉        | 6/32 [01:09<05:27, 12.61s/it]
rank0:  22%|██▏       | 7/32 [01:13<04:17, 10.29s/it]
rank2:  22%|██▏       | 7/32 [01:14<04:18, 10.33s/it]
rank3:  22%|██▏       | 7/32 [01:12<04:48, 11.55s/it]
rank1:  22%|██▏       | 7/32 [01:20<04:58, 11.94s/it]
rank2:  25%|██▌       | 8/32 [01:27<04:26, 11.10s/it]
rank1:  25%|██▌       | 8/32 [01:28<04:16, 10.70s/it]
rank0:  25%|██▌       | 8/32 [01:26<04:31, 11.31s/it]
rank3:  25%|██▌       | 8/32 [01:30<05:19, 13.33s/it]
rank0:  28%|██▊       | 9/32 [01:36<04:06, 10.74s/it]
rank2:  28%|██▊       | 9/32 [01:38<04:20, 11.31s/it]
rank1:  28%|██▊       | 9/32 [01:40<04:13, 11.01s/it]
rank3:  28%|██▊       | 9/32 [01:39<04:38, 12.12s/it]
rank0:  31%|███▏      | 10/32 [01:43<03:33,  9.69s/it]
rank1:  31%|███▏      | 10/32 [01:49<03:47, 10.35s/it]
rank2:  31%|███▏      | 10/32 [01:49<04:05, 11.18s/it]
rank3:  31%|███▏      | 10/32 [01:48<04:05, 11.15s/it]
rank0:  34%|███▍      | 11/32 [01:55<03:39, 10.46s/it]
rank1:  34%|███▍      | 11/32 [01:59<03:35, 10.26s/it]
rank2:  34%|███▍      | 11/32 [01:58<03:38, 10.39s/it]
rank3:  34%|███▍      | 11/32 [01:56<03:33, 10.18s/it]
rank0:  38%|███▊      | 12/32 [02:05<03:21, 10.10s/it]
rank1:  38%|███▊      | 12/32 [02:07<03:13,  9.68s/it]
rank2:  38%|███▊      | 12/32 [02:08<03:28, 10.43s/it]
rank0:  41%|████      | 13/32 [02:14<03:05,  9.79s/it]
rank1:  41%|████      | 13/32 [02:16<03:02,  9.59s/it]
rank3:  38%|███▊      | 12/32 [02:13<04:05, 12.30s/it]
rank1:  44%|████▍     | 14/32 [02:25<02:49,  9.40s/it]
rank3:  41%|████      | 13/32 [02:22<03:35, 11.35s/it]
rank2:  41%|████      | 13/32 [02:25<03:56, 12.47s/it]
rank0:  44%|████▍     | 14/32 [02:26<03:11, 10.62s/it]
rank1:  47%|████▋     | 15/32 [02:34<02:36,  9.21s/it]
rank3:  44%|████▍     | 14/32 [02:33<03:18, 11.02s/it]
rank0:  47%|████▋     | 15/32 [02:35<02:51, 10.11s/it]
rank2:  44%|████▍     | 14/32 [02:43<04:09, 13.86s/it]
rank1:  50%|█████     | 16/32 [02:45<02:36,  9.79s/it]
rank0:  50%|█████     | 16/32 [02:43<02:33,  9.59s/it]
rank1:  53%|█████▎    | 17/32 [02:53<02:16,  9.09s/it]
rank3:  47%|████▋     | 15/32 [02:50<03:37, 12.81s/it]
rank2:  47%|████▋     | 15/32 [02:53<03:35, 12.70s/it]
rank0:  53%|█████▎    | 17/32 [02:53<02:24,  9.65s/it]
rank1:  56%|█████▋    | 18/32 [03:01<02:02,  8.71s/it]
rank3:  50%|█████     | 16/32 [02:59<03:06, 11.68s/it]
rank2:  50%|█████     | 16/32 [03:02<03:06, 11.64s/it]
rank0:  56%|█████▋    | 18/32 [03:02<02:11,  9.36s/it]
rank3:  53%|█████▎    | 17/32 [03:08<02:43, 10.93s/it]
rank1:  59%|█████▉    | 19/32 [03:11<02:01,  9.31s/it]
rank0:  59%|█████▉    | 19/32 [03:11<01:58,  9.14s/it]
rank2:  53%|█████▎    | 17/32 [03:15<03:03, 12.24s/it]
rank1:  62%|██████▎   | 20/32 [03:19<01:46,  8.91s/it]
rank3:  56%|█████▋    | 18/32 [03:18<02:29, 10.71s/it]
rank0:  62%|██████▎   | 20/32 [03:19<01:48,  9.05s/it]
rank3:  59%|█████▉    | 19/32 [03:26<02:10, 10.02s/it]
rank2:  56%|█████▋    | 18/32 [03:29<02:55, 12.55s/it]
rank0:  66%|██████▌   | 21/32 [03:30<01:44,  9.54s/it]
rank1:  66%|██████▌   | 21/32 [03:35<02:01, 11.02s/it]
rank3:  62%|██████▎   | 20/32 [03:36<02:00, 10.02s/it]
rank0:  69%|██████▉   | 22/32 [03:38<01:29,  8.92s/it]
rank2:  59%|█████▉    | 19/32 [03:40<02:40, 12.33s/it]
rank1:  69%|██████▉   | 22/32 [03:43<01:41, 10.18s/it]
rank0:  72%|███████▏  | 23/32 [03:46<01:17,  8.65s/it]
rank3:  66%|██████▌   | 21/32 [03:46<01:50, 10.03s/it]
rank1:  72%|███████▏  | 23/32 [03:51<01:25,  9.51s/it]
rank2:  62%|██████▎   | 20/32 [03:51<02:20, 11.68s/it]
rank0:  75%|███████▌  | 24/32 [03:56<01:12,  9.08s/it]
rank3:  69%|██████▉   | 22/32 [03:58<01:44, 10.42s/it]
rank1:  75%|███████▌  | 24/32 [04:04<01:22, 10.32s/it]
rank2:  66%|██████▌   | 21/32 [04:03<02:09, 11.77s/it]
rank2:  69%|██████▉   | 22/32 [04:10<01:44, 10.43s/it]
rank3:  72%|███████▏  | 23/32 [04:08<01:33, 10.40s/it]
rank0:  78%|███████▊  | 25/32 [04:12<01:18, 11.26s/it]
rank1:  78%|███████▊  | 25/32 [04:16<01:16, 10.90s/it]
rank2:  72%|███████▏  | 23/32 [04:19<01:30, 10.11s/it]
rank3:  75%|███████▌  | 24/32 [04:17<01:20, 10.08s/it]
rank0:  81%|████████▏ | 26/32 [04:21<01:03, 10.52s/it]
rank1:  81%|████████▏ | 26/32 [04:25<01:02, 10.44s/it]
rank2:  75%|███████▌  | 24/32 [04:29<01:20, 10.02s/it]
rank3:  78%|███████▊  | 25/32 [04:29<01:13, 10.52s/it]
rank0:  84%|████████▍ | 27/32 [04:30<00:50, 10.19s/it]
rank1:  84%|████████▍ | 27/32 [04:42<01:01, 12.28s/it]
rank0:  88%|████████▊ | 28/32 [04:40<00:39,  9.98s/it]
rank2:  78%|███████▊  | 25/32 [04:42<01:16, 10.88s/it]
rank3:  81%|████████▏ | 26/32 [04:43<01:08, 11.44s/it]
rank1:  88%|████████▊ | 28/32 [04:51<00:45, 11.48s/it]
rank2:  81%|████████▏ | 26/32 [04:51<01:02, 10.36s/it]
rank0:  91%|█████████ | 29/32 [04:51<00:30, 10.26s/it]
rank3:  84%|████████▍ | 27/32 [04:51<00:52, 10.47s/it]
rank2:  84%|████████▍ | 27/32 [04:59<00:48,  9.66s/it]
rank3:  88%|████████▊ | 28/32 [05:00<00:40, 10.21s/it]
rank0:  94%|█████████▍| 30/32 [05:03<00:21, 10.78s/it]
rank1:  91%|█████████ | 29/32 [05:07<00:38, 12.79s/it]
rank2:  88%|████████▊ | 28/32 [05:09<00:38,  9.67s/it]
rank0:  97%|█████████▋| 31/32 [05:10<00:09,  9.86s/it]
rank1:  94%|█████████▍| 30/32 [05:16<00:23, 11.58s/it]
rank0: 100%|██████████| 32/32 [05:19<00:00,  9.40s/it]
rank0: 100%|██████████| 32/32 [05:19<00:00,  9.97s/it]

rank3:  91%|█████████ | 29/32 [05:18<00:37, 12.34s/it]/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once

rank2:  91%|█████████ | 29/32 [05:25<00:35, 11.73s/it]
rank3:  94%|█████████▍| 30/32 [05:29<00:23, 11.98s/it]
rank1:  97%|█████████▋| 31/32 [05:33<00:13, 13.07s/it]
rank2:  94%|█████████▍| 30/32 [05:35<00:21, 10.95s/it]
rank1: 100%|██████████| 32/32 [05:40<00:00, 11.44s/it]
rank1: 100%|██████████| 32/32 [05:40<00:00, 10.65s/it]
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once

rank2:  97%|█████████▋| 31/32 [05:42<00:09,  9.82s/it]
rank3:  97%|█████████▋| 31/32 [05:46<00:13, 13.64s/it]
rank2: 100%|██████████| 32/32 [05:52<00:00,  9.89s/it]
rank2: 100%|██████████| 32/32 [05:52<00:00, 11.01s/it]
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once

rank3: 100%|██████████| 32/32 [05:55<00:00, 12.01s/it]
rank3: 100%|██████████| 32/32 [05:55<00:00, 11.10s/it]
/home/zxd283/miniconda3/envs/backdoor/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. 
  warnings.warn(  # warn only once