LLM360

/

CrystalChat

like 37

Follow

LLM360 517

I'm trying to run the provided example of using the model, but I keep getting this error:

Traceback (most recent call last):
  File "/home/bandeze/Ethical_AI_Stuff/./crystal_chat.py", line 12, in <module>
    gen_tokens = model.generate(input_ids, do_sample=True, max_length=400)
  File "/home/bandeze/Ethical_AI_Stuff/ai_venv/lib/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/home/bandeze/Ethical_AI_Stuff/ai_venv/lib/python3.13/site-packages/transformers/generation/utils.py", line 2566, in generate
    result = decoding_method(
        self,
    ...<5 lines>...
        **model_kwargs,
    )
  File "/home/bandeze/Ethical_AI_Stuff/ai_venv/lib/python3.13/site-packages/transformers/generation/utils.py", line 2786, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/home/bandeze/Ethical_AI_Stuff/ai_venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/bandeze/Ethical_AI_Stuff/ai_venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/bandeze/.cache/huggingface/modules/transformers_modules/LLM360/CrystalChat/18091ea1d89f8c6c18d3a90eb86c8324bb09a6f6/modeling_crystalcoder.py", line 1271, in forward
    transformer_outputs = self.transformer(
        input_ids,
    ...<11 lines>...
        return_dict=return_dict,
    )
  File "/home/bandeze/Ethical_AI_Stuff/ai_venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/bandeze/Ethical_AI_Stuff/ai_venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/bandeze/.cache/huggingface/modules/transformers_modules/LLM360/CrystalChat/18091ea1d89f8c6c18d3a90eb86c8324bb09a6f6/modeling_crystalcoder.py", line 981, in forward
    past_length = past_key_values[0][0].size(-2)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'size'

I'm very new to this, so I don't know how to fix this.
I'm using Python 3.13.7, and the only change I made to the script is forcing it to use the CPU, since my NVIDIA card doesn't have enough VRAM (I have a good amount of DRAM, thankfully).
What do I do?