Ben Kelly PRO

How bad is the precision loss on the Q2_K_XL quants? I can run that with full GPU offload but I usually dont run lower than Q4, broadly.

all this popularity is going to my head. i'm getting a big opinion of myself as a thought leader among my army of 9

glad to see all 9 of my followers remain intact and human.

I run it on threadripper 3970x with 256gb system ram and offloading computation layers to a gtx 1660 6gb vram. Using llama.cpp with -nkvo -kvu and all MoE on CPU. With an amazing speed on 14/TpS generation speed using q8_0. I’m amazed

Leo, I am actually curating an entire collection of "Stuff that runs well on a GTX 1660 Super 6GB. I have a small cluster of them at the office that I use for light inference tasks / synthetic data, and I'm always looking for more use-cases for this exact GPU. https://huggingface.co/collections/YellowjacketGames/models-gtx-1660-super-6gb

14 TPS with CPU offload is remarkable. I have a 39xx threadripper with a bunch of RAM also, and i've never seen higher than like, 2 TPS, on a cpu offloaded model.

I wonder if thats just me not taking the time to customize it more in LM Studio .

I'm working on my orchestrator right now, and at some point I'll stop my GPUs from spamming Flux2Dev and actually try and get some coding work done on them :)

Question if i may:

I can run a higher precision quant via nvlink and 98gb shaerd memory.

OR

I can run a lower precision quant on a single card, and run a 2nd copy of the model on the second card.

I'm leaning towards "two models, smaller quants", but if you have any specific advice, i know i have a lot to learn