granite-4.1-30b-mxfp8-mlx

Brainwaves

         arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.456,0.572,0.897,0.621,0.444,0.757,0.616
mxfp4    0.453,0.565,0.892,0.624,0.442,0.759,0.585
qx86-hi  0.451,0.568,0.897,0.636,0.440,0.763,0.598
qx64-hi  0.462,0.582,0.896,0.642,0.448,0.769,0.600

Quant    Perplexity      Peak Memory   Tokens/sec
mxfp8   10.150 ± 0.120   33.36 GB      188
mxfp4   10.353 ± 0.119   18.93 GB      164
qx86-hi 10.040 ± 0.121   32.24 GB      158

This model granite-4.1-30b-mxfp8-mlx was converted to MLX format from ibm-granite/granite-4.1-30b using mlx-lm version 0.31.3.

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("granite-4.1-30b-mxfp8-mlx")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_dict=False,
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)