README.md

---
language:
- en
pipeline_tag: text-generation
tags:
- pytorch
- Mistral
---

## Model Details

We employ **Mistral-Base(7B)** as one of the base models to evaluate our proposed **Reward-Driven Selective Penalization for Preference Alignment Optimization (RSPO)** method. The model is trained for **one epoch** on the **UltraFeedback Binarized dataset** using **(RSPO)** method.

## How to use

#### Transformers AutoModelForCausalLM

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "li11111/Mistral-7B-Base-RSPO"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

terminators = [
    tokenizer.eos_token_id
]

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
```

## Experiment Parameters

| **Parameter**       | **Mistral-Base(7B)** |
| ------------------- | -------------------- |
| `GPU`               | 8×Ascend910B         |
| `beta`              | 0.01                 |
| `batch`             | 128                  |
| `learning_rate`     | 5e-7                 |
| `max_prompt_length` | 512                  |
| `max_length`        | 1024                 |
| `num_train_epochs`  | 1                    |
| `torch_dtype`       | `bfloat16`           |
| `warmup_ratio`      | 0.1                  |
| `β_w`               | 0.01                 |
| `β_l`               | 0.1                  |
| `λ`                 | 0.1                  |


## Training Data

We use the [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) dataset to train the Mistral Base model.  


## Benchmarks 

<table>
    <tr>
        <th>Method</th>
        <th colspan="3" style="text-align: center;">AlpacaEval 2.0</th>
    </tr>
    <tr>
        <th></th>
        <th>LC</th>
        <th>WR</th>
        <th>Avg. Len</th>
    </tr>
    <tr>
        <td><b>RSPO</b></td>
        <td><b>25.4</b></td>
        <td><b>23.7</b></td>
        <td>1873</td>
    </tr>
</table>



| **Method** | **GSM8K** | **ARC**   | **TQA**   | **MMLU**  | **IFEval** | **Avg.**  |
| ---------- | --------- | --------- | --------- | --------- | ---------- | --------- |
| **SFT**    | **42.61** | 55.97     | 28.15     | 57.17     | 36.59      | 44.10     |
| **DPO**    | 33.13     | 59.64     | 46.14     | 57.46     | 50.48      | 49.37     |
| **R-DPO**  | 30.10     | 56.06     | 40.64     | 58.48     | 53.24      | 47.70     |
| **SimPO**  | 33.59     | **60.15** | 43.45     | 58.25     | 52.98      | 49.68     |
| **WPO**    | 30.63     | 57.00     | 40.51     | 58.54     | **55.64**  | 48.46     |
| **RSPO**   | 37.45     | 57.94     | **47.25** | **58.58** | 55.04      | **51.25** |