NyayaLM: Qwen3 Legal LoRA Model
Model Card
Model Details
- Base Model:
unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit - Finetuned By: chhatramani
- License: Apache-2.0
- Language: English & Nepali (Legal domain)
- Frameworks: Unsloth, TRL, Transformers
LoRA Configuration
- LoRA Rank (r): 32
- Target Modules:
q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj - LoRA Alpha: 32
- LoRA Dropout: 0
- Bias: none
- Gradient Checkpointing: enabled (
unsloth) - Random State: 3407
Dataset
The model was trained on a mixed bilingual legal dataset:
- Legal Datasets:
- Nepal Civil Code (English)
- Muluki Dewani (Nepali)
- General Instruction Datasets (downsampled to 10% of legal size):
- Nepali Alpaca Instruction Dataset
- English Alpaca Instruction Dataset
Final Dataset Size:
- Training samples: ~3145
- Evaluation samples: ~350
Hyperparameters
- Max Sequence Length: 2048
- Batch Size (train): 4
- Batch Size (eval): 2
- Gradient Accumulation Steps: 4
- Warmup Ratio: 0.1
- Learning Rate: 1e-4
- Optimizer: AdamW (8-bit)
- Weight Decay: 0.01
- Scheduler: Cosine
- Epochs: 1
- Seed: 3407
- Evaluation Strategy: every 40 steps
Traning Loss
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1 \ /| Num examples = 3,145 | Num Epochs = 1 | Total steps = 197 O^O/ _/ \ Batch size per device = 4 | Gradient accumulation steps = 4 \ / Data Parallel GPUs = 1 | Total batch size (4 x 4 x 1) = 16 "-____-" Trainable parameters = 66,060,288 of 4,088,528,384 (1.62% trained) Unsloth: Will smartly offload gradients to save VRAM! [197/197 54:51, Epoch 1/1]
| Step | Training Loss | Validation Loss |
|---|---|---|
| 40 | 0.749500 | 0.827659 |
| 80 | 0.591000 | 0.742231 |
| 120 | 0.663600 | 0.718418 |
| 160 | 0.632600 | 0.707453 |
Evaluation Metrics
The model was evaluated on a held-out test set of ~350 samples.
Results: EVALUATION RESULTS
ROUGE-1: 0.5491 ROUGE-2: 0.2151 ROUGE-L: 0.4245 BERTScore F1: 0.9042
Intended Use
NyayaLM is designed as a legal assistant for Nepali law, capable of answering questions, summarizing statutes, and providing structured legal reasoning.
Limitations
- Focused on Nepali and English legal texts;
- LoRA adapters only; full model weights not included.
- Evaluation metrics are based on a small test set and may not generalize.
Uploaded model
- Developed by: chhatramani
- License: apache-2.0
- Finetuned from model : unsloth/qwen3-4b-instruct-2507-unsloth-bnb-4bit
This qwen3 model was trained 2x faster with Unsloth
