README.md

---
license: mit
base_model:
  - FacebookAI/xlm-roberta-large
language:
  - ru
tags:
  - reasoning
  - logical-analysis
  - text-classification
  - ai-safety
  - evaluation
  - judge-model
  - argumentation
pipeline_tag: text-classification
---

# RQA — Reasoning Quality Analyzer (R2)

**RQA-R2** is a **judge model** for reasoning-quality evaluation.  
It does **not** generate, rewrite, or explain text. Instead, it determines whether a text contains a reasoning problem, whether that problem is **hidden** or **explicit**, and which explicit error types are present.

> RQA is a judge, not a teacher and not a generator.

---

## What Is New in R2 Compared to R1

R2 is not just a retrain of R1. It is a full methodological upgrade.

### Core differences

- **R1** used a more limited 2-signal setup.
- **R2** uses a strict **3-head ontology**:
  - `has_issue`
  - `is_hidden`
  - `error_types`

### Key improvements in R2

- explicit hidden-problem modeling instead of weaker implicit logic
- strict `logical / hidden / explicit` inference contract
- honest `train / val / calib / test` split
- separate calibration split for temperatures and thresholds
- per-class thresholds for error types
- uncertainty-aware inference with `status=uncertain` and `review_required`
- duplicate and conflict-duplicate filtering in the loader
- truncation audit and richer evaluation reports
- better optimizer setup for transformer fine-tuning
- staged encoder fine-tuning with freeze/unfreeze
- stronger schema/version safety for inference artifacts

In short:

> **R1** was a strong prototype.  
> **R2** is the first version that behaves like a full training + calibration + inference pipeline.

---

## What Problem RQA-R2 Solves

Texts written by humans or LLMs can:

- sound coherent
- use correct vocabulary
- appear persuasive

...while still containing **reasoning problems** that are:

- subtle
- structural
- hidden in argumentation

RQA-R2 focuses specifically on **reasoning quality**, not on style, grammar, sentiment, or factual correctness.

---

## Model Overview

| Property | Value |
|---|---|
| Model Type | Judge / Evaluator |
| Base Encoder | [XLM-RoBERTa Large](https://huggingface.co/FacebookAI/xlm-roberta-large) |
| Pooling | Mean pooling |
| Heads | 3 (`has_issue`, `is_hidden`, `error_types`) |
| Language | Russian |
| License | MIT |

---

## What the Model Predicts

RQA-R2 predicts three connected outputs.

### 1. Logical Issue Detection

- `has_logical_issue ∈ {false, true}`
- calibrated probability available

### 2. Hidden Problem Detection

- `is_hidden_problem ∈ {false, true}`
- evaluated only when a reasoning issue exists

### 3. Explicit Error Type Classification

If the text is classified as `explicit`, the model may assign one or more of the following error types:

- `false_causality`
- `unsupported_claim`
- `overgeneralization`
- `missing_premise`
- `contradiction`
- `circular_reasoning`

This is a **multi-label** prediction head.

---

## Ontology

R2 uses a strict three-class reasoning ontology.

### `logical`

- no reasoning issue
- no hidden problem
- no explicit errors

### `hidden`

- reasoning problem exists
- no explicit labeled fallacy
- the issue is structural, implicit, or argumentative

### `explicit`

- reasoning problem exists
- at least one explicit error type is present

This ontology is enforced in both training and inference.

---

## Inference Contract

RQA-R2 uses gated inference:

- if `has_issue = false` -> class is `logical`, no errors are returned
- if `has_issue = true` and `is_hidden = true` -> class is `hidden`, no explicit errors are returned
- if `has_issue = true` and `is_hidden = false` -> class is `explicit`, explicit errors may be returned

R2 also supports:

- calibrated thresholds
- `uncertain` mode
- `review_required` for borderline cases

---

## Architecture

RQA-R2 is built on top of **XLM-RoBERTa Large** with:

- mean pooling
- separate projections per task
- separate dropout per head
- 3 task-specific heads
- uncertainty-weighted multi-task training

Training is hierarchical:

- `has_issue` is trained on all samples
- `is_hidden` is trained only on problem samples
- `error_types` are trained only on explicit samples

---

## Training and Calibration

R2 uses an honest experimental structure:

- `train` for fitting
- `val` for model selection
- `calib` for temperature scaling and threshold tuning
- `test` for final held-out evaluation

Calibration includes:

- issue temperature
- hidden temperature
- per-class error temperatures
- threshold selection for `has_issue`
- threshold selection for `is_hidden`
- per-class thresholds for error types

---

## Held-Out Synthetic Benchmark

The following metrics were obtained on the current held-out synthetic test split used for R2:

- `Issue`: `F1 = 0.988`, `FPR = 0.029`, `PR-AUC = 0.999`
- `Hidden`: `F1 = 0.960`, `PR-AUC = 0.994`
- `Errors`: `macro-F1 = 0.822`, `micro-F1 = 0.813`, `samples-F1 = 0.838`
- `Top-level class macro-F1 = 0.964`
- `Coverage = 95.6%`
- `Uncertain rate = 4.4%`

These are strong results for the current data regime.

Important:

> These metrics are measured on a held-out split from the current synthetic dataset.  
> They demonstrate that the R2 design works very well in-distribution, but they should not be interpreted as proof of universal real-world reasoning performance.

---

## Training Data

RQA-R2 was trained on a custom reasoning-quality dataset with:

- `7292` total samples
- `3150` logical texts
- `4142` problematic texts
- `1242` hidden problems
- `2900` explicit cases

Error-label counts:

- `false_causality`: `518`
- `unsupported_claim`: `524`
- `overgeneralization`: `599`
- `missing_premise`: `537`
- `contradiction`: `475`
- `circular_reasoning`: `540`

Multi-label explicit cases:

- `293`

The current dataset is useful and already strong enough for training and benchmarking R2, but it is still primarily **synthetic** and should be expanded with real-world data in future versions.

---

## Intended Use

### Recommended for

- reasoning-quality evaluation
- LLM output auditing
- AI safety pipelines
- judge/reranker pipelines
- pre-filtering for downstream review
- analytical tooling around argument structure

### Not intended for

- text generation
- explanation generation
- automatic rewriting or correction
- factual verification
- legal or scientific truth adjudication

---

## Output Example

```json
{
  "class": "explicit",
  "status": "ok",
  "review_required": false,
  "has_logical_issue": true,
  "has_issue_probability": 0.9993,
  "is_hidden_problem": false,
  "hidden_probability": 0.021,
  "errors": [
    {
      "type": "missing_premise",
      "probability": 0.923,
      "threshold": 0.54
    }
  ]
}
```

---

## Limitations

RQA-R2 still has important limits:

- it evaluates reasoning structure, not factual truth
- hidden problems remain partly subjective by nature
- the current benchmark is still synthetic and in-distribution
- real human-written texts and outputs from other LLMs may be harder
- the model should still be validated externally before being treated as a fully general reasoning judge

Also note:

- R2 was optimized toward low false positives, but on the current held-out synthetic test set the observed `Issue FPR` is `2.9%`, not `1.0%`
- if strict false-positive control is critical, threshold tuning may need to be tightened further for the target deployment environment

---

## Recommended Next Step

The best next step after R2 is external validation on:

- human-written argumentative texts
- outputs from other LLM families
- paraphrased and adversarially reworded samples
- harder hidden-problem cases

That is the correct way to turn a strong in-distribution result into a robust real-world system.

---

## Summary

RQA-R2 is a major upgrade over R1:

- better ontology
- better training logic
- better calibration
- better inference safety
- stronger held-out synthetic performance

R1 proved the idea.  
**R2 is the first version that fully validates it.**