README.md

---
library_name: transformers
license: apache-2.0
language:
- en
base_model:
- allenai/scibert_scivocab_uncased
pipeline_tag: text-classification
---

# Model Card for Model ID


This is a text classification model. 
It was fine-tuned to predict certainty ratings of scientific findings using a classification loss and a ranking loss. 
We fine-tuned an allenai/scibert_scivocab_uncased on the dataset made available by [Wurl et al (2024): Understanding Fine-Grained Distortions in Reports for Scientific Finding.](https://aclanthology.org/2024.findings-acl.369/).


## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

- **Developed by:** Researchers at UCI with the goal of obtaining a reliable certainty scoring function.
- **Model type:** BERT
- **Language(s) (NLP):** English
- **Finetuned from model:** allenai/scibert_scivocab_uncased

## Uses

The model is meant to be used for estimating certainty scores. Because it is trained on sentence-level academic findings, we suspect its reliability to be restricted to this domain. 
The original dataset had only moderate inter-annotator agreement (spearman correlation coefficient of 0.44), which suggests that predicting certainty scores is difficult even for humans. 
We recommend users of this model to validate that the model behaves as intended in a small portion of the data of interest before scaling evaluations. 
We also note that the per-class F1 scores ranged between (0.48-0.70), which reflects once again the difficulty in learning clear class boundaries.


## How to Get Started with the Model

Use the code below to get started with the model.

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("Cbelem/scibert-certainty-classif")
model = AutoModelForSequenceClassification.from_pretrained("Cbelem/scibert-certainty-classif")
model.eval()

texts = [
    "Compared with controls, taxi drivers had greater grey matter volume in the posterior hippocampi (Maguire et al.",
    "The study described in this paper focuses on gaze, but similar approaches can be used to understand the effects of other interactions that contribute to patient outcomes such as emotion.",
    '""The initial findings could have been explained by a correlation, that people with big hippocampi become taxi drivers,"" he says.',
    "We are less sure about a possible explanation for lower acceptance for mobile phone behaviors among professionals in the West.",
]

inputs_ids = tokenizer(texts, return_tensors="pt")
model(**inputs_ids)
```

## Training Details

### Training Data

TBD

### Training Procedure

TBD

#### Preprocessing [optional]

TBD


#### Training Hyperparameters

- **Training regime:** fp32

## Evaluation


### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

#### Metrics

TBD 

### Results

```
  "train/learning_rate": 6.869747470432602e-7,
  "train/loss": 0.562,
  "train/global_step": 3000,
  "eval/qwk": 0.5507,
  "eval/loss": 0.9391,
  "eval/accuracy": 0.6078,
  "eval/balanced_accuracy": 0.3980,
  "eval/f1_macro": 0.6006,
  "eval/f1_class_0": 0.6211,
  "eval/f1_class_1": 0.4932,
  "eval/f1_class_2": 0.6875,
  "eval/precision_macro": 0.6033,
  "eval/precision_class_0": 0.6410,
  "eval/precision_class_1": 0.5,
  "eval/precision_class_2": 0.6689,
  "eval/recall_macro": 0.5987,
  "eval/recall_class_0": 0.6024,
  "eval/recall_class_1": 0.4865,
  "eval/recall_class_2": 0.7071,
  "train_steps_per_second": 6.532,
```


#### Summary


## Technical Specifications [optional]

### Model Architecture and Objective

TBD

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

Transformers, Pytorch, Wandb for running the hyperparameter sweep

## Citation

TBD



## Model Card Authors

Catarina Belem (Cbelem)

## Model Card Contact

For more information contact cbelem@uci.edu.