SentenceTransformer based on hkunlp/instructor-xl

This is a sentence-transformers model finetuned from hkunlp/instructor-xl. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: hkunlp/instructor-xl
Maximum Sequence Length: 128 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'T5EncoderModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': False})
  (2): Dense({'in_features': 1024, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ahmedHamdi/ir-es-en-masked-instructor-xl")
# Run inference
sentences = [
    'Represent the plot: After leading the ORG team to the local ice hockey championship, coach PERSON returns to help them achieve something as seemingly impossible as winning ORG in GPE, an international championship where the young ORG will have the responsibility of representing GPE.',
    "Represent the plot: Former PERSON hockey coach PERSON is a star in the minor leagues, expected to reach ORG. However, a career-ending knee injury brings him back to the ORG district of GPE. GPE is offered a chance to coach a team representing GPE in the Junior Goodwill Games in GPE. He manages to reunite most of his former ORG players, while the ORG try to enact revenge for their humiliating loss two years earlier. Their plans are foiled by GPE, who leaves them tied up in their underpants. ORG consists of many of the old ORG, in addition to five new players with special talents. In GPE, the lure of celebrity distracts GPE, who begins to neglect the team for a luxurious lifestyle. The team wins easy victories over GPE and GPE and GPE in the double-elimination tournament. ORG and PERSON gain recognition for their enforcer skills, and are dubbed ORG. Backup goaltender PERSON asks GPE for a chance to play, but is told to wait as PERSON is on a hot streak. The team suffers an embarrassing 12–1 defeat against GPE, coached by ex-NHL player PERSON. USA plays badly, and star center PERSON is slashed in the wrist. Frustrated, GPE drives his players even harder, but they begin to suffer from complete exhaustion. Realizing the children are too tired to complete their school work or even stay awake in class, the team's tutor PERSON intervenes, cancelling practice and confronting GPE over his thoughtlessness. Once better rested, the players encounter a street hockey team who teaches them to play like the real ORG. GPE continues to suffer from the pressure until Jan, brother of ORG's mentor PERSON, visits and reminds him of his love for the game. In their match against GPE, GPE fails to arrive on time, forcing PERSON to tell the referee PERSON is the team's assistant coach. The team struggles, entering the third period tied, until GPE arrives and apologizes to the team for his behavior. Inspired by the true return of their coach, the players win the game with their signature ORG, and advance to the next round. The renewed GPE finally realizes PERSON's wrist injury and benches him despite his complaints. To fill the open roster spot, PERSON recruits street hockey player PERSON, whose unique knucklepuck – which rotates end over end rather than spinning around its centerline – secures GPE's victory over GPE, advancing them to the championship game for a rematch against GPE. PERSON's injury is healed only to find ORG with a full roster. Knowing the team needs PERSON's knucklepuck and PERSON's skill against GPE, PERSON gives up his own spot, cementing his leadership as true team captain. In the final game, the physically imposing GPE initially dominates as the ORG incur penalties: PERSON picks a fight with an opposing player, the ORG fight the entire GPE bench and PERSON an opposing player before he can check PERSON. An annoyed ORG observes, this isn't a hockey game, it's a circus. After a rousing locker room speech from GPE and new PERSON jerseys from Jan, the team emerges rejuvenated. The ORG tie the game with goals from ORG PERSON, and finally PERSON, who was targeted by GPE but disguised himself as PERSON to pull off a successful knucklepuck. The game is forced to go to a five-shot shootout. With a 4–3 score in favor of the ORG, PERSON, the tournament's leading scorer, is ORG final shooter. GPE replaces PERSON with PERSON, who has a faster glove. Gunnar fires a hard slapshot, and PERSON falls to the ice. The entire stadium waits in breathless anticipation as she opens her glove and drops the puck, revealing the game-winning save and the ORG’ triumph over GPE to win the tournament. The team returns to GPE, and sing PERSON's ORG around a campfire.",
    "Represent the plot: Elder PERSON, a young Mormon from GPE, GPE, is sent to GPE with three other missionaries to spread the faith. They move into an apartment next to openly gay party boy PERSON and his roommate PERSON, an aspiring singer. Christian and PERSON work as waiters at PERSON's, a trendy restaurant owned by retired actress PERSON. Christian makes a bet with his co-workers that he can seduce one of the ORG, and  soon comes to believe that PERSON, the least experienced missionary, is a closeted homosexual. PERSON and Christian become acquainted through several encounters in the apartment complex. When Christian accidentally injures himself, PERSON helps him indoors and cleans his wound. Christian attempts to seduce PERSON, but the hesitant ORG becomes upset by Christian's remark that sex doesn't have to mean anything. PERSON accuses him of being shallow and walks out. Worried that PERSON is correct, Christian joins a charity, delivering meals to people with AIDS.  He gains new insights through a friendship with one of the beneficiaries. PERSON meets and befriends PERSON, whose life partner has died, unaware of her connection to Christian. PERSON's fellow missionary, PERSON, has a cycling accident. Returning to his apartment, a distraught PERSON encounters Christian, who tries to comfort him with a hug. Both men are overwhelmed by their feelings and end up kissing, failing to notice the return of PERSON's roommates. PERSON is sent home in disgrace, leading Christian to confront ORG, who is angry that Christian corrupted PERSON for no reason. Christian admits that he initially just wanted to win a bet, but says it's not about that anymore. Seeing Christian's distress, ORG tells him that PERSON's flight has a five-hour layover in GPE. Christian finds PERSON standing outside the airport terminal. Christian confesses his love, and despite his misgivings, PERSON admits his own feelings of love. With all flights canceled due to a snowstorm, Christian and PERSON spend an intimate night in a motel. When Christian awakes, he finds PERSON gone. PERSON's pocket watch, a family heirloom, has been left behind. Christian returns to GPE. In GPE, PERSON is excommunicated by the church elders, led by his father, who is the stake president. PERSON is rejected by his father and scolded by his mother, who tells him that he must pray for forgiveness. When PERSON suggests that he might be gay, his mother slaps him. Overwhelmed by despair, PERSON attempts suicide. His parents send him to a treatment facility in an attempt to change his sexual orientation via conversion therapy. Christian locates PERSON's home address and phone number. PERSON's mother informs him that, Thanks to you, my son took a razor to his wrists; thanks to you I have lost my son. PERSON is dead, Christian spends the next few days thinking continually about PERSON. Christian travels to the PERSON home in GPE, where he tearfully returns PERSON's watch to his mother. After looking at the watch's inscription, which mentions charity, she tries to go after Christian, but he has already left. PERSON writes a song based on Christian's journal entry about the ordeal. PERSON shows Christian her video for the song, and he is upset that PERSON used his writings without his consent. PERSON tells Christian that she hoped something good would come from it. In the treatment facility, PERSON hears PERSON's song when her video airs on television. The video prompts PERSON to return to GPE in search of Christian. When a stranger answers the door to Christian's apartment, PERSON is heartbroken, thinking that Christian has returned to his party boy ways and moved on. Having nowhere else to go, PERSON makes his way to PERSON's restaurant and begins to recount his journey to her. Christian walks into the dining room, and is overjoyed to see PERSON alive. They reconcile and later celebrate Thanksgiving with Christian's co-workers. PERSON tells everyone that, no matter what, they will always have, a place at my table, and a place in my heart.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7188, 0.1909],
#         [0.7188, 1.0000, 0.1010],
#         [0.1909, 0.1010, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 2,336 training samples
Columns: sentence_0 and sentence_1
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1
type string string
details
min: 9 tokens
mean: 105.18 tokens
max: 128 tokens

min: 19 tokens
mean: 122.31 tokens
max: 128 tokens

	sentence_0	sentence_1
type	string	string
details	min: 9 tokens mean: 105.18 tokens max: 128 tokens	min: 19 tokens mean: 122.31 tokens max: 128 tokens

Samples:

sentence_0	sentence_1
Represent the plot: In GPE, during 1988, PERSON (PERSON) manages "The LOC," a nightclub frequented by Russian mobster and drug trafficker PERSON (PERSON). PERSON has become estranged from his father, PERSON (PERSON), and his brother, PERSON (PERSON), both prominent members of the ORG, and lives a hedonistic, lawless life with his girlfriend ORG (ORG). When a police unit led by PERSON raids "The LOC," hoping to capture ORG, PERSON refuses to cooperate. This incident further strains his relationship with his father and brother, who disapprove of his behavior, and during his time at the police station, he and PERSON get into a fight. The police operation against ORG is unsuccessful, and he, now free, decides to take revenge by killing PERSON. The assassination attempt fails, and PERSON survives, though his critical condition requires him to remain hospitalized for months. PERSON, deeply affected by what happened, changes his approach and decides to help the police by working undercover to...	Represent the plot: In GPE, GPE in November 1988, PERSON is the manager of the ORG nightclub in GPE, owned by his boss, fur importer PERSON, whose nephew, mobster PERSON, is a patron of the joint. Estranged from his father PERSON (PERSON), an NYPD Deputy Chief, and brother PERSON, a newly-minted Captain, he uses his late mother Carol's maiden name, PERSON, as his alias and hangs out with his mate PERSON and best friend PERSON, aiming to soon own a club in GPE. PERSON, recently appointed to be in charge of a newly-formed anti-drug unit, warns PERSON he will lead a bust on the spot, hoping to net ORG. PERSON is locked up for possessing drugs and resisting arrest in the raid on November 22, 1988, souring his relations with PERSON and PERSON, who bail him out the next morning; the brothers then come to blows in a harsh feud. That evening, a masked ORG, whom the police fail to convict, shoots PERSON in the face outside of his house and firebombs his car, hospitalizing him at ORG for 4 month...
Represent the plot: PERSON is an unsociable bank employee living in the small town of GPE, GPE. PERSON prefers to keep a low profile to hide his secret: he suffers from multiple personality disorder as a result of trauma stemming from abuse at the hands of his mother. His PERSON is a woman, PERSON, who does the housework and prepares his breakfast each morning before PERSON starts his day. One day, while he is in the yard dressed as ORG, a runaway train caboose crashes into PERSON's back garden. When his neighbors gather to observe the scene, "PERSON" enters the house and exposes PERSON's secret life, forcing him to tell his neighbors that PERSON is his wife and that they had secretly married. Compelled to make the town believe that he and his alter ego are husband and wife, PERSON and PERSON must maintain their secret while under public scrutiny. PERSON, a young single mother facing many financial difficulties, holds the key to PERSON's past and develops a battle between personali...	Represent the plot: PERSON (PERSON), a quiet bank clerk living alone in tiny GPE, GPE, prefers to live an invisible life in order to hide his secret: He has dissociative identity disorder, the implied result of childhood trauma inflicted by his abusive mother. His other identity is a woman, PERSON, who each morning does his chores and cooks him breakfast before he starts the day. One day while he is using the outside yard clothesline as ORG, a freight train caboose derails and crashes into PERSON's backyard. When his neighbors come to the scene, PERSON enters his house, putting PERSON's other life into the spotlight, so he is forced to tell his neighbors that PERSON is his wife, married in secrecy. Forced to fool the town into believing PERSON and PERSON are husband and wife, they must maintain their secret while in public view. The town mayor, PERSON (PERSON), and his wife PERSON (PERSON) come to see PERSON try to host a political rally in his backyard in support of the incumbent cand...
`Represent the plot: An American drug lord wants to retire and sell his marijuana empire in GPE, which causes conflicts with rivals, a boxing trainer, and a very meticulous private investigator.`	Represent the plot: Big Dave, editor of ORG tabloid, is snubbed by cannabis baron PERSON at a party and hires private detective PERSON to investigate Pearson's links to Lord Pressfield. Pressfield, a duke, has a heroin-addicted daughter named PERSON. Fletcher offers to sell his findings (typed up as a screenplay entitled PERSON) to Pearson's right-hand man, PERSON, for £20 million. Born into poverty in the GPE, Pearson won a scholarship to GPE, where he began selling PERSON before dropping out and building his criminal empire. He now plans to sell his business to American billionaire PERSON for £400m and retire peacefully with his wife, Rosalind. Pearson shows PERSON one of the labs where he grows cannabis under the estates of aristocratic landlords who need cash for the upkeep of their stately homes. Pearson is later approached by ORG, an underboss for Chinese gangster Lord George. Dry Eye offers to buy out Pearson's business, but he refuses. ORG's lab is raided by amateur ORG fighter...

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 3
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
1.7123	500	0.6445

Framework Versions

Python: 3.9.18
Sentence Transformers: 5.1.2
Transformers: 4.57.6
PyTorch: 2.8.0+cu128
Accelerate: 1.10.1
Datasets: 4.5.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}