Matches in SemOpenAlex for { <https://semopenalex.org/work/W4286959591> ?p ?o ?g. }
Showing items 1 to 54 of
54
with 100 items per page.
- W4286959591 abstract "In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g. sentence similarity and paraphrase identification). Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders. Bi-encoders produce fixed-dimensional sentence representations and are computationally efficient, however, they usually underperform cross-encoders. Cross-encoders can leverage their attention heads to exploit inter-sentence interactions for better performance but they require task fine-tuning and are computationally more expensive. In this paper, we present a completely unsupervised sentence representation model termed as Trans-Encoder that combines the two learning paradigms into an iterative joint framework to simultaneously learn enhanced bi- and cross-encoders. Specifically, on top of a pre-trained Language Model (PLM), we start with converting it to an unsupervised bi-encoder, and then alternate between the bi- and cross-encoder task formulations. In each alternation, one task formulation will produce pseudo-labels which are used as learning signals for the other task formulation. We then propose an extension to conduct such self-distillation approach on multiple PLMs in parallel and use the average of their pseudo-labels for mutual-distillation. Trans-Encoder creates, to the best of our knowledge, the first completely unsupervised cross-encoder and also a state-of-the-art unsupervised bi-encoder for sentence similarity. Both the bi-encoder and cross-encoder formulations of Trans-Encoder outperform recently proposed state-of-the-art unsupervised sentence encoders such as Mirror-BERT and SimCSE by up to 5% on the sentence similarity benchmarks." @default.
- W4286959591 created "2022-07-25" @default.
- W4286959591 creator A5023368832 @default.
- W4286959591 creator A5026154387 @default.
- W4286959591 creator A5056549528 @default.
- W4286959591 creator A5065341276 @default.
- W4286959591 creator A5076265623 @default.
- W4286959591 date "2021-09-27" @default.
- W4286959591 modified "2023-09-26" @default.
- W4286959591 title "Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations" @default.
- W4286959591 hasPublicationYear "2021" @default.
- W4286959591 type Work @default.
- W4286959591 citedByCount "0" @default.
- W4286959591 crossrefType "posted-content" @default.
- W4286959591 hasAuthorship W4286959591A5023368832 @default.
- W4286959591 hasAuthorship W4286959591A5026154387 @default.
- W4286959591 hasAuthorship W4286959591A5056549528 @default.
- W4286959591 hasAuthorship W4286959591A5065341276 @default.
- W4286959591 hasAuthorship W4286959591A5076265623 @default.
- W4286959591 hasBestOaLocation W42869595911 @default.
- W4286959591 hasConcept C111919701 @default.
- W4286959591 hasConcept C118505674 @default.
- W4286959591 hasConcept C153083717 @default.
- W4286959591 hasConcept C153180895 @default.
- W4286959591 hasConcept C154945302 @default.
- W4286959591 hasConcept C204321447 @default.
- W4286959591 hasConcept C2777530160 @default.
- W4286959591 hasConcept C28490314 @default.
- W4286959591 hasConcept C41008148 @default.
- W4286959591 hasConceptScore W4286959591C111919701 @default.
- W4286959591 hasConceptScore W4286959591C118505674 @default.
- W4286959591 hasConceptScore W4286959591C153083717 @default.
- W4286959591 hasConceptScore W4286959591C153180895 @default.
- W4286959591 hasConceptScore W4286959591C154945302 @default.
- W4286959591 hasConceptScore W4286959591C204321447 @default.
- W4286959591 hasConceptScore W4286959591C2777530160 @default.
- W4286959591 hasConceptScore W4286959591C28490314 @default.
- W4286959591 hasConceptScore W4286959591C41008148 @default.
- W4286959591 hasLocation W42869595911 @default.
- W4286959591 hasOpenAccess W4286959591 @default.
- W4286959591 hasPrimaryLocation W42869595911 @default.
- W4286959591 hasRelatedWork W12023667 @default.
- W4286959591 hasRelatedWork W1266808 @default.
- W4286959591 hasRelatedWork W13336246 @default.
- W4286959591 hasRelatedWork W13343246 @default.
- W4286959591 hasRelatedWork W13618705 @default.
- W4286959591 hasRelatedWork W13799842 @default.
- W4286959591 hasRelatedWork W14719935 @default.
- W4286959591 hasRelatedWork W3599679 @default.
- W4286959591 hasRelatedWork W4538592 @default.
- W4286959591 hasRelatedWork W5688477 @default.
- W4286959591 isParatext "false" @default.
- W4286959591 isRetracted "false" @default.
- W4286959591 workType "article" @default.