Matches in SemOpenAlex for { <https://semopenalex.org/work/W3202797376> ?p ?o ?g. }
Showing items 1 to 88 of
88
with 100 items per page.
- W3202797376 abstract "In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g. sentence similarity and paraphrase identification). Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders. Bi-encoders produce fixed-dimensional sentence representations and are computationally efficient, however, they usually underperform cross-encoders. Cross-encoders can leverage their attention heads to exploit inter-sentence interactions for better performance but they require task fine-tuning and are computationally more expensive. In this paper, we present a completely unsupervised sentence representation model termed as Trans-Encoder that combines the two learning paradigms into an iterative joint framework to simultaneously learn enhanced bi- and cross-encoders. Specifically, on top of a pre-trained Language Model (PLM), we start with converting it to an unsupervised bi-encoder, and then alternate between the bi- and cross-encoder task formulations. In each alternation, one task formulation will produce pseudo-labels which are used as learning signals for the other task formulation. We then propose an extension to conduct such self-distillation approach on multiple PLMs in parallel and use the average of their pseudo-labels for mutual-distillation. Trans-Encoder creates, to the best of our knowledge, the first completely unsupervised cross-encoder and also a state-of-the-art unsupervised bi-encoder for sentence similarity. Both the bi-encoder and cross-encoder formulations of Trans-Encoder outperform recently proposed state-of-the-art unsupervised sentence encoders such as Mirror-BERT and SimCSE by up to 5% on the sentence similarity benchmarks." @default.
- W3202797376 created "2021-10-11" @default.
- W3202797376 creator A5023368832 @default.
- W3202797376 creator A5026154387 @default.
- W3202797376 creator A5062423359 @default.
- W3202797376 creator A5065341276 @default.
- W3202797376 creator A5076265623 @default.
- W3202797376 date "2021-09-27" @default.
- W3202797376 modified "2023-09-26" @default.
- W3202797376 title "Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations" @default.
- W3202797376 cites W1486649854 @default.
- W3202797376 cites W1840435438 @default.
- W3202797376 cites W2126400076 @default.
- W3202797376 cites W2133458109 @default.
- W3202797376 cites W2152180407 @default.
- W3202797376 cites W2250790822 @default.
- W3202797376 cites W2251861449 @default.
- W3202797376 cites W2462305634 @default.
- W3202797376 cites W2842511635 @default.
- W3202797376 cites W2908510526 @default.
- W3202797376 cites W2963310665 @default.
- W3202797376 cites W2963644595 @default.
- W3202797376 cites W2963748441 @default.
- W3202797376 cites W2963804993 @default.
- W3202797376 cites W2964222566 @default.
- W3202797376 cites W2995289474 @default.
- W3202797376 cites W3006051380 @default.
- W3202797376 cites W3035524453 @default.
- W3202797376 cites W3104033643 @default.
- W3202797376 cites W3105816068 @default.
- W3202797376 cites W3122838366 @default.
- W3202797376 cites W3152670030 @default.
- W3202797376 cites W3155121067 @default.
- W3202797376 cites W3164054899 @default.
- W3202797376 cites W3168875417 @default.
- W3202797376 cites W3172806051 @default.
- W3202797376 cites W3173188814 @default.
- W3202797376 cites W3173783447 @default.
- W3202797376 cites W3176047188 @default.
- W3202797376 cites W3213189520 @default.
- W3202797376 cites W3213730158 @default.
- W3202797376 doi "https://doi.org/10.48550/arxiv.2109.13059" @default.
- W3202797376 hasPublicationYear "2021" @default.
- W3202797376 type Work @default.
- W3202797376 sameAs 3202797376 @default.
- W3202797376 citedByCount "0" @default.
- W3202797376 crossrefType "posted-content" @default.
- W3202797376 hasAuthorship W3202797376A5023368832 @default.
- W3202797376 hasAuthorship W3202797376A5026154387 @default.
- W3202797376 hasAuthorship W3202797376A5062423359 @default.
- W3202797376 hasAuthorship W3202797376A5065341276 @default.
- W3202797376 hasAuthorship W3202797376A5076265623 @default.
- W3202797376 hasBestOaLocation W32027973761 @default.
- W3202797376 hasConcept C111919701 @default.
- W3202797376 hasConcept C118505674 @default.
- W3202797376 hasConcept C153083717 @default.
- W3202797376 hasConcept C153180895 @default.
- W3202797376 hasConcept C154945302 @default.
- W3202797376 hasConcept C204321447 @default.
- W3202797376 hasConcept C2777530160 @default.
- W3202797376 hasConcept C28490314 @default.
- W3202797376 hasConcept C41008148 @default.
- W3202797376 hasConceptScore W3202797376C111919701 @default.
- W3202797376 hasConceptScore W3202797376C118505674 @default.
- W3202797376 hasConceptScore W3202797376C153083717 @default.
- W3202797376 hasConceptScore W3202797376C153180895 @default.
- W3202797376 hasConceptScore W3202797376C154945302 @default.
- W3202797376 hasConceptScore W3202797376C204321447 @default.
- W3202797376 hasConceptScore W3202797376C2777530160 @default.
- W3202797376 hasConceptScore W3202797376C28490314 @default.
- W3202797376 hasConceptScore W3202797376C41008148 @default.
- W3202797376 hasLocation W32027973761 @default.
- W3202797376 hasOpenAccess W3202797376 @default.
- W3202797376 hasPrimaryLocation W32027973761 @default.
- W3202797376 hasRelatedWork W1519302135 @default.
- W3202797376 hasRelatedWork W1544285860 @default.
- W3202797376 hasRelatedWork W1567338489 @default.
- W3202797376 hasRelatedWork W159132833 @default.
- W3202797376 hasRelatedWork W1978971213 @default.
- W3202797376 hasRelatedWork W2485476290 @default.
- W3202797376 hasRelatedWork W2803871820 @default.
- W3202797376 hasRelatedWork W2887799774 @default.
- W3202797376 hasRelatedWork W3211389291 @default.
- W3202797376 hasRelatedWork W397704218 @default.
- W3202797376 isParatext "false" @default.
- W3202797376 isRetracted "false" @default.
- W3202797376 magId "3202797376" @default.
- W3202797376 workType "article" @default.