Matches in SemOpenAlex for { <https://semopenalex.org/work/W4221167891> ?p ?o ?g. }
Showing items 1 to 79 of
79
with 100 items per page.
- W4221167891 abstract "Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information. Somewhat counter-intuitively, some of these studies also report that position embeddings appear to be crucial for models' good performance with shuffled text. We probe these language models for word order information and investigate what position embeddings learned from shuffled text encode, showing that these models retain information pertaining to the original, naturalistic word order. We show this is in part due to a subtlety in how shuffling is implemented in previous work -- before rather than after subword segmentation. Surprisingly, we find even Language models trained on text shuffled after subword segmentation retain some semblance of information about word order because of the statistical dependencies between sentence length and unigram probabilities. Finally, we show that beyond GLUE, a variety of language understanding tasks do require word order information, often to an extent that cannot be learned through fine-tuning." @default.
- W4221167891 created "2022-04-03" @default.
- W4221167891 creator A5003288044 @default.
- W4221167891 creator A5018138946 @default.
- W4221167891 creator A5021284917 @default.
- W4221167891 creator A5082194230 @default.
- W4221167891 date "2022-03-21" @default.
- W4221167891 modified "2023-09-25" @default.
- W4221167891 title "Word Order Does Matter (And Shuffled Language Models Know It)" @default.
- W4221167891 doi "https://doi.org/10.48550/arxiv.2203.10995" @default.
- W4221167891 hasPublicationYear "2022" @default.
- W4221167891 type Work @default.
- W4221167891 citedByCount "0" @default.
- W4221167891 crossrefType "posted-content" @default.
- W4221167891 hasAuthorship W4221167891A5003288044 @default.
- W4221167891 hasAuthorship W4221167891A5018138946 @default.
- W4221167891 hasAuthorship W4221167891A5021284917 @default.
- W4221167891 hasAuthorship W4221167891A5082194230 @default.
- W4221167891 hasBestOaLocation W42211678911 @default.
- W4221167891 hasConcept C10138342 @default.
- W4221167891 hasConcept C104317684 @default.
- W4221167891 hasConcept C136197465 @default.
- W4221167891 hasConcept C137293760 @default.
- W4221167891 hasConcept C138885662 @default.
- W4221167891 hasConcept C154945302 @default.
- W4221167891 hasConcept C162324750 @default.
- W4221167891 hasConcept C167927819 @default.
- W4221167891 hasConcept C182306322 @default.
- W4221167891 hasConcept C185592680 @default.
- W4221167891 hasConcept C198082294 @default.
- W4221167891 hasConcept C199360897 @default.
- W4221167891 hasConcept C204321447 @default.
- W4221167891 hasConcept C2777530160 @default.
- W4221167891 hasConcept C41008148 @default.
- W4221167891 hasConcept C41895202 @default.
- W4221167891 hasConcept C55493867 @default.
- W4221167891 hasConcept C66746571 @default.
- W4221167891 hasConcept C70777604 @default.
- W4221167891 hasConcept C89600930 @default.
- W4221167891 hasConcept C90805587 @default.
- W4221167891 hasConcept C98501671 @default.
- W4221167891 hasConceptScore W4221167891C10138342 @default.
- W4221167891 hasConceptScore W4221167891C104317684 @default.
- W4221167891 hasConceptScore W4221167891C136197465 @default.
- W4221167891 hasConceptScore W4221167891C137293760 @default.
- W4221167891 hasConceptScore W4221167891C138885662 @default.
- W4221167891 hasConceptScore W4221167891C154945302 @default.
- W4221167891 hasConceptScore W4221167891C162324750 @default.
- W4221167891 hasConceptScore W4221167891C167927819 @default.
- W4221167891 hasConceptScore W4221167891C182306322 @default.
- W4221167891 hasConceptScore W4221167891C185592680 @default.
- W4221167891 hasConceptScore W4221167891C198082294 @default.
- W4221167891 hasConceptScore W4221167891C199360897 @default.
- W4221167891 hasConceptScore W4221167891C204321447 @default.
- W4221167891 hasConceptScore W4221167891C2777530160 @default.
- W4221167891 hasConceptScore W4221167891C41008148 @default.
- W4221167891 hasConceptScore W4221167891C41895202 @default.
- W4221167891 hasConceptScore W4221167891C55493867 @default.
- W4221167891 hasConceptScore W4221167891C66746571 @default.
- W4221167891 hasConceptScore W4221167891C70777604 @default.
- W4221167891 hasConceptScore W4221167891C89600930 @default.
- W4221167891 hasConceptScore W4221167891C90805587 @default.
- W4221167891 hasConceptScore W4221167891C98501671 @default.
- W4221167891 hasLocation W42211678911 @default.
- W4221167891 hasOpenAccess W4221167891 @default.
- W4221167891 hasPrimaryLocation W42211678911 @default.
- W4221167891 hasRelatedWork W1594916913 @default.
- W4221167891 hasRelatedWork W2106036226 @default.
- W4221167891 hasRelatedWork W2252084125 @default.
- W4221167891 hasRelatedWork W2386836026 @default.
- W4221167891 hasRelatedWork W2804033347 @default.
- W4221167891 hasRelatedWork W2951061418 @default.
- W4221167891 hasRelatedWork W3107474891 @default.
- W4221167891 hasRelatedWork W38394648 @default.
- W4221167891 hasRelatedWork W4221167891 @default.
- W4221167891 hasRelatedWork W4365211491 @default.
- W4221167891 isParatext "false" @default.
- W4221167891 isRetracted "false" @default.
- W4221167891 workType "article" @default.