Matches in SemOpenAlex for { <https://semopenalex.org/work/W3207539211> ?p ?o ?g. }
- W3207539211 abstract "Pre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on large-scale corpora. Since PLMs capture word semantics in different contexts, the quality of word representations highly depends on word frequency, which usually follows a heavy-tailed distributions in the pre-training corpus. Therefore, the embeddings of rare words on the tail are usually poorly optimized. In this work, we focus on enhancing language model pre-training by leveraging definitions of the rare words in dictionaries (e.g., Wiktionary). To incorporate a rare word definition as a part of input, we fetch its definition from the dictionary and append it to the end of the input text sequence. In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word and sentence-level alignment between input text sequence and rare word definitions to enhance language modeling representation with dictionary. We evaluate the proposed Dict-BERT model on the language understanding benchmark GLUE and eight specialized domain benchmark datasets. Extensive experiments demonstrate that Dict-BERT can significantly improve the understanding of rare words and boost model performance on various NLP downstream tasks." @default.
- W3207539211 created "2021-10-25" @default.
- W3207539211 creator A5006654661 @default.
- W3207539211 creator A5020783463 @default.
- W3207539211 creator A5028597555 @default.
- W3207539211 creator A5034687114 @default.
- W3207539211 creator A5034826937 @default.
- W3207539211 creator A5039183544 @default.
- W3207539211 creator A5074821819 @default.
- W3207539211 creator A5089195158 @default.
- W3207539211 date "2022-01-01" @default.
- W3207539211 modified "2023-10-14" @default.
- W3207539211 title "Dict-BERT: Enhancing Language Model Pre-training with Dictionary" @default.
- W3207539211 cites W1682403713 @default.
- W3207539211 cites W2127795553 @default.
- W3207539211 cites W2626778328 @default.
- W3207539211 cites W2842511635 @default.
- W3207539211 cites W2911489562 @default.
- W3207539211 cites W2946006146 @default.
- W3207539211 cites W2953356739 @default.
- W3207539211 cites W2962883166 @default.
- W3207539211 cites W2963310665 @default.
- W3207539211 cites W2963341956 @default.
- W3207539211 cites W2965373594 @default.
- W3207539211 cites W2966610483 @default.
- W3207539211 cites W2981861606 @default.
- W3207539211 cites W2994915912 @default.
- W3207539211 cites W2995480165 @default.
- W3207539211 cites W2998385486 @default.
- W3207539211 cites W2998554035 @default.
- W3207539211 cites W3011574394 @default.
- W3207539211 cites W3014521650 @default.
- W3207539211 cites W3034238904 @default.
- W3207539211 cites W3034978746 @default.
- W3207539211 cites W3047312186 @default.
- W3207539211 cites W3082274269 @default.
- W3207539211 cites W3090656107 @default.
- W3207539211 cites W3092288641 @default.
- W3207539211 cites W3123123873 @default.
- W3207539211 cites W3175604467 @default.
- W3207539211 cites W3176750236 @default.
- W3207539211 cites W3197849207 @default.
- W3207539211 cites W3202807130 @default.
- W3207539211 cites W3213435953 @default.
- W3207539211 doi "https://doi.org/10.18653/v1/2022.findings-acl.150" @default.
- W3207539211 hasPublicationYear "2022" @default.
- W3207539211 type Work @default.
- W3207539211 sameAs 3207539211 @default.
- W3207539211 citedByCount "4" @default.
- W3207539211 countsByYear W32075392112022 @default.
- W3207539211 countsByYear W32075392112023 @default.
- W3207539211 crossrefType "proceedings-article" @default.
- W3207539211 hasAuthorship W3207539211A5006654661 @default.
- W3207539211 hasAuthorship W3207539211A5020783463 @default.
- W3207539211 hasAuthorship W3207539211A5028597555 @default.
- W3207539211 hasAuthorship W3207539211A5034687114 @default.
- W3207539211 hasAuthorship W3207539211A5034826937 @default.
- W3207539211 hasAuthorship W3207539211A5039183544 @default.
- W3207539211 hasAuthorship W3207539211A5074821819 @default.
- W3207539211 hasAuthorship W3207539211A5089195158 @default.
- W3207539211 hasBestOaLocation W32075392111 @default.
- W3207539211 hasConcept C120665830 @default.
- W3207539211 hasConcept C121332964 @default.
- W3207539211 hasConcept C13280743 @default.
- W3207539211 hasConcept C137293760 @default.
- W3207539211 hasConcept C138885662 @default.
- W3207539211 hasConcept C154945302 @default.
- W3207539211 hasConcept C162324750 @default.
- W3207539211 hasConcept C184337299 @default.
- W3207539211 hasConcept C185798385 @default.
- W3207539211 hasConcept C187736073 @default.
- W3207539211 hasConcept C192209626 @default.
- W3207539211 hasConcept C199360897 @default.
- W3207539211 hasConcept C204321447 @default.
- W3207539211 hasConcept C205649164 @default.
- W3207539211 hasConcept C2777530160 @default.
- W3207539211 hasConcept C2780451532 @default.
- W3207539211 hasConcept C28490314 @default.
- W3207539211 hasConcept C35639132 @default.
- W3207539211 hasConcept C41008148 @default.
- W3207539211 hasConcept C41895202 @default.
- W3207539211 hasConcept C90805587 @default.
- W3207539211 hasConceptScore W3207539211C120665830 @default.
- W3207539211 hasConceptScore W3207539211C121332964 @default.
- W3207539211 hasConceptScore W3207539211C13280743 @default.
- W3207539211 hasConceptScore W3207539211C137293760 @default.
- W3207539211 hasConceptScore W3207539211C138885662 @default.
- W3207539211 hasConceptScore W3207539211C154945302 @default.
- W3207539211 hasConceptScore W3207539211C162324750 @default.
- W3207539211 hasConceptScore W3207539211C184337299 @default.
- W3207539211 hasConceptScore W3207539211C185798385 @default.
- W3207539211 hasConceptScore W3207539211C187736073 @default.
- W3207539211 hasConceptScore W3207539211C192209626 @default.
- W3207539211 hasConceptScore W3207539211C199360897 @default.
- W3207539211 hasConceptScore W3207539211C204321447 @default.
- W3207539211 hasConceptScore W3207539211C205649164 @default.
- W3207539211 hasConceptScore W3207539211C2777530160 @default.
- W3207539211 hasConceptScore W3207539211C2780451532 @default.
- W3207539211 hasConceptScore W3207539211C28490314 @default.
- W3207539211 hasConceptScore W3207539211C35639132 @default.