Matches in SemOpenAlex for { <https://semopenalex.org/work/W4226258784> ?p ?o ?g. }
Showing items 1 to 61 of
61
with 100 items per page.
- W4226258784 abstract "Pre-trained Language Models (PLMs) have been widely used in various natural language processing (NLP) tasks, owing to their powerful text representations trained on large-scale corpora. In this paper, we propose a new PLM called PERT for natural language understanding (NLU). PERT is an auto-encoding model (like BERT) trained with Permuted Language Model (PerLM). The formulation of the proposed PerLM is straightforward. We permute a proportion of the input text, and the training objective is to predict the position of the original token. Moreover, we also apply whole word masking and N-gram masking to improve the performance of PERT. We carried out extensive experiments on both Chinese and English NLU benchmarks. The experimental results show that PERT can bring improvements over various comparable baselines on some of the tasks, while others are not. These results indicate that developing more diverse pre-training tasks is possible instead of masked language model variants. Several quantitative studies are carried out to better understand PERT, which might help design PLMs in the future. Resources are available: https://github.com/ymcui/PERT" @default.
- W4226258784 created "2022-05-05" @default.
- W4226258784 creator A5001241589 @default.
- W4226258784 creator A5051107002 @default.
- W4226258784 creator A5063089557 @default.
- W4226258784 date "2022-03-14" @default.
- W4226258784 modified "2023-10-18" @default.
- W4226258784 title "PERT: Pre-training BERT with Permuted Language Model" @default.
- W4226258784 doi "https://doi.org/10.48550/arxiv.2203.06906" @default.
- W4226258784 hasPublicationYear "2022" @default.
- W4226258784 type Work @default.
- W4226258784 citedByCount "0" @default.
- W4226258784 crossrefType "posted-content" @default.
- W4226258784 hasAuthorship W4226258784A5001241589 @default.
- W4226258784 hasAuthorship W4226258784A5051107002 @default.
- W4226258784 hasAuthorship W4226258784A5063089557 @default.
- W4226258784 hasBestOaLocation W42262587841 @default.
- W4226258784 hasConcept C137293760 @default.
- W4226258784 hasConcept C138885662 @default.
- W4226258784 hasConcept C142362112 @default.
- W4226258784 hasConcept C153349607 @default.
- W4226258784 hasConcept C154945302 @default.
- W4226258784 hasConcept C195324797 @default.
- W4226258784 hasConcept C204321447 @default.
- W4226258784 hasConcept C2777402240 @default.
- W4226258784 hasConcept C2779439875 @default.
- W4226258784 hasConcept C38652104 @default.
- W4226258784 hasConcept C41008148 @default.
- W4226258784 hasConcept C41895202 @default.
- W4226258784 hasConcept C48145219 @default.
- W4226258784 hasConcept C90805587 @default.
- W4226258784 hasConceptScore W4226258784C137293760 @default.
- W4226258784 hasConceptScore W4226258784C138885662 @default.
- W4226258784 hasConceptScore W4226258784C142362112 @default.
- W4226258784 hasConceptScore W4226258784C153349607 @default.
- W4226258784 hasConceptScore W4226258784C154945302 @default.
- W4226258784 hasConceptScore W4226258784C195324797 @default.
- W4226258784 hasConceptScore W4226258784C204321447 @default.
- W4226258784 hasConceptScore W4226258784C2777402240 @default.
- W4226258784 hasConceptScore W4226258784C2779439875 @default.
- W4226258784 hasConceptScore W4226258784C38652104 @default.
- W4226258784 hasConceptScore W4226258784C41008148 @default.
- W4226258784 hasConceptScore W4226258784C41895202 @default.
- W4226258784 hasConceptScore W4226258784C48145219 @default.
- W4226258784 hasConceptScore W4226258784C90805587 @default.
- W4226258784 hasLocation W42262587841 @default.
- W4226258784 hasOpenAccess W4226258784 @default.
- W4226258784 hasPrimaryLocation W42262587841 @default.
- W4226258784 hasRelatedWork W1538473846 @default.
- W4226258784 hasRelatedWork W1542956019 @default.
- W4226258784 hasRelatedWork W1563618553 @default.
- W4226258784 hasRelatedWork W1806995473 @default.
- W4226258784 hasRelatedWork W2602143361 @default.
- W4226258784 hasRelatedWork W2977842567 @default.
- W4226258784 hasRelatedWork W3018932980 @default.
- W4226258784 hasRelatedWork W3107474891 @default.
- W4226258784 hasRelatedWork W3184167880 @default.
- W4226258784 hasRelatedWork W4308854837 @default.
- W4226258784 isParatext "false" @default.
- W4226258784 isRetracted "false" @default.
- W4226258784 workType "article" @default.