Matches in SemOpenAlex for { <https://semopenalex.org/work/W4385573092> ?p ?o ?g. }
Showing items 1 to 53 of
53
with 100 items per page.
- W4385573092 abstract "To understand what kinds of linguistic knowledge are encoded by pretrained Chinese language models (LMs), we introduce the benchmark of Sino LINGuistics (SLING), which consists of 38K minimal sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. Each pair demonstrates the acceptability contrast of a specific syntactic or semantic phenomenon (e.g., The keys are lost vs. The keys is lost), and an LM should assign lower perplexity to the acceptable sentence. In contrast to the CLiMP dataset (Xiang et al., 2021), which also contains Chinese minimal pairs and was created by translating the vocabulary of the English BLiMP dataset, the minimal pairs in SLING are derived primarily by applying syntactic and lexical transformations to naturally-occurring, linguist-annotated sentences from the Chinese Treebank 9.0, thus addressing severe issues in CLiMP’s data generation process. We test 18 publicly available pretrained monolingual (e.g., BERT-base-zh, CPM) and multi-lingual (e.g., mT5, XLM) language models on SLING. Our experiments show that the average accuracy for LMs is far below human performance (69.7% vs. 97.1%), while BERT-base-zh achieves the highest accuracy (84.8%) of all tested LMs, even much larger ones. Additionally, we find that most LMs have a strong gender and number (singular/plural) bias, and they perform better on local phenomena than hierarchical ones." @default.
- W4385573092 created "2023-08-05" @default.
- W4385573092 creator A5022637213 @default.
- W4385573092 creator A5044579154 @default.
- W4385573092 creator A5078893115 @default.
- W4385573092 creator A5082767919 @default.
- W4385573092 date "2022-01-01" @default.
- W4385573092 modified "2023-10-16" @default.
- W4385573092 title "SLING: Sino Linguistic Evaluation of Large Language Models" @default.
- W4385573092 doi "https://doi.org/10.18653/v1/2022.emnlp-main.305" @default.
- W4385573092 hasPublicationYear "2022" @default.
- W4385573092 type Work @default.
- W4385573092 citedByCount "0" @default.
- W4385573092 crossrefType "proceedings-article" @default.
- W4385573092 hasAuthorship W4385573092A5022637213 @default.
- W4385573092 hasAuthorship W4385573092A5044579154 @default.
- W4385573092 hasAuthorship W4385573092A5078893115 @default.
- W4385573092 hasAuthorship W4385573092A5082767919 @default.
- W4385573092 hasBestOaLocation W43855730921 @default.
- W4385573092 hasConcept C138885662 @default.
- W4385573092 hasConcept C138954614 @default.
- W4385573092 hasConcept C154945302 @default.
- W4385573092 hasConcept C186644900 @default.
- W4385573092 hasConcept C204321447 @default.
- W4385573092 hasConcept C206134035 @default.
- W4385573092 hasConcept C2777530160 @default.
- W4385573092 hasConcept C41008148 @default.
- W4385573092 hasConcept C41895202 @default.
- W4385573092 hasConceptScore W4385573092C138885662 @default.
- W4385573092 hasConceptScore W4385573092C138954614 @default.
- W4385573092 hasConceptScore W4385573092C154945302 @default.
- W4385573092 hasConceptScore W4385573092C186644900 @default.
- W4385573092 hasConceptScore W4385573092C204321447 @default.
- W4385573092 hasConceptScore W4385573092C206134035 @default.
- W4385573092 hasConceptScore W4385573092C2777530160 @default.
- W4385573092 hasConceptScore W4385573092C41008148 @default.
- W4385573092 hasConceptScore W4385573092C41895202 @default.
- W4385573092 hasLocation W43855730921 @default.
- W4385573092 hasOpenAccess W4385573092 @default.
- W4385573092 hasPrimaryLocation W43855730921 @default.
- W4385573092 hasRelatedWork W1038817422 @default.
- W4385573092 hasRelatedWork W1818857488 @default.
- W4385573092 hasRelatedWork W1877285056 @default.
- W4385573092 hasRelatedWork W1963443923 @default.
- W4385573092 hasRelatedWork W2005229811 @default.
- W4385573092 hasRelatedWork W2095751497 @default.
- W4385573092 hasRelatedWork W2550024871 @default.
- W4385573092 hasRelatedWork W2789919619 @default.
- W4385573092 hasRelatedWork W2971623275 @default.
- W4385573092 hasRelatedWork W3217712442 @default.
- W4385573092 isParatext "false" @default.
- W4385573092 isRetracted "false" @default.
- W4385573092 workType "article" @default.