Matches in SemOpenAlex for { <https://semopenalex.org/work/W4309302569> ?p ?o ?g. }
Showing items 1 to 60 of
60
with 100 items per page.
- W4309302569 abstract "Multilingual pre-trained models exhibit zero-shot cross-lingual transfer, where a model fine-tuned on a source language achieves surprisingly good performance on a target language. While studies have attempted to understand transfer, they focus only on MLM, and the large number of differences between natural languages makes it hard to disentangle the importance of different properties. In this work, we specifically highlight the importance of word embedding alignment by proposing a pre-training objective (ALIGN-MLM) whose auxiliary loss guides similar words in different languages to have similar word embeddings. ALIGN-MLM either outperforms or matches three widely adopted objectives (MLM, XLM, DICT-MLM) when we evaluate transfer between pairs of natural languages and their counterparts created by systematically modifying specific properties like the script. In particular, ALIGN-MLM outperforms XLM and MLM by 35 and 30 F1 points on POS-tagging for transfer between languages that differ both in their script and word order (left-to-right v.s. right-to-left). We also show a strong correlation between alignment and transfer for all objectives (e.g., rho=0.727 for XNLI), which together with ALIGN-MLM's strong performance calls for explicitly aligning word embeddings for multilingual models." @default.
- W4309302569 created "2022-11-25" @default.
- W4309302569 creator A5025205227 @default.
- W4309302569 creator A5041777363 @default.
- W4309302569 creator A5054102866 @default.
- W4309302569 date "2022-11-15" @default.
- W4309302569 modified "2023-09-27" @default.
- W4309302569 title "ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-training" @default.
- W4309302569 doi "https://doi.org/10.48550/arxiv.2211.08547" @default.
- W4309302569 hasPublicationYear "2022" @default.
- W4309302569 type Work @default.
- W4309302569 citedByCount "0" @default.
- W4309302569 crossrefType "posted-content" @default.
- W4309302569 hasAuthorship W4309302569A5025205227 @default.
- W4309302569 hasAuthorship W4309302569A5041777363 @default.
- W4309302569 hasAuthorship W4309302569A5054102866 @default.
- W4309302569 hasBestOaLocation W43093025691 @default.
- W4309302569 hasConcept C120665830 @default.
- W4309302569 hasConcept C121332964 @default.
- W4309302569 hasConcept C138885662 @default.
- W4309302569 hasConcept C154945302 @default.
- W4309302569 hasConcept C173608175 @default.
- W4309302569 hasConcept C192209626 @default.
- W4309302569 hasConcept C204321447 @default.
- W4309302569 hasConcept C2776175482 @default.
- W4309302569 hasConcept C2777462759 @default.
- W4309302569 hasConcept C41008148 @default.
- W4309302569 hasConcept C41608201 @default.
- W4309302569 hasConcept C41895202 @default.
- W4309302569 hasConcept C90805587 @default.
- W4309302569 hasConceptScore W4309302569C120665830 @default.
- W4309302569 hasConceptScore W4309302569C121332964 @default.
- W4309302569 hasConceptScore W4309302569C138885662 @default.
- W4309302569 hasConceptScore W4309302569C154945302 @default.
- W4309302569 hasConceptScore W4309302569C173608175 @default.
- W4309302569 hasConceptScore W4309302569C192209626 @default.
- W4309302569 hasConceptScore W4309302569C204321447 @default.
- W4309302569 hasConceptScore W4309302569C2776175482 @default.
- W4309302569 hasConceptScore W4309302569C2777462759 @default.
- W4309302569 hasConceptScore W4309302569C41008148 @default.
- W4309302569 hasConceptScore W4309302569C41608201 @default.
- W4309302569 hasConceptScore W4309302569C41895202 @default.
- W4309302569 hasConceptScore W4309302569C90805587 @default.
- W4309302569 hasLocation W43093025691 @default.
- W4309302569 hasLocation W43093025692 @default.
- W4309302569 hasOpenAccess W4309302569 @default.
- W4309302569 hasPrimaryLocation W43093025691 @default.
- W4309302569 hasRelatedWork W2335882425 @default.
- W4309302569 hasRelatedWork W2620816324 @default.
- W4309302569 hasRelatedWork W2949267551 @default.
- W4309302569 hasRelatedWork W2993300079 @default.
- W4309302569 hasRelatedWork W3107679445 @default.
- W4309302569 hasRelatedWork W3134737443 @default.
- W4309302569 hasRelatedWork W3143412223 @default.
- W4309302569 hasRelatedWork W3202766982 @default.
- W4309302569 hasRelatedWork W4221011941 @default.
- W4309302569 hasRelatedWork W4307613132 @default.
- W4309302569 isParatext "false" @default.
- W4309302569 isRetracted "false" @default.
- W4309302569 workType "article" @default.