Matches in SemOpenAlex for { <https://semopenalex.org/work/W3152745207> ?p ?o ?g. }
Showing items 1 to 97 of
97
with 100 items per page.
- W3152745207 abstract "With the success of pre-trained language models in recent years, more and more researchers focus on opening the “black box” of these models. Following this interest, we carry out a qualitative and quantitative analysis of constituency grammar in attention heads of BERT and RoBERTa. We employ the syntactic distance method to extract implicit constituency grammar from the attention weights of each head. Our results show that there exist heads that can induce some grammar types much better than baselines, suggesting that some heads act as a proxy for constituency grammar. We also analyze how attention heads’ constituency grammar inducing (CGI) ability changes after fine-tuning with two kinds of tasks, including sentence meaning similarity (SMS) tasks and natural language inference (NLI) tasks. Our results suggest that SMS tasks decrease the average CGI ability of upper layers, while NLI tasks increase it. Lastly, we investigate the connections between CGI ability and natural language understanding ability on QQP and MNLI tasks." @default.
- W3152745207 created "2021-04-26" @default.
- W3152745207 creator A5055657033 @default.
- W3152745207 date "2021-01-01" @default.
- W3152745207 modified "2023-10-18" @default.
- W3152745207 title "Have Attention Heads in BERT Learned Constituency Grammar?" @default.
- W3152745207 cites W1632114991 @default.
- W3152745207 cites W2798569372 @default.
- W3152745207 cites W2910243263 @default.
- W3152745207 cites W2912206855 @default.
- W3152745207 cites W2946359678 @default.
- W3152745207 cites W2949433733 @default.
- W3152745207 cites W2962784628 @default.
- W3152745207 cites W2963310665 @default.
- W3152745207 cites W2963341956 @default.
- W3152745207 cites W2963403868 @default.
- W3152745207 cites W2963748441 @default.
- W3152745207 cites W2963846996 @default.
- W3152745207 cites W2965373594 @default.
- W3152745207 cites W2970120757 @default.
- W3152745207 cites W2970565456 @default.
- W3152745207 cites W2970597249 @default.
- W3152745207 cites W2972324944 @default.
- W3152745207 cites W2980282514 @default.
- W3152745207 cites W2991265431 @default.
- W3152745207 cites W2996428491 @default.
- W3152745207 cites W3004117589 @default.
- W3152745207 cites W3034779619 @default.
- W3152745207 cites W3049366647 @default.
- W3152745207 cites W3102085674 @default.
- W3152745207 cites W3104033643 @default.
- W3152745207 doi "https://doi.org/10.18653/v1/2021.eacl-srw.2" @default.
- W3152745207 hasPublicationYear "2021" @default.
- W3152745207 type Work @default.
- W3152745207 sameAs 3152745207 @default.
- W3152745207 citedByCount "0" @default.
- W3152745207 crossrefType "proceedings-article" @default.
- W3152745207 hasAuthorship W3152745207A5055657033 @default.
- W3152745207 hasBestOaLocation W31527452071 @default.
- W3152745207 hasConcept C103278499 @default.
- W3152745207 hasConcept C114793014 @default.
- W3152745207 hasConcept C115961682 @default.
- W3152745207 hasConcept C119857082 @default.
- W3152745207 hasConcept C120665830 @default.
- W3152745207 hasConcept C121332964 @default.
- W3152745207 hasConcept C127313418 @default.
- W3152745207 hasConcept C138885662 @default.
- W3152745207 hasConcept C154945302 @default.
- W3152745207 hasConcept C192209626 @default.
- W3152745207 hasConcept C195324797 @default.
- W3152745207 hasConcept C204321447 @default.
- W3152745207 hasConcept C26022165 @default.
- W3152745207 hasConcept C2776214188 @default.
- W3152745207 hasConcept C2777530160 @default.
- W3152745207 hasConcept C2780148112 @default.
- W3152745207 hasConcept C2780312720 @default.
- W3152745207 hasConcept C39890363 @default.
- W3152745207 hasConcept C41008148 @default.
- W3152745207 hasConcept C41895202 @default.
- W3152745207 hasConceptScore W3152745207C103278499 @default.
- W3152745207 hasConceptScore W3152745207C114793014 @default.
- W3152745207 hasConceptScore W3152745207C115961682 @default.
- W3152745207 hasConceptScore W3152745207C119857082 @default.
- W3152745207 hasConceptScore W3152745207C120665830 @default.
- W3152745207 hasConceptScore W3152745207C121332964 @default.
- W3152745207 hasConceptScore W3152745207C127313418 @default.
- W3152745207 hasConceptScore W3152745207C138885662 @default.
- W3152745207 hasConceptScore W3152745207C154945302 @default.
- W3152745207 hasConceptScore W3152745207C192209626 @default.
- W3152745207 hasConceptScore W3152745207C195324797 @default.
- W3152745207 hasConceptScore W3152745207C204321447 @default.
- W3152745207 hasConceptScore W3152745207C26022165 @default.
- W3152745207 hasConceptScore W3152745207C2776214188 @default.
- W3152745207 hasConceptScore W3152745207C2777530160 @default.
- W3152745207 hasConceptScore W3152745207C2780148112 @default.
- W3152745207 hasConceptScore W3152745207C2780312720 @default.
- W3152745207 hasConceptScore W3152745207C39890363 @default.
- W3152745207 hasConceptScore W3152745207C41008148 @default.
- W3152745207 hasConceptScore W3152745207C41895202 @default.
- W3152745207 hasLocation W31527452071 @default.
- W3152745207 hasLocation W31527452072 @default.
- W3152745207 hasOpenAccess W3152745207 @default.
- W3152745207 hasPrimaryLocation W31527452071 @default.
- W3152745207 hasRelatedWork W10043089 @default.
- W3152745207 hasRelatedWork W11095331 @default.
- W3152745207 hasRelatedWork W14378890 @default.
- W3152745207 hasRelatedWork W14519841 @default.
- W3152745207 hasRelatedWork W14574355 @default.
- W3152745207 hasRelatedWork W6961447 @default.
- W3152745207 hasRelatedWork W7875108 @default.
- W3152745207 hasRelatedWork W8407316 @default.
- W3152745207 hasRelatedWork W9452413 @default.
- W3152745207 hasRelatedWork W4098470 @default.
- W3152745207 isParatext "false" @default.
- W3152745207 isRetracted "false" @default.
- W3152745207 magId "3152745207" @default.
- W3152745207 workType "article" @default.