Matches in SemOpenAlex for { <https://semopenalex.org/work/W4288333864> ?p ?o ?g. }
Showing items 1 to 59 of
59
with 100 items per page.
- W4288333864 abstract "Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT's attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT's attention." @default.
- W4288333864 created "2022-07-28" @default.
- W4288333864 creator A5024311574 @default.
- W4288333864 creator A5031489667 @default.
- W4288333864 creator A5046006076 @default.
- W4288333864 creator A5088072227 @default.
- W4288333864 date "2019-06-10" @default.
- W4288333864 modified "2023-09-30" @default.
- W4288333864 title "What Does BERT Look At? An Analysis of BERT's Attention" @default.
- W4288333864 doi "https://doi.org/10.48550/arxiv.1906.04341" @default.
- W4288333864 hasPublicationYear "2019" @default.
- W4288333864 type Work @default.
- W4288333864 citedByCount "3" @default.
- W4288333864 countsByYear W42883338642022 @default.
- W4288333864 countsByYear W42883338642023 @default.
- W4288333864 crossrefType "posted-content" @default.
- W4288333864 hasAuthorship W4288333864A5024311574 @default.
- W4288333864 hasAuthorship W4288333864A5031489667 @default.
- W4288333864 hasAuthorship W4288333864A5046006076 @default.
- W4288333864 hasAuthorship W4288333864A5088072227 @default.
- W4288333864 hasBestOaLocation W42883338641 @default.
- W4288333864 hasConcept C121934690 @default.
- W4288333864 hasConcept C138268822 @default.
- W4288333864 hasConcept C138885662 @default.
- W4288333864 hasConcept C154945302 @default.
- W4288333864 hasConcept C204321447 @default.
- W4288333864 hasConcept C2777530160 @default.
- W4288333864 hasConcept C28076734 @default.
- W4288333864 hasConcept C41008148 @default.
- W4288333864 hasConcept C41895202 @default.
- W4288333864 hasConcept C60048249 @default.
- W4288333864 hasConcept C95623464 @default.
- W4288333864 hasConceptScore W4288333864C121934690 @default.
- W4288333864 hasConceptScore W4288333864C138268822 @default.
- W4288333864 hasConceptScore W4288333864C138885662 @default.
- W4288333864 hasConceptScore W4288333864C154945302 @default.
- W4288333864 hasConceptScore W4288333864C204321447 @default.
- W4288333864 hasConceptScore W4288333864C2777530160 @default.
- W4288333864 hasConceptScore W4288333864C28076734 @default.
- W4288333864 hasConceptScore W4288333864C41008148 @default.
- W4288333864 hasConceptScore W4288333864C41895202 @default.
- W4288333864 hasConceptScore W4288333864C60048249 @default.
- W4288333864 hasConceptScore W4288333864C95623464 @default.
- W4288333864 hasLocation W42883338641 @default.
- W4288333864 hasOpenAccess W4288333864 @default.
- W4288333864 hasPrimaryLocation W42883338641 @default.
- W4288333864 hasRelatedWork W1519302135 @default.
- W4288333864 hasRelatedWork W1978971213 @default.
- W4288333864 hasRelatedWork W2046738012 @default.
- W4288333864 hasRelatedWork W2295005279 @default.
- W4288333864 hasRelatedWork W2364327788 @default.
- W4288333864 hasRelatedWork W2513006088 @default.
- W4288333864 hasRelatedWork W2819889016 @default.
- W4288333864 hasRelatedWork W2925144130 @default.
- W4288333864 hasRelatedWork W3148951203 @default.
- W4288333864 hasRelatedWork W752907497 @default.
- W4288333864 isParatext "false" @default.
- W4288333864 isRetracted "false" @default.
- W4288333864 workType "article" @default.