Matches in SemOpenAlex for { <https://semopenalex.org/work/W4288258594> ?p ?o ?g. }
Showing items 1 to 54 of
54
with 100 items per page.
- W4288258594 abstract "BERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the current work, we focus on the interpretation of self-attention, which is one of the fundamental underlying components of BERT. Using a subset of GLUE tasks and a set of handcrafted features-of-interest, we propose the methodology and carry out a qualitative and quantitative analysis of the information encoded by the individual BERT's heads. Our findings suggest that there is a limited set of attention patterns that are repeated across different heads, indicating the overall model overparametrization. While different heads consistently use the same attention patterns, they have varying impact on performance across different tasks. We show that manually disabling attention in certain heads leads to a performance improvement over the regular fine-tuned BERT models." @default.
- W4288258594 created "2022-07-28" @default.
- W4288258594 creator A5001172800 @default.
- W4288258594 creator A5004127966 @default.
- W4288258594 creator A5029083907 @default.
- W4288258594 creator A5071360545 @default.
- W4288258594 date "2019-08-21" @default.
- W4288258594 modified "2023-10-02" @default.
- W4288258594 title "Revealing the Dark Secrets of BERT" @default.
- W4288258594 doi "https://doi.org/10.48550/arxiv.1908.08593" @default.
- W4288258594 hasPublicationYear "2019" @default.
- W4288258594 type Work @default.
- W4288258594 citedByCount "1" @default.
- W4288258594 countsByYear W42882585942023 @default.
- W4288258594 crossrefType "posted-content" @default.
- W4288258594 hasAuthorship W4288258594A5001172800 @default.
- W4288258594 hasAuthorship W4288258594A5004127966 @default.
- W4288258594 hasAuthorship W4288258594A5029083907 @default.
- W4288258594 hasAuthorship W4288258594A5071360545 @default.
- W4288258594 hasBestOaLocation W42882585941 @default.
- W4288258594 hasConcept C120665830 @default.
- W4288258594 hasConcept C121332964 @default.
- W4288258594 hasConcept C154945302 @default.
- W4288258594 hasConcept C177264268 @default.
- W4288258594 hasConcept C192209626 @default.
- W4288258594 hasConcept C199360897 @default.
- W4288258594 hasConcept C204321447 @default.
- W4288258594 hasConcept C41008148 @default.
- W4288258594 hasConcept C527412718 @default.
- W4288258594 hasConceptScore W4288258594C120665830 @default.
- W4288258594 hasConceptScore W4288258594C121332964 @default.
- W4288258594 hasConceptScore W4288258594C154945302 @default.
- W4288258594 hasConceptScore W4288258594C177264268 @default.
- W4288258594 hasConceptScore W4288258594C192209626 @default.
- W4288258594 hasConceptScore W4288258594C199360897 @default.
- W4288258594 hasConceptScore W4288258594C204321447 @default.
- W4288258594 hasConceptScore W4288258594C41008148 @default.
- W4288258594 hasConceptScore W4288258594C527412718 @default.
- W4288258594 hasLocation W42882585941 @default.
- W4288258594 hasOpenAccess W4288258594 @default.
- W4288258594 hasPrimaryLocation W42882585941 @default.
- W4288258594 hasRelatedWork W1517909231 @default.
- W4288258594 hasRelatedWork W1571404427 @default.
- W4288258594 hasRelatedWork W1859752461 @default.
- W4288258594 hasRelatedWork W2130575083 @default.
- W4288258594 hasRelatedWork W2173878312 @default.
- W4288258594 hasRelatedWork W2916492174 @default.
- W4288258594 hasRelatedWork W3037021823 @default.
- W4288258594 hasRelatedWork W3107474891 @default.
- W4288258594 hasRelatedWork W851785710 @default.
- W4288258594 hasRelatedWork W2613333037 @default.
- W4288258594 isParatext "false" @default.
- W4288258594 isRetracted "false" @default.
- W4288258594 workType "article" @default.