Matches in SemOpenAlex for { <https://semopenalex.org/work/W4385570922> ?p ?o ?g. }
Showing items 1 to 69 of
69
with 100 items per page.
- W4385570922 abstract "Recent studies have revealed some issues of Multi-Head Attention (MHA), e.g., redundancy and over-parameterization. Specifically, the heads of MHA were originally designed to attend to information from different representation subspaces, whereas prior studies found that some attention heads likely learn similar features and can be pruned without harming performance. Inspired by the minimum-redundancy feature selection, we assume that focusing on the most representative and distinctive features with minimum resources can mitigate the above issues and lead to more effective and efficient MHAs. In particular, we propose Grouped Head Attention, trained with a self-supervised group constraint that group attention heads, where each group focuses on an essential but distinctive feature subset. We additionally propose a Voting-to-Stay procedure to remove redundant heads, thus achieving a transformer with lighter weights. Extensive experiments are consistent with our hypothesis. Moreover, our method achieves significant performance gains on three well-established tasks while considerably compressing parameters." @default.
- W4385570922 created "2023-08-05" @default.
- W4385570922 creator A5019708503 @default.
- W4385570922 creator A5047828703 @default.
- W4385570922 creator A5072407590 @default.
- W4385570922 creator A5074934997 @default.
- W4385570922 creator A5087909151 @default.
- W4385570922 date "2023-01-01" @default.
- W4385570922 modified "2023-09-28" @default.
- W4385570922 title "Finding the Pillars of Strength for Multi-Head Attention" @default.
- W4385570922 doi "https://doi.org/10.18653/v1/2023.acl-long.812" @default.
- W4385570922 hasPublicationYear "2023" @default.
- W4385570922 type Work @default.
- W4385570922 citedByCount "0" @default.
- W4385570922 crossrefType "proceedings-article" @default.
- W4385570922 hasAuthorship W4385570922A5019708503 @default.
- W4385570922 hasAuthorship W4385570922A5047828703 @default.
- W4385570922 hasAuthorship W4385570922A5072407590 @default.
- W4385570922 hasAuthorship W4385570922A5074934997 @default.
- W4385570922 hasAuthorship W4385570922A5087909151 @default.
- W4385570922 hasBestOaLocation W43855709221 @default.
- W4385570922 hasConcept C111919701 @default.
- W4385570922 hasConcept C114793014 @default.
- W4385570922 hasConcept C119857082 @default.
- W4385570922 hasConcept C12362212 @default.
- W4385570922 hasConcept C127313418 @default.
- W4385570922 hasConcept C152124472 @default.
- W4385570922 hasConcept C153180895 @default.
- W4385570922 hasConcept C154945302 @default.
- W4385570922 hasConcept C17744445 @default.
- W4385570922 hasConcept C199539241 @default.
- W4385570922 hasConcept C2524010 @default.
- W4385570922 hasConcept C2780312720 @default.
- W4385570922 hasConcept C33923547 @default.
- W4385570922 hasConcept C41008148 @default.
- W4385570922 hasConcept C520049643 @default.
- W4385570922 hasConcept C94625758 @default.
- W4385570922 hasConceptScore W4385570922C111919701 @default.
- W4385570922 hasConceptScore W4385570922C114793014 @default.
- W4385570922 hasConceptScore W4385570922C119857082 @default.
- W4385570922 hasConceptScore W4385570922C12362212 @default.
- W4385570922 hasConceptScore W4385570922C127313418 @default.
- W4385570922 hasConceptScore W4385570922C152124472 @default.
- W4385570922 hasConceptScore W4385570922C153180895 @default.
- W4385570922 hasConceptScore W4385570922C154945302 @default.
- W4385570922 hasConceptScore W4385570922C17744445 @default.
- W4385570922 hasConceptScore W4385570922C199539241 @default.
- W4385570922 hasConceptScore W4385570922C2524010 @default.
- W4385570922 hasConceptScore W4385570922C2780312720 @default.
- W4385570922 hasConceptScore W4385570922C33923547 @default.
- W4385570922 hasConceptScore W4385570922C41008148 @default.
- W4385570922 hasConceptScore W4385570922C520049643 @default.
- W4385570922 hasConceptScore W4385570922C94625758 @default.
- W4385570922 hasLocation W43855709221 @default.
- W4385570922 hasOpenAccess W4385570922 @default.
- W4385570922 hasPrimaryLocation W43855709221 @default.
- W4385570922 hasRelatedWork W2961085424 @default.
- W4385570922 hasRelatedWork W3046775127 @default.
- W4385570922 hasRelatedWork W3170094116 @default.
- W4385570922 hasRelatedWork W3209574120 @default.
- W4385570922 hasRelatedWork W4205958290 @default.
- W4385570922 hasRelatedWork W4285260836 @default.
- W4385570922 hasRelatedWork W4286629047 @default.
- W4385570922 hasRelatedWork W4306321456 @default.
- W4385570922 hasRelatedWork W4306674287 @default.
- W4385570922 hasRelatedWork W4224009465 @default.
- W4385570922 isParatext "false" @default.
- W4385570922 isRetracted "false" @default.
- W4385570922 workType "article" @default.