Matches in SemOpenAlex for { <https://semopenalex.org/work/W4320342607> ?p ?o ?g. }
Showing items 1 to 73 of
73
with 100 items per page.
- W4320342607 abstract "Random-feature-based attention (RFA) is an efficient approximation of softmax attention with linear runtime and space complexity. However, the approximation gap between RFA and conventional softmax attention is not well studied. Built upon previous progress of RFA, we characterize this gap through the lens of control variates and show that RFA can be decomposed into a sum of multiple control variate estimators for each element in the sequence. This new framework reveals that exact softmax attention can be recovered from RFA by manipulating each control variate. Besides, it allows us to develop a more flexible form of control variates, resulting in a novel attention mechanism that significantly reduces the approximation gap while maintaining linear complexity. Extensive experiments demonstrate that our model outperforms state-of-the-art efficient attention mechanisms on both vision and language tasks." @default.
- W4320342607 created "2023-02-13" @default.
- W4320342607 creator A5007803202 @default.
- W4320342607 creator A5014554970 @default.
- W4320342607 creator A5025459071 @default.
- W4320342607 creator A5060197177 @default.
- W4320342607 date "2023-02-09" @default.
- W4320342607 modified "2023-10-14" @default.
- W4320342607 title "Efficient Attention via Control Variates" @default.
- W4320342607 doi "https://doi.org/10.48550/arxiv.2302.04542" @default.
- W4320342607 hasPublicationYear "2023" @default.
- W4320342607 type Work @default.
- W4320342607 citedByCount "0" @default.
- W4320342607 crossrefType "posted-content" @default.
- W4320342607 hasAuthorship W4320342607A5007803202 @default.
- W4320342607 hasAuthorship W4320342607A5014554970 @default.
- W4320342607 hasAuthorship W4320342607A5025459071 @default.
- W4320342607 hasAuthorship W4320342607A5060197177 @default.
- W4320342607 hasBestOaLocation W43203426071 @default.
- W4320342607 hasConcept C105795698 @default.
- W4320342607 hasConcept C107673813 @default.
- W4320342607 hasConcept C108583219 @default.
- W4320342607 hasConcept C111350023 @default.
- W4320342607 hasConcept C11413529 @default.
- W4320342607 hasConcept C121683094 @default.
- W4320342607 hasConcept C13153151 @default.
- W4320342607 hasConcept C138885662 @default.
- W4320342607 hasConcept C154945302 @default.
- W4320342607 hasConcept C185429906 @default.
- W4320342607 hasConcept C188441871 @default.
- W4320342607 hasConcept C2775924081 @default.
- W4320342607 hasConcept C2776401178 @default.
- W4320342607 hasConcept C2778112365 @default.
- W4320342607 hasConcept C33923547 @default.
- W4320342607 hasConcept C41008148 @default.
- W4320342607 hasConcept C41895202 @default.
- W4320342607 hasConcept C54355233 @default.
- W4320342607 hasConcept C86803240 @default.
- W4320342607 hasConceptScore W4320342607C105795698 @default.
- W4320342607 hasConceptScore W4320342607C107673813 @default.
- W4320342607 hasConceptScore W4320342607C108583219 @default.
- W4320342607 hasConceptScore W4320342607C111350023 @default.
- W4320342607 hasConceptScore W4320342607C11413529 @default.
- W4320342607 hasConceptScore W4320342607C121683094 @default.
- W4320342607 hasConceptScore W4320342607C13153151 @default.
- W4320342607 hasConceptScore W4320342607C138885662 @default.
- W4320342607 hasConceptScore W4320342607C154945302 @default.
- W4320342607 hasConceptScore W4320342607C185429906 @default.
- W4320342607 hasConceptScore W4320342607C188441871 @default.
- W4320342607 hasConceptScore W4320342607C2775924081 @default.
- W4320342607 hasConceptScore W4320342607C2776401178 @default.
- W4320342607 hasConceptScore W4320342607C2778112365 @default.
- W4320342607 hasConceptScore W4320342607C33923547 @default.
- W4320342607 hasConceptScore W4320342607C41008148 @default.
- W4320342607 hasConceptScore W4320342607C41895202 @default.
- W4320342607 hasConceptScore W4320342607C54355233 @default.
- W4320342607 hasConceptScore W4320342607C86803240 @default.
- W4320342607 hasLocation W43203426071 @default.
- W4320342607 hasOpenAccess W4320342607 @default.
- W4320342607 hasPrimaryLocation W43203426071 @default.
- W4320342607 hasRelatedWork W2088897998 @default.
- W4320342607 hasRelatedWork W2372527200 @default.
- W4320342607 hasRelatedWork W2469237301 @default.
- W4320342607 hasRelatedWork W2782592381 @default.
- W4320342607 hasRelatedWork W2888789309 @default.
- W4320342607 hasRelatedWork W2971416272 @default.
- W4320342607 hasRelatedWork W3095506574 @default.
- W4320342607 hasRelatedWork W3111562146 @default.
- W4320342607 hasRelatedWork W3202244193 @default.
- W4320342607 hasRelatedWork W4307834408 @default.
- W4320342607 isParatext "false" @default.
- W4320342607 isRetracted "false" @default.
- W4320342607 workType "article" @default.