Matches in SemOpenAlex for { <https://semopenalex.org/work/W2952136670> ?p ?o ?g. }
- W2952136670 abstract "Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning mid- and long-range dependencies. However, conventional attention mechanisms used in memory-augmented neural language models produce a single output vector per time step. This vector is used both for predicting the next token as well as for the key and value of a differentiable memory of a token history. In this paper, we propose a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution. This model outperforms existing memory-augmented neural language models on two corpora. Yet, we found that our method mainly utilizes a memory of the five most recent output representations. This led to the unexpected main finding that a much simpler model based only on the concatenation of recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models." @default.
- W2952136670 created "2019-06-27" @default.
- W2952136670 creator A5001151643 @default.
- W2952136670 creator A5037220915 @default.
- W2952136670 creator A5067022550 @default.
- W2952136670 creator A5079315903 @default.
- W2952136670 date "2017-02-15" @default.
- W2952136670 modified "2023-09-27" @default.
- W2952136670 title "Frustratingly Short Attention Spans in Neural Language Modeling" @default.
- W2952136670 cites W1514535095 @default.
- W2952136670 cites W1843891098 @default.
- W2952136670 cites W2024585065 @default.
- W2952136670 cites W2064675550 @default.
- W2952136670 cites W2118463056 @default.
- W2952136670 cites W2150355110 @default.
- W2952136670 cites W2178931739 @default.
- W2952136670 cites W2185726469 @default.
- W2952136670 cites W2194690080 @default.
- W2952136670 cites W2259472270 @default.
- W2952136670 cites W2293997542 @default.
- W2952136670 cites W2345668077 @default.
- W2952136670 cites W2409591106 @default.
- W2952136670 cites W2415755012 @default.
- W2952136670 cites W2416043263 @default.
- W2952136670 cites W2470713034 @default.
- W2952136670 cites W2477209458 @default.
- W2952136670 cites W2512457506 @default.
- W2952136670 cites W2535697732 @default.
- W2952136670 cites W2550448043 @default.
- W2952136670 cites W2766736793 @default.
- W2952136670 cites W2950527759 @default.
- W2952136670 cites W2951714314 @default.
- W2952136670 cites W2951793508 @default.
- W2952136670 cites W2952191002 @default.
- W2952136670 cites W2962819663 @default.
- W2952136670 cites W2963595025 @default.
- W2952136670 cites W2964121744 @default.
- W2952136670 cites W2964267515 @default.
- W2952136670 cites W2964308564 @default.
- W2952136670 cites W2584341106 @default.
- W2952136670 hasPublicationYear "2017" @default.
- W2952136670 type Work @default.
- W2952136670 sameAs 2952136670 @default.
- W2952136670 citedByCount "59" @default.
- W2952136670 countsByYear W29521366702017 @default.
- W2952136670 countsByYear W29521366702018 @default.
- W2952136670 countsByYear W29521366702019 @default.
- W2952136670 countsByYear W29521366702020 @default.
- W2952136670 countsByYear W29521366702021 @default.
- W2952136670 crossrefType "posted-content" @default.
- W2952136670 hasAuthorship W2952136670A5001151643 @default.
- W2952136670 hasAuthorship W2952136670A5037220915 @default.
- W2952136670 hasAuthorship W2952136670A5067022550 @default.
- W2952136670 hasAuthorship W2952136670A5079315903 @default.
- W2952136670 hasConcept C119857082 @default.
- W2952136670 hasConcept C125411270 @default.
- W2952136670 hasConcept C134306372 @default.
- W2952136670 hasConcept C137293760 @default.
- W2952136670 hasConcept C138885662 @default.
- W2952136670 hasConcept C154945302 @default.
- W2952136670 hasConcept C17744445 @default.
- W2952136670 hasConcept C199539241 @default.
- W2952136670 hasConcept C202615002 @default.
- W2952136670 hasConcept C204321447 @default.
- W2952136670 hasConcept C26517878 @default.
- W2952136670 hasConcept C2776291640 @default.
- W2952136670 hasConcept C2776359362 @default.
- W2952136670 hasConcept C33923547 @default.
- W2952136670 hasConcept C38652104 @default.
- W2952136670 hasConcept C41008148 @default.
- W2952136670 hasConcept C41895202 @default.
- W2952136670 hasConcept C48145219 @default.
- W2952136670 hasConcept C50644808 @default.
- W2952136670 hasConcept C80444323 @default.
- W2952136670 hasConcept C87619178 @default.
- W2952136670 hasConcept C90805587 @default.
- W2952136670 hasConcept C94375191 @default.
- W2952136670 hasConcept C94625758 @default.
- W2952136670 hasConceptScore W2952136670C119857082 @default.
- W2952136670 hasConceptScore W2952136670C125411270 @default.
- W2952136670 hasConceptScore W2952136670C134306372 @default.
- W2952136670 hasConceptScore W2952136670C137293760 @default.
- W2952136670 hasConceptScore W2952136670C138885662 @default.
- W2952136670 hasConceptScore W2952136670C154945302 @default.
- W2952136670 hasConceptScore W2952136670C17744445 @default.
- W2952136670 hasConceptScore W2952136670C199539241 @default.
- W2952136670 hasConceptScore W2952136670C202615002 @default.
- W2952136670 hasConceptScore W2952136670C204321447 @default.
- W2952136670 hasConceptScore W2952136670C26517878 @default.
- W2952136670 hasConceptScore W2952136670C2776291640 @default.
- W2952136670 hasConceptScore W2952136670C2776359362 @default.
- W2952136670 hasConceptScore W2952136670C33923547 @default.
- W2952136670 hasConceptScore W2952136670C38652104 @default.
- W2952136670 hasConceptScore W2952136670C41008148 @default.
- W2952136670 hasConceptScore W2952136670C41895202 @default.
- W2952136670 hasConceptScore W2952136670C48145219 @default.
- W2952136670 hasConceptScore W2952136670C50644808 @default.
- W2952136670 hasConceptScore W2952136670C80444323 @default.
- W2952136670 hasConceptScore W2952136670C87619178 @default.
- W2952136670 hasConceptScore W2952136670C90805587 @default.