Matches in SemOpenAlex for { <https://semopenalex.org/work/W2994590149> ?p ?o ?g. }
Showing items 1 to 94 of
94
with 100 items per page.
- W2994590149 abstract "In sequence learning tasks such as language modelling, Recurrent Neural Networks must learn relationships between input features separated by time. State of the art models such as LSTM and Transformer are trained by backpropagation of losses into prior hidden states and inputs held in memory. This allows gradients to flow from present to past and effectively learn with perfect hindsight, but at a significant memory cost. In this paper we show that it is possible to train high performance recurrent networks using information that is local in time, and thereby achieve a significantly reduced memory footprint. We describe a predictive autoencoder called bRSM featuring recurrent connections, sparse activations, and a boosting rule for improved cell utilization. The architecture demonstrates near optimal performance on a non-deterministic (stochastic) partially-observable sequence learning task consisting of high-Markov-order sequences of MNIST digits. We find that this model learns these sequences faster and more completely than an LSTM, and offer several possible explanations why the LSTM architecture might struggle with the partially observable sequence structure in this task. We also apply our model to a next word prediction task on the Penn Treebank (PTB) dataset. We show that a 'flattened' RSM network, when paired with a modern semantic word embedding and the addition of boosting, achieves 103.5 PPL (a 20-point improvement over the best N-gram models), beating ordinary RNNs trained with BPTT and approaching the scores of early LSTM implementations. This work provides encouraging evidence that strong results on challenging tasks such as language modelling may be possible using less memory intensive, biologically-plausible training regimes." @default.
- W2994590149 created "2019-12-13" @default.
- W2994590149 creator A5049553911 @default.
- W2994590149 creator A5055640606 @default.
- W2994590149 creator A5062820142 @default.
- W2994590149 date "2019-12-02" @default.
- W2994590149 modified "2023-09-27" @default.
- W2994590149 title "Long Distance Relationships without Time Travel: Boosting the Performance of a Sparse Predictive Autoencoder in Sequence Modeling" @default.
- W2994590149 cites W1853900790 @default.
- W2994590149 cites W2064675550 @default.
- W2994590149 cites W2093647425 @default.
- W2994590149 cites W2171928131 @default.
- W2994590149 cites W2185726469 @default.
- W2994590149 cites W2626778328 @default.
- W2994590149 cites W2792764867 @default.
- W2994590149 cites W2907127169 @default.
- W2994590149 cites W2922783569 @default.
- W2994590149 cites W2947369026 @default.
- W2994590149 cites W2949382160 @default.
- W2994590149 cites W2952566282 @default.
- W2994590149 cites W2972810859 @default.
- W2994590149 hasPublicationYear "2019" @default.
- W2994590149 type Work @default.
- W2994590149 sameAs 2994590149 @default.
- W2994590149 citedByCount "0" @default.
- W2994590149 crossrefType "posted-content" @default.
- W2994590149 hasAuthorship W2994590149A5049553911 @default.
- W2994590149 hasAuthorship W2994590149A5055640606 @default.
- W2994590149 hasAuthorship W2994590149A5062820142 @default.
- W2994590149 hasConcept C101738243 @default.
- W2994590149 hasConcept C108583219 @default.
- W2994590149 hasConcept C111919701 @default.
- W2994590149 hasConcept C119857082 @default.
- W2994590149 hasConcept C121332964 @default.
- W2994590149 hasConcept C137293760 @default.
- W2994590149 hasConcept C147168706 @default.
- W2994590149 hasConcept C154945302 @default.
- W2994590149 hasConcept C165801399 @default.
- W2994590149 hasConcept C190502265 @default.
- W2994590149 hasConcept C2777462759 @default.
- W2994590149 hasConcept C40506919 @default.
- W2994590149 hasConcept C41008148 @default.
- W2994590149 hasConcept C41608201 @default.
- W2994590149 hasConcept C46686674 @default.
- W2994590149 hasConcept C50644808 @default.
- W2994590149 hasConcept C62520636 @default.
- W2994590149 hasConcept C66322947 @default.
- W2994590149 hasConcept C74912251 @default.
- W2994590149 hasConceptScore W2994590149C101738243 @default.
- W2994590149 hasConceptScore W2994590149C108583219 @default.
- W2994590149 hasConceptScore W2994590149C111919701 @default.
- W2994590149 hasConceptScore W2994590149C119857082 @default.
- W2994590149 hasConceptScore W2994590149C121332964 @default.
- W2994590149 hasConceptScore W2994590149C137293760 @default.
- W2994590149 hasConceptScore W2994590149C147168706 @default.
- W2994590149 hasConceptScore W2994590149C154945302 @default.
- W2994590149 hasConceptScore W2994590149C165801399 @default.
- W2994590149 hasConceptScore W2994590149C190502265 @default.
- W2994590149 hasConceptScore W2994590149C2777462759 @default.
- W2994590149 hasConceptScore W2994590149C40506919 @default.
- W2994590149 hasConceptScore W2994590149C41008148 @default.
- W2994590149 hasConceptScore W2994590149C41608201 @default.
- W2994590149 hasConceptScore W2994590149C46686674 @default.
- W2994590149 hasConceptScore W2994590149C50644808 @default.
- W2994590149 hasConceptScore W2994590149C62520636 @default.
- W2994590149 hasConceptScore W2994590149C66322947 @default.
- W2994590149 hasConceptScore W2994590149C74912251 @default.
- W2994590149 hasLocation W29945901491 @default.
- W2994590149 hasOpenAccess W2994590149 @default.
- W2994590149 hasPrimaryLocation W29945901491 @default.
- W2994590149 hasRelatedWork W2345668077 @default.
- W2994590149 hasRelatedWork W2473934411 @default.
- W2994590149 hasRelatedWork W2566275344 @default.
- W2994590149 hasRelatedWork W2566895423 @default.
- W2994590149 hasRelatedWork W2586630418 @default.
- W2994590149 hasRelatedWork W2604158577 @default.
- W2994590149 hasRelatedWork W2754526845 @default.
- W2994590149 hasRelatedWork W2785138022 @default.
- W2994590149 hasRelatedWork W2903988544 @default.
- W2994590149 hasRelatedWork W2911013038 @default.
- W2994590149 hasRelatedWork W2936093102 @default.
- W2994590149 hasRelatedWork W2962832505 @default.
- W2994590149 hasRelatedWork W2964102284 @default.
- W2994590149 hasRelatedWork W3014997656 @default.
- W2994590149 hasRelatedWork W3090334579 @default.
- W2994590149 hasRelatedWork W3133102907 @default.
- W2994590149 hasRelatedWork W3158297926 @default.
- W2994590149 hasRelatedWork W3173414161 @default.
- W2994590149 hasRelatedWork W3200700388 @default.
- W2994590149 hasRelatedWork W3094674343 @default.
- W2994590149 isParatext "false" @default.
- W2994590149 isRetracted "false" @default.
- W2994590149 magId "2994590149" @default.
- W2994590149 workType "article" @default.