Matches in SemOpenAlex for { <https://semopenalex.org/work/W4288092922> ?p ?o ?g. }
Showing items 1 to 86 of
86
with 100 items per page.
- W4288092922 abstract "Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question: given that critical initialisation is crucial for training at large depth, if dropout limits the depth at which networks are trainable, does initialising critically still matter? We conduct a large-scale controlled experiment, and perform a statistical analysis of over $12000$ trained networks. We find that (1) trainable networks show no statistically significant difference in performance over a wide range of non-critical initialisations; (2) for initialisations that show a statistically significant difference, the net effect on performance is small; (3) only extreme initialisations (very small or very large) perform worse than criticality. These findings also apply to standard ReLU networks of moderate depth as a special case of zero dropout. Our results therefore suggest that, in the shallow-to-moderate depth setting, critical initialisation provides zero performance gains when compared to off-critical initialisations and that searching for off-critical initialisations that might improve training speed or generalisation, is likely to be a fruitless endeavour." @default.
- W4288092922 created "2022-07-28" @default.
- W4288092922 creator A5017043594 @default.
- W4288092922 creator A5020437129 @default.
- W4288092922 creator A5031038636 @default.
- W4288092922 creator A5040305929 @default.
- W4288092922 creator A5046382731 @default.
- W4288092922 creator A5047706306 @default.
- W4288092922 creator A5068297734 @default.
- W4288092922 creator A5076529776 @default.
- W4288092922 creator A5080963336 @default.
- W4288092922 date "2019-10-13" @default.
- W4288092922 modified "2023-09-29" @default.
- W4288092922 title "If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks" @default.
- W4288092922 doi "https://doi.org/10.48550/arxiv.1910.05725" @default.
- W4288092922 hasPublicationYear "2019" @default.
- W4288092922 type Work @default.
- W4288092922 citedByCount "0" @default.
- W4288092922 crossrefType "posted-content" @default.
- W4288092922 hasAuthorship W4288092922A5017043594 @default.
- W4288092922 hasAuthorship W4288092922A5020437129 @default.
- W4288092922 hasAuthorship W4288092922A5031038636 @default.
- W4288092922 hasAuthorship W4288092922A5040305929 @default.
- W4288092922 hasAuthorship W4288092922A5046382731 @default.
- W4288092922 hasAuthorship W4288092922A5047706306 @default.
- W4288092922 hasAuthorship W4288092922A5068297734 @default.
- W4288092922 hasAuthorship W4288092922A5076529776 @default.
- W4288092922 hasAuthorship W4288092922A5080963336 @default.
- W4288092922 hasBestOaLocation W42880929221 @default.
- W4288092922 hasConcept C11413529 @default.
- W4288092922 hasConcept C119857082 @default.
- W4288092922 hasConcept C121332964 @default.
- W4288092922 hasConcept C125611927 @default.
- W4288092922 hasConcept C127413603 @default.
- W4288092922 hasConcept C134306372 @default.
- W4288092922 hasConcept C138885662 @default.
- W4288092922 hasConcept C146978453 @default.
- W4288092922 hasConcept C151201525 @default.
- W4288092922 hasConcept C154945302 @default.
- W4288092922 hasConcept C185544564 @default.
- W4288092922 hasConcept C204323151 @default.
- W4288092922 hasConcept C2776145597 @default.
- W4288092922 hasConcept C2778755073 @default.
- W4288092922 hasConcept C2780813799 @default.
- W4288092922 hasConcept C33923547 @default.
- W4288092922 hasConcept C41008148 @default.
- W4288092922 hasConcept C41895202 @default.
- W4288092922 hasConcept C50644808 @default.
- W4288092922 hasConcept C62520636 @default.
- W4288092922 hasConceptScore W4288092922C11413529 @default.
- W4288092922 hasConceptScore W4288092922C119857082 @default.
- W4288092922 hasConceptScore W4288092922C121332964 @default.
- W4288092922 hasConceptScore W4288092922C125611927 @default.
- W4288092922 hasConceptScore W4288092922C127413603 @default.
- W4288092922 hasConceptScore W4288092922C134306372 @default.
- W4288092922 hasConceptScore W4288092922C138885662 @default.
- W4288092922 hasConceptScore W4288092922C146978453 @default.
- W4288092922 hasConceptScore W4288092922C151201525 @default.
- W4288092922 hasConceptScore W4288092922C154945302 @default.
- W4288092922 hasConceptScore W4288092922C185544564 @default.
- W4288092922 hasConceptScore W4288092922C204323151 @default.
- W4288092922 hasConceptScore W4288092922C2776145597 @default.
- W4288092922 hasConceptScore W4288092922C2778755073 @default.
- W4288092922 hasConceptScore W4288092922C2780813799 @default.
- W4288092922 hasConceptScore W4288092922C33923547 @default.
- W4288092922 hasConceptScore W4288092922C41008148 @default.
- W4288092922 hasConceptScore W4288092922C41895202 @default.
- W4288092922 hasConceptScore W4288092922C50644808 @default.
- W4288092922 hasConceptScore W4288092922C62520636 @default.
- W4288092922 hasLocation W42880929221 @default.
- W4288092922 hasLocation W42880929222 @default.
- W4288092922 hasOpenAccess W4288092922 @default.
- W4288092922 hasPrimaryLocation W42880929221 @default.
- W4288092922 hasRelatedWork W2961085424 @default.
- W4288092922 hasRelatedWork W3046775127 @default.
- W4288092922 hasRelatedWork W3110700750 @default.
- W4288092922 hasRelatedWork W3128220493 @default.
- W4288092922 hasRelatedWork W4205945679 @default.
- W4288092922 hasRelatedWork W4285260836 @default.
- W4288092922 hasRelatedWork W4286629047 @default.
- W4288092922 hasRelatedWork W4306321456 @default.
- W4288092922 hasRelatedWork W4306674287 @default.
- W4288092922 hasRelatedWork W4224009465 @default.
- W4288092922 isParatext "false" @default.
- W4288092922 isRetracted "false" @default.
- W4288092922 workType "article" @default.