Matches in SemOpenAlex for { <https://semopenalex.org/work/W3138350871> ?p ?o ?g. }
- W3138350871 abstract "Using a mean-field theory of signal propagation, we analyze the evolution of correlations between two signals propagating forward through a deep ReLU network with correlated weights. Signals become highly correlated in deep ReLU networks with uncorrelated weights. We show that ReLU networks with anti-correlated weights can avoid this fate and have a chaotic phase where the signal correlations saturate below unity. Consistent with this analysis, we find that networks initialized with anti-correlated weights can train faster (in a teacher-student setting) by taking advantage of the increased expressivity in the chaotic phase. Combining this with a previously proposed strategy of using an asymmetric initialization to reduce dead node probability, we propose an initialization scheme that allows faster training and learning than the best-known initializations." @default.
- W3138350871 created "2021-03-29" @default.
- W3138350871 creator A5052799543 @default.
- W3138350871 creator A5085315788 @default.
- W3138350871 date "2021-03-23" @default.
- W3138350871 modified "2023-09-27" @default.
- W3138350871 title "Initializing ReLU networks in an expressive subspace of weights" @default.
- W3138350871 cites W1665214252 @default.
- W3138350871 cites W1677182931 @default.
- W3138350871 cites W1836465849 @default.
- W3138350871 cites W197865394 @default.
- W3138350871 cites W2004227461 @default.
- W3138350871 cites W2058568633 @default.
- W3138350871 cites W2090614046 @default.
- W3138350871 cites W2097117768 @default.
- W3138350871 cites W2146502635 @default.
- W3138350871 cites W2156387975 @default.
- W3138350871 cites W2160815625 @default.
- W3138350871 cites W2163605009 @default.
- W3138350871 cites W2194775991 @default.
- W3138350871 cites W2257979135 @default.
- W3138350871 cites W2502312327 @default.
- W3138350871 cites W2745165998 @default.
- W3138350871 cites W2790814024 @default.
- W3138350871 cites W2803268044 @default.
- W3138350871 cites W2897097528 @default.
- W3138350871 cites W2899663614 @default.
- W3138350871 cites W2917668703 @default.
- W3138350871 cites W2919115771 @default.
- W3138350871 cites W2950621961 @default.
- W3138350871 cites W2962685937 @default.
- W3138350871 cites W2962698540 @default.
- W3138350871 cites W2962804662 @default.
- W3138350871 cites W2962972936 @default.
- W3138350871 cites W2963285578 @default.
- W3138350871 cites W2963454111 @default.
- W3138350871 cites W2963504252 @default.
- W3138350871 cites W2963568027 @default.
- W3138350871 cites W2963685250 @default.
- W3138350871 cites W2963966020 @default.
- W3138350871 cites W2963982496 @default.
- W3138350871 cites W2964052793 @default.
- W3138350871 cites W2964088238 @default.
- W3138350871 cites W2964121744 @default.
- W3138350871 cites W2964122761 @default.
- W3138350871 cites W2970338090 @default.
- W3138350871 cites W2970454961 @default.
- W3138350871 cites W2996141621 @default.
- W3138350871 cites W3037932933 @default.
- W3138350871 cites W3098586271 @default.
- W3138350871 cites W3099849883 @default.
- W3138350871 cites W3101398262 @default.
- W3138350871 cites W3107728769 @default.
- W3138350871 cites W3141595720 @default.
- W3138350871 cites W6908809 @default.
- W3138350871 hasPublicationYear "2021" @default.
- W3138350871 type Work @default.
- W3138350871 sameAs 3138350871 @default.
- W3138350871 citedByCount "0" @default.
- W3138350871 crossrefType "posted-content" @default.
- W3138350871 hasAuthorship W3138350871A5052799543 @default.
- W3138350871 hasAuthorship W3138350871A5085315788 @default.
- W3138350871 hasConcept C105795698 @default.
- W3138350871 hasConcept C11413529 @default.
- W3138350871 hasConcept C114466953 @default.
- W3138350871 hasConcept C121332964 @default.
- W3138350871 hasConcept C153180895 @default.
- W3138350871 hasConcept C154945302 @default.
- W3138350871 hasConcept C169345407 @default.
- W3138350871 hasConcept C199360897 @default.
- W3138350871 hasConcept C2777052490 @default.
- W3138350871 hasConcept C2779843651 @default.
- W3138350871 hasConcept C32834561 @default.
- W3138350871 hasConcept C33923547 @default.
- W3138350871 hasConcept C41008148 @default.
- W3138350871 hasConcept C62520636 @default.
- W3138350871 hasConcept C62611344 @default.
- W3138350871 hasConceptScore W3138350871C105795698 @default.
- W3138350871 hasConceptScore W3138350871C11413529 @default.
- W3138350871 hasConceptScore W3138350871C114466953 @default.
- W3138350871 hasConceptScore W3138350871C121332964 @default.
- W3138350871 hasConceptScore W3138350871C153180895 @default.
- W3138350871 hasConceptScore W3138350871C154945302 @default.
- W3138350871 hasConceptScore W3138350871C169345407 @default.
- W3138350871 hasConceptScore W3138350871C199360897 @default.
- W3138350871 hasConceptScore W3138350871C2777052490 @default.
- W3138350871 hasConceptScore W3138350871C2779843651 @default.
- W3138350871 hasConceptScore W3138350871C32834561 @default.
- W3138350871 hasConceptScore W3138350871C33923547 @default.
- W3138350871 hasConceptScore W3138350871C41008148 @default.
- W3138350871 hasConceptScore W3138350871C62520636 @default.
- W3138350871 hasConceptScore W3138350871C62611344 @default.
- W3138350871 hasLocation W31383508711 @default.
- W3138350871 hasOpenAccess W3138350871 @default.
- W3138350871 hasPrimaryLocation W31383508711 @default.
- W3138350871 hasRelatedWork W2125930537 @default.
- W3138350871 hasRelatedWork W2899686093 @default.
- W3138350871 hasRelatedWork W2899748887 @default.
- W3138350871 hasRelatedWork W2913473169 @default.
- W3138350871 hasRelatedWork W2948471475 @default.