Matches in SemOpenAlex for { <https://semopenalex.org/work/W3088274487> ?p ?o ?g. }
- W3088274487 abstract "In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become *independent* from the weights as the NN's widths tend to infinity, in the sense of asymptotic freeness in random matrix theory. We call this the Free Independence Principle (FIP), which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of gradient independence assumption used for calculating the Neural Tangent Kernel of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results." @default.
- W3088274487 created "2020-10-01" @default.
- W3088274487 creator A5061115157 @default.
- W3088274487 date "2020-09-22" @default.
- W3088274487 modified "2023-09-27" @default.
- W3088274487 title "Tensor Programs III: Neural Matrix Laws." @default.
- W3088274487 cites W1493832124 @default.
- W3088274487 cites W1567512734 @default.
- W3088274487 cites W1891181203 @default.
- W3088274487 cites W1973104253 @default.
- W3088274487 cites W2035612346 @default.
- W3088274487 cites W2043905980 @default.
- W3088274487 cites W2046172541 @default.
- W3088274487 cites W2060581589 @default.
- W3088274487 cites W2082029531 @default.
- W3088274487 cites W2093577844 @default.
- W3088274487 cites W2212676342 @default.
- W3088274487 cites W2328187444 @default.
- W3088274487 cites W2787173309 @default.
- W3088274487 cites W2789210533 @default.
- W3088274487 cites W2793904650 @default.
- W3088274487 cites W2803268044 @default.
- W3088274487 cites W2809090039 @default.
- W3088274487 cites W2887597596 @default.
- W3088274487 cites W2889560103 @default.
- W3088274487 cites W2889737445 @default.
- W3088274487 cites W2894604724 @default.
- W3088274487 cites W2899748887 @default.
- W3088274487 cites W2900103278 @default.
- W3088274487 cites W2900959181 @default.
- W3088274487 cites W2907747478 @default.
- W3088274487 cites W2910142656 @default.
- W3088274487 cites W2910655610 @default.
- W3088274487 cites W2913473169 @default.
- W3088274487 cites W2942052807 @default.
- W3088274487 cites W2949954798 @default.
- W3088274487 cites W2962685794 @default.
- W3088274487 cites W2962939986 @default.
- W3088274487 cites W2963323437 @default.
- W3088274487 cites W2963570896 @default.
- W3088274487 cites W2963679562 @default.
- W3088274487 cites W2964052793 @default.
- W3088274487 cites W2964065616 @default.
- W3088274487 cites W2964088238 @default.
- W3088274487 cites W2970474218 @default.
- W3088274487 cites W2991290085 @default.
- W3088274487 cites W3000384217 @default.
- W3088274487 cites W3013634494 @default.
- W3088274487 cites W3034979923 @default.
- W3088274487 cites W3036333857 @default.
- W3088274487 cites W3038074040 @default.
- W3088274487 cites W3098848552 @default.
- W3088274487 cites W3180313059 @default.
- W3088274487 cites W602904462 @default.
- W3088274487 cites W615589970 @default.
- W3088274487 hasPublicationYear "2020" @default.
- W3088274487 type Work @default.
- W3088274487 sameAs 3088274487 @default.
- W3088274487 citedByCount "13" @default.
- W3088274487 countsByYear W30882744872019 @default.
- W3088274487 countsByYear W30882744872020 @default.
- W3088274487 countsByYear W30882744872021 @default.
- W3088274487 crossrefType "posted-content" @default.
- W3088274487 hasAuthorship W3088274487A5061115157 @default.
- W3088274487 hasConcept C105795698 @default.
- W3088274487 hasConcept C106487976 @default.
- W3088274487 hasConcept C108710211 @default.
- W3088274487 hasConcept C121332964 @default.
- W3088274487 hasConcept C136119220 @default.
- W3088274487 hasConcept C138187205 @default.
- W3088274487 hasConcept C154945302 @default.
- W3088274487 hasConcept C155281189 @default.
- W3088274487 hasConcept C158622935 @default.
- W3088274487 hasConcept C159985019 @default.
- W3088274487 hasConcept C192562407 @default.
- W3088274487 hasConcept C200331156 @default.
- W3088274487 hasConcept C202444582 @default.
- W3088274487 hasConcept C2524010 @default.
- W3088274487 hasConcept C28826006 @default.
- W3088274487 hasConcept C33923547 @default.
- W3088274487 hasConcept C35651441 @default.
- W3088274487 hasConcept C41008148 @default.
- W3088274487 hasConcept C50644808 @default.
- W3088274487 hasConcept C62520636 @default.
- W3088274487 hasConcept C74193536 @default.
- W3088274487 hasConceptScore W3088274487C105795698 @default.
- W3088274487 hasConceptScore W3088274487C106487976 @default.
- W3088274487 hasConceptScore W3088274487C108710211 @default.
- W3088274487 hasConceptScore W3088274487C121332964 @default.
- W3088274487 hasConceptScore W3088274487C136119220 @default.
- W3088274487 hasConceptScore W3088274487C138187205 @default.
- W3088274487 hasConceptScore W3088274487C154945302 @default.
- W3088274487 hasConceptScore W3088274487C155281189 @default.
- W3088274487 hasConceptScore W3088274487C158622935 @default.
- W3088274487 hasConceptScore W3088274487C159985019 @default.
- W3088274487 hasConceptScore W3088274487C192562407 @default.
- W3088274487 hasConceptScore W3088274487C200331156 @default.
- W3088274487 hasConceptScore W3088274487C202444582 @default.
- W3088274487 hasConceptScore W3088274487C2524010 @default.
- W3088274487 hasConceptScore W3088274487C28826006 @default.