Matches in SemOpenAlex for { <https://semopenalex.org/work/W2971055146> ?p ?o ?g. }
Showing items 1 to 83 of
83
with 100 items per page.
- W2971055146 endingPage "8091" @default.
- W2971055146 startingPage "8080" @default.
- W2971055146 abstract "Natural gradient descent has proven very effective at mitigating the catastrophic effects of pathological curvature in the objective function, but little is known theoretically about its convergence properties, especially for emph{non-linear} networks. In this work, we analyze for the first time the speed of convergence to global optimum for natural gradient descent on non-linear neural networks with the squared error loss. We identify two conditions which guarantee the global convergence: (1) the Jacobian matrix (of network's output for all training cases w.r.t the parameters) is full row rank and (2) the Jacobian matrix is stable for small perturbations around the initialization. For two-layer ReLU neural networks (i.e. with one hidden layer), we prove that these two conditions do hold throughout the training under the assumptions that the inputs do not degenerate and the network is over-parameterized. We further extend our analysis to more general loss function with similar convergence property. Lastly, we show that K-FAC, an approximate natural gradient descent method, also converges to global minima under the same assumptions." @default.
- W2971055146 created "2019-09-05" @default.
- W2971055146 creator A5046229829 @default.
- W2971055146 creator A5067036768 @default.
- W2971055146 creator A5089378951 @default.
- W2971055146 date "2019-05-27" @default.
- W2971055146 modified "2023-10-18" @default.
- W2971055146 title "Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks" @default.
- W2971055146 hasPublicationYear "2019" @default.
- W2971055146 type Work @default.
- W2971055146 sameAs 2971055146 @default.
- W2971055146 citedByCount "27" @default.
- W2971055146 countsByYear W29710551462019 @default.
- W2971055146 countsByYear W29710551462020 @default.
- W2971055146 countsByYear W29710551462021 @default.
- W2971055146 crossrefType "proceedings-article" @default.
- W2971055146 hasAuthorship W2971055146A5046229829 @default.
- W2971055146 hasAuthorship W2971055146A5067036768 @default.
- W2971055146 hasAuthorship W2971055146A5089378951 @default.
- W2971055146 hasConcept C11413529 @default.
- W2971055146 hasConcept C114466953 @default.
- W2971055146 hasConcept C126255220 @default.
- W2971055146 hasConcept C134306372 @default.
- W2971055146 hasConcept C153258448 @default.
- W2971055146 hasConcept C154945302 @default.
- W2971055146 hasConcept C162324750 @default.
- W2971055146 hasConcept C165464430 @default.
- W2971055146 hasConcept C186633575 @default.
- W2971055146 hasConcept C199360897 @default.
- W2971055146 hasConcept C200331156 @default.
- W2971055146 hasConcept C2777303404 @default.
- W2971055146 hasConcept C28826006 @default.
- W2971055146 hasConcept C33923547 @default.
- W2971055146 hasConcept C41008148 @default.
- W2971055146 hasConcept C50522688 @default.
- W2971055146 hasConcept C50644808 @default.
- W2971055146 hasConceptScore W2971055146C11413529 @default.
- W2971055146 hasConceptScore W2971055146C114466953 @default.
- W2971055146 hasConceptScore W2971055146C126255220 @default.
- W2971055146 hasConceptScore W2971055146C134306372 @default.
- W2971055146 hasConceptScore W2971055146C153258448 @default.
- W2971055146 hasConceptScore W2971055146C154945302 @default.
- W2971055146 hasConceptScore W2971055146C162324750 @default.
- W2971055146 hasConceptScore W2971055146C165464430 @default.
- W2971055146 hasConceptScore W2971055146C186633575 @default.
- W2971055146 hasConceptScore W2971055146C199360897 @default.
- W2971055146 hasConceptScore W2971055146C200331156 @default.
- W2971055146 hasConceptScore W2971055146C2777303404 @default.
- W2971055146 hasConceptScore W2971055146C28826006 @default.
- W2971055146 hasConceptScore W2971055146C33923547 @default.
- W2971055146 hasConceptScore W2971055146C41008148 @default.
- W2971055146 hasConceptScore W2971055146C50522688 @default.
- W2971055146 hasConceptScore W2971055146C50644808 @default.
- W2971055146 hasLocation W29710551461 @default.
- W2971055146 hasOpenAccess W2971055146 @default.
- W2971055146 hasPrimaryLocation W29710551461 @default.
- W2971055146 hasRelatedWork W1970789124 @default.
- W2971055146 hasRelatedWork W2163605009 @default.
- W2971055146 hasRelatedWork W2194775991 @default.
- W2971055146 hasRelatedWork W2809090039 @default.
- W2971055146 hasRelatedWork W2886067286 @default.
- W2971055146 hasRelatedWork W2892218381 @default.
- W2971055146 hasRelatedWork W2894604724 @default.
- W2971055146 hasRelatedWork W2917744435 @default.
- W2971055146 hasRelatedWork W2938647293 @default.
- W2971055146 hasRelatedWork W2945554113 @default.
- W2971055146 hasRelatedWork W2947461788 @default.
- W2971055146 hasRelatedWork W2962698540 @default.
- W2971055146 hasRelatedWork W2962781217 @default.
- W2971055146 hasRelatedWork W2963239103 @default.
- W2971055146 hasRelatedWork W2963241285 @default.
- W2971055146 hasRelatedWork W2964121744 @default.
- W2971055146 hasRelatedWork W2964125128 @default.
- W2971055146 hasRelatedWork W2970217468 @default.
- W2971055146 hasRelatedWork W2971043187 @default.
- W2971055146 hasRelatedWork W3086499488 @default.
- W2971055146 hasVolume "32" @default.
- W2971055146 isParatext "false" @default.
- W2971055146 isRetracted "false" @default.
- W2971055146 magId "2971055146" @default.
- W2971055146 workType "article" @default.