Matches in SemOpenAlex for { <https://semopenalex.org/work/W3215268866> ?p ?o ?g. }
Showing items 1 to 92 of
92
with 100 items per page.
- W3215268866 abstract "In gradient descent, changing how we parametrize the model can lead to drastically different optimization trajectories, giving rise to a surprising range of meaningful inductive biases: identifying sparse classifiers or reconstructing low-rank matrices without explicit regularization. This implicit regularization has been hypothesised to be a contributing factor to good generalization in deep learning. However, natural gradient descent is approximately invariant to reparameterization, it always follows the same trajectory and finds the same optimum. The question naturally arises: What happens if we eliminate the role of parameterization, which solution will be found, what new properties occur? We characterize the behaviour of natural gradient flow in deep linear networks for separable classification under logistic loss and deep matrix factorization. Some of our findings extend to nonlinear neural networks with sufficient but finite over-parametrization. We demonstrate that there exist learning problems where natural gradient descent fails to generalize, while gradient descent with the right architecture performs well." @default.
- W3215268866 created "2021-12-06" @default.
- W3215268866 creator A5042446523 @default.
- W3215268866 creator A5044975358 @default.
- W3215268866 creator A5052054395 @default.
- W3215268866 date "2021-11-22" @default.
- W3215268866 modified "2023-09-28" @default.
- W3215268866 title "Depth Without the Magic: Inductive Bias of Natural Gradient Descent" @default.
- W3215268866 cites W1970789124 @default.
- W3215268866 cites W2053440286 @default.
- W3215268866 cites W2135046866 @default.
- W3215268866 cites W2167729035 @default.
- W3215268866 cites W2566079294 @default.
- W3215268866 cites W2790118436 @default.
- W3215268866 cites W2809090039 @default.
- W3215268866 cites W2889392686 @default.
- W3215268866 cites W2892218381 @default.
- W3215268866 cites W2899476926 @default.
- W3215268866 cites W2949650786 @default.
- W3215268866 cites W2951046202 @default.
- W3215268866 cites W2963208657 @default.
- W3215268866 cites W2963241285 @default.
- W3215268866 cites W2963403654 @default.
- W3215268866 cites W2963403868 @default.
- W3215268866 cites W2963645788 @default.
- W3215268866 cites W2963826371 @default.
- W3215268866 cites W2964084001 @default.
- W3215268866 cites W2964122761 @default.
- W3215268866 cites W2964125128 @default.
- W3215268866 cites W2964309400 @default.
- W3215268866 cites W2970170116 @default.
- W3215268866 cites W2970259623 @default.
- W3215268866 cites W2971055146 @default.
- W3215268866 cites W2981634459 @default.
- W3215268866 cites W2994848047 @default.
- W3215268866 cites W3035909932 @default.
- W3215268866 cites W3086499488 @default.
- W3215268866 cites W3089321292 @default.
- W3215268866 cites W3100200769 @default.
- W3215268866 cites W3169948375 @default.
- W3215268866 cites W3204187020 @default.
- W3215268866 hasPublicationYear "2021" @default.
- W3215268866 type Work @default.
- W3215268866 sameAs 3215268866 @default.
- W3215268866 citedByCount "0" @default.
- W3215268866 crossrefType "posted-content" @default.
- W3215268866 hasAuthorship W3215268866A5042446523 @default.
- W3215268866 hasAuthorship W3215268866A5044975358 @default.
- W3215268866 hasAuthorship W3215268866A5052054395 @default.
- W3215268866 hasConcept C108583219 @default.
- W3215268866 hasConcept C11413529 @default.
- W3215268866 hasConcept C115680565 @default.
- W3215268866 hasConcept C134306372 @default.
- W3215268866 hasConcept C153258448 @default.
- W3215268866 hasConcept C154945302 @default.
- W3215268866 hasConcept C162324750 @default.
- W3215268866 hasConcept C167879884 @default.
- W3215268866 hasConcept C187736073 @default.
- W3215268866 hasConcept C197352929 @default.
- W3215268866 hasConcept C206688291 @default.
- W3215268866 hasConcept C2776135515 @default.
- W3215268866 hasConcept C2780451532 @default.
- W3215268866 hasConcept C28006648 @default.
- W3215268866 hasConcept C28826006 @default.
- W3215268866 hasConcept C33923547 @default.
- W3215268866 hasConcept C41008148 @default.
- W3215268866 hasConcept C50644808 @default.
- W3215268866 hasConceptScore W3215268866C108583219 @default.
- W3215268866 hasConceptScore W3215268866C11413529 @default.
- W3215268866 hasConceptScore W3215268866C115680565 @default.
- W3215268866 hasConceptScore W3215268866C134306372 @default.
- W3215268866 hasConceptScore W3215268866C153258448 @default.
- W3215268866 hasConceptScore W3215268866C154945302 @default.
- W3215268866 hasConceptScore W3215268866C162324750 @default.
- W3215268866 hasConceptScore W3215268866C167879884 @default.
- W3215268866 hasConceptScore W3215268866C187736073 @default.
- W3215268866 hasConceptScore W3215268866C197352929 @default.
- W3215268866 hasConceptScore W3215268866C206688291 @default.
- W3215268866 hasConceptScore W3215268866C2776135515 @default.
- W3215268866 hasConceptScore W3215268866C2780451532 @default.
- W3215268866 hasConceptScore W3215268866C28006648 @default.
- W3215268866 hasConceptScore W3215268866C28826006 @default.
- W3215268866 hasConceptScore W3215268866C33923547 @default.
- W3215268866 hasConceptScore W3215268866C41008148 @default.
- W3215268866 hasConceptScore W3215268866C50644808 @default.
- W3215268866 hasLocation W32152688661 @default.
- W3215268866 hasOpenAccess W3215268866 @default.
- W3215268866 hasPrimaryLocation W32152688661 @default.
- W3215268866 isParatext "false" @default.
- W3215268866 isRetracted "false" @default.
- W3215268866 magId "3215268866" @default.
- W3215268866 workType "article" @default.