Matches in SemOpenAlex for { <https://semopenalex.org/work/W4287989672> ?p ?o ?g. }
Showing items 1 to 63 of
63
with 100 items per page.
- W4287989672 abstract "Learning in deep neural networks (DNNs) is implemented through minimizing a highly non-convex loss function, typically by a stochastic gradient descent (SGD) method. This learning process can effectively find good wide minima without being trapped in poor local ones. We present a novel account of how such effective deep learning emerges through the interactions of the SGD and the geometrical structure of the loss landscape. Rather than being a normal diffusion process (i.e. Brownian motion) as often assumed, we find that the SGD exhibits rich, complex dynamics when navigating through the loss landscape; initially, the SGD exhibits anomalous superdiffusion, which attenuates gradually and changes to subdiffusion at long times when the solution is reached. Such learning dynamics happen ubiquitously in different DNNs such as ResNet and VGG-like networks and are insensitive to batch size and learning rate. The anomalous superdiffusion process during the initial learning phase indicates that the motion of SGD along the loss landscape possesses intermittent, big jumps; this non-equilibrium property enables the SGD to escape from sharp local minima. By adapting the methods developed for studying energy landscapes in complex physical systems, we find that such superdiffusive learning dynamics are due to the interactions of the SGD and the fractal-like structure of the loss landscape. We further develop a simple model to demonstrate the mechanistic role of the fractal loss landscape in enabling the SGD to effectively find global minima. Our results thus reveal the effectiveness of deep learning from a novel perspective and have implications for designing efficient deep neural networks." @default.
- W4287989672 created "2022-07-26" @default.
- W4287989672 creator A5000919999 @default.
- W4287989672 creator A5060508343 @default.
- W4287989672 creator A5089531423 @default.
- W4287989672 date "2020-09-22" @default.
- W4287989672 modified "2023-10-17" @default.
- W4287989672 title "Anomalous diffusion dynamics of learning in deep neural networks" @default.
- W4287989672 doi "https://doi.org/10.48550/arxiv.2009.10588" @default.
- W4287989672 hasPublicationYear "2020" @default.
- W4287989672 type Work @default.
- W4287989672 citedByCount "0" @default.
- W4287989672 crossrefType "posted-content" @default.
- W4287989672 hasAuthorship W4287989672A5000919999 @default.
- W4287989672 hasAuthorship W4287989672A5060508343 @default.
- W4287989672 hasAuthorship W4287989672A5089531423 @default.
- W4287989672 hasBestOaLocation W42879896721 @default.
- W4287989672 hasConcept C108583219 @default.
- W4287989672 hasConcept C112401455 @default.
- W4287989672 hasConcept C119621388 @default.
- W4287989672 hasConcept C121332964 @default.
- W4287989672 hasConcept C121864883 @default.
- W4287989672 hasConcept C134306372 @default.
- W4287989672 hasConcept C154945302 @default.
- W4287989672 hasConcept C186633575 @default.
- W4287989672 hasConcept C206688291 @default.
- W4287989672 hasConcept C33923547 @default.
- W4287989672 hasConcept C40636538 @default.
- W4287989672 hasConcept C41008148 @default.
- W4287989672 hasConcept C50644808 @default.
- W4287989672 hasConcept C62520636 @default.
- W4287989672 hasConcept C97355855 @default.
- W4287989672 hasConceptScore W4287989672C108583219 @default.
- W4287989672 hasConceptScore W4287989672C112401455 @default.
- W4287989672 hasConceptScore W4287989672C119621388 @default.
- W4287989672 hasConceptScore W4287989672C121332964 @default.
- W4287989672 hasConceptScore W4287989672C121864883 @default.
- W4287989672 hasConceptScore W4287989672C134306372 @default.
- W4287989672 hasConceptScore W4287989672C154945302 @default.
- W4287989672 hasConceptScore W4287989672C186633575 @default.
- W4287989672 hasConceptScore W4287989672C206688291 @default.
- W4287989672 hasConceptScore W4287989672C33923547 @default.
- W4287989672 hasConceptScore W4287989672C40636538 @default.
- W4287989672 hasConceptScore W4287989672C41008148 @default.
- W4287989672 hasConceptScore W4287989672C50644808 @default.
- W4287989672 hasConceptScore W4287989672C62520636 @default.
- W4287989672 hasConceptScore W4287989672C97355855 @default.
- W4287989672 hasLocation W42879896721 @default.
- W4287989672 hasOpenAccess W4287989672 @default.
- W4287989672 hasPrimaryLocation W42879896721 @default.
- W4287989672 hasRelatedWork W1973164090 @default.
- W4287989672 hasRelatedWork W1976502508 @default.
- W4287989672 hasRelatedWork W1986336785 @default.
- W4287989672 hasRelatedWork W1999930243 @default.
- W4287989672 hasRelatedWork W2030225915 @default.
- W4287989672 hasRelatedWork W2085829122 @default.
- W4287989672 hasRelatedWork W2141593703 @default.
- W4287989672 hasRelatedWork W3087768334 @default.
- W4287989672 hasRelatedWork W3099105341 @default.
- W4287989672 hasRelatedWork W3166670227 @default.
- W4287989672 isParatext "false" @default.
- W4287989672 isRetracted "false" @default.
- W4287989672 workType "article" @default.