Matches in SemOpenAlex for { <https://semopenalex.org/work/W4281702319> ?p ?o ?g. }
Showing items 1 to 71 of
71
with 100 items per page.
- W4281702319 abstract "Understanding the relation between deep and shallow neural networks is extremely important for the theoretical study of deep learning. In this work, we discover an embedding principle in depth that loss landscape of an NN contains all critical points of the loss landscapes for shallower NNs. The key tool for our discovery is the critical lifting operator proposed in this work that maps any critical point of a network to critical manifolds of any deeper network while preserving the outputs. This principle provides new insights to many widely observed behaviors of DNNs. Regarding the easy training of deep networks, we show that local minimum of an NN can be lifted to strict saddle points of a deeper NN. Regarding the acceleration effect of batch normalization, we demonstrate that batch normalization helps avoid the critical manifolds lifted from shallower NNs by suppressing layer linearization. We also prove that increasing training data shrinks the lifted critical manifolds, which can result in acceleration of training as demonstrated in experiments. Overall, our discovery of the embedding principle in depth uncovers the depth-wise hierarchical structure of deep learning loss landscape, which serves as a solid foundation for the further study about the role of depth for DNNs." @default.
- W4281702319 created "2022-06-13" @default.
- W4281702319 creator A5002236438 @default.
- W4281702319 creator A5025402450 @default.
- W4281702319 creator A5033982342 @default.
- W4281702319 creator A5039481707 @default.
- W4281702319 date "2022-05-26" @default.
- W4281702319 modified "2023-10-18" @default.
- W4281702319 title "Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks" @default.
- W4281702319 doi "https://doi.org/10.48550/arxiv.2205.13283" @default.
- W4281702319 hasPublicationYear "2022" @default.
- W4281702319 type Work @default.
- W4281702319 citedByCount "0" @default.
- W4281702319 crossrefType "posted-content" @default.
- W4281702319 hasAuthorship W4281702319A5002236438 @default.
- W4281702319 hasAuthorship W4281702319A5025402450 @default.
- W4281702319 hasAuthorship W4281702319A5033982342 @default.
- W4281702319 hasAuthorship W4281702319A5039481707 @default.
- W4281702319 hasBestOaLocation W42817023191 @default.
- W4281702319 hasConcept C108583219 @default.
- W4281702319 hasConcept C11210021 @default.
- W4281702319 hasConcept C11413529 @default.
- W4281702319 hasConcept C114614502 @default.
- W4281702319 hasConcept C121332964 @default.
- W4281702319 hasConcept C136886441 @default.
- W4281702319 hasConcept C144024400 @default.
- W4281702319 hasConcept C154945302 @default.
- W4281702319 hasConcept C158622935 @default.
- W4281702319 hasConcept C184720557 @default.
- W4281702319 hasConcept C19165224 @default.
- W4281702319 hasConcept C2524010 @default.
- W4281702319 hasConcept C2681867 @default.
- W4281702319 hasConcept C33923547 @default.
- W4281702319 hasConcept C41008148 @default.
- W4281702319 hasConcept C41608201 @default.
- W4281702319 hasConcept C50644808 @default.
- W4281702319 hasConcept C62520636 @default.
- W4281702319 hasConceptScore W4281702319C108583219 @default.
- W4281702319 hasConceptScore W4281702319C11210021 @default.
- W4281702319 hasConceptScore W4281702319C11413529 @default.
- W4281702319 hasConceptScore W4281702319C114614502 @default.
- W4281702319 hasConceptScore W4281702319C121332964 @default.
- W4281702319 hasConceptScore W4281702319C136886441 @default.
- W4281702319 hasConceptScore W4281702319C144024400 @default.
- W4281702319 hasConceptScore W4281702319C154945302 @default.
- W4281702319 hasConceptScore W4281702319C158622935 @default.
- W4281702319 hasConceptScore W4281702319C184720557 @default.
- W4281702319 hasConceptScore W4281702319C19165224 @default.
- W4281702319 hasConceptScore W4281702319C2524010 @default.
- W4281702319 hasConceptScore W4281702319C2681867 @default.
- W4281702319 hasConceptScore W4281702319C33923547 @default.
- W4281702319 hasConceptScore W4281702319C41008148 @default.
- W4281702319 hasConceptScore W4281702319C41608201 @default.
- W4281702319 hasConceptScore W4281702319C50644808 @default.
- W4281702319 hasConceptScore W4281702319C62520636 @default.
- W4281702319 hasLocation W42817023191 @default.
- W4281702319 hasOpenAccess W4281702319 @default.
- W4281702319 hasPrimaryLocation W42817023191 @default.
- W4281702319 hasRelatedWork W2410085756 @default.
- W4281702319 hasRelatedWork W2731899572 @default.
- W4281702319 hasRelatedWork W2760944304 @default.
- W4281702319 hasRelatedWork W2767142767 @default.
- W4281702319 hasRelatedWork W2794115703 @default.
- W4281702319 hasRelatedWork W2963721882 @default.
- W4281702319 hasRelatedWork W2963958000 @default.
- W4281702319 hasRelatedWork W3156188347 @default.
- W4281702319 hasRelatedWork W4205989545 @default.
- W4281702319 hasRelatedWork W3126631784 @default.
- W4281702319 isParatext "false" @default.
- W4281702319 isRetracted "false" @default.
- W4281702319 workType "article" @default.