FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2950005703> ?p ?o ?g. }

• W2950005703 abstract "Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep networks. The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model. Several such singularities have been identified in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes. These singularities cause degenerate manifolds in the loss landscape that slow down learning. We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent. Moreover, for typical initializations, skip connections move the network away from the ghosts of these singularities and sculpt the landscape around them to alleviate the learning slow-down. These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets." @default.
• W2950005703 created "2019-06-27" @default.
• W2950005703 creator A5046369040 @default.
• W2950005703 creator A5057055829 @default.
• W2950005703 date "2017-01-31" @default.
• W2950005703 modified "2023-09-27" @default.
• W2950005703 title "Skip Connections Eliminate Singularities." @default.
• W2950005703 cites W1522301498 @default.
• W2950005703 cites W1552154247 @default.
• W2950005703 cites W1995842804 @default.
• W2950005703 cites W2001689396 @default.
• W2950005703 cites W2006903949 @default.
• W2950005703 cites W2040868835 @default.
• W2950005703 cites W2097998348 @default.
• W2950005703 cites W2107878631 @default.
• W2950005703 cites W2125911849 @default.
• W2950005703 cites W2125930537 @default.
• W2950005703 cites W2302255633 @default.
• W2950005703 cites W2337199865 @default.
• W2950005703 cites W2511730936 @default.
• W2950005703 cites W2556154603 @default.
• W2950005703 cites W2560977758 @default.
• W2950005703 cites W2591954064 @default.
• W2950005703 cites W2753926022 @default.
• W2950005703 cites W2914728119 @default.
• W2950005703 cites W2949117887 @default.
• W2950005703 cites W2949650786 @default.
• W2950005703 cites W2952574409 @default.
• W2950005703 cites W2953328958 @default.
• W2950005703 cites W577198184 @default.
• W2950005703 cites W194249466 @default.
• W2950005703 hasPublicationYear "2017" @default.
• W2950005703 type Work @default.
• W2950005703 sameAs 2950005703 @default.
• W2950005703 citedByCount "12" @default.
• W2950005703 countsByYear W29500057032017 @default.
• W2950005703 countsByYear W29500057032018 @default.
• W2950005703 countsByYear W29500057032019 @default.
• W2950005703 countsByYear W29500057032020 @default.
• W2950005703 countsByYear W29500057032021 @default.
• W2950005703 crossrefType "posted-content" @default.
• W2950005703 hasAuthorship W2950005703A5046369040 @default.
• W2950005703 hasAuthorship W2950005703A5057055829 @default.
• W2950005703 hasConcept C114614502 @default.
• W2950005703 hasConcept C119857082 @default.
• W2950005703 hasConcept C121332964 @default.
• W2950005703 hasConcept C122770356 @default.
• W2950005703 hasConcept C12843 @default.
• W2950005703 hasConcept C134306372 @default.
• W2950005703 hasConcept C136197465 @default.
• W2950005703 hasConcept C154945302 @default.
• W2950005703 hasConcept C184720557 @default.
• W2950005703 hasConcept C21308566 @default.
• W2950005703 hasConcept C24890656 @default.
• W2950005703 hasConcept C2524010 @default.
• W2950005703 hasConcept C2779886137 @default.
• W2950005703 hasConcept C33923547 @default.
• W2950005703 hasConcept C41008148 @default.
• W2950005703 hasConcept C62520636 @default.
• W2950005703 hasConcept C62611344 @default.
• W2950005703 hasConcept C72319582 @default.
• W2950005703 hasConceptScore W2950005703C114614502 @default.
• W2950005703 hasConceptScore W2950005703C119857082 @default.
• W2950005703 hasConceptScore W2950005703C121332964 @default.
• W2950005703 hasConceptScore W2950005703C122770356 @default.
• W2950005703 hasConceptScore W2950005703C12843 @default.
• W2950005703 hasConceptScore W2950005703C134306372 @default.
• W2950005703 hasConceptScore W2950005703C136197465 @default.
• W2950005703 hasConceptScore W2950005703C154945302 @default.
• W2950005703 hasConceptScore W2950005703C184720557 @default.
• W2950005703 hasConceptScore W2950005703C21308566 @default.
• W2950005703 hasConceptScore W2950005703C24890656 @default.
• W2950005703 hasConceptScore W2950005703C2524010 @default.
• W2950005703 hasConceptScore W2950005703C2779886137 @default.
• W2950005703 hasConceptScore W2950005703C33923547 @default.
• W2950005703 hasConceptScore W2950005703C41008148 @default.
• W2950005703 hasConceptScore W2950005703C62520636 @default.
• W2950005703 hasConceptScore W2950005703C62611344 @default.
• W2950005703 hasConceptScore W2950005703C72319582 @default.
• W2950005703 hasLocation W29500057031 @default.
• W2950005703 hasOpenAccess W2950005703 @default.
• W2950005703 hasPrimaryLocation W29500057031 @default.
• W2950005703 hasRelatedWork W116046179 @default.
• W2950005703 hasRelatedWork W1522301498 @default.
• W2950005703 hasRelatedWork W1533861849 @default.
• W2950005703 hasRelatedWork W1677182931 @default.
• W2950005703 hasRelatedWork W1901129140 @default.
• W2950005703 hasRelatedWork W2043530395 @default.
• W2950005703 hasRelatedWork W2059558823 @default.
• W2950005703 hasRelatedWork W2092886115 @default.
• W2950005703 hasRelatedWork W2097117768 @default.
• W2950005703 hasRelatedWork W2194775991 @default.
• W2950005703 hasRelatedWork W2541674938 @default.
• W2950005703 hasRelatedWork W2904130053 @default.
• W2950005703 hasRelatedWork W2921580130 @default.
• W2950005703 hasRelatedWork W2949117887 @default.
• W2950005703 hasRelatedWork W2949878810 @default.
• W2950005703 hasRelatedWork W2963427613 @default.
• W2950005703 hasRelatedWork W2963446712 @default.
• W2950005703 hasRelatedWork W2981491503 @default.

• next