Matches in SemOpenAlex for { <https://semopenalex.org/work/W2585476045> ?p ?o ?g. }
Showing items 1 to 82 of
82
with 100 items per page.
- W2585476045 abstract "Skip connections made the training of very deep neural networks possible and have become an indispendable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep neural networks. We argue that skip connections help break symmetries inherent in the loss landscapes of deep networks, leading to drastically simplified landscapes. In particular, skip connections between adjacent layers in a multilayer network break the permutation symmetry of nodes in a given layer, and the recently proposed DenseNet architecture, where each layer projects skip connections to every layer above it, also breaks the rescaling symmetry of connectivity matrices between different layers. This hypothesis is supported by evidence from a toy model with binary weights and from experiments with fully-connected networks suggesting (i) that skip connections do not necessarily improve training unless they help break symmetries and (ii) that alternative ways of breaking the symmetries also lead to significant performance improvements in training deep networks, hence there is nothing special about skip connections in this respect. We find, however, that skip connections confer additional benefits over and above symmetry-breaking, such as the ability to deal effectively with the vanishing gradients problem." @default.
- W2585476045 created "2017-02-10" @default.
- W2585476045 creator A5046369040 @default.
- W2585476045 date "2017-01-31" @default.
- W2585476045 modified "2023-09-27" @default.
- W2585476045 title "Skip Connections as Effective Symmetry-Breaking." @default.
- W2585476045 hasPublicationYear "2017" @default.
- W2585476045 type Work @default.
- W2585476045 sameAs 2585476045 @default.
- W2585476045 citedByCount "6" @default.
- W2585476045 countsByYear W25854760452017 @default.
- W2585476045 countsByYear W25854760452018 @default.
- W2585476045 countsByYear W25854760452020 @default.
- W2585476045 crossrefType "posted-content" @default.
- W2585476045 hasAuthorship W2585476045A5046369040 @default.
- W2585476045 hasConcept C109214941 @default.
- W2585476045 hasConcept C121332964 @default.
- W2585476045 hasConcept C136197465 @default.
- W2585476045 hasConcept C154945302 @default.
- W2585476045 hasConcept C178790620 @default.
- W2585476045 hasConcept C185592680 @default.
- W2585476045 hasConcept C204795200 @default.
- W2585476045 hasConcept C21308566 @default.
- W2585476045 hasConcept C24890656 @default.
- W2585476045 hasConcept C2524010 @default.
- W2585476045 hasConcept C2779227376 @default.
- W2585476045 hasConcept C2779886137 @default.
- W2585476045 hasConcept C2984842247 @default.
- W2585476045 hasConcept C33923547 @default.
- W2585476045 hasConcept C41008148 @default.
- W2585476045 hasConcept C48372109 @default.
- W2585476045 hasConcept C50644808 @default.
- W2585476045 hasConcept C80444323 @default.
- W2585476045 hasConcept C94375191 @default.
- W2585476045 hasConcept C96469262 @default.
- W2585476045 hasConceptScore W2585476045C109214941 @default.
- W2585476045 hasConceptScore W2585476045C121332964 @default.
- W2585476045 hasConceptScore W2585476045C136197465 @default.
- W2585476045 hasConceptScore W2585476045C154945302 @default.
- W2585476045 hasConceptScore W2585476045C178790620 @default.
- W2585476045 hasConceptScore W2585476045C185592680 @default.
- W2585476045 hasConceptScore W2585476045C204795200 @default.
- W2585476045 hasConceptScore W2585476045C21308566 @default.
- W2585476045 hasConceptScore W2585476045C24890656 @default.
- W2585476045 hasConceptScore W2585476045C2524010 @default.
- W2585476045 hasConceptScore W2585476045C2779227376 @default.
- W2585476045 hasConceptScore W2585476045C2779886137 @default.
- W2585476045 hasConceptScore W2585476045C2984842247 @default.
- W2585476045 hasConceptScore W2585476045C33923547 @default.
- W2585476045 hasConceptScore W2585476045C41008148 @default.
- W2585476045 hasConceptScore W2585476045C48372109 @default.
- W2585476045 hasConceptScore W2585476045C50644808 @default.
- W2585476045 hasConceptScore W2585476045C80444323 @default.
- W2585476045 hasConceptScore W2585476045C94375191 @default.
- W2585476045 hasConceptScore W2585476045C96469262 @default.
- W2585476045 hasLocation W25854760451 @default.
- W2585476045 hasOpenAccess W2585476045 @default.
- W2585476045 hasPrimaryLocation W25854760451 @default.
- W2585476045 hasRelatedWork W1522294902 @default.
- W2585476045 hasRelatedWork W1591558774 @default.
- W2585476045 hasRelatedWork W171992139 @default.
- W2585476045 hasRelatedWork W174785108 @default.
- W2585476045 hasRelatedWork W2054021332 @default.
- W2585476045 hasRelatedWork W2093218111 @default.
- W2585476045 hasRelatedWork W2123081773 @default.
- W2585476045 hasRelatedWork W2171902495 @default.
- W2585476045 hasRelatedWork W2188510695 @default.
- W2585476045 hasRelatedWork W2194775991 @default.
- W2585476045 hasRelatedWork W22057570 @default.
- W2585476045 hasRelatedWork W2293108845 @default.
- W2585476045 hasRelatedWork W2530414440 @default.
- W2585476045 hasRelatedWork W2783041130 @default.
- W2585476045 hasRelatedWork W2964121744 @default.
- W2585476045 hasRelatedWork W3116088813 @default.
- W2585476045 hasRelatedWork W3153303803 @default.
- W2585476045 hasRelatedWork W3166092007 @default.
- W2585476045 hasRelatedWork W3168793255 @default.
- W2585476045 hasRelatedWork W31840583 @default.
- W2585476045 isParatext "false" @default.
- W2585476045 isRetracted "false" @default.
- W2585476045 magId "2585476045" @default.
- W2585476045 workType "article" @default.