Matches in SemOpenAlex for { <https://semopenalex.org/work/W3099180329> ?p ?o ?g. }
Showing items 1 to 87 of
87
with 100 items per page.
- W3099180329 endingPage "9585" @default.
- W3099180329 startingPage "9573" @default.
- W3099180329 abstract "Several works have proposed Simplicity Bias (SB)---the tendency of standard training procedures such as Stochastic Gradient Descent (SGD) to find simple models---to justify why neural networks generalize well [Arpit et al. 2017, Nakkiran et al. 2019, Soudry et al. 2018]. However, the precise notion of simplicity remains vague. Furthermore, previous settings that use SB to theoretically justify why neural networks generalize well do not simultaneously capture the non-robustness of neural networks---a widely observed phenomenon in practice [Goodfellow et al. 2014, Jo and Bengio 2017]. We attempt to reconcile SB and the superior standard generalization of neural networks with the non-robustness observed in practice by designing datasets that (a) incorporate a precise notion of simplicity, (b) comprise multiple predictive features with varying levels of simplicity, and (c) capture the non-robustness of neural networks trained on real data. Through theory and empirics on these datasets, we make four observations: (i) SB of SGD and variants can be extreme: neural networks can exclusively rely on the simplest feature and remain invariant to all predictive complex features. (ii) The extreme aspect of SB could explain why seemingly benign distribution shifts and small adversarial perturbations significantly degrade model performance. (iii) Contrary to conventional wisdom, SB can also hurt generalization on the same data distribution, as SB persists even when the simplest feature has less predictive power than the more complex features. (iv) Common approaches to improve generalization and robustness---ensembles and adversarial training---can fail in mitigating SB and its pitfalls. Given the role of SB in training neural networks, we hope that the proposed datasets and methods serve as an effective testbed to evaluate novel algorithmic approaches aimed at avoiding the pitfalls of SB." @default.
- W3099180329 created "2020-11-23" @default.
- W3099180329 creator A5007012613 @default.
- W3099180329 creator A5031731960 @default.
- W3099180329 creator A5034432097 @default.
- W3099180329 creator A5048906669 @default.
- W3099180329 creator A5084889738 @default.
- W3099180329 date "2020-01-01" @default.
- W3099180329 modified "2023-09-24" @default.
- W3099180329 title "The Pitfalls of Simplicity Bias in Neural Networks" @default.
- W3099180329 hasPublicationYear "2020" @default.
- W3099180329 type Work @default.
- W3099180329 sameAs 3099180329 @default.
- W3099180329 citedByCount "18" @default.
- W3099180329 countsByYear W30991803292020 @default.
- W3099180329 countsByYear W30991803292021 @default.
- W3099180329 countsByYear W30991803292022 @default.
- W3099180329 crossrefType "proceedings-article" @default.
- W3099180329 hasAuthorship W3099180329A5007012613 @default.
- W3099180329 hasAuthorship W3099180329A5031731960 @default.
- W3099180329 hasAuthorship W3099180329A5034432097 @default.
- W3099180329 hasAuthorship W3099180329A5048906669 @default.
- W3099180329 hasAuthorship W3099180329A5084889738 @default.
- W3099180329 hasConcept C104317684 @default.
- W3099180329 hasConcept C111472728 @default.
- W3099180329 hasConcept C11413529 @default.
- W3099180329 hasConcept C119857082 @default.
- W3099180329 hasConcept C134306372 @default.
- W3099180329 hasConcept C138885662 @default.
- W3099180329 hasConcept C154945302 @default.
- W3099180329 hasConcept C177148314 @default.
- W3099180329 hasConcept C185592680 @default.
- W3099180329 hasConcept C2776372474 @default.
- W3099180329 hasConcept C2984842247 @default.
- W3099180329 hasConcept C33923547 @default.
- W3099180329 hasConcept C37736160 @default.
- W3099180329 hasConcept C41008148 @default.
- W3099180329 hasConcept C50644808 @default.
- W3099180329 hasConcept C55493867 @default.
- W3099180329 hasConcept C63479239 @default.
- W3099180329 hasConceptScore W3099180329C104317684 @default.
- W3099180329 hasConceptScore W3099180329C111472728 @default.
- W3099180329 hasConceptScore W3099180329C11413529 @default.
- W3099180329 hasConceptScore W3099180329C119857082 @default.
- W3099180329 hasConceptScore W3099180329C134306372 @default.
- W3099180329 hasConceptScore W3099180329C138885662 @default.
- W3099180329 hasConceptScore W3099180329C154945302 @default.
- W3099180329 hasConceptScore W3099180329C177148314 @default.
- W3099180329 hasConceptScore W3099180329C185592680 @default.
- W3099180329 hasConceptScore W3099180329C2776372474 @default.
- W3099180329 hasConceptScore W3099180329C2984842247 @default.
- W3099180329 hasConceptScore W3099180329C33923547 @default.
- W3099180329 hasConceptScore W3099180329C37736160 @default.
- W3099180329 hasConceptScore W3099180329C41008148 @default.
- W3099180329 hasConceptScore W3099180329C50644808 @default.
- W3099180329 hasConceptScore W3099180329C55493867 @default.
- W3099180329 hasConceptScore W3099180329C63479239 @default.
- W3099180329 hasLocation W30991803291 @default.
- W3099180329 hasOpenAccess W3099180329 @default.
- W3099180329 hasPrimaryLocation W30991803291 @default.
- W3099180329 hasRelatedWork W2108598243 @default.
- W3099180329 hasRelatedWork W2112796928 @default.
- W3099180329 hasRelatedWork W2194775991 @default.
- W3099180329 hasRelatedWork W2962835968 @default.
- W3099180329 hasRelatedWork W2963207607 @default.
- W3099180329 hasRelatedWork W2963446712 @default.
- W3099180329 hasRelatedWork W2964116600 @default.
- W3099180329 hasRelatedWork W2964122761 @default.
- W3099180329 hasRelatedWork W2964153729 @default.
- W3099180329 hasRelatedWork W2964253222 @default.
- W3099180329 hasRelatedWork W2971127900 @default.
- W3099180329 hasRelatedWork W2993049325 @default.
- W3099180329 hasRelatedWork W3041269312 @default.
- W3099180329 hasRelatedWork W3092696781 @default.
- W3099180329 hasRelatedWork W3100924069 @default.
- W3099180329 hasRelatedWork W3118608800 @default.
- W3099180329 hasRelatedWork W3120955712 @default.
- W3099180329 hasRelatedWork W3137695714 @default.
- W3099180329 hasRelatedWork W3160359250 @default.
- W3099180329 hasRelatedWork W3208923398 @default.
- W3099180329 hasVolume "33" @default.
- W3099180329 isParatext "false" @default.
- W3099180329 isRetracted "false" @default.
- W3099180329 magId "3099180329" @default.
- W3099180329 workType "article" @default.