Matches in SemOpenAlex for { <https://semopenalex.org/work/W3207404093> ?p ?o ?g. }
- W3207404093 abstract "Self-supervised learning (SSL) is a scalable way to learn general visual representations since it learns without labels. However, large-scale unlabeled datasets in the wild often have long-tailed label distributions, where we know little about the behavior of SSL. In this work, we systematically investigate self-supervised learning under dataset imbalance. First, we find out via extensive experiments that off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. The performance gap between balanced and imbalanced pre-training with SSL is significantly smaller than the gap with supervised learning, across sample sizes, for both in-domain and, especially, out-of-domain evaluation. Second, towards understanding the robustness of SSL, we hypothesize that SSL learns richer features from frequent data: it may learn label-irrelevant-but-transferable features that help classify the rare classes and downstream tasks. In contrast, supervised learning has no incentive to learn features irrelevant to the labels from frequent examples. We validate this hypothesis with semi-synthetic experiments and theoretical analyses on a simplified setting. Third, inspired by the theoretical insights, we devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets with several evaluation criteria, closing the small gap between balanced and imbalanced datasets with the same number of examples." @default.
- W3207404093 created "2021-10-25" @default.
- W3207404093 creator A5057287134 @default.
- W3207404093 creator A5060933081 @default.
- W3207404093 creator A5061905935 @default.
- W3207404093 creator A5075018873 @default.
- W3207404093 date "2021-10-11" @default.
- W3207404093 modified "2023-09-23" @default.
- W3207404093 title "Self-supervised Learning is More Robust to Dataset Imbalance" @default.
- W3207404093 cites W1797268635 @default.
- W3207404093 cites W1846799578 @default.
- W3207404093 cites W1977295328 @default.
- W3207404093 cites W1994410331 @default.
- W3207404093 cites W2023060295 @default.
- W3207404093 cites W2040010062 @default.
- W3207404093 cites W2095705004 @default.
- W3207404093 cites W2104933073 @default.
- W3207404093 cites W2117539524 @default.
- W3207404093 cites W2118858186 @default.
- W3207404093 cites W2118978333 @default.
- W3207404093 cites W2136256517 @default.
- W3207404093 cites W2138011018 @default.
- W3207404093 cites W2148143831 @default.
- W3207404093 cites W2194775991 @default.
- W3207404093 cites W2321533354 @default.
- W3207404093 cites W2338318698 @default.
- W3207404093 cites W2753300133 @default.
- W3207404093 cites W2767106145 @default.
- W3207404093 cites W2951585248 @default.
- W3207404093 cites W2962742544 @default.
- W3207404093 cites W2962858109 @default.
- W3207404093 cites W2962933664 @default.
- W3207404093 cites W2963212406 @default.
- W3207404093 cites W2963351448 @default.
- W3207404093 cites W2963371670 @default.
- W3207404093 cites W2963453204 @default.
- W3207404093 cites W2963469976 @default.
- W3207404093 cites W2963691377 @default.
- W3207404093 cites W2963749571 @default.
- W3207404093 cites W2963837241 @default.
- W3207404093 cites W2970711760 @default.
- W3207404093 cites W2970941190 @default.
- W3207404093 cites W2995197345 @default.
- W3207404093 cites W2995808739 @default.
- W3207404093 cites W3009571263 @default.
- W3207404093 cites W3034369739 @default.
- W3207404093 cites W3034840375 @default.
- W3207404093 cites W3034978746 @default.
- W3207404093 cites W3035054804 @default.
- W3207404093 cites W3035060554 @default.
- W3207404093 cites W3035342001 @default.
- W3207404093 cites W3035524453 @default.
- W3207404093 cites W3036224891 @default.
- W3207404093 cites W3045241369 @default.
- W3207404093 cites W3046882683 @default.
- W3207404093 cites W3093667565 @default.
- W3207404093 cites W3102572097 @default.
- W3207404093 cites W3102631365 @default.
- W3207404093 cites W3104182862 @default.
- W3207404093 cites W3118608800 @default.
- W3207404093 cites W3119113126 @default.
- W3207404093 cites W3122542623 @default.
- W3207404093 cites W3122855191 @default.
- W3207404093 cites W3126272270 @default.
- W3207404093 cites W3129532073 @default.
- W3207404093 cites W3131573008 @default.
- W3207404093 cites W3135715136 @default.
- W3207404093 cites W3149173402 @default.
- W3207404093 cites W3160314846 @default.
- W3207404093 cites W3166143997 @default.
- W3207404093 cites W3166596953 @default.
- W3207404093 cites W3167521527 @default.
- W3207404093 cites W3171007011 @default.
- W3207404093 cites W3177200443 @default.
- W3207404093 cites W3214375535 @default.
- W3207404093 cites W343636949 @default.
- W3207404093 cites W85350352 @default.
- W3207404093 doi "https://doi.org/10.48550/arxiv.2110.05025" @default.
- W3207404093 hasPublicationYear "2021" @default.
- W3207404093 type Work @default.
- W3207404093 sameAs 3207404093 @default.
- W3207404093 citedByCount "2" @default.
- W3207404093 countsByYear W32074040932022 @default.
- W3207404093 countsByYear W32074040932023 @default.
- W3207404093 crossrefType "posted-content" @default.
- W3207404093 hasAuthorship W3207404093A5057287134 @default.
- W3207404093 hasAuthorship W3207404093A5060933081 @default.
- W3207404093 hasAuthorship W3207404093A5061905935 @default.
- W3207404093 hasAuthorship W3207404093A5075018873 @default.
- W3207404093 hasBestOaLocation W32074040931 @default.
- W3207404093 hasConcept C104317684 @default.
- W3207404093 hasConcept C119857082 @default.
- W3207404093 hasConcept C136389625 @default.
- W3207404093 hasConcept C153180895 @default.
- W3207404093 hasConcept C154945302 @default.
- W3207404093 hasConcept C185592680 @default.
- W3207404093 hasConcept C2776135515 @default.
- W3207404093 hasConcept C2776145971 @default.
- W3207404093 hasConcept C41008148 @default.
- W3207404093 hasConcept C48044578 @default.