Matches in SemOpenAlex for { <https://semopenalex.org/work/W4387294098> ?p ?o ?g. }
Showing items 1 to 83 of
83
with 100 items per page.
- W4387294098 abstract "Pre-training on large-scale datasets and then fine-tuning on downstream tasks have become a standard practice in deep learning. However, pre-training data often contain label noise that may adversely affect the generalization of the model. This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. More specifically, through extensive experiments of supervised pre-training models on synthetic noisy ImageNet-1K and YFCC15M datasets, we demonstrate that while slight noise in pre-training can benefit in-domain (ID) transfer performance, where the training and testing data share the same distribution, it always deteriorates out-of-domain (OOD) performance, where training and testing data distribution are different. We empirically verify that the reason behind is noise in pre-training shapes the feature space differently. We then propose a lightweight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization on both ID and OOD tasks, considering one may not be able to fully fine-tune or even access the pre-trained models. We conduct practical experiments on popular vision and language models that are pre-trained on noisy data for evaluation of our approach. Our analysis and results show the importance of this interesting and novel research direction, which we term Noisy Model Learning." @default.
- W4387294098 created "2023-10-03" @default.
- W4387294098 creator A5016771874 @default.
- W4387294098 creator A5019541211 @default.
- W4387294098 creator A5020027500 @default.
- W4387294098 creator A5022499603 @default.
- W4387294098 creator A5049441085 @default.
- W4387294098 creator A5060805738 @default.
- W4387294098 creator A5067803447 @default.
- W4387294098 creator A5072744508 @default.
- W4387294098 date "2023-09-29" @default.
- W4387294098 modified "2023-10-04" @default.
- W4387294098 title "Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks" @default.
- W4387294098 doi "https://doi.org/10.48550/arxiv.2309.17002" @default.
- W4387294098 hasPublicationYear "2023" @default.
- W4387294098 type Work @default.
- W4387294098 citedByCount "0" @default.
- W4387294098 crossrefType "posted-content" @default.
- W4387294098 hasAuthorship W4387294098A5016771874 @default.
- W4387294098 hasAuthorship W4387294098A5019541211 @default.
- W4387294098 hasAuthorship W4387294098A5020027500 @default.
- W4387294098 hasAuthorship W4387294098A5022499603 @default.
- W4387294098 hasAuthorship W4387294098A5049441085 @default.
- W4387294098 hasAuthorship W4387294098A5060805738 @default.
- W4387294098 hasAuthorship W4387294098A5067803447 @default.
- W4387294098 hasAuthorship W4387294098A5072744508 @default.
- W4387294098 hasBestOaLocation W43872940981 @default.
- W4387294098 hasConcept C115961682 @default.
- W4387294098 hasConcept C119857082 @default.
- W4387294098 hasConcept C121332964 @default.
- W4387294098 hasConcept C134306372 @default.
- W4387294098 hasConcept C138885662 @default.
- W4387294098 hasConcept C150899416 @default.
- W4387294098 hasConcept C153294291 @default.
- W4387294098 hasConcept C154945302 @default.
- W4387294098 hasConcept C162324750 @default.
- W4387294098 hasConcept C177148314 @default.
- W4387294098 hasConcept C21547014 @default.
- W4387294098 hasConcept C2776207758 @default.
- W4387294098 hasConcept C2776401178 @default.
- W4387294098 hasConcept C2777211547 @default.
- W4387294098 hasConcept C33923547 @default.
- W4387294098 hasConcept C36503486 @default.
- W4387294098 hasConcept C41008148 @default.
- W4387294098 hasConcept C41895202 @default.
- W4387294098 hasConcept C51632099 @default.
- W4387294098 hasConcept C99498987 @default.
- W4387294098 hasConceptScore W4387294098C115961682 @default.
- W4387294098 hasConceptScore W4387294098C119857082 @default.
- W4387294098 hasConceptScore W4387294098C121332964 @default.
- W4387294098 hasConceptScore W4387294098C134306372 @default.
- W4387294098 hasConceptScore W4387294098C138885662 @default.
- W4387294098 hasConceptScore W4387294098C150899416 @default.
- W4387294098 hasConceptScore W4387294098C153294291 @default.
- W4387294098 hasConceptScore W4387294098C154945302 @default.
- W4387294098 hasConceptScore W4387294098C162324750 @default.
- W4387294098 hasConceptScore W4387294098C177148314 @default.
- W4387294098 hasConceptScore W4387294098C21547014 @default.
- W4387294098 hasConceptScore W4387294098C2776207758 @default.
- W4387294098 hasConceptScore W4387294098C2776401178 @default.
- W4387294098 hasConceptScore W4387294098C2777211547 @default.
- W4387294098 hasConceptScore W4387294098C33923547 @default.
- W4387294098 hasConceptScore W4387294098C36503486 @default.
- W4387294098 hasConceptScore W4387294098C41008148 @default.
- W4387294098 hasConceptScore W4387294098C41895202 @default.
- W4387294098 hasConceptScore W4387294098C51632099 @default.
- W4387294098 hasConceptScore W4387294098C99498987 @default.
- W4387294098 hasLocation W43872940981 @default.
- W4387294098 hasOpenAccess W4387294098 @default.
- W4387294098 hasPrimaryLocation W43872940981 @default.
- W4387294098 hasRelatedWork W2551012455 @default.
- W4387294098 hasRelatedWork W2960456850 @default.
- W4387294098 hasRelatedWork W3010482781 @default.
- W4387294098 hasRelatedWork W3021430260 @default.
- W4387294098 hasRelatedWork W4221167745 @default.
- W4387294098 hasRelatedWork W4281645081 @default.
- W4387294098 hasRelatedWork W4308262314 @default.
- W4387294098 hasRelatedWork W4312200629 @default.
- W4387294098 hasRelatedWork W4382286161 @default.
- W4387294098 hasRelatedWork W4386213806 @default.
- W4387294098 isParatext "false" @default.
- W4387294098 isRetracted "false" @default.
- W4387294098 workType "article" @default.