FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W3124384256> ?p ?o ?g. }

• W3124384256 abstract "Knowledge distillation is an effective approach to leverage a well-trained network or an ensemble of them, named as the teacher, to guide the training of a student network. The outputs from the teacher network are used as soft labels for supervising the training of a new network. Recent studies (M uller et al., 2019; Yuan et al., 2020) revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network. From the perspective of statistical learning, regularization aims to reduce the variance, however how bias and variance change is not clear for training with soft labels. In this paper, we investigate the bias-variance tradeoff brought by distillation with soft labels. Specifically, we observe that during training the bias-variance tradeoff varies sample-wisely. Further, under the same distillation temperature setting, we observe that the distillation performance is negatively associated with the number of some specific samples, which are named as regularization samples since these samples lead to bias increasing and variance decreasing. Nevertheless, we empirically find that completely filtering out regularization samples also deteriorates distillation performance. Our discoveries inspired us to propose the novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff. Experiments on standard evaluation benchmarks validate the effectiveness of our method. Our code is available in the supplementary." @default.
• W3124384256 created "2021-02-01" @default.
• W3124384256 creator A5014166482 @default.
• W3124384256 creator A5030945438 @default.
• W3124384256 creator A5047085626 @default.
• W3124384256 creator A5052441498 @default.
• W3124384256 creator A5079245916 @default.
• W3124384256 creator A5080366019 @default.
• W3124384256 creator A5085245110 @default.
• W3124384256 date "2021-05-03" @default.
• W3124384256 modified "2023-09-26" @default.
• W3124384256 title "Rethinking Soft Labels for Knowledge Distillation: A Bias–Variance Tradeoff Perspective" @default.
• W3124384256 cites W126840538 @default.
• W3124384256 cites W1516193414 @default.
• W3124384256 cites W1540358749 @default.
• W3124384256 cites W1592410721 @default.
• W3124384256 cites W1663973292 @default.
• W3124384256 cites W1821462560 @default.
• W3124384256 cites W2076118331 @default.
• W3124384256 cites W2094051685 @default.
• W3124384256 cites W2108598243 @default.
• W3124384256 cites W2134797427 @default.
• W3124384256 cites W2183341477 @default.
• W3124384256 cites W2194775991 @default.
• W3124384256 cites W2581377246 @default.
• W3124384256 cites W2613678836 @default.
• W3124384256 cites W2731516819 @default.
• W3124384256 cites W2739879705 @default.
• W3124384256 cites W2887783173 @default.
• W3124384256 cites W2897025488 @default.
• W3124384256 cites W2908467437 @default.
• W3124384256 cites W2936864631 @default.
• W3124384256 cites W2945289329 @default.
• W3124384256 cites W2963140444 @default.
• W3124384256 cites W2963351448 @default.
• W3124384256 cites W2963518130 @default.
• W3124384256 cites W2963534679 @default.
• W3124384256 cites W2964082701 @default.
• W3124384256 cites W2964118293 @default.
• W3124384256 cites W2965100203 @default.
• W3124384256 cites W2970206392 @default.
• W3124384256 cites W2970290137 @default.
• W3124384256 cites W2970454332 @default.
• W3124384256 cites W2982157312 @default.
• W3124384256 cites W2982242214 @default.
• W3124384256 cites W2984898826 @default.
• W3124384256 cites W2986015886 @default.
• W3124384256 cites W2995607862 @default.
• W3124384256 cites W2996514524 @default.
• W3124384256 cites W3008906732 @default.
• W3124384256 cites W3034756453 @default.
• W3124384256 cites W3118608800 @default.
• W3124384256 hasPublicationYear "2021" @default.
• W3124384256 type Work @default.
• W3124384256 sameAs 3124384256 @default.
• W3124384256 citedByCount "6" @default.
• W3124384256 countsByYear W31243842562021 @default.
• W3124384256 crossrefType "proceedings-article" @default.
• W3124384256 hasAuthorship W3124384256A5014166482 @default.
• W3124384256 hasAuthorship W3124384256A5030945438 @default.
• W3124384256 hasAuthorship W3124384256A5047085626 @default.
• W3124384256 hasAuthorship W3124384256A5052441498 @default.
• W3124384256 hasAuthorship W3124384256A5079245916 @default.
• W3124384256 hasAuthorship W3124384256A5080366019 @default.
• W3124384256 hasAuthorship W3124384256A5085245110 @default.
• W3124384256 hasConcept C111919701 @default.
• W3124384256 hasConcept C115575686 @default.
• W3124384256 hasConcept C119857082 @default.
• W3124384256 hasConcept C121955636 @default.
• W3124384256 hasConcept C12713177 @default.
• W3124384256 hasConcept C144133560 @default.
• W3124384256 hasConcept C153083717 @default.
• W3124384256 hasConcept C154945302 @default.
• W3124384256 hasConcept C178790620 @default.
• W3124384256 hasConcept C185592680 @default.
• W3124384256 hasConcept C196083921 @default.
• W3124384256 hasConcept C204030448 @default.
• W3124384256 hasConcept C2776135515 @default.
• W3124384256 hasConcept C41008148 @default.
• W3124384256 hasConcept C98045186 @default.
• W3124384256 hasConceptScore W3124384256C111919701 @default.
• W3124384256 hasConceptScore W3124384256C115575686 @default.
• W3124384256 hasConceptScore W3124384256C119857082 @default.
• W3124384256 hasConceptScore W3124384256C121955636 @default.
• W3124384256 hasConceptScore W3124384256C12713177 @default.
• W3124384256 hasConceptScore W3124384256C144133560 @default.
• W3124384256 hasConceptScore W3124384256C153083717 @default.
• W3124384256 hasConceptScore W3124384256C154945302 @default.
• W3124384256 hasConceptScore W3124384256C178790620 @default.
• W3124384256 hasConceptScore W3124384256C185592680 @default.
• W3124384256 hasConceptScore W3124384256C196083921 @default.
• W3124384256 hasConceptScore W3124384256C204030448 @default.
• W3124384256 hasConceptScore W3124384256C2776135515 @default.
• W3124384256 hasConceptScore W3124384256C41008148 @default.
• W3124384256 hasConceptScore W3124384256C98045186 @default.
• W3124384256 hasOpenAccess W3124384256 @default.
• W3124384256 hasRelatedWork W1821462560 @default.
• W3124384256 hasRelatedWork W1967030981 @default.
• W3124384256 hasRelatedWork W1983429559 @default.
• W3124384256 hasRelatedWork W2118930608 @default.

• next