Matches in SemOpenAlex for { <https://semopenalex.org/work/W4285410153> ?p ?o ?g. }
Showing items 1 to 70 of
70
with 100 items per page.
- W4285410153 abstract "State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the difficulty of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megvii-research/mdistiller." @default.
- W4285410153 created "2022-07-14" @default.
- W4285410153 creator A5025594373 @default.
- W4285410153 creator A5035488063 @default.
- W4285410153 creator A5044029777 @default.
- W4285410153 creator A5055782213 @default.
- W4285410153 creator A5070740938 @default.
- W4285410153 date "2022-03-16" @default.
- W4285410153 modified "2023-09-29" @default.
- W4285410153 title "Decoupled Knowledge Distillation" @default.
- W4285410153 doi "https://doi.org/10.48550/arxiv.2203.08679" @default.
- W4285410153 hasPublicationYear "2022" @default.
- W4285410153 type Work @default.
- W4285410153 citedByCount "1" @default.
- W4285410153 countsByYear W42854101532022 @default.
- W4285410153 crossrefType "posted-content" @default.
- W4285410153 hasAuthorship W4285410153A5025594373 @default.
- W4285410153 hasAuthorship W4285410153A5035488063 @default.
- W4285410153 hasAuthorship W4285410153A5044029777 @default.
- W4285410153 hasAuthorship W4285410153A5055782213 @default.
- W4285410153 hasAuthorship W4285410153A5070740938 @default.
- W4285410153 hasBestOaLocation W42854101531 @default.
- W4285410153 hasConcept C105795698 @default.
- W4285410153 hasConcept C115961682 @default.
- W4285410153 hasConcept C119857082 @default.
- W4285410153 hasConcept C138885662 @default.
- W4285410153 hasConcept C140331021 @default.
- W4285410153 hasConcept C154945302 @default.
- W4285410153 hasConcept C185592680 @default.
- W4285410153 hasConcept C204030448 @default.
- W4285410153 hasConcept C2776401178 @default.
- W4285410153 hasConcept C2777212361 @default.
- W4285410153 hasConcept C2780598303 @default.
- W4285410153 hasConcept C2781238097 @default.
- W4285410153 hasConcept C33923547 @default.
- W4285410153 hasConcept C41008148 @default.
- W4285410153 hasConcept C41895202 @default.
- W4285410153 hasConcept C43617362 @default.
- W4285410153 hasConceptScore W4285410153C105795698 @default.
- W4285410153 hasConceptScore W4285410153C115961682 @default.
- W4285410153 hasConceptScore W4285410153C119857082 @default.
- W4285410153 hasConceptScore W4285410153C138885662 @default.
- W4285410153 hasConceptScore W4285410153C140331021 @default.
- W4285410153 hasConceptScore W4285410153C154945302 @default.
- W4285410153 hasConceptScore W4285410153C185592680 @default.
- W4285410153 hasConceptScore W4285410153C204030448 @default.
- W4285410153 hasConceptScore W4285410153C2776401178 @default.
- W4285410153 hasConceptScore W4285410153C2777212361 @default.
- W4285410153 hasConceptScore W4285410153C2780598303 @default.
- W4285410153 hasConceptScore W4285410153C2781238097 @default.
- W4285410153 hasConceptScore W4285410153C33923547 @default.
- W4285410153 hasConceptScore W4285410153C41008148 @default.
- W4285410153 hasConceptScore W4285410153C41895202 @default.
- W4285410153 hasConceptScore W4285410153C43617362 @default.
- W4285410153 hasLocation W42854101531 @default.
- W4285410153 hasOpenAccess W4285410153 @default.
- W4285410153 hasPrimaryLocation W42854101531 @default.
- W4285410153 hasRelatedWork W2079759540 @default.
- W4285410153 hasRelatedWork W2092957489 @default.
- W4285410153 hasRelatedWork W2332301053 @default.
- W4285410153 hasRelatedWork W2961085424 @default.
- W4285410153 hasRelatedWork W3046775127 @default.
- W4285410153 hasRelatedWork W3108556207 @default.
- W4285410153 hasRelatedWork W4286629047 @default.
- W4285410153 hasRelatedWork W4306321456 @default.
- W4285410153 hasRelatedWork W4306674287 @default.
- W4285410153 hasRelatedWork W4224009465 @default.
- W4285410153 isParatext "false" @default.
- W4285410153 isRetracted "false" @default.
- W4285410153 workType "article" @default.