Matches in SemOpenAlex for { <https://semopenalex.org/work/W3169549154> ?p ?o ?g. }
- W3169549154 abstract "Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher -- and that more closely matching the teacher paradoxically does not always lead to better student generalization." @default.
- W3169549154 created "2021-06-22" @default.
- W3169549154 creator A5036325834 @default.
- W3169549154 creator A5043360924 @default.
- W3169549154 creator A5066297851 @default.
- W3169549154 creator A5069234607 @default.
- W3169549154 creator A5084173427 @default.
- W3169549154 date "2021-06-10" @default.
- W3169549154 modified "2023-09-27" @default.
- W3169549154 title "Does Knowledge Distillation Really Work" @default.
- W3169549154 cites W1574427581 @default.
- W3169549154 cites W1836465849 @default.
- W3169549154 cites W2099001231 @default.
- W3169549154 cites W2114766824 @default.
- W3169549154 cites W2147800946 @default.
- W3169549154 cites W2194775991 @default.
- W3169549154 cites W2335728318 @default.
- W3169549154 cites W2508418541 @default.
- W3169549154 cites W2561238782 @default.
- W3169549154 cites W2565304260 @default.
- W3169549154 cites W2734358244 @default.
- W3169549154 cites W2739879705 @default.
- W3169549154 cites W2743289088 @default.
- W3169549154 cites W2750784772 @default.
- W3169549154 cites W2795867901 @default.
- W3169549154 cites W2962835968 @default.
- W3169549154 cites W2963023528 @default.
- W3169549154 cites W2963031676 @default.
- W3169549154 cites W2963397674 @default.
- W3169549154 cites W2963399829 @default.
- W3169549154 cites W2963723401 @default.
- W3169549154 cites W2963736842 @default.
- W3169549154 cites W2963823954 @default.
- W3169549154 cites W2963836885 @default.
- W3169549154 cites W2964082701 @default.
- W3169549154 cites W2964111476 @default.
- W3169549154 cites W2964118293 @default.
- W3169549154 cites W2964121744 @default.
- W3169549154 cites W2964212410 @default.
- W3169549154 cites W2964222566 @default.
- W3169549154 cites W2964299589 @default.
- W3169549154 cites W2964306570 @default.
- W3169549154 cites W2970971581 @default.
- W3169549154 cites W2996184071 @default.
- W3169549154 cites W2996970889 @default.
- W3169549154 cites W3004127093 @default.
- W3169549154 cites W3099502074 @default.
- W3169549154 cites W3100400430 @default.
- W3169549154 cites W3103361051 @default.
- W3169549154 cites W3104310207 @default.
- W3169549154 cites W3118608800 @default.
- W3169549154 cites W3170224286 @default.
- W3169549154 cites W3209893611 @default.
- W3169549154 hasPublicationYear "2021" @default.
- W3169549154 type Work @default.
- W3169549154 sameAs 3169549154 @default.
- W3169549154 citedByCount "5" @default.
- W3169549154 countsByYear W31695491542021 @default.
- W3169549154 crossrefType "posted-content" @default.
- W3169549154 hasAuthorship W3169549154A5036325834 @default.
- W3169549154 hasAuthorship W3169549154A5043360924 @default.
- W3169549154 hasAuthorship W3169549154A5066297851 @default.
- W3169549154 hasAuthorship W3169549154A5069234607 @default.
- W3169549154 hasAuthorship W3169549154A5084173427 @default.
- W3169549154 hasConcept C105795698 @default.
- W3169549154 hasConcept C119857082 @default.
- W3169549154 hasConcept C127413603 @default.
- W3169549154 hasConcept C134306372 @default.
- W3169549154 hasConcept C145420912 @default.
- W3169549154 hasConcept C15744967 @default.
- W3169549154 hasConcept C165064840 @default.
- W3169549154 hasConcept C177148314 @default.
- W3169549154 hasConcept C185592680 @default.
- W3169549154 hasConcept C18762648 @default.
- W3169549154 hasConcept C204030448 @default.
- W3169549154 hasConcept C26517878 @default.
- W3169549154 hasConcept C33923547 @default.
- W3169549154 hasConcept C38652104 @default.
- W3169549154 hasConcept C41008148 @default.
- W3169549154 hasConcept C43617362 @default.
- W3169549154 hasConcept C78519656 @default.
- W3169549154 hasConceptScore W3169549154C105795698 @default.
- W3169549154 hasConceptScore W3169549154C119857082 @default.
- W3169549154 hasConceptScore W3169549154C127413603 @default.
- W3169549154 hasConceptScore W3169549154C134306372 @default.
- W3169549154 hasConceptScore W3169549154C145420912 @default.
- W3169549154 hasConceptScore W3169549154C15744967 @default.
- W3169549154 hasConceptScore W3169549154C165064840 @default.
- W3169549154 hasConceptScore W3169549154C177148314 @default.
- W3169549154 hasConceptScore W3169549154C185592680 @default.
- W3169549154 hasConceptScore W3169549154C18762648 @default.
- W3169549154 hasConceptScore W3169549154C204030448 @default.
- W3169549154 hasConceptScore W3169549154C26517878 @default.
- W3169549154 hasConceptScore W3169549154C33923547 @default.
- W3169549154 hasConceptScore W3169549154C38652104 @default.
- W3169549154 hasConceptScore W3169549154C41008148 @default.
- W3169549154 hasConceptScore W3169549154C43617362 @default.
- W3169549154 hasConceptScore W3169549154C78519656 @default.
- W3169549154 hasLocation W31695491541 @default.
- W3169549154 hasOpenAccess W3169549154 @default.