FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W4313015748> ?p ?o ?g. }

• W4313015748 endingPage "119" @default.
• W4313015748 startingPage "104" @default.
• W4313015748 abstract "Self-distillation exploits non-uniform soft supervision from itself during training and improves performance without any runtime cost. However, the overhead during training is often overlooked, and yet reducing time and memory overhead during training is increasingly important in the giant models’ era. This paper proposes an efficient self-distillation method named Zipf’s Label Smoothing (Zipf’s LS), which uses the on-the-fly prediction of a network to generate soft supervision that conforms to Zipf distribution without using any contrastive samples or auxiliary parameters. Our idea comes from an empirical observation that when the network is duly trained the output values of a network’s final softmax layer, after sorting by the magnitude and averaged across samples, should follow a distribution reminiscent to Zipf’s Law in the word frequency statistics of natural languages. By enforcing this property on the sample level and throughout the whole training period, we find that the prediction accuracy can be greatly improved. Using ResNet50 on the INAT21 fine-grained classification dataset, our technique achieves +3.61% accuracy gain compared to the vanilla baseline, and 0.88% more gain against the previous label smoothing or self-distillation strategies. The implementation is publicly available at https://github.com/megvii-research/zipfls ." @default.
• W4313015748 created "2023-01-05" @default.
• W4313015748 creator A5000109841 @default.
• W4313015748 creator A5005043562 @default.
• W4313015748 creator A5007289699 @default.
• W4313015748 creator A5014611666 @default.
• W4313015748 creator A5070740938 @default.
• W4313015748 creator A5080414420 @default.
• W4313015748 creator A5080795625 @default.
• W4313015748 date "2022-01-01" @default.
• W4313015748 modified "2023-10-16" @default.
• W4313015748 title "Efficient One Pass Self-distillation with Zipf’s Label Smoothing" @default.
• W4313015748 cites W1677182931 @default.
• W4313015748 cites W1968625547 @default.
• W4313015748 cites W2108598243 @default.
• W4313015748 cites W2183341477 @default.
• W4313015748 cites W2194775991 @default.
• W4313015748 cites W2549139847 @default.
• W4313015748 cites W2904170036 @default.
• W4313015748 cites W2963163009 @default.
• W4313015748 cites W2963855133 @default.
• W4313015748 cites W2987861506 @default.
• W4313015748 cites W2996970889 @default.
• W4313015748 cites W2997006708 @default.
• W4313015748 cites W3004127093 @default.
• W4313015748 cites W3034695001 @default.
• W4313015748 cites W3034756453 @default.
• W4313015748 cites W3035321581 @default.
• W4313015748 cites W3170224286 @default.
• W4313015748 cites W3173196825 @default.
• W4313015748 cites W3182707920 @default.
• W4313015748 cites W3185613252 @default.
• W4313015748 cites W3204610735 @default.
• W4313015748 cites W3204703845 @default.
• W4313015748 doi "https://doi.org/10.1007/978-3-031-20083-0_7" @default.
• W4313015748 hasPublicationYear "2022" @default.
• W4313015748 type Work @default.
• W4313015748 citedByCount "1" @default.
• W4313015748 countsByYear W43130157482023 @default.
• W4313015748 crossrefType "book-chapter" @default.
• W4313015748 hasAuthorship W4313015748A5000109841 @default.
• W4313015748 hasAuthorship W4313015748A5005043562 @default.
• W4313015748 hasAuthorship W4313015748A5007289699 @default.
• W4313015748 hasAuthorship W4313015748A5014611666 @default.
• W4313015748 hasAuthorship W4313015748A5070740938 @default.
• W4313015748 hasAuthorship W4313015748A5080414420 @default.
• W4313015748 hasAuthorship W4313015748A5080795625 @default.
• W4313015748 hasBestOaLocation W43130157482 @default.
• W4313015748 hasConcept C105795698 @default.
• W4313015748 hasConcept C111696304 @default.
• W4313015748 hasConcept C111919701 @default.
• W4313015748 hasConcept C11413529 @default.
• W4313015748 hasConcept C119857082 @default.
• W4313015748 hasConcept C125932096 @default.
• W4313015748 hasConcept C154945302 @default.
• W4313015748 hasConcept C165696696 @default.
• W4313015748 hasConcept C178790620 @default.
• W4313015748 hasConcept C185592680 @default.
• W4313015748 hasConcept C188441871 @default.
• W4313015748 hasConcept C204030448 @default.
• W4313015748 hasConcept C2779960059 @default.
• W4313015748 hasConcept C31972630 @default.
• W4313015748 hasConcept C33923547 @default.
• W4313015748 hasConcept C3770464 @default.
• W4313015748 hasConcept C38652104 @default.
• W4313015748 hasConcept C41008148 @default.
• W4313015748 hasConcept C50644808 @default.
• W4313015748 hasConceptScore W4313015748C105795698 @default.
• W4313015748 hasConceptScore W4313015748C111696304 @default.
• W4313015748 hasConceptScore W4313015748C111919701 @default.
• W4313015748 hasConceptScore W4313015748C11413529 @default.
• W4313015748 hasConceptScore W4313015748C119857082 @default.
• W4313015748 hasConceptScore W4313015748C125932096 @default.
• W4313015748 hasConceptScore W4313015748C154945302 @default.
• W4313015748 hasConceptScore W4313015748C165696696 @default.
• W4313015748 hasConceptScore W4313015748C178790620 @default.
• W4313015748 hasConceptScore W4313015748C185592680 @default.
• W4313015748 hasConceptScore W4313015748C188441871 @default.
• W4313015748 hasConceptScore W4313015748C204030448 @default.
• W4313015748 hasConceptScore W4313015748C2779960059 @default.
• W4313015748 hasConceptScore W4313015748C31972630 @default.
• W4313015748 hasConceptScore W4313015748C33923547 @default.
• W4313015748 hasConceptScore W4313015748C3770464 @default.
• W4313015748 hasConceptScore W4313015748C38652104 @default.
• W4313015748 hasConceptScore W4313015748C41008148 @default.
• W4313015748 hasConceptScore W4313015748C50644808 @default.
• W4313015748 hasLocation W43130157481 @default.
• W4313015748 hasLocation W43130157482 @default.
• W4313015748 hasOpenAccess W4313015748 @default.
• W4313015748 hasPrimaryLocation W43130157481 @default.
• W4313015748 hasRelatedWork W2331043530 @default.
• W4313015748 hasRelatedWork W2393933887 @default.
• W4313015748 hasRelatedWork W2888789309 @default.
• W4313015748 hasRelatedWork W2961085424 @default.
• W4313015748 hasRelatedWork W2971416272 @default.
• W4313015748 hasRelatedWork W2997512100 @default.
• W4313015748 hasRelatedWork W4211165872 @default.
• W4313015748 hasRelatedWork W4295681619 @default.

• next