Matches in SemOpenAlex for { <https://semopenalex.org/work/W4384648207> ?p ?o ?g. }
Showing items 1 to 63 of
63
with 100 items per page.
- W4384648207 abstract "Distilling knowledge from convolutional neural networks (CNNs) is a double-edged sword for vision transformers (ViTs). It boosts the performance since the image-friendly local-inductive bias of CNN helps ViT learn faster and better, but leading to two problems: (1) Network designs of CNN and ViT are completely different, which leads to different semantic levels of intermediate features, making spatial-wise knowledge transfer methods (e.g., feature mimicking) inefficient. (2) Distilling knowledge from CNN limits the network convergence in the later training period since ViT's capability of integrating global information is suppressed by CNN's local-inductive-bias supervision. To this end, we present Cumulative Spatial Knowledge Distillation (CSKD). CSKD distills spatial-wise knowledge to all patch tokens of ViT from the corresponding spatial responses of CNN, without introducing intermediate features. Furthermore, CSKD exploits a Cumulative Knowledge Fusion (CKF) module, which introduces the global response of CNN and increasingly emphasizes its importance during the training. Applying CKF leverages CNN's local inductive bias in the early training period and gives full play to ViT's global capability in the later one. Extensive experiments and analysis on ImageNet-1k and downstream datasets demonstrate the superiority of our CSKD. Code will be publicly available." @default.
- W4384648207 created "2023-07-19" @default.
- W4384648207 creator A5035488063 @default.
- W4384648207 creator A5055782213 @default.
- W4384648207 creator A5070740938 @default.
- W4384648207 date "2023-07-17" @default.
- W4384648207 modified "2023-09-26" @default.
- W4384648207 title "Cumulative Spatial Knowledge Distillation for Vision Transformers" @default.
- W4384648207 doi "https://doi.org/10.48550/arxiv.2307.08500" @default.
- W4384648207 hasPublicationYear "2023" @default.
- W4384648207 type Work @default.
- W4384648207 citedByCount "0" @default.
- W4384648207 crossrefType "posted-content" @default.
- W4384648207 hasAuthorship W4384648207A5035488063 @default.
- W4384648207 hasAuthorship W4384648207A5055782213 @default.
- W4384648207 hasAuthorship W4384648207A5070740938 @default.
- W4384648207 hasBestOaLocation W43846482071 @default.
- W4384648207 hasConcept C119599485 @default.
- W4384648207 hasConcept C119857082 @default.
- W4384648207 hasConcept C127413603 @default.
- W4384648207 hasConcept C153180895 @default.
- W4384648207 hasConcept C154945302 @default.
- W4384648207 hasConcept C165696696 @default.
- W4384648207 hasConcept C165801399 @default.
- W4384648207 hasConcept C197352929 @default.
- W4384648207 hasConcept C201995342 @default.
- W4384648207 hasConcept C2780451532 @default.
- W4384648207 hasConcept C28006648 @default.
- W4384648207 hasConcept C38652104 @default.
- W4384648207 hasConcept C41008148 @default.
- W4384648207 hasConcept C66322947 @default.
- W4384648207 hasConcept C81363708 @default.
- W4384648207 hasConceptScore W4384648207C119599485 @default.
- W4384648207 hasConceptScore W4384648207C119857082 @default.
- W4384648207 hasConceptScore W4384648207C127413603 @default.
- W4384648207 hasConceptScore W4384648207C153180895 @default.
- W4384648207 hasConceptScore W4384648207C154945302 @default.
- W4384648207 hasConceptScore W4384648207C165696696 @default.
- W4384648207 hasConceptScore W4384648207C165801399 @default.
- W4384648207 hasConceptScore W4384648207C197352929 @default.
- W4384648207 hasConceptScore W4384648207C201995342 @default.
- W4384648207 hasConceptScore W4384648207C2780451532 @default.
- W4384648207 hasConceptScore W4384648207C28006648 @default.
- W4384648207 hasConceptScore W4384648207C38652104 @default.
- W4384648207 hasConceptScore W4384648207C41008148 @default.
- W4384648207 hasConceptScore W4384648207C66322947 @default.
- W4384648207 hasConceptScore W4384648207C81363708 @default.
- W4384648207 hasLocation W43846482071 @default.
- W4384648207 hasOpenAccess W4384648207 @default.
- W4384648207 hasPrimaryLocation W43846482071 @default.
- W4384648207 hasRelatedWork W2175746458 @default.
- W4384648207 hasRelatedWork W2732542196 @default.
- W4384648207 hasRelatedWork W2738221750 @default.
- W4384648207 hasRelatedWork W2760085659 @default.
- W4384648207 hasRelatedWork W2767651786 @default.
- W4384648207 hasRelatedWork W2883200793 @default.
- W4384648207 hasRelatedWork W3027997911 @default.
- W4384648207 hasRelatedWork W3093612317 @default.
- W4384648207 hasRelatedWork W4281789486 @default.
- W4384648207 hasRelatedWork W4287776258 @default.
- W4384648207 isParatext "false" @default.
- W4384648207 isRetracted "false" @default.
- W4384648207 workType "article" @default.