Matches in SemOpenAlex for { <https://semopenalex.org/work/W4366198740> ?p ?o ?g. }
Showing items 1 to 71 of
71
with 100 items per page.
- W4366198740 abstract "Learning new classes without forgetting is crucial for real-world applications for a classification model. Vision Transformers (ViT) recently achieve remarkable performance in Class Incremental Learning (CIL). Previous works mainly focus on block design and model expansion for ViTs. However, in this paper, we find that when the ViT is incrementally trained, the attention layers gradually lose concentration on local features. We call this interesting phenomenon as emph{Locality Degradation} in ViTs for CIL. Since the low-level local information is crucial to the transferability of the representation, it is beneficial to preserve the locality in attention layers. In this paper, we encourage the model to preserve more local information as the training procedure goes on and devise a Locality-Preserved Attention (LPA) layer to emphasize the importance of local features. Specifically, we incorporate the local information directly into the vanilla attention and control the initial gradients of the vanilla attention by weighting it with a small initial value. Extensive experiments show that the representations facilitated by LPA capture more low-level general information which is easier to transfer to follow-up tasks. The improved model gets consistently better performance on CIFAR100 and ImageNet100." @default.
- W4366198740 created "2023-04-19" @default.
- W4366198740 creator A5027946213 @default.
- W4366198740 creator A5030194599 @default.
- W4366198740 creator A5065180062 @default.
- W4366198740 creator A5073912249 @default.
- W4366198740 date "2023-04-14" @default.
- W4366198740 modified "2023-09-23" @default.
- W4366198740 title "Preserving Locality in Vision Transformers for Class Incremental Learning" @default.
- W4366198740 doi "https://doi.org/10.48550/arxiv.2304.06971" @default.
- W4366198740 hasPublicationYear "2023" @default.
- W4366198740 type Work @default.
- W4366198740 citedByCount "0" @default.
- W4366198740 crossrefType "posted-content" @default.
- W4366198740 hasAuthorship W4366198740A5027946213 @default.
- W4366198740 hasAuthorship W4366198740A5030194599 @default.
- W4366198740 hasAuthorship W4366198740A5065180062 @default.
- W4366198740 hasAuthorship W4366198740A5073912249 @default.
- W4366198740 hasBestOaLocation W43661987401 @default.
- W4366198740 hasConcept C119599485 @default.
- W4366198740 hasConcept C119857082 @default.
- W4366198740 hasConcept C126838900 @default.
- W4366198740 hasConcept C127413603 @default.
- W4366198740 hasConcept C138885662 @default.
- W4366198740 hasConcept C140331021 @default.
- W4366198740 hasConcept C154945302 @default.
- W4366198740 hasConcept C165801399 @default.
- W4366198740 hasConcept C183115368 @default.
- W4366198740 hasConcept C2777212361 @default.
- W4366198740 hasConcept C2779808786 @default.
- W4366198740 hasConcept C41008148 @default.
- W4366198740 hasConcept C41895202 @default.
- W4366198740 hasConcept C61272859 @default.
- W4366198740 hasConcept C66322947 @default.
- W4366198740 hasConcept C7149132 @default.
- W4366198740 hasConcept C71924100 @default.
- W4366198740 hasConcept C80444323 @default.
- W4366198740 hasConceptScore W4366198740C119599485 @default.
- W4366198740 hasConceptScore W4366198740C119857082 @default.
- W4366198740 hasConceptScore W4366198740C126838900 @default.
- W4366198740 hasConceptScore W4366198740C127413603 @default.
- W4366198740 hasConceptScore W4366198740C138885662 @default.
- W4366198740 hasConceptScore W4366198740C140331021 @default.
- W4366198740 hasConceptScore W4366198740C154945302 @default.
- W4366198740 hasConceptScore W4366198740C165801399 @default.
- W4366198740 hasConceptScore W4366198740C183115368 @default.
- W4366198740 hasConceptScore W4366198740C2777212361 @default.
- W4366198740 hasConceptScore W4366198740C2779808786 @default.
- W4366198740 hasConceptScore W4366198740C41008148 @default.
- W4366198740 hasConceptScore W4366198740C41895202 @default.
- W4366198740 hasConceptScore W4366198740C61272859 @default.
- W4366198740 hasConceptScore W4366198740C66322947 @default.
- W4366198740 hasConceptScore W4366198740C7149132 @default.
- W4366198740 hasConceptScore W4366198740C71924100 @default.
- W4366198740 hasConceptScore W4366198740C80444323 @default.
- W4366198740 hasLocation W43661987401 @default.
- W4366198740 hasOpenAccess W4366198740 @default.
- W4366198740 hasPrimaryLocation W43661987401 @default.
- W4366198740 hasRelatedWork W2363648756 @default.
- W4366198740 hasRelatedWork W2381880241 @default.
- W4366198740 hasRelatedWork W2576147416 @default.
- W4366198740 hasRelatedWork W2626799276 @default.
- W4366198740 hasRelatedWork W2886384632 @default.
- W4366198740 hasRelatedWork W2954850701 @default.
- W4366198740 hasRelatedWork W3126776133 @default.
- W4366198740 hasRelatedWork W3174101157 @default.
- W4366198740 hasRelatedWork W4287234591 @default.
- W4366198740 hasRelatedWork W4313050734 @default.
- W4366198740 isParatext "false" @default.
- W4366198740 isRetracted "false" @default.
- W4366198740 workType "article" @default.