Matches in SemOpenAlex for { <https://semopenalex.org/work/W4387687161> ?p ?o ?g. }
Showing items 1 to 67 of
67
with 100 items per page.
- W4387687161 abstract "Continual learning entails learning a sequence of tasks and balancing their knowledge appropriately. With limited access to old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide an exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to mitigate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis of its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting on these particular challenges, we propose Adaptive Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics, corresponding to the training and testing stages. By implementing BN in a continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (e.g., up to 7.68%, 6.86% and 4.26% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our code is available at https://github.com/lvyilin/AdaB2N." @default.
- W4387687161 created "2023-10-17" @default.
- W4387687161 creator A5017102094 @default.
- W4387687161 creator A5050652350 @default.
- W4387687161 creator A5062903761 @default.
- W4387687161 creator A5069599527 @default.
- W4387687161 creator A5069749738 @default.
- W4387687161 creator A5080071218 @default.
- W4387687161 creator A5090845370 @default.
- W4387687161 date "2023-10-13" @default.
- W4387687161 modified "2023-10-18" @default.
- W4387687161 title "Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation" @default.
- W4387687161 doi "https://doi.org/10.48550/arxiv.2310.08855" @default.
- W4387687161 hasPublicationYear "2023" @default.
- W4387687161 type Work @default.
- W4387687161 citedByCount "0" @default.
- W4387687161 crossrefType "posted-content" @default.
- W4387687161 hasAuthorship W4387687161A5017102094 @default.
- W4387687161 hasAuthorship W4387687161A5050652350 @default.
- W4387687161 hasAuthorship W4387687161A5062903761 @default.
- W4387687161 hasAuthorship W4387687161A5069599527 @default.
- W4387687161 hasAuthorship W4387687161A5069749738 @default.
- W4387687161 hasAuthorship W4387687161A5080071218 @default.
- W4387687161 hasAuthorship W4387687161A5090845370 @default.
- W4387687161 hasBestOaLocation W43876871611 @default.
- W4387687161 hasConcept C119857082 @default.
- W4387687161 hasConcept C120665830 @default.
- W4387687161 hasConcept C121332964 @default.
- W4387687161 hasConcept C136886441 @default.
- W4387687161 hasConcept C138885662 @default.
- W4387687161 hasConcept C139807058 @default.
- W4387687161 hasConcept C144024400 @default.
- W4387687161 hasConcept C154945302 @default.
- W4387687161 hasConcept C19165224 @default.
- W4387687161 hasConcept C41008148 @default.
- W4387687161 hasConcept C41895202 @default.
- W4387687161 hasConcept C7149132 @default.
- W4387687161 hasConcept C97541855 @default.
- W4387687161 hasConceptScore W4387687161C119857082 @default.
- W4387687161 hasConceptScore W4387687161C120665830 @default.
- W4387687161 hasConceptScore W4387687161C121332964 @default.
- W4387687161 hasConceptScore W4387687161C136886441 @default.
- W4387687161 hasConceptScore W4387687161C138885662 @default.
- W4387687161 hasConceptScore W4387687161C139807058 @default.
- W4387687161 hasConceptScore W4387687161C144024400 @default.
- W4387687161 hasConceptScore W4387687161C154945302 @default.
- W4387687161 hasConceptScore W4387687161C19165224 @default.
- W4387687161 hasConceptScore W4387687161C41008148 @default.
- W4387687161 hasConceptScore W4387687161C41895202 @default.
- W4387687161 hasConceptScore W4387687161C7149132 @default.
- W4387687161 hasConceptScore W4387687161C97541855 @default.
- W4387687161 hasLocation W43876871611 @default.
- W4387687161 hasOpenAccess W4387687161 @default.
- W4387687161 hasPrimaryLocation W43876871611 @default.
- W4387687161 hasRelatedWork W2104218666 @default.
- W4387687161 hasRelatedWork W2145559838 @default.
- W4387687161 hasRelatedWork W2164121020 @default.
- W4387687161 hasRelatedWork W2794885965 @default.
- W4387687161 hasRelatedWork W2959635497 @default.
- W4387687161 hasRelatedWork W3116498279 @default.
- W4387687161 hasRelatedWork W3183027292 @default.
- W4387687161 hasRelatedWork W4287549553 @default.
- W4387687161 hasRelatedWork W4289718052 @default.
- W4387687161 hasRelatedWork W4310285384 @default.
- W4387687161 isParatext "false" @default.
- W4387687161 isRetracted "false" @default.
- W4387687161 workType "article" @default.