Matches in SemOpenAlex for { <https://semopenalex.org/work/W4320343010> ?p ?o ?g. }
Showing items 1 to 67 of
67
with 100 items per page.
- W4320343010 abstract "Self-predictive unsupervised learning methods such as BYOL or SimSiam have shown impressive results, and counter-intuitively, do not collapse to trivial representations. In this work, we aim at exploring the simplest possible mathematical arguments towards explaining the underlying mechanisms behind self-predictive unsupervised learning. We start with the observation that those methods crucially rely on the presence of a predictor network (and stop-gradient). With simple linear algebra, we show that when using a linear predictor, the optimal predictor is close to an orthogonal projection, and propose a general framework based on orthonormalization that enables to interpret and give intuition on why BYOL works. In addition, this framework demonstrates the crucial role of the exponential moving average and stop-gradient operator in BYOL as an efficient orthonormalization mechanism. We use these insights to propose four new emph{closed-form predictor} variants of BYOL to support our analysis. Our closed-form predictors outperform standard linear trainable predictor BYOL at $100$ and $300$ epochs (top-$1$ linear accuracy on ImageNet)." @default.
- W4320343010 created "2023-02-13" @default.
- W4320343010 creator A5002692470 @default.
- W4320343010 creator A5010396788 @default.
- W4320343010 creator A5024394972 @default.
- W4320343010 creator A5038979833 @default.
- W4320343010 creator A5051968502 @default.
- W4320343010 creator A5091349569 @default.
- W4320343010 date "2023-02-09" @default.
- W4320343010 modified "2023-09-23" @default.
- W4320343010 title "The Edge of Orthogonality: A Simple View of What Makes BYOL Tick" @default.
- W4320343010 doi "https://doi.org/10.48550/arxiv.2302.04817" @default.
- W4320343010 hasPublicationYear "2023" @default.
- W4320343010 type Work @default.
- W4320343010 citedByCount "0" @default.
- W4320343010 crossrefType "posted-content" @default.
- W4320343010 hasAuthorship W4320343010A5002692470 @default.
- W4320343010 hasAuthorship W4320343010A5010396788 @default.
- W4320343010 hasAuthorship W4320343010A5024394972 @default.
- W4320343010 hasAuthorship W4320343010A5038979833 @default.
- W4320343010 hasAuthorship W4320343010A5051968502 @default.
- W4320343010 hasAuthorship W4320343010A5091349569 @default.
- W4320343010 hasBestOaLocation W43203430101 @default.
- W4320343010 hasConcept C111472728 @default.
- W4320343010 hasConcept C11413529 @default.
- W4320343010 hasConcept C119857082 @default.
- W4320343010 hasConcept C132010649 @default.
- W4320343010 hasConcept C138885662 @default.
- W4320343010 hasConcept C139352143 @default.
- W4320343010 hasConcept C154945302 @default.
- W4320343010 hasConcept C17137986 @default.
- W4320343010 hasConcept C202444582 @default.
- W4320343010 hasConcept C2524010 @default.
- W4320343010 hasConcept C2780586882 @default.
- W4320343010 hasConcept C33923547 @default.
- W4320343010 hasConcept C41008148 @default.
- W4320343010 hasConcept C49766605 @default.
- W4320343010 hasConceptScore W4320343010C111472728 @default.
- W4320343010 hasConceptScore W4320343010C11413529 @default.
- W4320343010 hasConceptScore W4320343010C119857082 @default.
- W4320343010 hasConceptScore W4320343010C132010649 @default.
- W4320343010 hasConceptScore W4320343010C138885662 @default.
- W4320343010 hasConceptScore W4320343010C139352143 @default.
- W4320343010 hasConceptScore W4320343010C154945302 @default.
- W4320343010 hasConceptScore W4320343010C17137986 @default.
- W4320343010 hasConceptScore W4320343010C202444582 @default.
- W4320343010 hasConceptScore W4320343010C2524010 @default.
- W4320343010 hasConceptScore W4320343010C2780586882 @default.
- W4320343010 hasConceptScore W4320343010C33923547 @default.
- W4320343010 hasConceptScore W4320343010C41008148 @default.
- W4320343010 hasConceptScore W4320343010C49766605 @default.
- W4320343010 hasLocation W43203430101 @default.
- W4320343010 hasOpenAccess W4320343010 @default.
- W4320343010 hasPrimaryLocation W43203430101 @default.
- W4320343010 hasRelatedWork W1963934847 @default.
- W4320343010 hasRelatedWork W2188059687 @default.
- W4320343010 hasRelatedWork W2302519229 @default.
- W4320343010 hasRelatedWork W3021393819 @default.
- W4320343010 hasRelatedWork W3128755792 @default.
- W4320343010 hasRelatedWork W3198296966 @default.
- W4320343010 hasRelatedWork W4206493799 @default.
- W4320343010 hasRelatedWork W583117815 @default.
- W4320343010 hasRelatedWork W599420020 @default.
- W4320343010 hasRelatedWork W618248042 @default.
- W4320343010 isParatext "false" @default.
- W4320343010 isRetracted "false" @default.
- W4320343010 workType "article" @default.