FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2895876826> ?p ?o ?g. }

• W2895876826 abstract "We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information $I(X;T)$ between the input $X$ and internal representations $T$ decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true $I(X;T)$ over these networks is provably either constant (discrete $X$) or infinite (continuous $X$). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which $I(X;T)$ is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for $I(X;T)$ in noisy DNNs and observe compression in various models. By relating $I(X;T)$ in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the $T$ space. Finally, we return to the estimator of $I(X;T)$ employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest." @default.
• W2895876826 created "2018-10-26" @default.
• W2895876826 creator A5002525343 @default.
• W2895876826 creator A5003725957 @default.
• W2895876826 creator A5031031216 @default.
• W2895876826 creator A5035670477 @default.
• W2895876826 creator A5045421508 @default.
• W2895876826 creator A5071112095 @default.
• W2895876826 creator A5091640094 @default.
• W2895876826 date "2018-10-12" @default.
• W2895876826 modified "2023-09-27" @default.
• W2895876826 title "Estimating Information Flow in Deep Neural Networks" @default.
• W2895876826 cites W2011133724 @default.
• W2895876826 cites W2029140344 @default.
• W2895876826 cites W2052044664 @default.
• W2895876826 cites W2070204917 @default.
• W2895876826 cites W2076608847 @default.
• W2895876826 cites W2092939357 @default.
• W2895876826 cites W2097854052 @default.
• W2895876826 cites W2112796928 @default.
• W2895876826 cites W2114771311 @default.
• W2895876826 cites W2122925692 @default.
• W2895876826 cites W2188155116 @default.
• W2895876826 cites W2622255292 @default.
• W2895876826 cites W2803832867 @default.
• W2895876826 cites W2880214242 @default.
• W2895876826 cites W2887997457 @default.
• W2895876826 cites W2899771611 @default.
• W2895876826 cites W2963180359 @default.
• W2895876826 cites W2964041897 @default.
• W2895876826 cites W2964294232 @default.
• W2895876826 cites W2974929241 @default.
• W2895876826 cites W3106022083 @default.
• W2895876826 cites W3110763182 @default.
• W2895876826 hasPublicationYear "2018" @default.
• W2895876826 type Work @default.
• W2895876826 sameAs 2895876826 @default.
• W2895876826 citedByCount "2" @default.
• W2895876826 countsByYear W28958768262019 @default.
• W2895876826 countsByYear W28958768262021 @default.
• W2895876826 crossrefType "posted-content" @default.
• W2895876826 hasAuthorship W2895876826A5002525343 @default.
• W2895876826 hasAuthorship W2895876826A5003725957 @default.
• W2895876826 hasAuthorship W2895876826A5031031216 @default.
• W2895876826 hasAuthorship W2895876826A5035670477 @default.
• W2895876826 hasAuthorship W2895876826A5045421508 @default.
• W2895876826 hasAuthorship W2895876826A5071112095 @default.
• W2895876826 hasAuthorship W2895876826A5091640094 @default.
• W2895876826 hasConcept C105795698 @default.
• W2895876826 hasConcept C11413529 @default.
• W2895876826 hasConcept C115961682 @default.
• W2895876826 hasConcept C121332964 @default.
• W2895876826 hasConcept C13481523 @default.
• W2895876826 hasConcept C138885662 @default.
• W2895876826 hasConcept C149635348 @default.
• W2895876826 hasConcept C152139883 @default.
• W2895876826 hasConcept C154945302 @default.
• W2895876826 hasConcept C180016635 @default.
• W2895876826 hasConcept C185429906 @default.
• W2895876826 hasConcept C2779136372 @default.
• W2895876826 hasConcept C2780513914 @default.
• W2895876826 hasConcept C2984842247 @default.
• W2895876826 hasConcept C33923547 @default.
• W2895876826 hasConcept C41008148 @default.
• W2895876826 hasConcept C41895202 @default.
• W2895876826 hasConcept C50644808 @default.
• W2895876826 hasConcept C52622258 @default.
• W2895876826 hasConcept C60008888 @default.
• W2895876826 hasConcept C73555534 @default.
• W2895876826 hasConcept C80444323 @default.
• W2895876826 hasConcept C9417928 @default.
• W2895876826 hasConcept C94835093 @default.
• W2895876826 hasConcept C97355855 @default.
• W2895876826 hasConceptScore W2895876826C105795698 @default.
• W2895876826 hasConceptScore W2895876826C11413529 @default.
• W2895876826 hasConceptScore W2895876826C115961682 @default.
• W2895876826 hasConceptScore W2895876826C121332964 @default.
• W2895876826 hasConceptScore W2895876826C13481523 @default.
• W2895876826 hasConceptScore W2895876826C138885662 @default.
• W2895876826 hasConceptScore W2895876826C149635348 @default.
• W2895876826 hasConceptScore W2895876826C152139883 @default.
• W2895876826 hasConceptScore W2895876826C154945302 @default.
• W2895876826 hasConceptScore W2895876826C180016635 @default.
• W2895876826 hasConceptScore W2895876826C185429906 @default.
• W2895876826 hasConceptScore W2895876826C2779136372 @default.
• W2895876826 hasConceptScore W2895876826C2780513914 @default.
• W2895876826 hasConceptScore W2895876826C2984842247 @default.
• W2895876826 hasConceptScore W2895876826C33923547 @default.
• W2895876826 hasConceptScore W2895876826C41008148 @default.
• W2895876826 hasConceptScore W2895876826C41895202 @default.
• W2895876826 hasConceptScore W2895876826C50644808 @default.
• W2895876826 hasConceptScore W2895876826C52622258 @default.
• W2895876826 hasConceptScore W2895876826C60008888 @default.
• W2895876826 hasConceptScore W2895876826C73555534 @default.
• W2895876826 hasConceptScore W2895876826C80444323 @default.
• W2895876826 hasConceptScore W2895876826C9417928 @default.
• W2895876826 hasConceptScore W2895876826C94835093 @default.
• W2895876826 hasConceptScore W2895876826C97355855 @default.
• W2895876826 hasLocation W28958768261 @default.
• W2895876826 hasOpenAccess W2895876826 @default.

• next