FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W3199446656> ?p ?o ?g. }

• W3199446656 abstract "In recent years, knowledge distillation has been widely used in the field of deep learning in order to reduce the model size and save time and space. The student-teacher paradigm is a framework for knowledge distillation, and knowledge distillation proposed to minimize the KL divergence between the probabilistic outputs of a teacher and student network. However, apart from the probabilistic outputs, there are much valuable information contained in the middle layers of the teacher network. As for NLP tasks, the hidden vectors from different layers of a model have different semantic information, but the vectors' dimension of the student network is different from that of the teacher network in many cases, which makes hidden layer distillation hard to be performed directly. We propose to simply use a transition matrix to project the student's vector to a space of the same dimension as the teacher's vector, and we theoretically prove the effectiveness of this method. Our analysis shows how the transition matrix preserve important semantic information, which is closely related to the vector's characteristic in Euclidean space. We provide a geometric method for the interpretability of shared knowledge space for student-teacher architectures. Our experiments show that this method can significantly improve the performance of a small model in different tasks with different models." @default.
• W3199446656 created "2021-09-27" @default.
• W3199446656 creator A5001006462 @default.
• W3199446656 creator A5004609196 @default.
• W3199446656 creator A5025780527 @default.
• W3199446656 creator A5074965982 @default.
• W3199446656 creator A5085695760 @default.
• W3199446656 date "2021-07-18" @default.
• W3199446656 modified "2023-09-23" @default.
• W3199446656 title "Bridging the Gap of Dimensions in Distillation: Understanding the knowledge transfer between different-dimensional semantic spaces" @default.
• W3199446656 cites W1591801644 @default.
• W3199446656 cites W1724438581 @default.
• W3199446656 cites W1821462560 @default.
• W3199446656 cites W1832693441 @default.
• W3199446656 cites W1902237438 @default.
• W3199446656 cites W2014902591 @default.
• W3199446656 cites W2070246124 @default.
• W3199446656 cites W2114524997 @default.
• W3199446656 cites W2160660844 @default.
• W3199446656 cites W2163455955 @default.
• W3199446656 cites W2251939518 @default.
• W3199446656 cites W2561238782 @default.
• W3199446656 cites W2739879705 @default.
• W3199446656 cites W2741613777 @default.
• W3199446656 cites W2904759072 @default.
• W3199446656 cites W2962965870 @default.
• W3199446656 cites W2963350559 @default.
• W3199446656 cites W2963403868 @default.
• W3199446656 cites W2963736842 @default.
• W3199446656 cites W2963982496 @default.
• W3199446656 cites W2964118293 @default.
• W3199446656 cites W2964121744 @default.
• W3199446656 cites W2964222566 @default.
• W3199446656 cites W2995607862 @default.
• W3199446656 cites W3091643389 @default.
• W3199446656 cites W3105966348 @default.
• W3199446656 doi "https://doi.org/10.1109/ijcnn52387.2021.9534452" @default.
• W3199446656 hasPublicationYear "2021" @default.
• W3199446656 type Work @default.
• W3199446656 sameAs 3199446656 @default.
• W3199446656 citedByCount "0" @default.
• W3199446656 crossrefType "proceedings-article" @default.
• W3199446656 hasAuthorship W3199446656A5001006462 @default.
• W3199446656 hasAuthorship W3199446656A5004609196 @default.
• W3199446656 hasAuthorship W3199446656A5025780527 @default.
• W3199446656 hasAuthorship W3199446656A5074965982 @default.
• W3199446656 hasAuthorship W3199446656A5085695760 @default.
• W3199446656 hasConcept C111919701 @default.
• W3199446656 hasConcept C119857082 @default.
• W3199446656 hasConcept C138885662 @default.
• W3199446656 hasConcept C154945302 @default.
• W3199446656 hasConcept C174348530 @default.
• W3199446656 hasConcept C178790620 @default.
• W3199446656 hasConcept C185592680 @default.
• W3199446656 hasConcept C186450821 @default.
• W3199446656 hasConcept C202444582 @default.
• W3199446656 hasConcept C204030448 @default.
• W3199446656 hasConcept C207390915 @default.
• W3199446656 hasConcept C2778572836 @default.
• W3199446656 hasConcept C2781067378 @default.
• W3199446656 hasConcept C2986420190 @default.
• W3199446656 hasConcept C31258907 @default.
• W3199446656 hasConcept C33676613 @default.
• W3199446656 hasConcept C33923547 @default.
• W3199446656 hasConcept C41008148 @default.
• W3199446656 hasConcept C41895202 @default.
• W3199446656 hasConcept C49937458 @default.
• W3199446656 hasConcept C50644808 @default.
• W3199446656 hasConcept C80444323 @default.
• W3199446656 hasConceptScore W3199446656C111919701 @default.
• W3199446656 hasConceptScore W3199446656C119857082 @default.
• W3199446656 hasConceptScore W3199446656C138885662 @default.
• W3199446656 hasConceptScore W3199446656C154945302 @default.
• W3199446656 hasConceptScore W3199446656C174348530 @default.
• W3199446656 hasConceptScore W3199446656C178790620 @default.
• W3199446656 hasConceptScore W3199446656C185592680 @default.
• W3199446656 hasConceptScore W3199446656C186450821 @default.
• W3199446656 hasConceptScore W3199446656C202444582 @default.
• W3199446656 hasConceptScore W3199446656C204030448 @default.
• W3199446656 hasConceptScore W3199446656C207390915 @default.
• W3199446656 hasConceptScore W3199446656C2778572836 @default.
• W3199446656 hasConceptScore W3199446656C2781067378 @default.
• W3199446656 hasConceptScore W3199446656C2986420190 @default.
• W3199446656 hasConceptScore W3199446656C31258907 @default.
• W3199446656 hasConceptScore W3199446656C33676613 @default.
• W3199446656 hasConceptScore W3199446656C33923547 @default.
• W3199446656 hasConceptScore W3199446656C41008148 @default.
• W3199446656 hasConceptScore W3199446656C41895202 @default.
• W3199446656 hasConceptScore W3199446656C49937458 @default.
• W3199446656 hasConceptScore W3199446656C50644808 @default.
• W3199446656 hasConceptScore W3199446656C80444323 @default.
• W3199446656 hasFunder F4320321001 @default.
• W3199446656 hasLocation W31994466561 @default.
• W3199446656 hasOpenAccess W3199446656 @default.
• W3199446656 hasPrimaryLocation W31994466561 @default.
• W3199446656 hasRelatedWork W2605281151 @default.
• W3199446656 hasRelatedWork W3006943036 @default.
• W3199446656 hasRelatedWork W3012234327 @default.
• W3199446656 hasRelatedWork W3119715496 @default.
• W3199446656 hasRelatedWork W3191046242 @default.

• next