Matches in SemOpenAlex for { <https://semopenalex.org/work/W4385436426> ?p ?o ?g. }
Showing items 1 to 77 of
77
with 100 items per page.
- W4385436426 abstract "Knowledge distillation (KD) is the process of transferring knowledge from a large model to a small one. It has gained increasing attention in the natural language processing community, driven by the demands of compressing ever-growing language models. In this work, we propose an f-DISTILL framework, which formulates sequence-level knowledge distillation as minimizing a generalized f-divergence function. We propose four distilling variants under our framework and show that existing SeqKD and ENGINE approaches are approximations of our f-DISTILL methods. We further derive step-wise decomposition for our f-DISTILL, reducing intractable sequence-level divergence to word-level losses that can be computed in a tractable manner. Experiments across four datasets show that our methods outperform existing KD approaches, and that our symmetric distilling losses can better force the student to learn from the teacher distribution." @default.
- W4385436426 created "2023-08-01" @default.
- W4385436426 creator A5009277091 @default.
- W4385436426 creator A5024821632 @default.
- W4385436426 creator A5077548608 @default.
- W4385436426 creator A5079011877 @default.
- W4385436426 date "2023-07-27" @default.
- W4385436426 modified "2023-10-16" @default.
- W4385436426 title "f-Divergence Minimization for Sequence-Level Knowledge Distillation" @default.
- W4385436426 doi "https://doi.org/10.48550/arxiv.2307.15190" @default.
- W4385436426 hasPublicationYear "2023" @default.
- W4385436426 type Work @default.
- W4385436426 citedByCount "0" @default.
- W4385436426 crossrefType "posted-content" @default.
- W4385436426 hasAuthorship W4385436426A5009277091 @default.
- W4385436426 hasAuthorship W4385436426A5024821632 @default.
- W4385436426 hasAuthorship W4385436426A5077548608 @default.
- W4385436426 hasAuthorship W4385436426A5079011877 @default.
- W4385436426 hasBestOaLocation W43854364261 @default.
- W4385436426 hasConcept C124681953 @default.
- W4385436426 hasConcept C138885662 @default.
- W4385436426 hasConcept C14036430 @default.
- W4385436426 hasConcept C147764199 @default.
- W4385436426 hasConcept C154945302 @default.
- W4385436426 hasConcept C178790620 @default.
- W4385436426 hasConcept C185592680 @default.
- W4385436426 hasConcept C18903297 @default.
- W4385436426 hasConcept C199360897 @default.
- W4385436426 hasConcept C204030448 @default.
- W4385436426 hasConcept C207390915 @default.
- W4385436426 hasConcept C2524010 @default.
- W4385436426 hasConcept C2778112365 @default.
- W4385436426 hasConcept C33923547 @default.
- W4385436426 hasConcept C41008148 @default.
- W4385436426 hasConcept C41895202 @default.
- W4385436426 hasConcept C54355233 @default.
- W4385436426 hasConcept C78458016 @default.
- W4385436426 hasConcept C86803240 @default.
- W4385436426 hasConcept C90805587 @default.
- W4385436426 hasConcept C98045186 @default.
- W4385436426 hasConceptScore W4385436426C124681953 @default.
- W4385436426 hasConceptScore W4385436426C138885662 @default.
- W4385436426 hasConceptScore W4385436426C14036430 @default.
- W4385436426 hasConceptScore W4385436426C147764199 @default.
- W4385436426 hasConceptScore W4385436426C154945302 @default.
- W4385436426 hasConceptScore W4385436426C178790620 @default.
- W4385436426 hasConceptScore W4385436426C185592680 @default.
- W4385436426 hasConceptScore W4385436426C18903297 @default.
- W4385436426 hasConceptScore W4385436426C199360897 @default.
- W4385436426 hasConceptScore W4385436426C204030448 @default.
- W4385436426 hasConceptScore W4385436426C207390915 @default.
- W4385436426 hasConceptScore W4385436426C2524010 @default.
- W4385436426 hasConceptScore W4385436426C2778112365 @default.
- W4385436426 hasConceptScore W4385436426C33923547 @default.
- W4385436426 hasConceptScore W4385436426C41008148 @default.
- W4385436426 hasConceptScore W4385436426C41895202 @default.
- W4385436426 hasConceptScore W4385436426C54355233 @default.
- W4385436426 hasConceptScore W4385436426C78458016 @default.
- W4385436426 hasConceptScore W4385436426C86803240 @default.
- W4385436426 hasConceptScore W4385436426C90805587 @default.
- W4385436426 hasConceptScore W4385436426C98045186 @default.
- W4385436426 hasLocation W43854364261 @default.
- W4385436426 hasOpenAccess W4385436426 @default.
- W4385436426 hasPrimaryLocation W43854364261 @default.
- W4385436426 hasRelatedWork W2348111516 @default.
- W4385436426 hasRelatedWork W2353329674 @default.
- W4385436426 hasRelatedWork W2358216765 @default.
- W4385436426 hasRelatedWork W2360025963 @default.
- W4385436426 hasRelatedWork W2360785147 @default.
- W4385436426 hasRelatedWork W2361443205 @default.
- W4385436426 hasRelatedWork W2362702465 @default.
- W4385436426 hasRelatedWork W2368517087 @default.
- W4385436426 hasRelatedWork W2379589241 @default.
- W4385436426 hasRelatedWork W3034071520 @default.
- W4385436426 isParatext "false" @default.
- W4385436426 isRetracted "false" @default.
- W4385436426 workType "article" @default.