Matches in SemOpenAlex for { <https://semopenalex.org/work/W4385570159> ?p ?o ?g. }
Showing items 1 to 71 of
71
with 100 items per page.
- W4385570159 abstract "Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance. Previous studies have shown that DWT can be effective in the vision domain and natural language processing (NLP) pre-training stage. Specifically, DWT shows promise in practical scenarios, such as enhancing new generation or larger models using pre-trained yet older or smaller models and lacking a resource budget. However, the optimal conditions for using DWT have yet to be fully investigated in NLP pre-training. Therefore, this study examines three key factors to optimize DWT, distinct from those used in the vision domain or traditional knowledge distillation. These factors are:(i) the impact of teacher model quality on DWT effectiveness, (ii) guidelines for adjusting the weighting value for DWT loss, and (iii) the impact of parameter remapping as a student model initialization technique for DWT." @default.
- W4385570159 created "2023-08-05" @default.
- W4385570159 creator A5010636698 @default.
- W4385570159 creator A5022908478 @default.
- W4385570159 creator A5061000448 @default.
- W4385570159 creator A5070302452 @default.
- W4385570159 creator A5079193147 @default.
- W4385570159 creator A5091672910 @default.
- W4385570159 date "2023-01-01" @default.
- W4385570159 modified "2023-09-27" @default.
- W4385570159 title "A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models" @default.
- W4385570159 doi "https://doi.org/10.18653/v1/2023.findings-acl.714" @default.
- W4385570159 hasPublicationYear "2023" @default.
- W4385570159 type Work @default.
- W4385570159 citedByCount "0" @default.
- W4385570159 crossrefType "proceedings-article" @default.
- W4385570159 hasAuthorship W4385570159A5010636698 @default.
- W4385570159 hasAuthorship W4385570159A5022908478 @default.
- W4385570159 hasAuthorship W4385570159A5061000448 @default.
- W4385570159 hasAuthorship W4385570159A5070302452 @default.
- W4385570159 hasAuthorship W4385570159A5079193147 @default.
- W4385570159 hasAuthorship W4385570159A5091672910 @default.
- W4385570159 hasBestOaLocation W43855701591 @default.
- W4385570159 hasConcept C114466953 @default.
- W4385570159 hasConcept C119857082 @default.
- W4385570159 hasConcept C126838900 @default.
- W4385570159 hasConcept C134306372 @default.
- W4385570159 hasConcept C137293760 @default.
- W4385570159 hasConcept C154945302 @default.
- W4385570159 hasConcept C178790620 @default.
- W4385570159 hasConcept C183115368 @default.
- W4385570159 hasConcept C185592680 @default.
- W4385570159 hasConcept C199360897 @default.
- W4385570159 hasConcept C204030448 @default.
- W4385570159 hasConcept C204321447 @default.
- W4385570159 hasConcept C33923547 @default.
- W4385570159 hasConcept C36503486 @default.
- W4385570159 hasConcept C41008148 @default.
- W4385570159 hasConcept C71924100 @default.
- W4385570159 hasConceptScore W4385570159C114466953 @default.
- W4385570159 hasConceptScore W4385570159C119857082 @default.
- W4385570159 hasConceptScore W4385570159C126838900 @default.
- W4385570159 hasConceptScore W4385570159C134306372 @default.
- W4385570159 hasConceptScore W4385570159C137293760 @default.
- W4385570159 hasConceptScore W4385570159C154945302 @default.
- W4385570159 hasConceptScore W4385570159C178790620 @default.
- W4385570159 hasConceptScore W4385570159C183115368 @default.
- W4385570159 hasConceptScore W4385570159C185592680 @default.
- W4385570159 hasConceptScore W4385570159C199360897 @default.
- W4385570159 hasConceptScore W4385570159C204030448 @default.
- W4385570159 hasConceptScore W4385570159C204321447 @default.
- W4385570159 hasConceptScore W4385570159C33923547 @default.
- W4385570159 hasConceptScore W4385570159C36503486 @default.
- W4385570159 hasConceptScore W4385570159C41008148 @default.
- W4385570159 hasConceptScore W4385570159C71924100 @default.
- W4385570159 hasLocation W43855701591 @default.
- W4385570159 hasOpenAccess W4385570159 @default.
- W4385570159 hasPrimaryLocation W43855701591 @default.
- W4385570159 hasRelatedWork W2002577756 @default.
- W4385570159 hasRelatedWork W2060370854 @default.
- W4385570159 hasRelatedWork W2359001871 @default.
- W4385570159 hasRelatedWork W2374512474 @default.
- W4385570159 hasRelatedWork W2949992439 @default.
- W4385570159 hasRelatedWork W2957565102 @default.
- W4385570159 hasRelatedWork W3093097038 @default.
- W4385570159 hasRelatedWork W3184365942 @default.
- W4385570159 hasRelatedWork W4312304159 @default.
- W4385570159 hasRelatedWork W4362508197 @default.
- W4385570159 isParatext "false" @default.
- W4385570159 isRetracted "false" @default.
- W4385570159 workType "article" @default.