Matches in SemOpenAlex for { <https://semopenalex.org/work/W4312050685> ?p ?o ?g. }
Showing items 1 to 95 of
95
with 100 items per page.
- W4312050685 abstract "Commonsense capabilities of pre-trained language models dramatically improve with scale, leading many to believe that scale is the only winning recipe. But is it? Here, we investigate an alternative that a priori seems impossible: can smaller language models (e.g., GPT-2) win over models that are orders of magnitude larger and better (e.g., GPT-3), if powered with novel commonsense distillation algorithms? The key intellectual challenge is to design a learning algorithm that achieve a competitive level of commonsense acquisition, without relying on the benefits of scale. In particular, we study generative models of commonsense knowledge, focusing on the task of generating generics, statements of commonsense facts about everyday concepts, e.g., birds can fly. We introduce I2D2, a novel commonsense distillation framework that loosely follows the Symbolic Knowledge Distillation of West et al. but breaks the dependence on the extreme-scale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the model's own enhanced commonsense acquisition capabilities. Empirical results suggest that scale is not the only way, as novel algorithms can be a promising alternative. Moreover, our study leads to a new corpus of generics, Gen-A-tomic, that is the largest and highest quality available to date." @default.
- W4312050685 created "2023-01-04" @default.
- W4312050685 creator A5024879161 @default.
- W4312050685 creator A5035760052 @default.
- W4312050685 creator A5043450042 @default.
- W4312050685 creator A5044250030 @default.
- W4312050685 creator A5045464993 @default.
- W4312050685 creator A5076059888 @default.
- W4312050685 creator A5076880940 @default.
- W4312050685 creator A5080544237 @default.
- W4312050685 creator A5081420816 @default.
- W4312050685 date "2022-12-18" @default.
- W4312050685 modified "2023-09-27" @default.
- W4312050685 title "I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation" @default.
- W4312050685 doi "https://doi.org/10.48550/arxiv.2212.09246" @default.
- W4312050685 hasPublicationYear "2022" @default.
- W4312050685 type Work @default.
- W4312050685 citedByCount "0" @default.
- W4312050685 crossrefType "posted-content" @default.
- W4312050685 hasAuthorship W4312050685A5024879161 @default.
- W4312050685 hasAuthorship W4312050685A5035760052 @default.
- W4312050685 hasAuthorship W4312050685A5043450042 @default.
- W4312050685 hasAuthorship W4312050685A5044250030 @default.
- W4312050685 hasAuthorship W4312050685A5045464993 @default.
- W4312050685 hasAuthorship W4312050685A5076059888 @default.
- W4312050685 hasAuthorship W4312050685A5076880940 @default.
- W4312050685 hasAuthorship W4312050685A5080544237 @default.
- W4312050685 hasAuthorship W4312050685A5081420816 @default.
- W4312050685 hasBestOaLocation W43120506851 @default.
- W4312050685 hasConcept C111472728 @default.
- W4312050685 hasConcept C119857082 @default.
- W4312050685 hasConcept C121332964 @default.
- W4312050685 hasConcept C126388530 @default.
- W4312050685 hasConcept C137293760 @default.
- W4312050685 hasConcept C138885662 @default.
- W4312050685 hasConcept C154945302 @default.
- W4312050685 hasConcept C15744967 @default.
- W4312050685 hasConcept C162324750 @default.
- W4312050685 hasConcept C178790620 @default.
- W4312050685 hasConcept C185592680 @default.
- W4312050685 hasConcept C187736073 @default.
- W4312050685 hasConcept C193221554 @default.
- W4312050685 hasConcept C204030448 @default.
- W4312050685 hasConcept C2778755073 @default.
- W4312050685 hasConcept C2779530757 @default.
- W4312050685 hasConcept C2780451532 @default.
- W4312050685 hasConcept C30542707 @default.
- W4312050685 hasConcept C41008148 @default.
- W4312050685 hasConcept C4554734 @default.
- W4312050685 hasConcept C55493867 @default.
- W4312050685 hasConcept C62520636 @default.
- W4312050685 hasConcept C75553542 @default.
- W4312050685 hasConcept C77805123 @default.
- W4312050685 hasConcept C98184364 @default.
- W4312050685 hasConceptScore W4312050685C111472728 @default.
- W4312050685 hasConceptScore W4312050685C119857082 @default.
- W4312050685 hasConceptScore W4312050685C121332964 @default.
- W4312050685 hasConceptScore W4312050685C126388530 @default.
- W4312050685 hasConceptScore W4312050685C137293760 @default.
- W4312050685 hasConceptScore W4312050685C138885662 @default.
- W4312050685 hasConceptScore W4312050685C154945302 @default.
- W4312050685 hasConceptScore W4312050685C15744967 @default.
- W4312050685 hasConceptScore W4312050685C162324750 @default.
- W4312050685 hasConceptScore W4312050685C178790620 @default.
- W4312050685 hasConceptScore W4312050685C185592680 @default.
- W4312050685 hasConceptScore W4312050685C187736073 @default.
- W4312050685 hasConceptScore W4312050685C193221554 @default.
- W4312050685 hasConceptScore W4312050685C204030448 @default.
- W4312050685 hasConceptScore W4312050685C2778755073 @default.
- W4312050685 hasConceptScore W4312050685C2779530757 @default.
- W4312050685 hasConceptScore W4312050685C2780451532 @default.
- W4312050685 hasConceptScore W4312050685C30542707 @default.
- W4312050685 hasConceptScore W4312050685C41008148 @default.
- W4312050685 hasConceptScore W4312050685C4554734 @default.
- W4312050685 hasConceptScore W4312050685C55493867 @default.
- W4312050685 hasConceptScore W4312050685C62520636 @default.
- W4312050685 hasConceptScore W4312050685C75553542 @default.
- W4312050685 hasConceptScore W4312050685C77805123 @default.
- W4312050685 hasConceptScore W4312050685C98184364 @default.
- W4312050685 hasLocation W43120506851 @default.
- W4312050685 hasOpenAccess W4312050685 @default.
- W4312050685 hasPrimaryLocation W43120506851 @default.
- W4312050685 hasRelatedWork W2968908603 @default.
- W4312050685 hasRelatedWork W3026899555 @default.
- W4312050685 hasRelatedWork W3114176606 @default.
- W4312050685 hasRelatedWork W3115157649 @default.
- W4312050685 hasRelatedWork W3116645451 @default.
- W4312050685 hasRelatedWork W3117841010 @default.
- W4312050685 hasRelatedWork W3160008796 @default.
- W4312050685 hasRelatedWork W4292409471 @default.
- W4312050685 hasRelatedWork W4312050685 @default.
- W4312050685 hasRelatedWork W4323824390 @default.
- W4312050685 isParatext "false" @default.
- W4312050685 isRetracted "false" @default.
- W4312050685 workType "article" @default.