Matches in SemOpenAlex for { <https://semopenalex.org/work/W4387427488> ?p ?o ?g. }
Showing items 1 to 69 of
69
with 100 items per page.
- W4387427488 abstract "Pretrained language models (LMs) encode implicit representations of knowledge in their parameters. However, localizing these representations and disentangling them from each other remains an open problem. In this work, we investigate whether pretrained language models contain various knowledge-critical subnetworks: particular sparse computational subgraphs responsible for encoding specific knowledge the model has memorized. We propose a multi-objective differentiable weight masking scheme to discover these subnetworks and show that we can use them to precisely remove specific knowledge from models while minimizing adverse effects on the behavior of the original language model. We demonstrate our method on multiple GPT2 variants, uncovering highly sparse subnetworks (98%+) that are solely responsible for specific collections of relational knowledge. When these subnetworks are removed, the remaining network maintains most of its initial capacity (modeling language and other memorized relational knowledge) but struggles to express the removed knowledge, and suffers performance drops on examples needing this removed knowledge on downstream tasks after finetuning." @default.
- W4387427488 created "2023-10-08" @default.
- W4387427488 creator A5018905679 @default.
- W4387427488 creator A5021574351 @default.
- W4387427488 creator A5027070511 @default.
- W4387427488 creator A5055878811 @default.
- W4387427488 creator A5088410008 @default.
- W4387427488 date "2023-10-04" @default.
- W4387427488 modified "2023-10-09" @default.
- W4387427488 title "Discovering Knowledge-Critical Subnetworks in Pretrained Language Models" @default.
- W4387427488 doi "https://doi.org/10.48550/arxiv.2310.03084" @default.
- W4387427488 hasPublicationYear "2023" @default.
- W4387427488 type Work @default.
- W4387427488 citedByCount "0" @default.
- W4387427488 crossrefType "posted-content" @default.
- W4387427488 hasAuthorship W4387427488A5018905679 @default.
- W4387427488 hasAuthorship W4387427488A5021574351 @default.
- W4387427488 hasAuthorship W4387427488A5027070511 @default.
- W4387427488 hasAuthorship W4387427488A5055878811 @default.
- W4387427488 hasAuthorship W4387427488A5088410008 @default.
- W4387427488 hasBestOaLocation W43874274881 @default.
- W4387427488 hasConcept C104317684 @default.
- W4387427488 hasConcept C119857082 @default.
- W4387427488 hasConcept C125411270 @default.
- W4387427488 hasConcept C134306372 @default.
- W4387427488 hasConcept C137293760 @default.
- W4387427488 hasConcept C142362112 @default.
- W4387427488 hasConcept C153349607 @default.
- W4387427488 hasConcept C154945302 @default.
- W4387427488 hasConcept C185592680 @default.
- W4387427488 hasConcept C204321447 @default.
- W4387427488 hasConcept C2777402240 @default.
- W4387427488 hasConcept C33923547 @default.
- W4387427488 hasConcept C41008148 @default.
- W4387427488 hasConcept C55493867 @default.
- W4387427488 hasConcept C66746571 @default.
- W4387427488 hasConcept C77618280 @default.
- W4387427488 hasConceptScore W4387427488C104317684 @default.
- W4387427488 hasConceptScore W4387427488C119857082 @default.
- W4387427488 hasConceptScore W4387427488C125411270 @default.
- W4387427488 hasConceptScore W4387427488C134306372 @default.
- W4387427488 hasConceptScore W4387427488C137293760 @default.
- W4387427488 hasConceptScore W4387427488C142362112 @default.
- W4387427488 hasConceptScore W4387427488C153349607 @default.
- W4387427488 hasConceptScore W4387427488C154945302 @default.
- W4387427488 hasConceptScore W4387427488C185592680 @default.
- W4387427488 hasConceptScore W4387427488C204321447 @default.
- W4387427488 hasConceptScore W4387427488C2777402240 @default.
- W4387427488 hasConceptScore W4387427488C33923547 @default.
- W4387427488 hasConceptScore W4387427488C41008148 @default.
- W4387427488 hasConceptScore W4387427488C55493867 @default.
- W4387427488 hasConceptScore W4387427488C66746571 @default.
- W4387427488 hasConceptScore W4387427488C77618280 @default.
- W4387427488 hasLocation W43874274881 @default.
- W4387427488 hasOpenAccess W4387427488 @default.
- W4387427488 hasPrimaryLocation W43874274881 @default.
- W4387427488 hasRelatedWork W140709781 @default.
- W4387427488 hasRelatedWork W1510159504 @default.
- W4387427488 hasRelatedWork W1581723585 @default.
- W4387427488 hasRelatedWork W2017583614 @default.
- W4387427488 hasRelatedWork W2156531654 @default.
- W4387427488 hasRelatedWork W2253069048 @default.
- W4387427488 hasRelatedWork W2294330161 @default.
- W4387427488 hasRelatedWork W2372020181 @default.
- W4387427488 hasRelatedWork W2804553224 @default.
- W4387427488 hasRelatedWork W4378714697 @default.
- W4387427488 isParatext "false" @default.
- W4387427488 isRetracted "false" @default.
- W4387427488 workType "article" @default.