Matches in SemOpenAlex for { <https://semopenalex.org/work/W4304195093> ?p ?o ?g. }
Showing items 1 to 79 of
79
with 100 items per page.
- W4304195093 abstract "To produce accurate predictions, language models (LMs) must balance between generalization and memorization. Yet, little is known about the mechanism by which transformer LMs employ their memorization capacity. When does a model decide to output a memorized phrase, and how is this phrase then retrieved from memory? In this work, we offer the first methodological framework for probing and characterizing recall of memorized sequences in transformer LMs. First, we lay out criteria for detecting model inputs that trigger memory recall, and propose idioms as inputs that typically fulfill these criteria. Next, we construct a dataset of English idioms and use it to compare model behavior on memorized vs. non-memorized inputs. Specifically, we analyze the internal prediction construction process by interpreting the model's hidden representations as a gradual refinement of the output probability distribution. We find that across different model sizes and architectures, memorized predictions are a two-step process: early layers promote the predicted token to the top of the output distribution, and upper layers increase model confidence. This suggests that memorized information is stored and retrieved in the early layers of the network. Last, we demonstrate the utility of our methodology beyond idioms in memorized factual statements. Overall, our work makes a first step towards understanding memory recall, and provides a methodological basis for future studies of transformer memorization." @default.
- W4304195093 created "2022-10-11" @default.
- W4304195093 creator A5009457488 @default.
- W4304195093 creator A5011769908 @default.
- W4304195093 creator A5028476919 @default.
- W4304195093 creator A5065238853 @default.
- W4304195093 creator A5065717258 @default.
- W4304195093 creator A5087870256 @default.
- W4304195093 date "2022-10-07" @default.
- W4304195093 modified "2023-10-16" @default.
- W4304195093 title "Understanding Transformer Memorization Recall Through Idioms" @default.
- W4304195093 doi "https://doi.org/10.48550/arxiv.2210.03588" @default.
- W4304195093 hasPublicationYear "2022" @default.
- W4304195093 type Work @default.
- W4304195093 citedByCount "0" @default.
- W4304195093 crossrefType "posted-content" @default.
- W4304195093 hasAuthorship W4304195093A5009457488 @default.
- W4304195093 hasAuthorship W4304195093A5011769908 @default.
- W4304195093 hasAuthorship W4304195093A5028476919 @default.
- W4304195093 hasAuthorship W4304195093A5065238853 @default.
- W4304195093 hasAuthorship W4304195093A5065717258 @default.
- W4304195093 hasAuthorship W4304195093A5087870256 @default.
- W4304195093 hasBestOaLocation W43041950931 @default.
- W4304195093 hasConcept C100660578 @default.
- W4304195093 hasConcept C119857082 @default.
- W4304195093 hasConcept C121332964 @default.
- W4304195093 hasConcept C134306372 @default.
- W4304195093 hasConcept C154945302 @default.
- W4304195093 hasConcept C15744967 @default.
- W4304195093 hasConcept C165801399 @default.
- W4304195093 hasConcept C177148314 @default.
- W4304195093 hasConcept C180747234 @default.
- W4304195093 hasConcept C204321447 @default.
- W4304195093 hasConcept C2776224158 @default.
- W4304195093 hasConcept C28490314 @default.
- W4304195093 hasConcept C30038468 @default.
- W4304195093 hasConcept C33923547 @default.
- W4304195093 hasConcept C38652104 @default.
- W4304195093 hasConcept C41008148 @default.
- W4304195093 hasConcept C48145219 @default.
- W4304195093 hasConcept C5274069 @default.
- W4304195093 hasConcept C62520636 @default.
- W4304195093 hasConcept C66322947 @default.
- W4304195093 hasConceptScore W4304195093C100660578 @default.
- W4304195093 hasConceptScore W4304195093C119857082 @default.
- W4304195093 hasConceptScore W4304195093C121332964 @default.
- W4304195093 hasConceptScore W4304195093C134306372 @default.
- W4304195093 hasConceptScore W4304195093C154945302 @default.
- W4304195093 hasConceptScore W4304195093C15744967 @default.
- W4304195093 hasConceptScore W4304195093C165801399 @default.
- W4304195093 hasConceptScore W4304195093C177148314 @default.
- W4304195093 hasConceptScore W4304195093C180747234 @default.
- W4304195093 hasConceptScore W4304195093C204321447 @default.
- W4304195093 hasConceptScore W4304195093C2776224158 @default.
- W4304195093 hasConceptScore W4304195093C28490314 @default.
- W4304195093 hasConceptScore W4304195093C30038468 @default.
- W4304195093 hasConceptScore W4304195093C33923547 @default.
- W4304195093 hasConceptScore W4304195093C38652104 @default.
- W4304195093 hasConceptScore W4304195093C41008148 @default.
- W4304195093 hasConceptScore W4304195093C48145219 @default.
- W4304195093 hasConceptScore W4304195093C5274069 @default.
- W4304195093 hasConceptScore W4304195093C62520636 @default.
- W4304195093 hasConceptScore W4304195093C66322947 @default.
- W4304195093 hasLocation W43041950931 @default.
- W4304195093 hasOpenAccess W4304195093 @default.
- W4304195093 hasPrimaryLocation W43041950931 @default.
- W4304195093 hasRelatedWork W1538473846 @default.
- W4304195093 hasRelatedWork W1586984800 @default.
- W4304195093 hasRelatedWork W1697423248 @default.
- W4304195093 hasRelatedWork W2063375830 @default.
- W4304195093 hasRelatedWork W2369308426 @default.
- W4304195093 hasRelatedWork W2989932438 @default.
- W4304195093 hasRelatedWork W3107474891 @default.
- W4304195093 hasRelatedWork W4220812971 @default.
- W4304195093 hasRelatedWork W4304195093 @default.
- W4304195093 hasRelatedWork W86096423 @default.
- W4304195093 isParatext "false" @default.
- W4304195093 isRetracted "false" @default.
- W4304195093 workType "article" @default.