Matches in SemOpenAlex for { <https://semopenalex.org/work/W4385571306> ?p ?o ?g. }
Showing items 1 to 57 of
57
with 100 items per page.
- W4385571306 abstract "While there is much recent interest in studying why Transformer-based large language models make predictions the way they do, the complex computations performed within each layer have made their behavior somewhat opaque. To mitigate this opacity, this work presents a linear decomposition of final hidden states from autoregressive language models based on each initial input token, which is exact for virtually all contemporary Transformer architectures. This decomposition allows the definition of probability distributions that ablate the contribution of specific input tokens, which can be used to analyze their influence on model probabilities over a sequence of upcoming words with only one forward pass from the model. Using the change in next-word probability as a measure of importance, this work first examines which context words make the biggest contribution to language model predictions. Regression experiments suggest that Transformer-based language models rely primarily on collocational associations, followed by linguistic factors such as syntactic dependencies and coreference relationships in making next-word predictions. Additionally, analyses using these measures to predict syntactic dependencies and coreferent mention spans show that collocational association and repetitions of the same token largely explain the language models’ predictions on these tasks." @default.
- W4385571306 created "2023-08-05" @default.
- W4385571306 creator A5066248885 @default.
- W4385571306 creator A5086362177 @default.
- W4385571306 date "2023-01-01" @default.
- W4385571306 modified "2023-09-24" @default.
- W4385571306 title "Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions" @default.
- W4385571306 doi "https://doi.org/10.18653/v1/2023.acl-long.562" @default.
- W4385571306 hasPublicationYear "2023" @default.
- W4385571306 type Work @default.
- W4385571306 citedByCount "0" @default.
- W4385571306 crossrefType "proceedings-article" @default.
- W4385571306 hasAuthorship W4385571306A5066248885 @default.
- W4385571306 hasAuthorship W4385571306A5086362177 @default.
- W4385571306 hasBestOaLocation W43855713061 @default.
- W4385571306 hasConcept C105795698 @default.
- W4385571306 hasConcept C121332964 @default.
- W4385571306 hasConcept C137293760 @default.
- W4385571306 hasConcept C154945302 @default.
- W4385571306 hasConcept C159877910 @default.
- W4385571306 hasConcept C165801399 @default.
- W4385571306 hasConcept C204321447 @default.
- W4385571306 hasConcept C33923547 @default.
- W4385571306 hasConcept C38652104 @default.
- W4385571306 hasConcept C41008148 @default.
- W4385571306 hasConcept C48145219 @default.
- W4385571306 hasConcept C62520636 @default.
- W4385571306 hasConcept C66322947 @default.
- W4385571306 hasConceptScore W4385571306C105795698 @default.
- W4385571306 hasConceptScore W4385571306C121332964 @default.
- W4385571306 hasConceptScore W4385571306C137293760 @default.
- W4385571306 hasConceptScore W4385571306C154945302 @default.
- W4385571306 hasConceptScore W4385571306C159877910 @default.
- W4385571306 hasConceptScore W4385571306C165801399 @default.
- W4385571306 hasConceptScore W4385571306C204321447 @default.
- W4385571306 hasConceptScore W4385571306C33923547 @default.
- W4385571306 hasConceptScore W4385571306C38652104 @default.
- W4385571306 hasConceptScore W4385571306C41008148 @default.
- W4385571306 hasConceptScore W4385571306C48145219 @default.
- W4385571306 hasConceptScore W4385571306C62520636 @default.
- W4385571306 hasConceptScore W4385571306C66322947 @default.
- W4385571306 hasLocation W43855713061 @default.
- W4385571306 hasOpenAccess W4385571306 @default.
- W4385571306 hasPrimaryLocation W43855713061 @default.
- W4385571306 hasRelatedWork W2359001871 @default.
- W4385571306 hasRelatedWork W3016124757 @default.
- W4385571306 hasRelatedWork W3033862527 @default.
- W4385571306 hasRelatedWork W3097571385 @default.
- W4385571306 hasRelatedWork W3112776819 @default.
- W4385571306 hasRelatedWork W3156902660 @default.
- W4385571306 hasRelatedWork W3174977793 @default.
- W4385571306 hasRelatedWork W3196747313 @default.
- W4385571306 hasRelatedWork W4226082499 @default.
- W4385571306 hasRelatedWork W4287761227 @default.
- W4385571306 isParatext "false" @default.
- W4385571306 isRetracted "false" @default.
- W4385571306 workType "article" @default.