SemOpenAlex |

SemOpenAlex

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2157804791> ?p ?o ?g. }

Showing items 1 to 100 of ±205 with 100 items per page.

W2157804791 abstract "This thesis concerns the form-meaning mapping of multimodal communicative actions consisting of speech signals and improvised co-speech gestures, produced spontaneously with the hand. The interaction between speech and speech-accompanying gestures has been standardly addressed from a cognitive perspective to establish the underlying cognitive mechanisms for the synchronous speech and gesture production, and also from a computational perspective to build computer systems that communicate through multiple modalities. Based on the findings of this previous research, we advance a new theory in which the mapping from the form of the combined speech-and-gesture signal to its meaning is analysed in a constraint-based multimodal grammar. We propose several construction rules about multimodal well-formedness that we motivate empirically from an extensive and detailed corpus study. In particular, the construction rules use the prosody, syntax and semantics of speech, the form and meaning of the gesture signal, as well as the temporal performance of the speech relative to the temporal performance of the gesture to constrain the derivation of a single multimodal syntax tree which in turn determines a meaning representation via standard mechanisms for semantic composition. Gestural form often underspecifies its meaning, and so the output of our grammar is underspecified logical formulae that support the range of possible interpretations of the multimodal act in its final context-of-use, given the current models of the semantics/pragmatics interface. It is standardly held in the gesture community that the co-expressivity of speech and gesture is determined on the basis of their temporal co-occurrence: that is, a gesture signal is semantically related to the speech signal that happened at the same time as the gesture. Whereas this is usually taken for granted, we propose a methodology of establishing in a systematic and domain-independent way which spoken element(s) gesture can be semantically related to, based on their form, so as to yield a meaning representation that supports the intended interpretation(s) in context. The ‘semantic’ alignment of speech and gesture is thus driven not from the temporal co-occurrence alone, but also from the linguistic properties of the speech signal gesture overlaps with. In so doing, we contribute a fine-grained system for articulating the form-meaning mapping of multimodal actions that uses standard methods from linguistics. We show that just as language exhibits ambiguity in both form and meaning, so do multimodal actions: for instance, the integration of gesture is not restricted to a unique speech phrase but rather speech and gesture can be aligned in multiple multimodal iii syntax trees thus yielding distinct meaning representations. These multiple mappings stem from the fact that the meaning as derived from gesture form is highly incomplete even in context. An overall challenge is thus to account for the range of possible interpretations of the multimodal action in context using standard methods from linguistics for syntactic derivation and semantic composition." @default.
W2157804791 created "2016-06-24" @default.
W2157804791 creator A5039211209 @default.
W2157804791 creator A5058001431 @default.
W2157804791 creator A5069073392 @default.
W2157804791 date "2013-07-02" @default.
W2157804791 modified "2023-09-23" @default.
W2157804791 title "Alignment of Speech and Co-speech Gesture in a Constraint-based Grammar" @default.
W2157804791 cites W112381816 @default.
W2157804791 cites W1493636870 @default.
W2157804791 cites W1498883201 @default.
W2157804791 cites W1504188396 @default.
W2157804791 cites W1504344236 @default.
W2157804791 cites W1510929985 @default.
W2157804791 cites W1519642432 @default.
W2157804791 cites W153699105 @default.
W2157804791 cites W1539850931 @default.
W2157804791 cites W1553000749 @default.
W2157804791 cites W1553203434 @default.
W2157804791 cites W1553846935 @default.
W2157804791 cites W1567277581 @default.
W2157804791 cites W1572948005 @default.
W2157804791 cites W1574440611 @default.
W2157804791 cites W1581670392 @default.
W2157804791 cites W1581889689 @default.
W2157804791 cites W1586232984 @default.
W2157804791 cites W159413648 @default.
W2157804791 cites W1598122162 @default.
W2157804791 cites W1605174196 @default.
W2157804791 cites W164929510 @default.
W2157804791 cites W1708911126 @default.
W2157804791 cites W1935857547 @default.
W2157804791 cites W1947579273 @default.
W2157804791 cites W1974257828 @default.
W2157804791 cites W1976212489 @default.
W2157804791 cites W1976913920 @default.
W2157804791 cites W1988519408 @default.
W2157804791 cites W1988604523 @default.
W2157804791 cites W1989590930 @default.
W2157804791 cites W1991568221 @default.
W2157804791 cites W1991626023 @default.
W2157804791 cites W1999090433 @default.
W2157804791 cites W2001087363 @default.
W2157804791 cites W2007465845 @default.
W2157804791 cites W2008208299 @default.
W2157804791 cites W2011611983 @default.
W2157804791 cites W2011623898 @default.
W2157804791 cites W2013489329 @default.
W2157804791 cites W2013631598 @default.
W2157804791 cites W2020010749 @default.
W2157804791 cites W2025835932 @default.
W2157804791 cites W203491554 @default.
W2157804791 cites W2038248725 @default.
W2157804791 cites W2042822564 @default.
W2157804791 cites W2043136118 @default.
W2157804791 cites W2045220037 @default.
W2157804791 cites W204690852 @default.
W2157804791 cites W2050530001 @default.
W2157804791 cites W2052417512 @default.
W2157804791 cites W2054052728 @default.
W2157804791 cites W2054823774 @default.
W2157804791 cites W2061270248 @default.
W2157804791 cites W2069753856 @default.
W2157804791 cites W2070883187 @default.
W2157804791 cites W2072819621 @default.
W2157804791 cites W2078226637 @default.
W2157804791 cites W2079503396 @default.
W2157804791 cites W2079563482 @default.
W2157804791 cites W2083058451 @default.
W2157804791 cites W2083325968 @default.
W2157804791 cites W2085574295 @default.
W2157804791 cites W2087392613 @default.
W2157804791 cites W2091312385 @default.
W2157804791 cites W2091818076 @default.
W2157804791 cites W2095589793 @default.
W2157804791 cites W2096871981 @default.
W2157804791 cites W2097606805 @default.
W2157804791 cites W2099915323 @default.
W2157804791 cites W2102980895 @default.
W2157804791 cites W2111907704 @default.
W2157804791 cites W2113928197 @default.
W2157804791 cites W2117299073 @default.
W2157804791 cites W2118163921 @default.
W2157804791 cites W2118452738 @default.
W2157804791 cites W2123013114 @default.
W2157804791 cites W2127508752 @default.
W2157804791 cites W2133168789 @default.
W2157804791 cites W2135978202 @default.
W2157804791 cites W2137904396 @default.
W2157804791 cites W2138105680 @default.
W2157804791 cites W2138230411 @default.
W2157804791 cites W2139974778 @default.
W2157804791 cites W2140588956 @default.
W2157804791 cites W2143385847 @default.
W2157804791 cites W2147994465 @default.
W2157804791 cites W2150421899 @default.
W2157804791 cites W2151157246 @default.
W2157804791 cites W2152197045 @default.
W2157804791 cites W2153382679 @default.
W2157804791 cites W2153804780 @default.