0000000000963598
AUTHOR
Benjamin Straube
Gesture's body orientation modulates the N400 for visual sentences primed by gestures
Abstract Body orientation of gesture entails social‐communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face‐to‐face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an electroencephalography (EEG) experiment presenting to participants (n = 21) videos of frontal and lateral communicative hand gestures of 5 s (e.g., raising a hand), followed by visually presented …
The facilitative effect of gestures on the neural processing of semantic complexity in a continuous narrative
© 2019 Elsevier Inc. Gestures are elemental components of social communication and aid comprehension of verbal messages; however, little is known about the potential role of gestures in facilitating processing of semantic complexity in an ecologically valid setting. The goal of this study was to investigate whether cognitive load, as indexed by semantic complexity, is modulated by the presentation of gestures accompanying speech. Twenty healthy participants watched 16 video clips of a short narrative while instructed to carefully listen to and watch the narrator while functional magnetic resonance imaging (fMRI) data were acquired. The videos contained passages with and without various co-s…
Antagonism between brain regions relevant for cognitive control and emotional memory facilitates the generation of humorous ideas
Abstract The ability to generate humor gives rise to positive emotions and thus facilitate the successful resolution of adversity. Although there is consensus that inhibitory processes might be related to broaden the way of thinking, the neural underpinnings of these mechanisms are largely unknown. Here, we use functional Magnetic Resonance Imaging, a humorous alternative uses task and a stroop task, to investigate the brain mechanisms underlying the emergence of humorous ideas in 24 subjects. Neuroimaging results indicate that greater cognitive control abilities are associated with increased activation in the amygdala, the hippocampus and the superior and medial frontal gyrus during the ge…
Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia
Highlights • Social/non-social information processing in three modalities was investigated in SZ. • SZ showed reduced activation for social information only in gesture modality. • Reduced activation in SZ was observed for non-social information only in speech. • Neural Neural processing in bimodal condition is not different between patients and controls.
Evidence for gesture-speech mismatch detection impairments in schizophrenia.
Patients with schizophrenia suffer from impairments in the perception and production of gestures. The extent to which patients can access the semantic association between speech and co-verbal gestures in concrete or abstract/metaphorical meaning contexts is unknown. We investigated 1) how patients differ from controls in gesture matching performance, 2) how performance differs in the context of abstract versus concrete meaning, and 3) whether formal thought disorder (FTD) symptom severity predicts task impairment. Forty-five patients with schizophrenia spectrum disorders (two subgroups, mild and severe) took part in this study. Participants were presented with video clips, each showing an a…
Feeling addressed! The neural processing of social communicative cues in patients with major depression
Abstract The feeling of being addressed is the first step in a complex processing stream enabling successful social communication. Social impairments are a relevant characteristic of patients with major depressive disorder (MDD). Here, we investigated a mechanism which—if impaired—might contribute to withdrawal or isolation in MDD, namely, the neural processing of social cues such as body orientation and gesture. During funtional magnetic resonance imaging (fMRI) data acquisition, 33 patients with MDD and 43 healthy control subjects watched video clips of a speaking actor: one version with a gesture accompanying the speech and one without gesture. Videos were filmed simultaneously from two …
Gesture’s body orientation modulates the N400 during semantic integration of gesture and visual sentence
AbstractBody orientation of gesture entails social-communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face-to-face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an EEG experiment presenting to participants (n=21) videos of frontal and lateral hand gestures of five-seconds (e.g., raising a hand), followed by visually presented sentences that are either congrue…
Neural Basis of Speech-Gesture Mismatch Detection in Schizophrenia Spectrum Disorders
AbstractPatients with schizophrenia spectrum disorders (SSD) exhibit an aberrant perception and comprehension of abstract speech-gesture combinations associated with dysfunctional activation of the left inferior frontal gyrus (IFG). Recently, a significant deficit of speech-gesture mismatch detection was identified in SSD, but the underlying neural mechanisms have not yet been examined. A novel mismatch-detection fMRI paradigm was implemented manipulating speech-gesture abstractness (abstract/concrete) and relatedness (related/unrelated). During fMRI data acquisition, 42 SSD patients (schizophrenia, schizoaffective disorder, or other non-organic psychotic disorder [ICD-10: F20, F25, F28; DS…
Characterizing the Theory of Mind Network in Schizophrenia Reveals a Sparser Network Structure
AbstractImpaired social functioning is a hallmark of schizophrenia and altered functional integration between distant brain regions are expected to account for signs and symptoms of the disorder. The functional neuroarchitecture of a network relevant for social functioning, the mentalizing network, is however poorly understood. In this study we examined dysfunctions of the mentalizing network in patients with schizophrenia compared to healthy controls via dynamic causal modelling and an interactive social decision-making game. Network characteristics were analyzed on a single subject basis whereas graph theoretic metrics such as in-degree, out-degree and edge-connectivity per network node w…
Hand Gestures Alert Auditory Cortices
When acquiring a foreign language, the first challenge is to break into the speech stream to identify basic linguistic units. The present study tested the hypothesis that hand gestures facilitate this process by alerting auditory cortices to attend to and identify meaningful phonemic information. During fMRI data acquisition, participants watched videos of an actor speaking in Russian under three conditions. Sentences were produced with just speech alone or were accompanied by two types of hand gestures: 1) metaphoric gesture and 2) free gesture. The main finding was that there was increased auditory cortex activation when both types of gestures accompanied speech compared to speech alone, …
The EEG and fMRI signatures of neural integration: An investigation of meaningful gestures and corresponding speech
Abstract One of the key features of human interpersonal communication is our ability to integrate information communicated by speech and accompanying gestures. However, it is still not fully understood how this essential combinatory process is represented in the human brain. Functional magnetic resonance imaging (fMRI) studies have unanimously attested the relevance of activation in the posterior superior temporal sulcus/middle temporal gyrus (pSTS/MTG), while electroencephalography (EEG) studies have shown oscillatory activity in specific frequency bands to be associated with multisensory integration. In the current study, we used fMRI and EEG to separately investigate the anatomical and o…
The Role of Gamma Oscillations During Integration of Metaphoric Gestures and Abstract Speech
Metaphoric (MP) co-speech gestures are commonly used during daily communication. They communicate about abstract information by referring to gestures that are clearly concrete (e.g., raising a hand for "the level of the football game is high"). To understand MP co-speech gestures, a multisensory integration at semantic level is necessary between abstract speech and concrete gestures. While semantic gesture-speech integration has been extensively investigated using functional magnetic resonance imaging, evidence from electroencephalography (EEG) is rare. In the current study, we set out an EEG experiment, investigating the processing of MP vs. iconic (IC) co-speech gestures in different cont…
Cognitive and emotional empathy in patients with schizophrenia spectrum disorders: A replication and extension study.
Impairments of social cognition are defined as core features in the pathology of schizophrenia. In a study by Lehmann and colleagues (2014), patients with schizophrenia have been shown to demonstrate a diminished capacity to understand others' emotions (i.e. cognitive empathy), but a preserved ability to share or feel the emotional states of others (i.e. emotional empathy). Here, we report on an independent replication study investigating cognitive and emotional empathy in 35 patients with schizophrenia spectrum disorders and a matched control group, which 1) confirms that patients demonstrate preserved emotional empathy in self-report and behavioural measurements, and 2) reveals associatio…
Supramodal neural processing of abstract information conveyed by speech and gesture
Abstractness and modality of interpersonal communication have a considerable impact on comprehension. They are relevant for determining thoughts and constituting internal models of the environment. Whereas concrete object-related information can be represented in mind irrespective of language, abstract concepts require a representation in speech. Consequently, modality-independent processing of abstract information can be expected. Here we investigated the neural correlates of abstractness (abstract vs. concrete) and modality (speech vs. gestures), to identify an abstractness-specific supramodal neural network. During fMRI data acquisition 20 participants were presented with videos of an ac…
Modality-specific dysfunctional neural processing of social and non-social information in schizophrenia
Abstract Background Schizophrenia (SZ) is characterized by marked social dysfunctions encompassing potential deficits in the processing of social and non-social information, especially in everyday settings where multiple modalities are present. To date, the neurobiological basis of these deficits remains elusive. Methods In a functional magnetic resonance imaging (fMRI) study, 17 patients with schizophrenia or schizoaffective disorder, and 18 matched controls watched videos of an actor speaking, gesturing (unimodal), and both speaking and gesturing (bimodal) about social or non-social events in a naturalistic way. Participants had to judge whether each video contains person-related (social)…
A Multimodal Speech-Gesture Training Intervention for Patients With Schizophrenia and Its Neural Underpinnings – the Study Protocol of a Randomized Controlled Pilot Trial
Dysfunctional social communication is one of the most stable characteristics in patients with schizophrenia that also affects quality of life. Interpreting abstract speech and integrating nonverbal modalities is particularly affected. Considering the impact of communication on social life but failure to treat communication dysfunctions with usual treatment, we will investigate the possibility to improve verbal and non-verbal communication in schizophrenia by applying a multimodal speech-gesture training (MSG training). Here we describe the newly developed MSG training program and the study design for the first clinical investigation. The intervention contains perceptive rating (match/mismat…
Impaired Right Temporoparietal Junction–Hippocampus Connectivity in Schizophrenia and Its Relevance for Generating Representations of Other Minds
Schizophrenia is associated with impaired and exaggerated Theory of Mind processes, pointing on alterations in generating a representation of another person's mind. Despite recent work on healthy subjects suggesting that a coupling between the right temporoparietal junction (rTPJ) and the hippocampus is relevant for building representations of others' intentions, the neural basis of related dysfunctions in patients with schizophrenia remains unclear. Therefore, we used structural and functional magnetic resonance imaging together with a modified prisoner's dilemma game to test the hypotheses, that patients show dysfunctional social updating on behavioral level accompanied by altered rTPJ-hi…
Neural processing of nouns and verbs in spontaneous speech of patients with schizophrenia.
Previous fMRI-studies investigating the production of nouns and verbs in healthy participants reported predominantly activation in the left inferior frontal gyrus (IFG) for both classes of words with increased neural responses for verbs. To date, comparable imaging data for spontaneous speech in patients with schizophrenia is missing. These results are novel and may contribute to understand the neural basis of noun and verb production in a "natural" environment. Fifteen patients with schizophrenia and fifteen healthy control participants described pictures for one minute each while BOLD signal changes were measured with fMRI. In an event-related design, activations related to noun and verb …
The role of the right temporo-parietal junction in social decision-making
Identifying someone else's noncooperative intentions can prevent exploitation in social interactions. Hence, the inference of another person's mental state might be most pronounced in order to improve social decision‐making. Here, we tested the hypothesis that brain regions associated with Theory of Mind (ToM), particularly the right temporo–parietal junction (rTPJ), show higher neural responses when interacting with a selfish person and that the rTPJ‐activity as well as cooperative tendencies will change over time. We used functional magnetic resonance imaging (fMRI) and a modified prisoner's dilemma game in which 20 participants interacted with three fictive playing partners who behaved a…
The Neural Basis of Idea Density During Natural Spoken Language
Idea density (ID) evolved as a quantification of propositional base structure. Besides its function as a measure of linguistic complexity, ID has also been used as an index of general linguistic ability. In order to find the neural basis for the processing of high or low ID during spontaneous speech, a sample of healthy adults was assessed using the functional resonance imaging (fMRI) technique; participants described pictures presented to them while in the scanner. Differential patterns of activation were observed for the low- and high-ID conditions, providing new insights into the processing correlates of ID.
Frontal hypoactivation and alterations in the reward-system during humor processing in patients with schizophrenia spectrum disorders
Humor is a ubiquitous human ability with important implications for both social and emotional functioning. Patients with neuropsychiatric disorders, such as schizophrenia, experience difficulties in the comprehension and appreciation of humor. However, the specific neural mechanisms underlying these deficits are unknown. In the current study, we sought to elucidate the neural correlates of humor processing in patients with schizophrenia spectrum disorders (SSD). Functional Magnetic Resonance Imaging (fMRI) was used in thirty-one patients with SSD and a control group, performing a humor processing paradigm. Both regional brain activation and parametric modulation of brain responses via subje…