1. The GREED project: Keeping the result in sight – effects of grammar on event encoding in Estonian vs. Dutch
This project, which was the MA thesis of Maria Sakarias from Utrecht university, investigates how grammar influences the way in which we perceive and memorize causative actions (see example below) in verbal and nonverbal encoding conditions.
We were interested in the language pair Estonian and Dutch, since Estonian marks the presence or absence of a result for such actions via grammatical case-marking on object nouns. Dutch and Estonian Participants were eye-tracked while viewing videos of two types of causative events (resultative/non-resultative) which were depicted as either finished or unfinished. In the verbal encoding condition, participants were asked to describe each video clip in one sentence (e.g., “the woman drew a flower”). In the nonverbal encoding condition, they performed a distracter task, which involved detecting sound cues while watching the videos.
In a subsequent surprise memory test, participants in both conditions were tested on their recall of the end state (+/- finished) of the events. For resultative actions, Estonian speakers marked the presence or absence of a result with the appropriate grammatical case-marker on the object noun. In this condition, this group also allocated significantly more attention to the visual area of the action/object (action-bias) during the final frames of the videos than Dutch participants. Estonian participants showed no such action-bias in the nonverbal condition; Dutch participants’ attention was generally not influenced by task condition.
Looking at memory data, Estonian participants were more accurate at recalling the end state of the events in the verbal than the nonverbal condition. Interestingly, this group showed higher accuracy for resultative compared to non-resultative events in both verbal and nonverbal conditions, while no effect of resultativity of the event could be detected in Dutch participants.
The observed visual attention patterns and memory test results for Estonian participants are consistent with the hypothesis that language effects emerge most clearly under linguistic encoding instructions. In addition, improved performance for resultative events by Estonians in the nonverbal condition is taken to reveal a global influence of language on how events are habitually processed and memorized.
2. The OWLIE project: To stand or to lie, that’s the question! Semantic categories guide verbal and nonverbal processing
In a joint project with Geertje van Bergen (also MPI) (www.geertjevanbergen.com) we target how language-specific semantics guide core language processing mechanisms (i.e. predictive sentence processing), as well as visual processing in nonlinguistic tasks. Our research focuses on a cross-linguistic contrast in the semantics of verbs describing locative states and caused motion (i.e., placement events), which by default encode the position of objects in Dutch and German, but not in, for example, French and English.
In a first study (van Bergen & Flecken, 2017), we examined anticipatory looks to objects in a display upon encountering placement verbs in sentences (visual world paradigm, cf. Altmann & Kamide, 1999). The sentences described placement events in Dutch (e.g., De jongen zette/legde kort geleden een fles op de tafel ‘the boy put.STAND/put.LIE recently a bottle on the table’). We measured the extent to which people already started looking at the objects depicted on the screen (4 in total, see picture below) as soon as they heard the verb (“zette” put.STAND or “legde” put.LIE) and before they heard the noun “fles” (bottle). We found that German second language (L2) users of Dutch anticipated objects that matched the verbally encoded position, i.e., they looked more often at the bottle in an upright position on the table, than the bottle that was lying on the table (similar to native Dutch listeners); a sample of equally proficient English and French L2 users of Dutch did not predict objects on the basis of verbal semantics. Findings could be attributed to the transfer of native language processing routines: German, like Dutch, encodes the position of objects in placement verbs (hinstellen/hinlegen). Given their lifetime of experience in attending to object position when describing placement events, the German L2 users of Dutch could process this information rapidly, and thus use it to already anticipate what object was going to be mentioned next.
In a second study (Flecken & van Bergen, in prep.) we targeted the extent to which cross-linguistic diversity in verbal semantics biases visual processing in a nonlinguistic task. We contrasted Dutch and English native speakers on a picture-matching task, while their EEG was recorded. Participants saw pairs of pictures, each showing an object on a surface (e.g., a bottle on a table). Objects varied across two spatial dimensions, i.e., rotated 90 degrees along the horizontal (standing/lying orientation) or the vertical (left/right orientation) axis. Event related potentials show early increased attention to the position manipulation only in Dutch participants; this language effect on perception emerged within 200ms after stimulus presentation, before any verbal information could be lexically accessed.
In all, findings of the OWLIE project show that our native language strongly influences the way in which we process both verbal and nonverbal information. In the course of acquisition, processing routines are shaped and biased by the concepts encoded in that specific language; these routines are recruited by default in various tasks.
3. Thinking and talking about events: a cross-linguistic approach to L1 and L2 processing
In my NWO-funded VENI-project (2012-2015) I looked at the extent to which language affects how people process, perceive and attend to events, actions and situations. I used a range of psycho- and neuro-linguistic research methods (eye tracking, ERPs) to investigate event processing in mono- and bilingual speakers of Dutch, German, English, Spanish and Turkish.
First of all, I was interested in how the use of specific linguistic constructions modulates people’s perception and understanding of events. In Flecken & Gerwien (2013), for example, we were able to show that the use of progressive aspect in a sentence affects how people estimate the duration of everyday events and actions.
Secondly, I addressed the question, whether the conceptual distinctions encoded in a specific language lead certain features or dimensions of events or objects (e.g., manner of motion – given cross-linguistic differences in motion verb semantics; motion event endpoints – given differences in grammatical aspect systems) to be perceptually more salient to its native speakers.
Thirdly, I focused on how learning new ways of talking about events or actions through the acquisition of a second language, affects perception and attention patterns.
4. Event conceptualization in language production
My research focuses on cross-linguistic differences in the way in which every-day events and actions are conceptualized and described (cf. Thinking for Speaking (TfS), Slobin, 1996, and Seeing for Speaking (SfS), von Stutterheim & Nuese, 2003). Such cross-linguistic differences represent the basis for analyses of event construal by bilingual speakers: Do they adhere to patterns from 1, or both of their languages, or are there cases of bilingual-specific event conceptualization patterns? (my dissertation work)
I am specifically interested in the interrelation between visual and linguistic processing in language production, using eye tracking as a research method. I investigate whether language differences are ‘superficial’ and ‘merely’ form-related, or whether they already play a role during early phases of processing, in, for example, scene apprehension and message generation (conceptualization) processes. Do speakers of different languages pay attention to different aspects of events, early on, when generating a conceptual representation of the event?
In Flecken & Gerwien (2012), for example, we analyzed first fixations on visual stimuli of events, comparing native speakers of Spanish and German. The languages differ grammatically in the domain of agent-hood: Spanish is a pro-drop language and provides options to leave agents of events implicit or unspecific (e.g., ‘the vase broke itself’, see also Fausey & Boroditsky, 2011). German speakers, on the other hand, conceptualize and describe events with a great deal of specificity in relation to entities (Flecken et al., in revision). Given brief exposure time to the stimuli (300ms, 500ms) , we find that the two groups of speakers show different starting points for visual and linguistic processing; whereas German speakers fixate specific regions of the stimuli (in particular, regions in which agents are displayed), Spanish speakers do so to a lesser extent.
As another example, in a cross-linguistic analysis of motion event construal in French, Italian and German native speakers, we find that, in line with directional concepts expressed in motion verbs in French and Italian, speakers allocate early visual attention to the moving entity as well as the endpoint in motion event scenes, i.e., before utterance onset (Flecken et al. submitted). German speakers only direct their attention towards endpoints after they have started articulating their sentences.
With respect to the domain of event conceptualization, I am also interested in the factors underlying the production of complex syntactic structures (i.e., progressive aspect) in mono- and bilingual speakers (see for a recent sentence priming study, Flecken & Gerwien, 2013).
5. The processing implications of grammatical aspect
Another research area (which is also part of the two above mentioned themes) is the investigation of how grammatical aspect affects how speakers of different languages and different types of language users perceive, understand, conceptualize and describe actions, events and situations.
The psycholinguistic investigation of aspect is very interesting, given the fact that aspect is an abstract grammatical category, which still has quite solid semantic content, in that it expresses a specific perspective or viewpoint on a given state-of-affairs. The function is rather ‘semantic’, however, when grammaticalized, use is highly systematic and aspect marking is obligatory in specific contexts. Furthermore, languages vary widely in the extent to which they encode aspectual distinctions grammatically or lexically.
For example, imagine a situation in which a person asks you about the where-abouts of John, “Where is John? What is he up to right now?”
A speaker of English will most likely say something like: “John is playing football” (indicating that he is out there on the football field right now)
A speaker of Dutch may say: “Jan is aan het voetballen” or “Jan voetbalt (nu)”
A speaker of German will say: “Jan spielt (gerade) Fussball”
Note that, whereas in Dutch a speaker can choose to use the available progressive marker (‘is aan het’) to describe this situation in this context, a speaker of standard German would probably not use similar constructions which are available but are not considered grammatical (‘ist am/beim’). A speaker of English, on the other hand, is required to use progressive aspect when describing an ongoing activity in the here-and-now. The statement “John plays football” would imply reference to a habitual situation, the fact that John is member of a football club.
I am interested in how the use of this aspectual viewpoint, or the lack thereof, affects event perception.
In sum, my research takes an interdisciplinary approach: I aim to bridge the gap between comparative linguistics, language typology and second language acquisition, on the one hand, and cognitive psychology and cognitive neuroscience on the other hand.