Third Party Funds Group - Sub project
Acronym: GRK 2839 Project 4
Start date : 01.10.2022
End date : 30.09.2027
The main objective of this project is to study the representational and computational principles of
construction storage and processing, and their structural and functional implementation in the
brain.
Recent experimental findings in neuroscience provide converging evidence for the neural and
psychological plausibility of network-like representations of hierarchical linguistic structures such
as constructions (Herbst 2018a, 2020) on abstract cognitive maps. Cognitive maps are mental
representations that serve an organism to acquire, code, store, recall, and decode information
about the relative locations and features of objects (Tolman 1948, O’Keefe & Nadel 1978).
Electrophysiological research in rodents suggests that the hippocampus (O’Keefe & Nadel 1978)
and the entorhinal cortex (Moser, Moser & McNaughton 2017) are the neurological basis of
cognitive maps. There, highly specialised neurons including place (O’Keefe & Dostrovsky 1971)
and grid cells (Hafting et al. 2005) support map-like spatial codes, and thus enable spatial
navigation (Moser, Kropff & Moser 2008). Furthermore, human fMRI studies during virtual
navigation tasks have shown that the hippocampal and entorhinal spatial codes, together with
areas in the frontal lobe, enable route planning during navigation (Spiers & Maguire 2006, Spiers
& Gilbert 2015, Hartley et al. 2003, Balaguer et al. 2016) based on distance preserving
representations (Morgan et al. 2011). Recent human fMRI studies even suggest that these maplike
representations extend beyond physical space to more abstract social and conceptual spaces
(Epstein et al. 2017), thereby contributing broadly to other cognitive domains (Schiller et al. 2015),
and thus enabling navigation and route planning in arbitrary abstract cognitive spaces (Bellmund
et al. 2018). Besides contributing to spatial navigation, the hippocampus also plays a crucial role
in episodic and declarative memory (Tulving & Markowitsch 1998) by receiving highly processed
information via direct and indirect pathways from a large number of multi-modal areas of the
cerebral cortex (Battaglia et al. 2011) including language related areas (Hickok & Poeppel 2004).
Finally, some findings indicate that the hippocampus even contributes to the coding of narrative
context (Milivojevic et al. 2016). These findings provide a novel theoretical framework of language
representation and processing. There, hippocampal coding would enable flexible representational
mapping of linguistic structures across a wide range of scales and hierarchical levels, from phonemes, single words and collocations (Evert 2008), through valency patterns (Herbst & Uhrig
2019, Herbst et al. 2004), to idioms and abstract argument structure constructions (Herbst 2018,
Herbst & Uhrig 2019).
In particular, the project is to explore the hypothesis that linguistic constructions are represented
as multi-scale network-like maps, and that constructions are combined in the process of
formulating an utterance by navigating on these maps, i.e. a certain utterance would correspond
to a certain route. Furthermore, overlap and blending (Herbst 2018b) of constructions are realized
and guided as and correspond to switching between different levels of the multi-scale maps.
In contrast to the brain, computational models have the decisive advantage that they are fully
accessible, i.e. the temporal evolution of all model variables can be read out for further analysis
at any time. In addition, we can perform arbitrary experimental manipulations on these models.
Therefore, starting from contemporary machine learning approaches for the representation and
processing of natural language (e.g. Devlin et al. 2019), computational models will be constructed
that can build and navigate on hierarchical map-like multi-scale representations of language,
thereby being capable of representing natural language input, transforming these representations
according to pre-defined tasks (e.g. re-phrasing), and producing modified natural language
output. Using evolutionary optimization, the biological fidelity of the initially constructed models
will be increased iteratively by applying the concepts of variation and selection. In order to select
those candidate models that best fit to measured brain activity, internal model activity and brain
activity will be compared using an advanced methodology comprising multivariate statistics
(Kriegeskorte, Mur & Bandettini 2008, Krauss et al. 2018, Schilling et al. 2021) and Bayesian
model selection (Mark et al. 2018). After each selection step, new “child models” will be created
by adjusting their architectures and parameters using machine learning approaches such as
neural architecture search (Elsken, Metzen & Hutter 2018, Wistuba, Rawat & Pedapati 2019, Liu
et al. 2018, Zoph & Le 2016, Chen et al. 2019) and genetic algorithms (Gerum et al. 2020).
So far, neuroscientific studies of language have mostly employed over-simplified experimental
paradigms, e.g. by focussing on single word processing or sentences in isolation. Very recently,
the advantages of using natural, connected language such as narratives for neuroimaging studies
have been discussed (Willems, Nastase & Milivojevic 2020, Jääskeläinen et al. 2020, Hamilton &
Huth 2020, Hauk & Weiss 2020, Schilling et al. 2021). Therefore, multi-modal (fMRI, MEG, EEG,
iEEG and ECoG) measurements during continuous speech perception (listening to audiobooks)
will be performed, as described in detail in our preliminary work (Schilling et al. 2021).