Seminar: Vision and Language in Brains and Machines

Abstract: 

Research in the brain and cognitive sciences attempts to uncover the neural mechanisms underlying intelligent behavior. Due to the complexities of brain processing, studies necessarily had to start with a narrow scope of experimental investigation and computational modeling. I will argue that it is time for our field to take the next step: build system models that capture neural mechanisms and supported behaviors in entire domains of intelligence. To make progress on system models, we are developing the Brain-Score platform which, to date, hosts over 50 benchmarks of neural and behavioral experiments that models can be tested on. By systematically evaluating a wide variety of model candidates, we not only identify models beginning to match a range of brain data (~50% explained variance), but also discover key relationships: Models' brain scores are predicted by their object categorization performance in vision and their next-word prediction performance in language. The better models predict internal neural activity, the better they match human behavioral outputs, with architecture substantially contributing to brain-like representations. Using the integrative benchmarks, we develop improved state-of-the-art system models that more closely match shallow recurrent neuroanatomy, predict primate temporal processing, and are more robust to changes in the input distribution. Finally I will argue that our newest generation of models can be used to predict the behavioral effects of neural interventions, and to drive new causal experiments.

Bio: 

Martin is a tenure-track assistant professor at EPFL where he builds artificial intelligence models of the brain. To achieve this goal, he bridges research in Machine Learning, Neuroscience, and Cognitive Science. He initiated the community-wide Brain-Score platform for evaluating models on their brain and behavioral alignment, and built state-of-the-art models such as CORnet and VOneNet. Martin completed his PhD at MIT with Jim DiCarlo, following Bachelor's and Master's degrees in computer science at TUM, LMU, and UNA. Previously he worked at Harvard, MetaMind/Salesforce, Oracle, and co-founded two startups. Among others, his work has been recognized in the news at Science magazine, MIT News, and Scientific American; and with awards such as the Neuro-Irv and Helga Cooper Open Science Prize, the Google.org Impact Challenge prize, and the Takeda fellowship in AI + Health.