Brain Roads

Brain Roads is an interdisciplinary French-German project for the visualization and interactive exploration of the brain. It brings together designers, engineers and researchers in neurosurgery and humanities. Its ambition is to develop visual representations and interactive tools for exploring white matter. We wish to bridge the analog and the digital in a cooperation regime between the specific knowledge of some outstanding neurosurgeons, which we will try to translate into visual forms, and new representations of brain models, which allow us to explore dense, digitalized matter, and to visualize its changes of state, but also its zones of uncertainty.

The project consists of two complementary approaches: one that is driven by data visualization and design practice, and one that is based on the observation and description of surgical practices of excellence in their milieu. We aim to elaborate new visual forms to represent the functional model of the brain, focusing on the available information within the datasets. Here, we develop the visual forms of these networks and further advance their current prototype of immersive data visualization to explore patient MRI data at multiple scales, ranging dynamically from the voxel-level to global brain representations. Additionally, an analysis of the methods and tacit knowledge of neurosurgeons within a conceptual framework between design theory and anthropology, offers new perspectives on the crafting of digital matter in image-based surgery and virtual reality design.

Mapping and representing ›brain roads‹ can be achieved in different ways: through real-time dialogue with awake patients during surgery for example, as well as through various imaging devices and methods, which help the neurosurgeons extend their senses and their knowledge of the cerebral anatomy and activity. Nowadays, the surgeon’s perception relies largely on imaging processes, which need to be supported by developing new designed interfaces to aid navigating through data and consolidating numerous inputs. To advance such digital representations, we develop functional models and prototypes that introduce new paradigms both in image viewing and analysis in an interdisciplinary setting, consisting of graphic design, software design, cognitive anthropology, software engineering and neurosurgery.

The Data Visualization Program, developed by the Department of Graphic Design at ESAD de Reims with computer engineers of JIN (video games and digital interactions) department at Télécom SudParis (TSP), aims to develop innovative data filtering to enable the analysis and implementation of large datasets in real-time. The underlying reality of digital technologies is decreasingly perceived and understood by its users, due to the increasing complexity of technology. The joint team of designers and engineers proposes effective, open visual concepts and interface prototypes based on state-of-the-art interfaces, such as Virtual Reality, Augmented Reality, Mixed Reality or Multi-modal interfaces to allow new interactions to be experienced.

Brain Roads I: Color

The first step of the research consists in analyzing and decomposing the images currently circulating in neurosurgery departments and neuroscience laboratories by analyzing their modes of fabrication and their »formative« processes (Ribault, 2022). Our objective is to question the current graphic modalities - MRI (Fig.1), tractographies (Fig.2), connectome maps (Fig.3) - by focusing on their visual communication properties: what do these images tell us about the reality they propose to describe? Are they effective in this respect? To which eyes are they addressed? How do they account for areas of uncertainty?

The first step of our research concerns the very specific use of color in tractography images . The color is used here as a signaling system of localization in space. Each fiber of the white matter is colored according to its orientation in space. The three-dimensional system is referenced according to the 3 planes Axial, Coronal and Sagittal (Fig. 4). The parametric use of color generates very beautiful multicolored images that remain difficult to interpret, not only because of the abundant density of intertwined fibers but also because of the color reference used: RGB (Red, Blue, Green).

When interpreting the secondary colors to decompose their primary origin - which is required by the spatial signaling system -, it appears that the RGB reference frame is much less powerful than the CMY (Cyan, Magenta, Yellow) reference frame. We thus move the primary colors on the axis of the chromatic circle to create tractographies which employ the CMY reference frame while reducing the initial color space to desaturate it, which allows the visual highlighting of details which were already present but less visible in RGB (Fig. 5). The prototype is then published as a Github branch of the open-source software MRtrix, widely used by the neuroscience community to produce tractography images.

Brain Roads II: Forms

During the second phase of our research, we question the forms of the 3D models used and their adequacy to reality.

We note that the use of 3D envelopes of organic inspiration struggles to account for the different states of matter that the surgeon encounters during surgery (Fig. 6). The tumor is not a clearly delimited foreign body; it is an alteration of the brain matter itself. This alteration is not uniform but progressive, making the tumor a diffuse entity that is difficult to circumscribe. Current images therefore pose a problem of accuracy of representation and can lead to misinterpretation. How then to represent the uncertainty and the information that is really available?

We try to remove the 3D shapes of organic inspiration to return to a system of representation based on the volumetric pixels of the MRI scan to work a matrix of voxels, which constitute the real information recorded and available (Fig. 7). By leaving the realism of 3D shapes, we enter an aesthetically abstract model, a kind of mental space that expresses with the same intensity the recorded information and the unknown that surrounds it, thus offering a representation that is closer to the real level of information that we have on the observed brain. In other words, this shift in representation mode allows us to move from ›realism‹ to reality.

The challenge for us is to analyze and work on the shape of the voxel in order to perceive the volumetric aspect of the information without necessarily proceeding to cuts that would act as filters of the information. In short, it is an attempt to see through the material. We proceed by representing only the geometrical center of the voxel. Once this principle is established, visual tests are carried out to test different graphic vocabularies: what degree of transparency is possible, what diameters for the voxel centers, what distance between each voxel, what graphic variations to signify significant changes of state?

The next phase consists in integrating the movement in space of the observer's point of view (Fig. 8). This allows us to think about the needs of a navigation interface that we simulate.

Brain Roads III: Dynamics

In the third phase of our research, we step back from the use of digital technologies, returning to drawing and sensitive expression to represent neuroplasticity as perceived and analyzed by French neurosurgeon Prof. Hugues Duffau during his clinical practice (Fig. 9). Together with two groups of art and design students — one from weißensee kunsthochschule berlin, and one from École supérieure d'art et de design de Reims — we work on visual models of representation of his specific dynamic mode of operating.

If the brain is an organ in constant evolution, medical imagery consists mainly in snapshots taken by machines and devices, which struggle to account for the movement and evolution of the functional networks and meta-networks of cognition of the brain. Professor Duffau's patients suffer from low-grade gliomas, tumors that grow diffusely, making them difficult to resect. The surgeon operates on his patients in an awake state to be guided in real-time by their cerebral abilities, using very little imagery in the OR (Fig. 10). It is this rare practical knowledge of the dynamic properties of the brain that we try to understand, to ›de-specialize‹ and to communicate to a broader audience through design.

The project is based on a long-term dialogue and interviews with Prof. Duffau, as well as on the observation of the latter's practice during awake surgeries at the Hôpital Gui de Chauliac in Montpellier. First, we observe, analyze and draw during awake surgeries, then we give graphic forms to our observations and to Duffau’s ideas on how the brain operates. With this different way to perform neurosurgery, we also hope to improve our voxel model.