In recent months, EEMCS scientists Elmar Eisemann and Ruben Wiersma# attracted a lot of attention with their three-dimensional augmented reality experience of a work by 17th century painter Pieter de Hooch. This is just the beginning of how mathematics and computer graphics will enhance our perception of art.
‘I like to tell stories, using film or plain text,’ Ruben Wiersma says, ‘but the medium of computer graphics adds entirely new possibilities.’ The essence of a story is not so much in the beauty or realism of its setting, but in the message it conveys. It would therefore be detracting to characterise their 3D visualisation of the painting Woman with a child in a pantry as a gimmick, as a modern-day twist applied to a 17th century piece of art. ‘Pieter de Hooch paid a lot of attention to linear perspective to add depth to his paintings, just like his contemporary, Johannes Vermeer,’ Wiersma explains. ‘We specifically highlight this aspect of his work. We use computer visualisation to tell the story of De Hooch and his use of perspective.’
The overlap between art and science
The idea for their visualisation is rooted in the organisation, by the Foundation Delft in-Ovatie, of an exhibition of the work by Pieter de Hooch, who resided in Delft from 1652 to 1661. Elmar Eisemann, professor in the Computer Graphics and Visualisation group at TU Delft, was one of the scientists approached by the foundation for a symposium to be held in parallel to the exhibition. He more or less succeeded in combining his love of mathematics with his childhood dream of becoming a Disney illustrator. ‘There are many similarities between the work of the Dutch Masters from the Golden Age and the research that we do in our group,’ Eisemann explains. ‘They were masters of light: the incidence of light, shadows, indirect lighting of surfaces, transparency, the accurate portrayal of complex materials such as clothing or metal objects. Within our research group, we aim to achieve a similar realism using computers, and to achieve this as efficiently as possible. Sometimes we have to develop completely new physics models.’
Varying rules of perspective
De Hooch often painted the same scene twice. ‘At first, our idea was to make a 3D reconstruction of a room using both paintings,’ Eisemann says. ‘That didn’t work out well, as it turned out that De Hooch didn’t always apply linear perspective very strictly. Sometimes he painted somewhat freely or at least applied different rules of perspective. We then decided to specifically focus on his use of perspective for adding depth. It fit well with our ongoing work in computer graphics in which we, for example, showed that you don’t need to have maximum accuracy in depth for a scene to be perceived as being realistic. We had also already developed software to transform a two-dimensional image into an interactive three-dimensional scene, requiring only very little user input. That’s how we came up with the idea of developing an augmented reality experience for the De Hooch exhibition.’
Pinning down the perspective
The characteristic feature of linear perspective is that it uses one or more vanishing points located on a real or imaginary horizon (see figure). Lines that run parallel in the real world, such as the seams between tiles or the edges of a table, appear to recede to the horizon towards the same vanishing point. ‘To strictly apply this perspective, De Hooch marked such a vanishing point with a pin to which he attached a piece of string serving as a construction line,’ Eisemann says. ‘In some paintings you can uncover this tiny hole underneath the layers of paint using x-ray imaging techniques.’ At first, the researchers tried to reconstruct these vanishing points by hand, just like art historians still do. This turned out to be very time consuming as it required extreme precision. They therefore wrote an algorithm that automatically extracts lines and vanishing points from a painting. ‘It turned out that, next to multiple vanishing points, De Hooch also used multiple horizons,’ Eisemann explains. ‘A room appears to be much larger when the central vanishing point of the floor is located below the “real” horizon. It also increases the perceived distance between the characters in the painting and you as an observer. We think De Hooch may have applied this false perspective intentionally.’
A different perspective on art
The researchers also noticed that De Hooch often had his virtual horizons coincide with objects in the painting, such as the seat of a chair, a table top or a windowsill. The algorithm also exposed composition lines presumably used by the painter to arrange the scene and to direct the observer’s gaze – a nobleman’s pipe pointing at the eyes of the servant maid near the opposite edge of the painting, or a lower leg pointing at the pinnacle of a church tower. Quite often, De Hooch marked such composition lines with light and dark accents. ‘All this information came to light using a very simple line detection algorithm, and it already tells us an intricate story of how the artist composed his paintings,’ Eisemann explains. The augmented reality experience is only a simple application of the actual scientific research performed in the Computer Graphics and Visualisation Group. Based on their reaction, the visitors of the exhibition and the symposium weren’t any less impressed. ‘Some visitors told us that they can no longer look at a painting without wanting to find the imaginary horizons and vanishing points,’ Eisemann says. ‘And a photographer who didn’t like the paintings by De Hooch, because he intuitively grasped the perspective to be off, now wondered how to use false perspective in his own work. Imagine that we employ the full power of artificial intelligence and computer visualisation to enrich the experience and understanding of art lovers.’ That is exactly what Wiersma and Eisemann intend to do.
How did the De Hooch visualisation come to be
First, each pixel was assigned a value for its depth, using an existing machine learning tool and the software for manual depth annotation as developed by the researchers themselves. They subsequently transformed this depth map into a three-dimensional model applying the perspective as used by De Hooch. The model had to be simplified as the augmented reality toolkit of the phone would otherwise not be able to provide a smooth experience. Using the phone’s camera, this toolkit projects the three-dimensional model on top of (the image of) the painting, allowing the observer to move around the scene.
Combined with x-ray imaging techniques
Immediately after obtaining his cum laude master’s degree, in October last year, Wiersma started with his doctoral research, again in the group of Eisemann. This research is partially financed by the Delft University Fund and specifically focusses on the use of computer graphics for the analysis of paintings. ‘I am a frequent visitor of museums and this research is a good fit to my general interest in art,’ Wiersma says. ‘We will use modern data analysis techniques, such as machine learning, to increase our understanding of paintings. Think of an even more profound analysis of the use of perspective, of the underlying layers of paint and of the colour palette used. The researchers teamed up with professor Joris Dik, professor of Materials in Art and Archeology at the ME faculty. Over the past decade he has used and improved x-ray techniques to non-destructively dissect paintings, layer by layer. Such an analysis results in massive data, which Wiersma intends to reduce to a more manageable size using mathematical techniques. He will also apply advanced computer visualisation techniques to present his scientific findings to both experts and laypeople. ‘The end goal of my doctoral research is the development of algorithms that can be used for any painting by any painter, and in any museum, such as the Mauritshuis and the Rijksmuseum.’
Imagine that we employ the full power of artificial intelligence and computer visualisation to enrich the experience and understanding of art lovers.
Art, not artificial
The question remains whether the research of Eisemann and Wiersma may somehow diminish the magic of art. ‘Actually, we add magic,’ Wiersma says. ‘The presentation of art nowadays is very factual and scientific. They show you an x-ray image or a sample of the paint that was used. We can present this, and so much more, in an exciting manner, increasing the observer’s connection to the piece of art.’ Eisemann adds that their research is certainly not purely driven by the technical challenge. ‘I find it interesting to apply mathematical precision in an area that, typically, is a bit less exact, a bit more elusive. And we do want our research to be relevant and alluring. Our De Hooch visualisation stimulated the observer to have a deeper experience of his use of perspective.’ Both researchers stress the importance of having a continual discussion with the art community as to the use and added value of their research. Should you still have doubts about their use of computers to analyse paintings then, perhaps, Wiersma’s personal opinion will reassure you: ‘I believe that art is communication between people. It is meaningful only because the person observing it has emotions, memories, experiences, that allow him or her to empathise. This is why art, by definition, is something uniquely human. A computer cannot diminish that.’
# The research into the use of perspective by Pieter de Hooch was a collaboration between Elmar Eisemann (professor), Ricardo Marroquim (assistant professor), Ruben Wiersma (doctoral student) en Yoann Coudert-Osmont (visiting student), all within the Computer Graphics and Visualisation group at TU Delft.
