Immersive tour into Rembrandt’s painting from 1632

In the interactive diorama, the seven doctors present at the original sitting in the The Anatomy Lesson of Dr. Nicolaes Tulp painting have been re-created in 3D avatar placeholders displaying gestures, motion and speech. The setting of the anatomy lesson, that reputedly took place at the meeting space of the Barber’s Surgeon Guild of Amsterdam, has been rendered through the study of the 18th century images and using photogrammetry. The diorama deconstructs this important moment when the history of science and art converged in spectacle. The work created in the Media Lab, the Department of Media in Aalto University, was presented in Ars Electronica Festival 7-11 September in Linz, Austria.

A peek into the future

Technology used is at the forefront of mainstream. The HTC Vive hardware and accompanying STEAM VR software used in the project was released two years ago. This means that there is not much content done with it yet, anywhere.

"There are not too many pieces in the world like ours at the moment. In the whole of Ars Electronica 2017, we counted three artworks that used HTC Vive technology and virtual reality, including our own. To our knowledge, ours was the only one that simultaneously allowed for full immersion while allowing full user interaction", says professor in new media at Aalto University Lily Díaz-Kommonen, the initiator, producer and the author of the project.

There are some existing works that try to show the “inside of a painting”. Many of the works done use a diversity of techniques including pseudo-VR 360 degree stereoscopic panoramas. These are not considered to be ‘true’ VR because it involves the projection of images onto a screen, like in traditional film. These works include world-known classics, such as Archeological Reminiscence of Millet’s Angelus, by Salvador Dalí; Garden of Earthly Delights, by Hieronymus Bosch--using mostly 2D animation, to move within the space circumscribed by the canvas--and Guernica, by Pablo Picasso--a version that uses 3D modelling to separate the figures and the background from foreground, allowing you to move within the space of the canvas.

Project brings together various experts

The concept and research for the interactive diorama began in 2011 while Díaz-Kommonen was on a brief sabbatical at Carnegie Mellon University’s Entertainment Technology Centre (ETC) in the United States. Díaz-Kommonen’s aim is to use virtual networked environments and combined use of anthropology and new media to promote human-centered design. Her interest is in European research and development projects in the area of digital technology and cultural heritage. She says her inspiration came from a visit to Venice and upon reading a recently published book by Cynthia Klestinec: Theaters of Anatomy Students, Teachers, and Traditions of Dissection in Renaissance Venice. Production started in the summer of 2013.

In the beginning, the project team was formed by Díaz-Kommonen and animator, game designer Juha Koppström. Professor Sofia Pantouvaki and Johanna Ilmarinen from the Department of Film and Scenography at Aalto University helped with dress design by creating samples which the working group could study closely. The installation features a true 3D computer-generated environment in which objects have been modelled and rendered to include all the geometry. There are the 3D avatars of the doctors that display life-like gestures created using 3D animation and motion capture techniques. This work was realised in LUME studio with Max Mäkinen, Toni Tolin and Shareef Askar from the Department of Media. In the installation the objects have been programmed to display characteristics of the real world. Ling Chen from the Department of Media has programmed the interface and interaction design and can tell for example about the physical behaviour of the book and the scissors and other details in the installation.

Regarding the space depicted, there is the installation itself in which the user enters into a 3 X 3 meters interaction space framed by transparent printouts from Andrea Vesalius 1543 book, De humani corporis fabrica. These prints were created for the installation by Angela Hernández from the Department of Design. The design of the installation was done by Díaz-Kommonen, with suggestions from Karola Sahi from the Department of Design and realised by Mercedes Said from the Department of Media. Markku Reunanen from the Department of Media provided VR and interaction design consultancy for the project.

A thorough reconstruction as a base

The virtual space rendered inside the VR application is a reconstruction of the anatomic theatre that used to exist at Waag Society in Amsterdam (since 1691). The replica was created through a combination of photogrammetry techniques by Dr. Judith van der Elst, an independent scholar residing in Amsterdam, using 18th century prints. Waag Society was the place where the painting was originally created, having served as the site of the Amsterdam Guild of Barbers’ Surgeons. Díaz-Kommonen got architectural materials with accurate measurements and depictions of the actual site.

"Still, I could not ascertain exactly in which room at Waag it had been created. However, I decided to go for the reconstruction of the anatomical theatre because I thought this would be of interest to both historians of art and science, as well as the public", she says.  

Professor Díaz-Kommonen together with the Systems of Representation research group have done other virtual reality simulations in which the work has called for as much historical accuracy as possible. That is not the case with this work, since here the aim was for a more artistic result. Also, Díaz-Kommonen notes, the piece is not a game but an art installation. It is not about clicking here and there or completing tasks, but rather taking your time and reflecting. The typical length of a visit is between five to ten minutes. This means that everyday around seventy-five people at the Ars Electronica Festival got to interact with the piece directly and many more could vicariously experience the work by watching the interactant’s engagement on a projection screen.

Interactive Diorama is a work done with virtual reality but it is also an investigation into new forms of computer mediated communications tools. “We are interested in exploring new media formats emerging from the synthesis of high-end computer graphics and telecommunications. What is their relation to more established audio-visual genres such as film? In which ways are they similar? And where do they differ? These are some of the questions that we can now focus on”, Díaz-Kommonen explains.

The team has been invited to showcase the interactive diorama at Waag Society in Amsterdam. There have been other invitations for possible exhibitions well. Among the items that the team would like to realise is the possibility for having multiple visitors to access the space simultaneously and for them to be able to literally “wear” the avatar’s skins while interacting in the virtual space.