Being There - immersive documentaries

A little side bar on image, perspective and emotion in this week’s news about the arrest of IMF boss Daniel Strauss Kahn for alleged rape of a 32-year old hotel worker from Guinee (in his 3000$ a night hotel suite in New York). I am surprised by my own gut feelings about the case and try to understand the role of image and immersion in constituting my reaction.

Although I would generally tend to support a woman’s point of view in most alleged rape cases, my heart breaks when I see the video images of this distinguished French man, who has undoubtedly shown dedication, persuasion and professionalism in reinventing the IMF in the past three years (even if you do not agree with the IMF’s policies - I don’t)  All this doesn’t matter, I suppose, if he really did rape her, which is a criminal act deserving punishment. No exceptions should be made for white men in powerful positions.

On the other hand, I feel surprisingly  little sympathy for the hotel worker and I do intuitively mistrust her story. I even wonder if she has been paid to set him up. The accounts of her being ‘very confused and upset’,  ‘not knowing who he is’ and ‘not being able to express herself clearly’ are not eliciting my support and understanding for her, but making things worse. By not showing herself and by not talking, she’s looking even more suspicious in my eyes.

Why is that? Why do I not sympathize with an alleged victim, who is a poor, powerless person and a woman like me? At least one of the reasons is that I have not seen her, but I have seen him. Even though he is a man of a kind that I generally mistrust and dislike, seeing him in this humiliating and possibly unjust position is really painful for me. It makes me wish strongly that I had the means to end his suffering. On the other hand, the woman is  unknown to me. I have not seen her at all, because she has been shielded from the media as an alleged rape victim. This missing image of the hotel worker is undeniably influencing my opinion and emotional judgement here, in favor of the alleged rapist.

Secondly, I do not know what went on in that hotel room. Of course, I have not been there myself, there is not even an amateur video available and I have not seen or heard an independent  and trustworthy eyewitness account of the alleged rape. This is another crucial missing image, for which my brain fills in creatively, thereby favoring the defendant, whom I have seen on TV. It is certainly easier to sympathize with an alleged rapist if you have not witnessed the rape and only seem him as a ‘victim’ rather than an attacker himself.

In Taiwan, they found a simple solution to this problem. TV news needs images to tell stories. And since there was not even a mobile phone account of the events in the hotel room, the special effects department of the TV station just made an animated version of what they think has happened in the room. The professional animation immerses the viewer in the actual moment of the alleged rape, by providing a virtual room and a scenario with two avatars, with whom viewers can identify. Just look at it and judge for yourself what happens to your sympathy and how the images are influencing your opinion.

I think this kind of ‘immersive journalism’ is potentially very influential and also dangerous, especially when dealing with cases still on trial. Of course, the problem is that spatial information and a scenario are provided as ‘facts’ in a newscast, although the story inside the hotel room has been invented based on partial information. As Computer Generated Imaging (CGI) software improves and becomes more widely and cheaply available, we will surely see more ‘immersive reconstructions’ of news stories as they are unfolding. They will also be increasingly life-like and soon include real life photography and video images, rather than animations. We can only start to imagine what this means for our relation to reality.

On my way back from a recent tourist outing with friends to ‘The Peak’ in Hong Kong, I stumbled upon a talented salesman of ‘Exclusive’ video sunglasses. The sunglasses looked liked regular sunglasses to me, but concealed an invisible video camera inside the frame. Of course, this cool gadget opened the door to my wildest fantasies about ‘point of view’ filmmaking, and in true Hong Kong style, I had to buy it right away (for about 200 Euro).

May I record your point of view, please?

Think about it! How many times in the past didn’t I wish that I could have filmed with my own eyes? The thought of having a hard disk inside my brain, a camera built into my forehead, and a USB stick protruding from one of my fingers has always been very appealing to me. Something that would come to the world inevitably, but maybe not in time for me to make good use of it.

Just imagine, if only we could record what we see when we see it, and then edit… Life would be so much easier and documentary films would be so much more interesting. In our films, we would be able to engage with life as it is happening in front of our very eyes, rather than with staged and uncomfortable reenactments of it.

Fly-on-your-face films

Incredibly small cameras like MUVI (discussed earlier on this blog) already offer something quite close to the direct cinema ideal of the camera as a ‘fly-on-the-wall’, but a camera that is visibly attached to your forehead - no matter how small - still takes away some of the natural interaction. And mounting it on someone else’s forehead feels close to a medical procedure.

Video sunglasses promise that all this will be over and I can now just walk up to my subject and kindly ask: ‘Would you mind wearing these sunglasses today? I would like to get your point of view on this matter’. And a few hours later, I would be able to see the events of that day from my subject’s perspective on my screen. If that’s what technology can do for documentary film, it seems a very valuable contribution to me. After all, many problems in the world arise from the fact that we are not capable of see matters from ‘the other’ person’s perspective.

Well, so much for the potential, now the harsh reality. My first trial with the  video sunglasses involved simply leaving my apartment, walking to the park across the street and playing soccer with my son. I will show and discuss the results tomorrow.

Video sunglasses, Trial 1: leaving the apartment from Bregtje van der Haak on Vimeo.

One of the underlying assumptions of this research blog is that media exposure can produce identification and empathy in viewers. But is that true? What is the psychological effect of  immersion in mediated stories? For example, do media images of suffering in far-away places really produce identification and empathy in viewers? And when will they resort to action? These are relevant questions for storytellers who are interested in using new technologies to connect viewers to the social issues and political struggles of our time.  

In a recent talk at City University, the eminent sociologist and scholar of media effects Elihu Katz surveyed his his own 60-year career and recalled being struck intellectually by the introduction of the concept of “empathy” by Daniel Lerner in the 1950s. Lerner proposed that media affect modernization by expanding individual horizons, inviting identification with remote others and the trying on of new identities.  Katz says:”Lerner attributed empathy to the kind of media exposure which produced an ablility to provide a substantive reply to a survey question like: “What would you do if you were editor of the local newspaper?” Or:”What would you do if you were Prime Minister?” Rather than answering, as most people did, “Who, me, editor of a newspaper?”‘ there were other, more media literate respondents who could imagine themselves in these roles. Lerner argued these were the newly modernizing individuals, who had psychological access to the world outside the village; they were the harbingers of a radical transformation.”

These survey questions of the early days of radio (and the very start of television) resonate remarkably with current questions about immersive journalism and the psychological effects of using interaction and first person perspective in new media storytelling. Have these types of studies been reproduced for television and new media ? Scanning the field, 84-year old Katz concludes that empathy and identification are central concepts in Film Studies, but are largely absent form the catalogue of studied media effects in communication reserach. Katz:” Most research effort has gone into the persuasive effects of political communciation and advertising and, ironically, what we have to show is only how surprisingly ineffective the media are in this domain. Even though we hardly believe it ourselves, the truth is that propaganda and advertising don’t persuade very well.”

Katz feels that more communication research on empathy and identification in the current (new) media landscape is needed and points out the emergence of a new field of communication research focused on “distant suffering”, which includes work by Luc Boltanski, Lilie Chouliaraki, Paul Frosh, Amit Pinchevski and others. Katz: ”There is a sudden rush of interest in the emotional, cognitive and, especially, moral aspects of mediated witnessing of widespread tragedy. Under what conditions, these authors ask, do people rise to the challenge of doing something to right a wrong, or save a life, in response to what they see on the nightly news or learn about from internet? When do people get up from their TV sets to demand that their governments intervene? When do people mobilize to donate money to cope with a far-away disaster?”

With ubiquitous online news and information, moral questions resurface with increased urgency.  And, I would add, with increasing immersive media storytelling, reality literally flies in your face.  Katz: “We have run out of ways of saying: ‘I didn’t know’. That’s why the concepts of empathy and compassion have been resurrected in order to explain everyday arousal and action, and to revive the idea that the media have the power to empower.” Maybe the news iscurrently  not brought to us in ways which allow for identification and empathy. I guess it’s  is my job to work on improving that.

Honey, it’s me

Are you feeling lonely? Talk to your virtual girlfriend, recommends Seoul-based developer Nabix. It released an app for the iPhone last month called “Honey, it’s me“, which includes 100 recorded video conversations with two actresses, who will respond to questions and engage in dialogue with the caller. A virtual model called “Mina” can also call users four times a day at her own initiative and say things like “Are you still sleeping?”, “ Time for breakfast!” or “Good night, sweet dreams”. A real Korean model posed for the videos, says Kim Yoon-Kak, head of Nabix, who has kind motives: “I’ve developed this application to console people for their loneliness.”

Cold winter

The “Honey, it’s me” app has been downloaded over 80,000 times per day during an initial free launch period and users seem to be content. “I wish I could meet Mina before I die,” a user wrote. “I recommend this application to lonely single guys who will struggle through the cold winter this year without a girlfriend because Mina will make them forget the fact that they are single.” 

The South China Morning Post reports that in Hong Kong,  many youth are feeling lonely in December and are more open to start a relationship in this time of the year. They fear being alone at Christmas and would rather be with a ’second choice’ boyfriend or girlfriend than single when going through celebrations with their extensive families. So much for the reputedly strong Chinese sense of family bonding.

Love Plus

There are a number of games and apps on the market to make single men feel less lonely, including a night time app that plays the sound of a female sleeping peacefully on the next pillow. Boys rather than  girls seem to be the prime market  for these types of toys. “Love Plus”, designed by Konami Digital Entertainment,  is a simulation game for the Nintendo DS console, which presents cartoon girlfriend Rinko as a playmate to lonely boys. The game  is only released in Japan and has sold 430,000 copies last year, making it the most widely used ’dating game’ according to research firm Enterbrain.

   

Interestingly, Love Plus is spilling over from the virtual into the real world, since Konami is also offering schoolgirl Rinko on site in the beach town and honey moon destination Atami, Japan through Augmented Reality (AR)  features. Coming from the bus in Atami, young men with iPhones may head straight for the statue of Kanichi and Omiya, depicting an old love story set in Atami. By pointing their smart phones at a barcode on the statue, Rinko materializes - as an image at least - and kindly allows her enamored vistors to have their picture taken with her.  In total, there are thirteen ‘romantic locations’ in Tatami, which can be overlaid with AR pictures of Rinko and her friends Nene and Manaka. The Ohnoya hotel even offers traditional rooms to the unusual couples, with two futon beds and a barcode allowing men to visualize Rinko sitting on her bed in an elegant summer kimono (rather than her usual school uniform).

Building a relationship

“In conventional love games, you went up levels to make a virtual girl fall in love with you, so she would accept you as a boyfriend,” Konami spokesman Kunio Ishihara says. “But players of Love Plus are in a scenario where they’re a highschool boy who’s already dating one of the virtual girls. The goal is to see if you can build a relationship.”  Technically, Love Plus is an open ended ‘communication game’ with voice recognition software and a screen clock that keeps real time to make players feel like they are really sharing their life with the girlfriend. That means the girl can also get moody when neglected by a player and demands attention when she feels unwell. (source: AFP) In the latest version of the game, facial recognition software is introduced to prevent the virtual girlfriends from cheating on their boyfriends.

A lonely generation 

In a recent talk at City University, Kai-Fu Lee,  CEO of tech investment company Innovation Works, stressed the commercial success of new applications designed to meet the needs of ‘the lonely generation’ in China. Not only is there a generation of lonely children born under the one-child-policy, the boys born since the 1980s are also facing increasing difficulties in finding a girlfriend because of  the scarcity of women. With only one child allowed, many parents have found ways to have a boy rather than a girl, which has resulted in a surplus of lonely males of marriageable age. In addition, 50 - 60  hour work weeks are the rule rather than the exception for university students and starting professionals, which is not very conducive to getting to know a life partner and building a stable relationship. Kai-Fu Lee, who quit as president of Google China to start Innovation Works, which invests in Chinese technology start-ups, predicts that virtual solutions to loneliness will be increasingly accompanied by technologically assisted real world dating opportunities, leading to mixed reality living and dating. 

My colleague Olli Tapio Leino wrote his PhD on  ’Emotions in Play’ at the Center for Computer Games Research at the University of Copenhagen and was kind enough to share some of his insights with me. I am interested in exploring non-fiction storytelling in a gaming environment, since gamers seem to be very immersed and I am interested to find out if we could reproduce that intense experience with documentary content.

Olli explains that the Copenhagen School in game research - to which he belongs - believes that gaming has its own logic, which is focused on ‘play’ and therefore incompatible with ’story’. The idea that gaming represents some kind of new ’storytelling’ belongs to Film Studies and is discarded by the Copenhagen-ers, who believe that actively trying to reach a goal - essential to the gaming logic - is mutually exclusive with experiencing ’story’, which implies ‘being carried away’ and giving up agency to the storyteller.

Escape from Woomera

This being said, Dr. Leino comes up with a number of interesting examples of ‘news games’ which relate to the real world and provide user interaction inside a game world. Whether these games provide ’situations’, ‘worlds’ or ’stories’ is not so important to me. I’m willing to be very pragmatic about it. Just show me something that works! Unfortunately, Olli thinks that empathy is not an emotion that can be easily experienced during game play. Typically, game players experience frustration (and fulfilment) related to their performance in the game and even if they do feel empathy for certain characters, it is unlikely it will carry over into the real world. For example, in the news game Escape from Woomera, illegal immigrants try to escape from an existing prison in Australia. One can intensely enjoy this game and identify with the escapists, without feeling any empathy for illegal immigrants in the real world afterwards.

Olli and I agree that educational news games, which tend to put the player in the ‘objective’ position of  journalist or investigator, are somewhat lame. First of all, those so-called ‘observer’ positions aren’t neutral or objective in real life at all and, secondly, the excitement of gaming comes not with being an observer, but with being an actor, whose acts have real consequences  - inside the game world at least. ”The player of a game is locked up in the gamer logic and can not relate to the story in any other way than wanting to win it”, says Olli, “so the only way to critically confront a game player with a documentary story inside a gaming world would be to break that logic”.

September 12th

That’s exactly what happens in September 12, a news game about the War on terror developed by Montevideo based computer researcher Gonzalo Frasca and the collective Newsgaming.com. When trying to kill terrorists in a crowded town represented in typical game aesthetic, the player will only kill civilians and continue to create more terrorists by doing so. In this game, the players’ acts have real but intended consequences, just as in real life.  The game logic is defied, as there is no way to ‘win’ this game, and the authors prefer to call it a ’simulation’ for that reason.

Madrid

In Madrid, a game by the same makers that was released two days after the terrorist attacks in March 2004, players have to push the buttons in an attempt to keep candles burning and pay homage to the victims of terrorist attacks all over the world. Again, winning is impossible.

Both games are powerful, addictive and painful to play. I would not say they are immersive, however, since I am not so much drawn into a world, but rather confronted with a logic. I guess that confrontation has to be preceded by me entering the game world somehow, but definitely not in a way which makes me forget the current one. In an interview, Frasca compares these ‘critical games’ with cartoons: “We see the concept of newsgaming as a 21st century equivalent to traditional printed political cartoons: short, controversial satirical pieces that convey biased ideological messages. Video games are a perfect medium for this since they can model complex situations”. Frasca, a former journalist with CNN Espanol, used to write the game research blog ludoloy.org (which closed shop last May) and currently runs games studio Powerful Robot . He’s also a researcher at the Copenhagen Computer Games Research Center. So yes, indeed, critical gaming exists. How immersive it is - or can be - deserves further experimentation.

“Capricious and profound, the experience definitely captures what it is to escape from the world a little bit, then return and find that you see things just a bit differently.” This is a participant of a so called SUBTLEMOB speaking and it sounds like a clear case of ‘being there’ to me. It describes exactly the kind of immersive experience I am looking for: the one that makes you look differently at the real world after you come back from it. Here’s a clip of some of the other survivors, talking about their subtlemob experience:

What is a ’subtlemob’? I would say it is a clever, poetic and artistic adaptation of the flashmob, which takes inspiration from location theater, performance art,  and media experiments, like those by Willem de Ridder and Moniek Toebosch. It has a contemporary ’social media’ touch to it though, since the real meeting is now convocated through email, Twitter and Facebook.

This weekend, we were lucky to have one - as part of the Microwave New Media Arts Festival - in ‘a secret location’  in Tsuen Muen, one of the new towns in the northern part of Hong Kong. Participation required signing up on the website two days before the event and bringing an MP3 player, an iPod, an iPhone or some other device that plays podcasts - to a location revealed just hours before the meeting. Once there, participants were invited to download the podcast, start listening and be taken to another world inside the current one.  It reminded me of Soundtrack City, a wonderful artistic project in Amsterdam last year, offering nine soundtracks turning walks through different parts of the city into movies in which the listener is playing a part (only in Dutch, unfortunately). Potentially, this could really be a revealing, consciousness raising and mind-changing experience.

This is what subtlemob creator Duncan Speakman says about ’subtlemob’:

We’re not sure if we know yet (what it is), but this is what we think it might be . . .

Imagine walking through a film, but it’s happening on the streets you live in

Subtlemobs usually happen in public spaces

This is music composed for those spaces

This is about trying to make films without cameras

It’s about integrating with a social or physical space, not taking it over

The audience listen on headphones, a mix of music, story and instructions

Sometimes they just watch, sometimes they perform scenes for each other

A subtlemob is not a flashmob

try to remain invisible . . .

Sign up here if you want to be informed of a subtlemob coming near you. Or follow on Facebook.

For my new documentary about the global rise of the Pentecostal religion - which teaches that ‘body and mind are one’ - I am exploring storytelling options to bring viewers closer to the intense encounters with the Holy Spirit experienced by Pentecosta believers. I am considering using 3D or 360 degree photography to shoot in churches and I recently had the privilege to join a shoot with the 360 ‘Ladybug’ camera by City University students enrolled in the ‘Immersive Museums‘ course by Sarah Kenderdine.

Ladybug 3

The ‘Ladybug’ camera is a rather slick looking device with five cameras pointing sideways and a sixth one on top, recording 360 degree video footage, or rather a collection  of six separate video images that can be stitched together in the computer during post-production to form a 360 degree moving image. We are familiar with this kind of viewpoints of the world thanks to Google street view, travel agents and real estate brokers, who use this type of video on their websites.

Google street view

Google has been a motor behind the development of this type of technology by constantly upgrading its cameras, which are mounted on the rooftop of cars to ultimately record every single street in the world. It is now using the fourth generation street view camera and also recently introduced a snowmobile (2010) and a tricycle (2010) to film roads where cars can not travel.

Ladybug storytelling

The ‘Ladybug‘ camera is slightly different. It has firewire and a higher resolution, allowing for projection on dome- and semi-dome shaped screens, and six rather than nine cameras to record a surround view. In my imagination, watching a Pentecosta believer meet the devil in person in a dome shaped movie theater, would be an awesome, important and frightful documentary experience, but the field trip with the students to Yau Ma Tei is a rather chilling experience for a documentary director like me. The students have planned to shoot just one shot of the Temple Street market, which requires about one hour of preparations, leading to the ‘director’ sitting in a wheelchair with a laptop on his knees and a sound engineer recording ambient sound on a separate recorder. The fixed focus point for the Ladybug is at 2,5 meter from the camera and no adjustment is possible. Also, the camera has to be at a certain height (about 1,5 meter) to allow the director - in the wheelchair - to be present at his own shoot, but not visible in the picture. And the wheel chair has to be motorized to make a traveling shot of the market, as anyone pushing would be in the picture frame… To a ‘content driven’ documentary storyteller like me, this all seems like a lot of hassle for very little ’story’.

I can still imagine that a visceral Pentecosta church service would be interesting to catch with the Ladybug camera. But the problem with all 360 degree video productions, is that the story has to take place in one - very exciting - location. If not, having the 360o perspective is pointless. Why would you want to look behind you, if nothing is happening?  In documentary film, I can think of Raymond Depardon’s Délits Flagrants, set in the examination room of a police station, as inspiration for this type of shooting. Coincidentally, Depardon also made a film in Africa, structured by various 360 degree shots recorded with a regular camera on various locations in Africa. Or the documentary box office hit Etre et Avoir by Nicholas Philibert, set largely in one class room in rural France. Or the documentary films of  Fred Wiseman, which deal with institutions and display unusual unity of place (a hospital, a high school, a welfare counter). Wiseman often shot over hundred hours of film though (spending up to a year in the editing room) and at this point in time,  that would be technically quite challenging with a Ladybug, as one minute of 3600 footage takes more than an hour to render… Unfortunately, I must admit that after the field trip, it is easier to think of the limitations of the Ladybug (one location, lack of movement, fixed focus, no sync sound) than to think of exciting new opportunities. To be continued.

By the way, Yellowbird in The Netherlands also offers interesting 360 degree video recording packages, but they seem to focus on online productions and do not sell the camera. The result can be seen in Highrise: Out my Window (2010) directed by Katerina Cizek and presented at IDA Doclab.

This is what CNN did in terms of ‘immersive journalism’, using the Google street view camera to take viewers into ravaged Haiti. The uncut footage provides a 360 look on the human disaster, filmed from the roof of a driving car. For me, browsing these interactive videos definitely provokes a new and strong sense of ‘being there’ - but painfully so. Being confronted with the surround view, I dearly miss ‘perspective’. Who is looking through the camera? Who is looking at whom? What for? Why am I here? Gazing around from the top of a car makes me feel like a voyeur or a tourist, rather than a connected and empathic citizen. Here, the interactivity and 360 do not produce a sense of connection, but rather a confrontation with a reality that I am clearly not a part of, because I can manipulate it at will. Unlike the persons shown in the video. A lot of pixels and little storytelling.

This is just a Youtube rendering, but look on the CNN website for the real deal, with full control over your point of view. It’s a rather alienating way to relate to the world, if you ask me. Ffrom the CNN website: “Haiti: 360. Use your mouse to click and drag around the video to change the view. You can also zoom in and out. Pause and explore at any time by pressing the play/pause button under the video to stop and look around. The video was shot on Monday, January 18, at 9:52 a.m. EST in Port-au-Prince, Haiti.” I wish the Haitians in the picture were able to ‘pause and explore’ their world ‘ at any time.

My most intense encounter with immersive documentary so far involved a face-to-face meeting with cosmonaut Yuri Gagarin, the Russian hero and man of my dreams, who happened to be the first human being to travel in space and orbit the earth, five years before I was born. The meeting with the space traveller took place in Macau, one hour by ferry from Hong Kong in the impressive and somewhat eery Macau Science Center (designed by Japanese architect I.M.Pei, of the Louvre pyramid). The building is located on a huge lot of empty and probably very expensive land adjacent to the Macau-Hong Kong-China ferry terminal, facing the sea on one side and the giant casinos of Asia’s gambling metropolis Macau on the other.
In December 2009, it was opened officially in the presence of no one less than Hu Jintao, president of the People’s Republic of China. In his speech, Hu said that “the Center showcases a number of exhibits featuring current technologies, as well as exhibits and models, where young people can learn about the remarkable progress and achievements in aerospace technology development made by our home country. The Center shall become the hub of science education in Macao, and an important platform for civil education on patriotism.”

In addition to a patriotic celebration of Chinese achievements in aerospace technology and robotics, the building features a Planetarium with one of the most advanced 3D/360o movie theaters in the world, being the first one with a 8000 x 8000 pixel resolution. It has a tilted semi-dome screen of 15 meters in diameter, supported by high resolution 3D digital projectors, 135 seats equipped with interactive controls for a choice of language (Cantonese, Mandarin, English and Portugese are offered) and so-called ‘4D’, meaning that the chairs rock and tilt in line with the story on the screen to increase the immersive experience for viewers.

Although I entered the dome sceptically, I was totally won over by the sense of Yuri’s presence and the truly immersive experience of the ‘Dawn of he Space Age’ film. For the first time, I felt that I was part of a documentary story that was not just a demonstration of visual effects and entertainment, but a technologically supported invitation to re-live a very important episode of our history: the race between Russia and the US to conquer space. It still takes special glasses to see 3D, but the powerful database coupled with projection effects takes viewers on something like a journey traveling freely into space.
The fact that this race was actually run by men, rather than machines, made it compelling and emotionally engaging for me, and the 3D-storytellers had managed to keep just enough of the human aspects of the story to connect me to the space race as a major political event. Although framed here in the chauvinistic and official discourse of China’s conquests beyond the earth, this is definitely the kind of new storytelling that I am looking for. As I watched the credits roll, the film turned out to be made in The Netherlands. Now, I will continue looking for the first 3D Indie films….