Being There - immersive documentaries

Honey, it’s me

Are you feeling lonely? Talk to your virtual girlfriend, recommends Seoul-based developer Nabix. It released an app for the iPhone last month called “Honey, it’s me“, which includes 100 recorded video conversations with two actresses, who will respond to questions and engage in dialogue with the caller. A virtual model called “Mina” can also call users four times a day at her own initiative and say things like “Are you still sleeping?”, “ Time for breakfast!” or “Good night, sweet dreams”. A real Korean model posed for the videos, says Kim Yoon-Kak, head of Nabix, who has kind motives: “I’ve developed this application to console people for their loneliness.”

Cold winter

The “Honey, it’s me” app has been downloaded over 80,000 times per day during an initial free launch period and users seem to be content. “I wish I could meet Mina before I die,” a user wrote. “I recommend this application to lonely single guys who will struggle through the cold winter this year without a girlfriend because Mina will make them forget the fact that they are single.” 

The South China Morning Post reports that in Hong Kong,  many youth are feeling lonely in December and are more open to start a relationship in this time of the year. They fear being alone at Christmas and would rather be with a ’second choice’ boyfriend or girlfriend than single when going through celebrations with their extensive families. So much for the reputedly strong Chinese sense of family bonding.

Love Plus

There are a number of games and apps on the market to make single men feel less lonely, including a night time app that plays the sound of a female sleeping peacefully on the next pillow. Boys rather than  girls seem to be the prime market  for these types of toys. “Love Plus”, designed by Konami Digital Entertainment,  is a simulation game for the Nintendo DS console, which presents cartoon girlfriend Rinko as a playmate to lonely boys. The game  is only released in Japan and has sold 430,000 copies last year, making it the most widely used ’dating game’ according to research firm Enterbrain.

   

Interestingly, Love Plus is spilling over from the virtual into the real world, since Konami is also offering schoolgirl Rinko on site in the beach town and honey moon destination Atami, Japan through Augmented Reality (AR)  features. Coming from the bus in Atami, young men with iPhones may head straight for the statue of Kanichi and Omiya, depicting an old love story set in Atami. By pointing their smart phones at a barcode on the statue, Rinko materializes - as an image at least - and kindly allows her enamored vistors to have their picture taken with her.  In total, there are thirteen ‘romantic locations’ in Tatami, which can be overlaid with AR pictures of Rinko and her friends Nene and Manaka. The Ohnoya hotel even offers traditional rooms to the unusual couples, with two futon beds and a barcode allowing men to visualize Rinko sitting on her bed in an elegant summer kimono (rather than her usual school uniform).

Building a relationship

“In conventional love games, you went up levels to make a virtual girl fall in love with you, so she would accept you as a boyfriend,” Konami spokesman Kunio Ishihara says. “But players of Love Plus are in a scenario where they’re a highschool boy who’s already dating one of the virtual girls. The goal is to see if you can build a relationship.”  Technically, Love Plus is an open ended ‘communication game’ with voice recognition software and a screen clock that keeps real time to make players feel like they are really sharing their life with the girlfriend. That means the girl can also get moody when neglected by a player and demands attention when she feels unwell. (source: AFP) In the latest version of the game, facial recognition software is introduced to prevent the virtual girlfriends from cheating on their boyfriends.

A lonely generation 

In a recent talk at City University, Kai-Fu Lee,  CEO of tech investment company Innovation Works, stressed the commercial success of new applications designed to meet the needs of ‘the lonely generation’ in China. Not only is there a generation of lonely children born under the one-child-policy, the boys born since the 1980s are also facing increasing difficulties in finding a girlfriend because of  the scarcity of women. With only one child allowed, many parents have found ways to have a boy rather than a girl, which has resulted in a surplus of lonely males of marriageable age. In addition, 50 - 60  hour work weeks are the rule rather than the exception for university students and starting professionals, which is not very conducive to getting to know a life partner and building a stable relationship. Kai-Fu Lee, who quit as president of Google China to start Innovation Works, which invests in Chinese technology start-ups, predicts that virtual solutions to loneliness will be increasingly accompanied by technologically assisted real world dating opportunities, leading to mixed reality living and dating. 

My colleague Olli Tapio Leino wrote his PhD on  ’Emotions in Play’ at the Center for Computer Games Research at the University of Copenhagen and was kind enough to share some of his insights with me. I am interested in exploring non-fiction storytelling in a gaming environment, since gamers seem to be very immersed and I am interested to find out if we could reproduce that intense experience with documentary content.

Olli explains that the Copenhagen School in game research - to which he belongs - believes that gaming has its own logic, which is focused on ‘play’ and therefore incompatible with ’story’. The idea that gaming represents some kind of new ’storytelling’ belongs to Film Studies and is discarded by the Copenhagen-ers, who believe that actively trying to reach a goal - essential to the gaming logic - is mutually exclusive with experiencing ’story’, which implies ‘being carried away’ and giving up agency to the storyteller.

Escape from Woomera

This being said, Dr. Leino comes up with a number of interesting examples of ‘news games’ which relate to the real world and provide user interaction inside a game world. Whether these games provide ’situations’, ‘worlds’ or ’stories’ is not so important to me. I’m willing to be very pragmatic about it. Just show me something that works! Unfortunately, Olli thinks that empathy is not an emotion that can be easily experienced during game play. Typically, game players experience frustration (and fulfilment) related to their performance in the game and even if they do feel empathy for certain characters, it is unlikely it will carry over into the real world. For example, in the news game Escape from Woomera, illegal immigrants try to escape from an existing prison in Australia. One can intensely enjoy this game and identify with the escapists, without feeling any empathy for illegal immigrants in the real world afterwards.

Olli and I agree that educational news games, which tend to put the player in the ‘objective’ position of  journalist or investigator, are somewhat lame. First of all, those so-called ‘observer’ positions aren’t neutral or objective in real life at all and, secondly, the excitement of gaming comes not with being an observer, but with being an actor, whose acts have real consequences  - inside the game world at least. ”The player of a game is locked up in the gamer logic and can not relate to the story in any other way than wanting to win it”, says Olli, “so the only way to critically confront a game player with a documentary story inside a gaming world would be to break that logic”.

September 12th

That’s exactly what happens in September 12, a news game about the War on terror developed by Montevideo based computer researcher Gonzalo Frasca and the collective Newsgaming.com. When trying to kill terrorists in a crowded town represented in typical game aesthetic, the player will only kill civilians and continue to create more terrorists by doing so. In this game, the players’ acts have real but intended consequences, just as in real life.  The game logic is defied, as there is no way to ‘win’ this game, and the authors prefer to call it a ’simulation’ for that reason.

Madrid

In Madrid, a game by the same makers that was released two days after the terrorist attacks in March 2004, players have to push the buttons in an attempt to keep candles burning and pay homage to the victims of terrorist attacks all over the world. Again, winning is impossible.

Both games are powerful, addictive and painful to play. I would not say they are immersive, however, since I am not so much drawn into a world, but rather confronted with a logic. I guess that confrontation has to be preceded by me entering the game world somehow, but definitely not in a way which makes me forget the current one. In an interview, Frasca compares these ‘critical games’ with cartoons: “We see the concept of newsgaming as a 21st century equivalent to traditional printed political cartoons: short, controversial satirical pieces that convey biased ideological messages. Video games are a perfect medium for this since they can model complex situations”. Frasca, a former journalist with CNN Espanol, used to write the game research blog ludoloy.org (which closed shop last May) and currently runs games studio Powerful Robot . He’s also a researcher at the Copenhagen Computer Games Research Center. So yes, indeed, critical gaming exists. How immersive it is - or can be - deserves further experimentation.

Foxconn is the world’s largest electronics manufacturer, producing iPhones, iPods, Nokias, etc. It employs more than 900,000 workers in China. In the first five months of 2010, the world was shocked to see a continuous series of suicides happening in Foxconn. What actually went wrong? This video is part of the report “Workers as Machines: Military management in Foxconn” released by SACOM.

The most interesting new documentary filmmaker I met in Hong Kong is  Jack Linchuan Qiu. He is, in fact, an outstanding social scientist, author of  Working Class Network Society,  professor at The Chinese University in Hong Kong and, until very recently, not a filmmaker at all. Last spring, he decided to pick up a camera incited by the wave of worker suicides at tech-giant Foxconn, which has close to one million employees in China and produces most of our iPads and iPhones. It also produces something like 40,000 broken fingers annually, as Qiu and other researchers had found out earlier from field research in hospitals in the Pearl River Delta.

Network

Frustrated by the lack of access and information provided by Foxconn, Qiu and  colleagues of universities in Hong Kong, Taiwan and Sjanghai joined forces with student network SACOM and decided to resort to new collaborative methods. They used the internet, mobile and video technology to organize a massive, networked data collection effort, in order to better understand the working conditions in the Foxconn factories connected to the suicides.

Pun Ngai - an acclaimed ethnographer who also wrote Made in China on the basis of her experiences as a factory worker - was among the initiators of the project, which involved students successfully applying for summer jobs at Foxconn factories and taking photos and video on the job with cell phone cameras. Qiu had been using video, photography and audio recordings in his fieldwork for years, but resorted to i-movie editing and documentary filmmaking for the first time with the Foxconn project. He started to edit the visual material gathered by the researchers (photos, video and audio) and made a poetic, angry yet very factual twenty minute film, which might evolve into a feature length documentary called ‘Deconstructing Foxconn‘.

Taipei

Even more moving is his black and white short film about a visit to Taipei on a rainy day, where he reads a poem for the deceased workers in front of the Foxconn headquarters until he’s interrupted by a security guard. The way the camera keeps recording while it’s being handed to the taxi driver reminds me of Jean Rouch‘ best films. On the way back to the airport, Qiu gets involved in a lengthy discussion with the taxi driver, which he keeps entirely in the film.

Progress

The work is still in progress, but what I have seen so far is very touching and immersive in the sense that it takes the viewer deep inside Foxconn and provides a view from within through video, photography and audio recordings collected over a long period of time. In this case, the ‘author’ of the film is a group of researchers armed with mobile phones and dressed in Foxconn uniforms, who are able to mix with the factory workers and report unusually rich data on feelings and atmospheres, in addition to quantitative data about wages and working hours.  Without photo, video and audio functions on mobile phones, this would have been simply impossible. 

Twenty five years ago, journalist and writer Gunter Wallraff did something similar when he went to work as a migrant worker in Germany and wrote a book about it (Lowest of the Low, 1985). Barbara Ehrenreich reported on living the hard life in the US from her own experience in Nickel and Dimed: on (not) getting by in America (2001) and Ngai did it in China more recently, armed with a theoretical perspective. But they did not have cameras.

I do feel that the visual dimension is adding a rich and essential layer to this type of work, which explores new realities immersively, from the inside outward. In Qiu’s film, we meet a seventeen year old girl who jumped and survived but will never be able to walk again. She does not need to say much. Just by seeing her in the hospital bed, we start to understand how she feels ad what she did. I think this kind of rich ‘visual information’ is among the real gains of the pervasive visual culture that is emerging. Many intellectuals, journalists and academics lament the decline of reading and the diminished impact of words. But is that really necessary? When we just stop for a minute, look and think about the depth of information provided in those video frames, I feel grateful for the possibility to watch them. And I wish I could help her get up and go.

Is critical immersion possible? Media archeologist and art historian Erkki Huhtamo of UCLA fame thinks that the suspension of disbelief required for deep and complete immersion works well for fantasy stories such as Avatar or Alice in Wonderland, but not necessarily for creating engagement with pressing real world issues. On a visit to Hong Kong, Huhtamo says:” I’m not sure if we need technical immersion to engage with social  and political issues. If the content is compelling, seeing a frame is not a problem.’ In order to activate the viewer, we need to attract the viewer’s attention to reality, he says, but we also need a distancing act to induce reflection on that reality.  This is a viewpoint shared by many media artists and critical filmmakers of my generation (and older): alienation - not immersion - leads to awareness, reflection and, ultimately, social change.

As an example of a succesful immersive media work with political content, Huhtamo mentions ‘World Skin’  (1997) by French artists Maurice Benayoun and Jean-Baptiste Barriere, who used 3D CAVE technology (three wall video projection), to present a kind of  battle field animation based on 2D newspaper cut outs of war scenes. Standing in the middle of the CAVE, the viewer is invited to act as a photojournalist, taking pictures of the cruelties surrounding him. Huhtamo particularly likes the fact that the immersive effects produced by CAVE are counterbalanced by the flatness and ‘unreality’ of the war photographs in the animation, which he assumes will stimulate the viewer-turned-war photographer to reflect on the madness of war.

 

Unfortunately, I have not seen World Skin myself, but it seems to be a very succesful art work. The impact of the installation, however, is limited to museum visitors who have already decided to be thoughtful, attentive and probably critical. I am interested in finding out how immersion in everyday media can be compelling (young) viewers/users to think about reality in unexpected ways and put their minds in motion. 

Huhtamo points out that the artificiality - which is always part of the act of total, technical immersion - may be counterproductive for relating to real life horrors. For example, would it really be more moving to see refugees struggling on rafts to reach our shores in stereoscopic 3D with special glasses than to see them on TV in the grainy images shot on a mobile phone ? Probably not. As with the CNN footage of the Haiti disaster, it could be turning a disturbing reality into a kind of reassuring fantasy,  farther removed from us mentally although we may feel temporarily immersed in it physically.

Still, the way media are fusing with everyday life and practically all social relations, I wonder if we can rest assured on the effectiveness of the alienation-thesis. After all, the only time my students really woke up this semester is when I presented them examples of interactive documentaries, gaming and non-fiction stories in Second Life. Huhtamo’s students in LA are also constantly connected and can not live without checking with their friends on their mobiles every fifteen minutes. With documentary (and news) increasingly perceived as ‘boring’ and newspapers on the way out for most people aged under twenty five, I really doubt that there will be many opportunities for succesful and productive ‘alienation’. Getting attention, by using methods that are somehow, in some way  ’immersive’ and pull you viewers into a non-fiction story seems to me an urgent and challenging task for journalist and storytellers.  What we need though is good, independent, immersive examples.

I’m not at all convinced that immersion itself excludes raising the kind of social and political awareness, which leads to action. After all, my own most radical immersive experience this fall has been reading - or rather ‘deep reading’ as Huhtamo calls it appealingly - ‘La Carte et le Territoire’, the new novel by Michel Houellebecq. It sucked me in, I could not put it down, it put my mind into motion and it probably is influencing my acts right now. So if immersion in a novel can change my mind, why coudn’t full immersion in a non-fiction film do the same?

My students want to do a short documentary about biking in Hong Kong and I recommended them to get this little camera (100 Euro) to tie to the head of the biker and get his point of view as an additional camera angle:

The Muvi is a pretty cool, very small and incredibly cheap camera with reasonable image quality. It represents a new step in the continuous development of ever smaller and lighter cameras, allowing for a growing range of viewpoints and camera angles in documentary films. Just as direct cinema came with the liberation of the camera from the tripod, one can easily imagine new cinematic styles with the availability of these cameras the size of a thumb. In the current global media society, however, the impact would be far beyond aesthetics.

For example, it’s now literally child’s play to record the world from a child’s perspective - or that of any moving object or subject - adding more points of view to the current scale. Imagine the possible impact of all those emerging voices (and eyes) taking the stage. Will children get voting rights because their points of view can no longer be ignored? Will the current fringe Party for Animal Rights rise to power as a result of the increased representation of pets’ points of view in the flow of media images?

This camera is marketed as a cool gadget for ‘extreme sports’ lovers, but I can imagine much more interesting applications for undercover journalists and NGOs. This little camera can handle a 8GB memory card, which records four hours of video and, even more surprising, it is voice activated. You could go into any workplace with it attached to your forehead, and it will start recording automatically whenever the boss starts yelling at you. Any kind of (loud) abuse could be registered on video this way and serve as instant proof in a court room, which is quite scary when you start to think about it. It is as if we would be carrying a personal security and surveillance system on our heads.

Back to storytelling. The films of Leonard Retel Helmrich come to mind as a more poetic catalog of unusual and immersive camera angles. With the handy cam, we could already film inside a refrigerator, but with the Muvi, we can film inside a bag and its successor may allow for filming inside the body… Personally, I would like to use this camera to make a short film from a child’s perspective, simply called ‘UM’, which is short for ‘unaccompanied minor’,  a term used by airlines to refer to the countless children traveling the world in airplanes without their parents for various reasons.

Watch this review, which includes some sample footage. There are many more similar test videos out there. I will share my first trials later.

This is what CNN did in terms of ‘immersive journalism’, using the Google street view camera to take viewers into ravaged Haiti. The uncut footage provides a 360 look on the human disaster, filmed from the roof of a driving car. For me, browsing these interactive videos definitely provokes a new and strong sense of ‘being there’ - but painfully so. Being confronted with the surround view, I dearly miss ‘perspective’. Who is looking through the camera? Who is looking at whom? What for? Why am I here? Gazing around from the top of a car makes me feel like a voyeur or a tourist, rather than a connected and empathic citizen. Here, the interactivity and 360 do not produce a sense of connection, but rather a confrontation with a reality that I am clearly not a part of, because I can manipulate it at will. Unlike the persons shown in the video. A lot of pixels and little storytelling.

This is just a Youtube rendering, but look on the CNN website for the real deal, with full control over your point of view. It’s a rather alienating way to relate to the world, if you ask me. Ffrom the CNN website: “Haiti: 360. Use your mouse to click and drag around the video to change the view. You can also zoom in and out. Pause and explore at any time by pressing the play/pause button under the video to stop and look around. The video was shot on Monday, January 18, at 9:52 a.m. EST in Port-au-Prince, Haiti.” I wish the Haitians in the picture were able to ‘pause and explore’ their world ‘ at any time.

Just imagine a 1000 m2 black box  filled with stereoscopic 3D machines, interactive video and 360 projections, located in the science fiction like setting of the brand new Hong Kong Science Park in Shatin. This is where Philips also recently moved its Consumer Electronics division and this is where City University of Hong Kong opened the spectacular ALiVE lab last summer. ALiVE stands for ‘Applied Laboratory for Interactive Visualization and Embodiment’ and is the baby of director and dean of the School of Creative Media Jeffrey Shaw and research director Sarah Kenderdine. Both have extensive experience with immersive installations as art works and museum displays and more recently they have ventured into 3D stereoscopic representations of cultural heritage sites such as Hampi in India. There are three installations which have possible uses as documentary storytelling machines.

The first one is the iDome, a half round screen which accommodates content shot with a 360o camera like the one used for Google street view. In this iDome, one viewer can manipulate a mouse to see a 360o film from all angles. The current content is limited and shows mostly ‘place’ (e.g. a street in Australia and a mosque in Istanbul)  and very little ’story’. It seems to me that storytelling could be introduced most convincingly by constructing a sound scape of interviews and location sounds recorded separately. Visually, I could imagine experiments in observational, ‘fly-on-the-wall’ style documentary if the scene is extremely interesting and happening in one room (e.g. a court room, an interrogation room, an IC unit at the hospital, a class room challenging the teacher). Shooting and editing are still extremely limted with the 360 camera (called ‘Ladybug‘), because rendering takes about one hour per minute and footage takes up enormous amounts of memory.

In the ‘Media Dome’, viewers can lie on their backs and watch stereoscopic photos projected onto the domed ceiling, so this limits the non-fiction storytelling potential to anything taking place in the sky. The media dome has a high ‘lounge-feel’ to it and could potentially work with a documentary story taking place in the top of a tree for example. I saw it with a slide show of photos of ornate ceilings from all over the world (and without spoken words).

Thirdly, T_visionarium is a wonderful and extremely engaging cylinder of 12 meter diameter based on six powerful 3D video projections, which can host around 24,000 video clips at once and has the most powerful storytelling potential. In terms of content, it originally is an archive of random clips from 24 hours of Australian television, which could be replaced by any other type of content that is properly clipped and tagged to fit the software. The effect on a viewer of this machine is truly impressive and ‘immersive’ and invites further thinking about the development of database-stories. For example, the Asia Art Archive in Hong Kong has an extensive collection of video interviews with artists and critics about the recent history of Cantonese art, which could potentially be transformed into a mesmerizing interactive and alternative history of contemporary Cantonese art as told by its creators. The drawbacks are the production cost and the complexity of the installation. It takes considerable man power to cut tens of thousands of clips and tag them manually, which is needed to enable viewer interaction and meaningful rearrangements of the clips in the space.  However, if this can be mastered and overcome, T-visionarium is much better than imax and could potentially host emotionally engaging and interactive documentary storytelling.