Towards a full body narrative: a communicational approach to techno-interactions in virtual reality

The essay approaches the sociocultural and technological issues linked to the communicational process that exists in the user’s relationship with immersive digital environments, specifically: Virtual Reality (VR). A critical look at VR as a communication interface is launched, precisely because it indicates assumptions of enhancing the creation of meanings on media surfaces by leveraging multisensory stimuli that cause the immersion of individual’s body in a 360° framed experience. In order to instigate the problematization about socio-semio-technical transformations resulting from these techno-experience, empirical explorations are conducted to two Brazilian immersive environments. In addition, data are qualitatively analyzed based on Kerckhove (1995), Gumbrecht (2003), Slater et al. (2013), Sodre (2014), Flusser (2017) and others ideas to illuminate the debate of technical-images, digital narratives, materialities and appropriations on technological devices, media languages and coding in the digital age. The results suggest that audiovisualities predominate in the VR techno-interactions but from a new state of media understanding, where the user, guided by 3D computational models and gestures interactions, is placed inside the stage of content enunciation and no longer in front of a flat screen.


Starting Point: An Introduction
When we talk about mediatic surfaces we can seek the thoughts of authors like McLuhan, Flusser, Dubois, Gumbrecht, Baudrillard, Sodre, Ferreira, Kilpp, Kerckhove and others. Indeed, it is the study of the media field that brings these author's ideas closer to this work. For example, if McLuhan (1964) indicated that the medium is the message due to its sociotechnical influence transforming communicational practices and, consequently, sociocultural, market, labor and economic; Flusser (2017) pointed out about the transformations that technical images exert on electronic platforms, catapulting the coded (and immaterialized) world of computers to a kind of almost materiality; or, yet, Sodre (2014) warned us of the importance of understanding the influence of techno-interactions not only for technical navigation issues in digital interfaces, but for all their impact on the culture and behavior of pos-contemporary society.
Before proceeding with the debate on ways to build a narrative of the full/ whole body, specifically through techno-interactions in Virtual Reality (VR), let's clarify important concepts that, in some way, illuminates the reflections presented in this work, such as: mediatic surfaces, perceptual process and the virtual reality experiences.

Theoretical Point: Conceptual Lines, Imagery Surfaces and Computational Modes
On mediatic surfaces, Flusser (2017) indicates a tendency of those kind of interfaces to make use of imagery resources for the representation of facts, especially when dealing with electronic platforms (television, cinema, mobile phone screens).
The surface thinking makes use of figures, paintings or videos to encourage the interpretation of the meanings contained therein; while the line thinking, on other hand, makes use of texts to explain concepts. Thus, the facts would be represented by the imagery of surface thinking in a more complete form or, on the other hand, by the written lines of conceptual thinking in a less complete, but clearer form. "The messages from the imagery media are richer and the messages from the conceptual media are clearer" (FLUSSER, 2017, p.111). The Czech thinker goes further suggesting the image in communication gained more prominence in the media processes when the masses became interested about information. If previously the information Towards a full body narrative: a communicational approach to technointeractions in virtual reality was restricted to the lines of texts written in books that only the literate elite could understand (conceptual thinking), from the rise of the mass media the image became an appealing media resource to make get the facts to the society (imagery thinking for the masses).
In fact, imagery resources have, until today, been the main sensory human principle to be reproduced by surfaces. However, the truth is that the human is a multisensory being, because, yes, she/he absorbs images to understand the world, but, yes, she/he is also influenced by other senses that make up the perception of reality process (sound, smells, textures, temperatures, etc.). Davidoff (2001) explains that in order to understand the contexts in which we are inserted our body acts as a multisensory interface capable of mediating interactions and experiences with spaces, objects or even other people. This understanding of contexts (or realities, or spaces, or objects, whatever) is guided by the ability of our sensory organs to capture energy patterns that can influence our world experiences, for example: the lights, the sound, the pressions, the temperatures. The five senses -sight, hear, touch, smell and taste -are agents responsible for detecting and bringing the most varied sensory stimuli (objective data) to our brain, where the subjectivity of each person is added with the purpose of perform an interpretation and, consequently, consolidate our perception of something (subjective data). This mental consolidation of the perceptual process indicates that the human being has the body as an interface for the collection of objective data from the world and the brain as a machine of subjectivity. After all, each of us can interpret a sensory stimulus from a memory, a preference or a very particular past experience. This process of understanding the physical world, then, involves a sensorineural phenomenon which has two particularities: a) people have a similar sensory experience because we capture the stimuli that surround us through our sensory organs (the body-interface has objective experiences); but b) each one of us build an own interpretative resolution for the lived experiences, as our memories and preferences are combined with the objective data collected to create a perception of reality (the brain-interface has subjective experiences). In turn, the way we use artistic, communication or, simply mediatic resources to represent realities in any platform is something that has been with us since we became rational (FLUSSER, 2017). Whether through a text, a rock painting, a photographic portrait or a 3D model in a digital tablet screen the expansion of our knowledge of reality, objects or phenomena reveals attempts to take physical contexts into the non-hermeneutic field, the universe of code or the virtual (BAUDRILLARD, 1994;GUMBRECHT, 2003;SODRE, 2014;FLUSSER, 2017). Anyway, regardless of the mediatic surface used to portray or record stories and facts, it is undeniable that the representations of their original versions become more convincing when they reach a level of multisensory simulation. For example, the three-dimensional image of a shoe has greater similarities with its physical version than a photograph on paper and a painting on a canvas (two-dimensional surfaces) or a textual description (one-dimensional line), precisely because it deceives our view by presenting illusions of form and materiality when using algorithmic techniques of digital machines for the construction of synthetic models. After all, in this example, the shapes, perspectives, depths, textures, shadows and lights of the 3D virtual shoe really mimic imagery patterns of the original object in a similar way we would look at a shoe in the physical reality.
At this point it is highlighted the potential for using VR as communication platform that provides multisensory narratives (for the whole body); in a computational reality supported by algorithms capable to construct 3D model interfaces responsible for mediating advanced interactions between human and machine. Advanced, precisely because it is an interface that provides the visualization, interaction and manipulation of objects, spaces or even characters (avatars) in a digital environment with aesthetic and functional similarities to the experience we have with the physical contexts (KIRNER; TORI, 2004). Indeed, when producing simulations that can imitate physical reality (or even give shapes and models to imaginary realities), VR builds scenarios and situations that stimulate the user to feel the sensation of inhabiting the communicational context. That is, though the communicational environment (2013) suggest that in these cases a more complete and complex notion of perception of reality is awakened. When mind and body do not seem to distinguish the difference between real (physical) and virtual (digital), the authors indicate that the user is facing a plausibility of a new reality configuration, as her/his perceptual process was convinced that the synthetic world is about a new reality. Based on the ideas of Pausch et al (1997) and Slater et al. (2013), adding immersion and presence concepts of Burdea Zuffo et al. (2006) and Thom (2008)   However, something that also catch our attention is the extent to which audiovisuality remains present in experiences with content mediated by different devices and their frames. As an example of continuity and discontinuity of media elements in other ones, we take a ride on the thought of Kilpp (2015, p.17) when claiming that "on all screens at the end of the successful search for 'TV', television content is broadcast.
However, in the wake of McLuhan, we insist that the medium is the message". That is, "it is not the content that defines the media that disseminates it", (KILPP, 2015, p.17).
Following with the author's ideas we quote: The most obvious example of this, in our view, is still the film shown on television: it is not cinema, but a TV program: the content is cinematographic (comes from a previous media -the cinema), but the final meaning we attribute to such a content depends much more on the meanings attributed to it by the TV media (as a TV program and as a state-television) than on those attributed to It by the cinema media (as a film and as a state-cinema) (KILPP, 2015, p. 17-18) [4].
While we do not believe in a purely technical explanation, in which we could simply claim that new technology devices change the way audiovisual content reaches viewers; we also do not believe in an explanation purely linked to the remediation of the audiovisual to other mediatic surfaces. In other words, dissatisfied with these two possibilities, we prefer to take a socio-semio-technical look that, perhaps (and this is a hypothesis), will allow us to understand that something changes but somethings also remains and, that, adopting Heraclitus thought, we must to look at the becoming

Exploration Point: Report, Perception and Data Analysis
Here the two techno-experiences in VR are reported and analyzed, in order to reflect on the narratives that are present in these communicational practices and processes. In addition to describing the user's feelings, the thoughts from researchers in the area are also crossed in order to deepen the analysis about the characteristics, peculiarities and possibilities of configuring multisensory narratives.

Experiment 1: A Synthetic Garden to Test a Product
The first experience consisted of using VR devices that allowed the users to immerse themselves through visual stimuli (VR goggles), audible (headphones) and Towards a full body narrative: a communicational approach to technointeractions in virtual reality motor interactions (motion sensors). This experience was created by a brand that sells garden maintenance equipment, in a marketing action that aimed to encourage potential consumers to carry out tests with a grass and shrub trimmer. In this case, more than stimulating immersion in a virtual garden to actually use the product and test its attributes, the action was carried out at a sales point (a store), in which a large circulation of people raised the interest of participating in the experience. Also, television screens were set up on the walls of the promotional stand inside the store.
So, people could visualize in real-time what the user was exploring in VR.
It is important to emphasize that the audiovisual experience was very realistic, in the sense that images of the bushes, fences, flowers, insects and grasses built with 3D by Gumbrecht (2003), in the spectrum of imagination (or in Plato's ideas world), but the true is, in the universe of VR technology simulations, it seems to reach a new level.
The quasi-thing of Flusser (2017) theories related to digital modeling images looks like to reach a whole new level, where the virtual thing begins to be understood as a new reality, as a new context. So, a new truth for virtual things was configured, as if they were their own original versions. As much as the immateriality of objects and spaces in the experiences is a truth -after all, they aren't atomic elements, but bits grouping pixels for the visualization of synthetic images -it is necessary to face these technical images as a new composition of materiality and spatiality of communication environments. As Kerckhove (1995) or Sodre (2014)  as if it were reality itself. In other words, the immateriality sustained by the binary code that produces realistic 3D models in the VR context supposes that user interpret this immersion and/or presence in the same way as her/his perception of materiality works with physical objects, spaces and people. As say Zilles Borba (2017), on the flat screens we can see an experience of the she/he/it-avatar (in third person), in which there is a symbiosis between user and character, but clearly one is inside the enunciation space (the avatar) while the other is outside of it (the user). In turn, the experience of the self-avatar indicates a profound plausibility of inhabiting the virtual context, with no longer a third-party synthetic body, but the feeling that the organic body itself has been transposed to the scene of techno-interactions.
During the experience with the shrub and grass pruning machine, important aspects of the sound experience could also be noted. The sounds of space had a power to generate sensory stimuli similar to those experienced in physical spaces. After all, the singing of birds, the humming of mosquitoes or, of course, the sound emitted by the engine of the pruning machine were mapped three-dimensionally in the VR garden. This technique of emitting sounds from the locations of the elements that propagate them makes the experience more realistic, precisely because the intensity, volume, duration and reverberation allow the interpretation of depths, distances and movements existing in objects and spaces which the self-avatar was inserted (DAVIDOFF, 2001;ZILLES BORBA, 2017).
In turn, the motor interactions experiences also sought to stimulate a feeling of symbiosis between the user's body and the avatar's body. Through motion sensors placed specifically in the real product (in the pruning machine) and, also, in the HMD allowed any movement of the arms and body positioning of the person in the physical were real-time converted to her/his avatar. Clearly, this made it possible to include natural human operations in VR, making the subject's interactions and command intensions more intuitive. S, the user could use the product in an empirical way to verify its qualities and uses for gardening practices (it all without cutting a real plant).
Indeed, more than that, due to the freedom to move the physical body with orders issued by the brain (the cognitive knowledge of being a human being in the physical world), but to perceive these motor operations feedback occurring in harmony with the self-avatar, a kind of user-character symbiosis was established, creating even for just a few seconds the plausibility to inhabit the virtual context.

Experiment 2: A Shoe Production Process in VR
The second experience consisted on using VR devices for training factory employees to produce footwear through visual stimuli (VR goggles), audible Towards a full body narrative: a communicational approach to technointeractions in virtual reality (headphones) and interactivity with input devices (joysticks). This simulation model was created by a company with the main objective of optimizing the production process of only one of the several stages of footwear manufacturing.
Different from the experience previously reported, the audiovisual content in this one was created entirely by capturing 360° video, and not through the creation of 3D scenes. So, this experience presents a linear script in which the user followed a predefined journey: she/he learned how to turn on an industrial machine, selected personal protective equipment, knew the procedures for taking the upper of the shoe on a treadmill and placing it on the machine and, finally, use the buttons necessary to point the toe of the shoe.
The audiovisual experience supported by the 360° video format presented a realistic scenario, objects (machines, shoes, etc.) and coworkers in the factory sphere.
After all, it was a video projecting real scenes and activities of the factory routines into a first-person perspective field of vision supported by the HMD. However, precisely because it was a recording, and not a 3D creation, the user could not move around the enunciation space and, consequently, the realism of exploring the scenario even only visually cancelled some perception of realism during the experience. Here, it is imperative to note that, yes, there was an immersion generated by visual stimuli related to the form, scales and proportion of objects around the user (360° video).
Anyway, because it was a video format, even with a 5K quality resolution, the visual narrative was compromised when talking about full immersion. There seemed the aesthetics are more similar to a state-video or state-television, adding to the user the first-person perspective. In other words, a feeling of being within the context had been created, but the aesthetics of the content generated by the 360° video resembled more images than places the user had been (ZILLES BORBA, 2018).
The audio experiences were very real. Due to the fact it was captured with binaural audio techniques, the spatialization of sounds in the 360° scenes had a high performance with regard to the realism of the experience (machines noises, people talking, etc.). In short, the sounds were loud and shrill, reflection the reality of the noises of an industry.
The elements of interactivity with the virtual context were limited to the possibility of interaction with a pre-recorded video. So, it was not possible to the user take personal decisions in order to have a personalized experience, since the interactions with the virtual objects were limited to direct a cursor to the shoes or to the machine buttons to active another action. To make the interactivity experience more realistic for the user, the cursor movements were controlled by the user gestures with a joystick. But, in the end it creates only a sense of motion sensor (via Bluetooth) that looked more like a mouse working in three dimensions (axis X, Y and Z  Based on the theoreticians and the two experiments carried out, it is possible to point out that sensory stimuli are important to awaken the subject's beginning of diving in the digital context. This includes the use of multisensory narratives and the specificities related to this kind of media: the stage interactions in 360°, the perspective of the first-person view, the natural gestures of arms, legs and head, the sounds mapped in the space, etc. Also, the mental experience proved to be important, because more than feeling the virtual objects, the mental connections and the user's attention transfer with the plot were fundamental aspects to encouraged the user engagement with the techno-experience. For example, in the second experiment, although there was less immersion, a high sense of presence was created due to the mental stimuli launched as a challenge (a gamification) for the worker to learn how to produce a footwear with that specific industrial machine.
In short, the results achieved with this work were satisfactory in terms of reaching initial answers about a creation of a full body narrative experience through VR interfaces. As future work, indeed, it is intended to make two approaches based on what was found here. The first one is a natural continuation of this research with a bigger corpus of analysis. The second one, is to expand the exploration of a full body narrative exploring VR interfaces that also provide smell and taste inputs, in order to study how the user perceives those experiences and how much new dimensions for the mediatic narratives in VR simulation models increases immersion, presence and/ or plausibility in techno-experiences.

Notes
[1] Randy Pausch was a professor and researcher in Computer Science at Carnegie Mellon University known also for leading Disney's simulators projects.
[2] Zuffo et al. (2006) and Burdea (2003) highlight immersive experiences in VR must provide realism and interactivity from the user's perspective. In turn, Thom (2008) suggests that even more important is the rate of attention transfer of the subject to the experience, which could be stimulated psychologically due to the plot, plot or involvement of the story.
[3] It is understood here that the use of hands when holding a joystick, gamepad, mouse or keyboard guarantees the user control of the content on the screen. This would be closer to the sense of touch, despite being a metaphor for our gestures, touches or movements. That is, the device allows interactions with objects in the context of the screen to be carried out in real-time as a metaphorical representation of the real movements.
[5] Despite being an atypical practice, the food tasting experience can be performed in VR. Some restaurants have already created multisensory experiences where the customer visualizes things they would never think of eating (cloud, rainbow, etc.).