Book review: Digital reality in education, art galleries, and museums (with AR, VR, MR)
By Shalin Hai-Jew, Kansas State University
Virtual and Augmented Reality in Education, Art, and Museums
Giuliana Guazzaroni and Anitha Pillai
IGI Global
2020 384 pp.
Any new technologies on the horizon need to find ways to be adopted in various practices and sectors. First movers tend to be those with a level of vision, daring, skills, and funding to use the new technologies before prices have come down in ways that make the tools more available more broadly. In the so-called “creative” sector, including art and museums, and the education sector, digital realities have been applied in rich ways. Giuliana Guazzaroni and Anitha Pillai’s edited collection Virtual and Augmented Reality in Education, Art, and Museums (2020) may seed future generations of harnessing augmented reality (AR), virtual reality (VR), and mixed reality (MR) tools in the respective spaces.
Digital reality (DR) may add dimensionality and experiential richness to in-person and virtual experiences; it may layer in information in a more seamless way. Certainly, DR may be used to add seamless layers of surrealism or fantasy (so high-fidelity to the real is not de rigueur). In this collection are creative integrations of AR and VR, with re-imaginings of ancient cultures and societies, dramatic re-enactments of historical events and bygone lifestyles, reconstructions of archaeological digs, high-resolution digital images of lost art, and rich multimodal expressions of complex meanings. Clearly, what is conceptualized and designed and deployed within the constraints of the technologies is broad but requires hard thinking and technologically complex effort. For all the excitement around the cutting-edge technologies, required expertise is rare, and costs are prohibitive. Also, several works address the challenges of accessibility with AR but without clear applied paths to achieve. (Immersive virtual worlds and other forms of “VR” have some built-in adaptations for accessibility.) Some experience vertigo and other effects while experiencing AR through headsets and head-mounted displays.
In the Foreword, titled “Give Me a Lever and I Can Lift the World,” Nicola Bandoni (of VR Land in Italy) suggests that “immersive technologies have the power to create a thought, a vision, a common mission shared by all of us to release a healthy culture of scientific and/or academic matrix towards professionals, entrepreneurs and educators who want to improve their understanding of the strategic role offered by immersive technologies in the fields of education, art and museum environments” (p. xvii). In the Preface, the co-editors laud the full-sensory experiences possible using virtual reality, including “sight, touch, hearing, and smell” (p. xix). As such, these have been used for professional trainings and research; as yet, immersive technologies may be harnessed in combination with the Internet of Things (IoT), artificial intelligence (AI), and machine learning, among others, given the simultaneous advances in these technologies in prior decades and recombinatorial potential.
Designing VR Environments for Teaching and Learning
Ali Geris and Nesrin Ӧzdener’s “Design Models for Developing Educational Virtual Reality Environments: A Systematic Review” (Ch. 1) involves a selection of 44 articles in the space to extract themes and some best practices. The researchers used targeted keywords used to seed the searches (“Virtual Reality & Instruction,” “Virtual Reality & Model,” “Virtual Reality & Designing,” “Virtual Reality & Design Model,” and “Virtual Reality & Instructional Design”) for articles from mainline databases, including Web of Science, Science Direct, Scopus, ERIC, and IEEE Explore (p. 7). They reviewed the abstracts and engaged selected articles in depth. They emerged with a three-part top-level theme (and related subthemes): (1) an educational framework (including “problem and objectives, scenario, learning, socialization, and evaluation”), (2) a design framework (including “authenticity, immersion, presence, fidelity, and interaction”), and (3) a technical framework (including “peripheral devices, modeling, and programming”) (p. 9). They explore the importance of the authenticity of the virtual designed space and the representation of the problems being addressed. They discuss the need for presence, defined as “the feeling of being in real life in a realistically designed virtual reality environment” (p. 12). They point to a model with macro and micro strategies for instructional design of virtual reality environments, with micro focuses on the principles of multimedia design. They point to the relevance of usability and learning studies for understanding the actual efficacy of the virtual reality designed space (p. 10). Beyond the basics, it seems that other methods for designing will depend on the particular case and would benefit from an adapted ADDIE model approach.
Improving Healthcare Trainings with Technologies
Figure 1: Augmented Reality and Health Sciences (by zedinteractive on Pixabay)
Prabha Susy Mathew and Anitha S. Pillai’s “Role of Immersive (XR) Technologies in Improving Healthcare Competencies: A Review” (Ch. 2) offers an in-depth and engaging sense of “extended reality” (XR) technologies are used in healthcare, one of the front-runner sectors in the uses of AR/VR/MR.
Mathew and Pillai suggest that the gaming industry were the early adopters of such technologies but also “mining, healthcare, and medicine, retail, education, automotive, manufacturing” sectors (p. 23). This work offers an overloaded figure with some of the name brands of various immersive technologies (p. 25) that blend the physical and the virtual. They describe various instantiations of XR: one which enables people to point their smart phone cameras at animal skeletons that suddenly come to life in full visual form. Another is a VR tool that enables a medical practitioner to see the veins underneath the skin during Botox treatments. Yet another shows a simulation of a surgical process to inform potential patients (p. 26). Various multisensory simulators enable learning and practice of various types of surgeries (p. 27).
The real strength of this work is in their summary of healthcare applications. Extended reality applications are applied to various phases: preoperative planning, surgery (including cosmetic, laparoscopic, orthopedic, neuro-, ocular, spinal, cardiac, and dental, and robotic surgery). They share research which suggests improvements in performance. Beyond surgical procedures, there are blended reality trainings in anatomy and clinical skills. There are tools to help with training on various equipment. They add a section on telemedicine platforms, given the shortage of medical facilities and staff in rural areas. The various technologies enable cost savings and training efficiencies; there is research showing higher learner retention in some learning (Mathew & Pillai, 2020, p. 36) and improved skills in practice (p. 37).
However, there are some down sides as well. A foremost one is a “lack in quality content and its availability for engaging the user in VR applications” (p. 37). Some have experienced discomfort with some of the peripheral and wearable gear (p. 37). This work offers a list of technological glitches mentioned in the academic literature, such as low battery life, VR latency, a limited field-of-view, a “vergence accommodation conflict” (“the difference between the physical surface of the screen and the focal point of the simulated world the user views”), interoperability between various AR and VR systems, privacy concerns, and security concerns (p. 38). There are combined strengths in the healthcare field with extended reality tools, for all the gaps. Perhaps those who are interested in moving into this space would do well to learn from these various applications.
Nursing Education and Virtuality
Derya Uzelli Yilmaz, Sevil Hamarat Tuncali, and Yusuf Yilmaz’s “Nursing Education in the Era of Virtual Reality” (Ch. 3) begins with a basic assumption that contemporary Gen Y and Gen Z learners expect some level of multimedia in their learning. So, too, in nursing education, which requires “cognitive, affective, and psychomotor learning,” the coauthors suggest that full immersion in virtual reality may be just the thing (p. 47). VR can provide contextual richness—for simulations of particular events (re-enactments), role-based individuals, environments, problems, and cases. The co-authors describe some high-technology simulations: “screen-based simulations; realistic, high-fidelity procedural simulators; realistic high-tech interactive human simulators; (and) virtual reality (VR), augmented reality (AR), and haptic systems” as compared to more low-tech approaches like 3D organ models, plastic mannequins, simulated patients, and “human cadavers” (Yilmaz, Tuncali, & Yilmaz, 2020, p. 52). They offer a fairly comprehensive review of the literature about VR-enriched teaching and learning with different levels of immersiveness, learning complexity, and learning outcomes. With less costly technologies over time, increasing familiarity to nurse educators, perhaps simpler ways to program and customize the learning, VR has an important place in nursing learning.
Exploring Virtual Radio
If people are mostly thinking of sound and voices in radio programming, stemming from the 1900s, they have not caught up with modern online radio. Nazime Tuncay’s “Constructing Virtual Radio Center: Providing Visuality to Sounds” (Ch. 4) proposes various ways to make sounds visible and visual for mixed-sensory experiences. Visualizing sound involves “providing visual platforms as an alternative to the real-world platforms” (Tuncay, 2020, p. 71). The author explains the vision early on:
Virtual Radios are a way of simulating or replicating an environment and giving the user a sense of being there, taking control, and personally interacting with that environment with his/her own body…Virtual reality makes reality more abstract and easier to be visually acceptable. Radio technology lacking visuality may be taken as an abstract resource for education (Tuncay, 2020, p. 72).
This work offers the basic requirements to set up a virtual radio center, which is seen as a way to “bring coding, inventing and imagining to the students” (p. 73). Virtual reality may be created with a variety of technologies, from online spaces to three-dimensional video to interactive platforms to immersive virtual worlds, and other instantiations (p. 74). While there are various ways to construct a virtual radio station, the researcher here used Second Life as the structure and a program called ooVoo (an app for instant voice and text messaging). Tuncay shares evocative screenshots of the various learners collaborating around radio station issues in virtual rooms in SL via their own digital avatars. Some lessons learned from this experience are shared.
Researching Gamification for Actual Learning Outcomes
Biancamaria Mori’s “Gamification: To Engage is to Learn” (Ch. 5) explores two learning cases in which games are employed in order to see if they have a verifiable positive learning effect. One involves Toca Boca for children, and one involves Assassin’s Creed Oringis. She points to research that suggests playing videogames for learning may enable “more mnemonic learning skills” and faster reactions to situations (p. 83). A critical element in game design involves the rules guiding its gameplay. This author provides a walk-through of some of the vast literature related to gamification. Gamification has been applied to various sectors: education, business, museum displays, and others. In business, gamification has been harnessed for customer retention, brand management, advertising, and marketing (p. 88). Gamification may provide incentives for learners to engage with the learning contents, their peers, and their instructor. They may vicariously experience various learning designs.
Mori (2020) highlights the gamification in more cutting-edge applications which provide full-sensory (or most-sensory) inputs to the learners. She writes: “Interactions between humans and digital worlds have now been temporarily classified into 3 types: Virtual Reality, Augmented Virtuality, Augmented Reality, the combination of these three technologies is called Mixed Reality” (p. 89). Refreshingly, and using physics concepts, she suggests augmented reality is superpositioning of virtual data in the real analog world, that enables “real hands that can interact with the simulation” (p. 89).
Figure 2: “Gamification in Business” (by Disertel on April 28, 2018, on Wikipedia)
Gamification had a hot moment when it was seen as a way to motivate learners and enable effective learning. At this point, some aspects of gamification have been built into online learning systems and other learning applications, but there is a sense that its moment may have passed. Perhaps integrating these approaches into newer technologies may revive the approach. Hers is a light review of gamification, with some major thinkers missing, but it suffices for a setting the context for the book.
Selfies Everywhere
Giorgio Cipolletta’s “Ubiquitous Self: From Self-portrait to Selfie” (Ch. 6) opens with a reflection on the importance people place on faces—their own and that of others. Biologically, a particular part of the human brain (the fusiform gyrus) is a part devoted to facial recognition. The face (as metonymy) stands in for the whole body and the self. This tendency explains why so much human attention is paid to making an attractive public-facing face, shaping it for various types of expression, and how people paint and modify it into masks that can represent different characteristics and social personalities (p. 93). The author writes:
All visual culture revolves around the body and the body par excellence is the face. The 21st century portrait represents a kind of black mirror where we project ourselves into a kind of blindness. Mask and face are confused by an omnipresent multividuality in which the shield reveals itself and reveals other possible worlds. The face-mask melts in between Real and Virtual and the self becomes augmented (Cipolletta, 2020, p. 93).
The author offers a bravura and thickly theorized meditation on the roles of faces throughout history, in artful expressions, in death masks, in film representations, up to the current moment of self-generated selfies (self-portraiture) on social media. He writes: “The viewer looks at the screen and consumes faces on which society projects its power structure. The public face has produced its own mask. Photography and cinema, despite their claim to exact reproduction, only create other masks” (p. 97). People reveal and hide something about themselves in how they self-portray. And yet, people are entranced by each other’s self-complimentary self-inflating self-portrayals and socially performative acts, and such portrays evoke various forms of parasocial followership relationships. Into this space comes “touchscreen aesthetics” where the screen “becomes the new skin through which we touch the other and empathy is replaced by screen-empathy” (p. 102). He describes an intimacy of people with each other through technological devices, which enable them to consume each other’s selves:
The screen becomes wearable, portable and plays with the body that embodies it. In other words, the screen is the new flesh, while the body becomes a device, it arranges the bodily system in relation to new technologies and is placed in a process of communication by activation information (info-aesthetics), data, downloads that generate continuous connections and exchanges between users. Contemporary man produces technology, but at the same time he is its technological ‘product’. The diffusion of new digital trends involves all disciplines and influences the status of the body at various levels, both psychological and physical, cultural and biological. A new subjectivity is born from this digital experiential contamination based on the possibility of being connected and of ‘touching’ us virtually. (Cipolletta, 2020, p. 103)
He weaves in an entrancing work by French artist Laurent Mignonneau in Portrait on a Fly, described as an interactive installation “consisting of a monitor showing a swarm of a few thousand flies. When a person stands in front of the monitor, the insects build the contour of the person. They begin to organize and reorganize themselves continuously, thus creating a recognizable resemblance of the individual. Posing in front of the monitor attracts flies. In a few seconds, they invade the face—but even the slightest movement of the head or parts of the face pushes them away (Kluszcyński, 2012, as cited in Cipolletta, 2020, p. 105). With so much obsessional focus with the self, the fulfillment of ego and identity, the individual is insinuated into the museum experience. This work is a type of simultaneous artwalk and musing about the self and self portrayal through time and into the current age.
He writes of how contorting selfies are in terms of people’s representations:
As we have seen, the increased self, the multiplied self, the fragmented self are the new conditions of existence of digital and ubiquity. The ubiquitous self becomes the driving self-portrait of the twenty-first century. The contemporary self-portrait is the digital portrait of an increasingly facial and “eXcessed’ society. The self is hidden, collaborative, ambiguous, transvestite, fleeting. The self is the new performance of the contemporary always present, but difficult to define. The self is elusive and ubiquitous. What is a selfie if no one in the photo is the real self? (Cipolletta, 2020, p. 111)
In the modern era, people are trapped by the technologies around them, augmented by the digital and the virtual, and few can see their way out, much less free themselves from the allure.
21st Century Museums with AR and VR and Social Media
Given the digital ecology, museums are turning more to VR + AR + social media in a digital convergence as a matter of course, according to Antonios Kargas, Nikoletta Karitsioti, and Georgios Loumos in “Reinventing Museums in 21st Century: Implementing Augmented Reality and Virtual Reality Technologies alongside Social Media’s Logics” (Ch. 7). More precisely, there is the integration of “emoticons, comments, like-dislike, chatting, social networking” from “social media operational logic” in VR and AR integrations at museums (p. 119), as cultural organizations.
Figure 3: Octagonic Cloud of Social Media Connections (adapted from GDJ on Pixabay)
Of the various technologies, mixed reality applications are not foremost as part of a digital communications strategy. Rather, the uptake of social media is “the most widespread” among museums (Kargas, Karitsioti, & Loumos, 2020, p. 120), which makes sense given that these are the proverbial low-hanging fruit. Social media have built-in user bases and technologies that are easy to learn and that enable a wide range of digital sharing across a range of modalities (text, imagery, video, audio, and others). The coauthoring team cites various museums on Instagram, like the Louvre, the Acropolis, Van Gogh, Natural History Museum, Guggenheim, the Victoria and Albert, and others.
While many museums depict their physical locations or reimagined historical locations virtually, online affordances have enabled many Instagram-only sorts of museums. These include the Museum of Ice Cream (2016), 29ROOMS, Color Factory, Candytopia, The Beach on Snarkitecture, whose evocative names suggest something of the various exhibits (Kargas, Karitsioti, & Loumos, 2020, pp. 129 – 131). Various VR and AR tours of museums (National Museum of Natural History; Renwick Gallery, Smithsonian American Art Museum, The Louvre, The British Museum, the Metropolitan Museum of Art, Hintze Hall of NHM London) (p. 131). The fanciful online versions are described evocatively and seem like they warrant an online visit.
Global Assessment of Digital Reality for Art Exhibitions and Museums
Figure 4: Feather Art Sculpture (by Yuri_B on Pixabay)
Prior to COVID-19, in 2019, it was suggested that the global augmented reality, mixed reality, and virtual reality markets (including panoramic video) would reach $40.6 billion and perhaps that the momentum of the growth would continue. 2020 has been deeply marked by COVID-19, with some suggesting that this global crisis would be an accelerator of particular advances, particularly in technology. What the impacts may be will be left to the future to see. Museums have gone virtual and online, and others have worked to create de-densified sparse in-person visits between the walls. It may be a while longer before the live traditional museum and art gallery experiences of the guards, the lighting, the velvet ropes, the people watching, the learning in-person, the visits with family and friends, the in-building café dining, and the shop visit for books and toys, will return fully. Meanwhile, virtual atmospherics and resources are the order of the day.
Figure 5: Picture Frames (by Glamazon on Pixabay)
Yowei Kang and Kenneth C.C. Yang’s “Employing Digital Reality Technologies in Art Exhibitions and Museums: A Global Survey of Best Practices and Implications” (Ch. 8) points to various platform capabilities for visualization, interactivity, visual realism, social engagement and others. The art and museum sectors are placed squarely in “the (creative) cultural industry…made of thirteen different industries that comprise the production and dissemination of a variety of cultural contents, ranging from ‘(1) advertising, (2) architecture, (3) art & antiques market, (4) crafts, (5) design, (6) designer fashion, (7) film & video, (8) interactive leisure software, (9) music, (10) performing arts, (11) publishing, (12) software and computer services, and (13) television and radio” (Yoshimoto, 2003, p. 1, as cited in Kang & Yang, 2020, p. 141).
Digital advances enable the distribution of various types of cultural contents. Digital technology systems generally include: “1) a graphic rendering system; 2) gloves, trackers, and user interface to sense and input users’ movements; 3) output devices to enable aural, haptic, and visual interactions in a virtual environment; 4) a software to model virtual objectives (sic) and to construct databases; 5) a system to deliver sensory stimuli such as visual display technology to offer users interactive and immersive experiences…” (Kang & Yang, 2020, pp. 141-142). This research team then conducted a review of the literature to capture an early history of the uses of AR/VR/MR in the art and museum space globally to better understand the smart technologies used and the apparent best practices. They describe collaborations with industry in order to fund some of these efforts. They observe narrative overlays applied to the artifacts for presentations. They describe various research deployed to understand the efficacy of such installations in the physical and the virtual and the combined.
The chapter includes evocative photos showing motorized chairs and VR headsets and various mobile augmented reality apps in the National Museum of Natural Science in Taiwan. In the National Museum of History, in Taiwan, there is a virtual reconstruction in VR of the Peinan People and an AR overlay over a Peinan archeological remains display. The Metropolitan Museum of Art and British Museum use 360o videos to enable experiencing of their physical space and various displays. Another project involves AR overlays at the Rose Fitzgerald Kennedy Greenway. While this work is not comprehensive, not fully systematic, and takes an outside-looking-in perspective, it serves to spark interest in the uses of digital reality technologies in the cultural space based on what has been achieved and is extant.
Figure 6: Art Gallery (by ZhuWei06191973 on Pixabay)
Mobile Augmented Reality in Museums and Art Galleries…for All: Bridging Different Accessibilities
Ajinkya Rajendrakumar Kunjir and Krutika Ravindra Patil’s “Challenges of Mobile Augmented Reality in Museums and Art Galleries for Visitors Suffering from Vision, Speech, and Learning Disabilities” (Ch. 9) focuses ostensibly on accessibility, but it undercuts its own arguments with its use of able-list language. First, in the title, visitors “suffering from” various disabilities might be offensive or at minimum insensitive and impolitic. The authors write about “healthy people without any disabilities” (Kunjir & Patil, 2020, p. 162), as if those with different abilities are unhealthy by comparison. Later on, the coauthors refer to “H.I. people” in reference to the “hearing impaired” (p. 165), which in a more politically correct approach may be referred to as “persons with hearing impairments,” “the deaf,” or “the Deaf,” depending on nuances. This is not just a critique of the language but the fact that language can be used to justify mistreatment of others and social-cultural callousness and disregard.
What follows next is a grab bag of details about Mobile Augmented Reality (MAR). MAR is defined as “a carry-and-go Augmented Reality in your pockets via smart devices such as mobile phones, tablets, P.D.A.’s and wearable devices” (Kunjir & Patil, 2020, p. 163) This approach offers some “11 vital elements which include Usability, Design, Motivation, Interaction, Perceived control, Satisfaction, Attention,” and others including Engagement (p. 162). [They use a visual of an AR Pokemon game played in different locations.] For all the benefits, they cite a curious downside—that “one out of ten M.A.R. devices is non-repairable” as one of the disadvantages of M.A.R. (p. 164).
The coauthors provide an outsider description of the evolution of Google’s Tango platform revealed in 2014 but later supplanted by Google ARcore, with location detection capabilities. If nothing else, there are fast-moving developments in the AR space. The newer technology has perception of environmental lighting and enabled adjustment of the AR of the captured image on-screen. The reliance of computer vision and deep learning enables AR devices to do without markers in enclosures for functionality (Kunjir & Patil, 2020, p. 167). The researchers write that they used web sources for their research. They are providing a regurgitation of technology documentation instead of firsthand insights or actual depth of understanding. They offer a rough description of the modeling of a physical object using photogrammetry techniques to capture details about the object, its size, its surface, and its textures and other features, then using the modeling tools and techniques to create the AR visualization and the interfaces to engage with the object (p. 169).
This work is not really about accommodating the needs of people with varying perceptual and cognitive abilities and combinations of abilities. After the awkward summaries, the coauthors suggest haptic devices and interfaces could be used to enable broader access: “To increase the interaction between disabled users and the A.R. environment, haptic devices, which are categorized into ‘tactile’ and ‘kinesthetic’ groups based on human sensing capabilities can be merged with the proposed system” (Kunjir & Patil, 2020, p. 170). A reader is left feeling like the accessibility angle was a pretense for the publication because there is no actual analytical follow-through. It is not clear what standing the authors have to approach the topic. A mere review of some technologies is not grounds for a relevant academic work. It is unclear as to whether they have the direct expertise to actualize an MAR project in the arts based on this work.
Harnessing Emotions as Part of the Interactive Museum Experience
Part of people’s intelligence makeup involves emotions and affect, which enable smoother decision-making and analysis in some cases but also some “noise” in decision-making in others. Giuliana Guazzaroni’s “Role of Emotions in Interactive Museums: How Art and Virtual Reality Affect Emotions” (Ch. 10) highlights the centrality of emotions in engaging art. She writes:
Brain research highlights the role of emotions in the fruition of art. Damasio (2000) uses the term emotion to refer to internal changes in the state of human body (e.g. chemical, visceral, muscular etc.) and the resulting changes in the nervous system. They can create a specific emotional state, which can be a stimulus for the action. Emotions play a crucial role in an aesthetic experience. The visual act is not a passive recording of the physical environment, but an active construction that involves elaboration and analysis processes. Complex cognitive and affective psychic processes are involved when people visit an art gallery. Authors separate emotion from cognition, based on a differentiation of the cerebral hemispheres, placing the processing of emotions on the right hemisphere. When individuals live an aesthetic emotion, the action consists in the interest aroused by the artwork. The interest produces a mobilization of the whole organism based on the exclusive role of the perceived aesthetic object. (Guazzaroni, 2020, p. 175)
People are prewired for responding to aesthetic beauty. They have a range of emotions that are responses to various depictions, from calm to violence. People may not be conscious of their own emotions, or they may only be partially aware. In an interactive museum context, with digital overlays, people experience different layers of reality in a hybrid and multisensory experience. She points to the work of “multiple neuron systems that specialize in executing and understanding the actions of others and their intentions, the social meaning of their behavior and their emotions” (p. 177), so there is a sociality aspect. Artworks, artifacts, performances, information, and social interactions—in the museum and gallery spaces—may be more powerfully experienced if human emotions are brought into play.
The author goes on to describe various VR experiences, such as at the Monastery of Santa Maria delle Rose, Sant’Angelo in Pontano in Italy. One interactive installation, by artist Luca Tomassini, in Tolentino, in the Marche Region of Italy, is called the “Walking Eight”. Guazzaroni (2020) writes:
The characters are real people and they are taken from old photos of individuals living in Tolentino and performing their own life. The oil painted canvas is 13 meters long and partially wrapped in a pipe to represent time or a story that unfolds from an ancient papyrus. The second part of the space is a performative place where Google Cardboards and other Virtual reality visors (e.g. Oculus Rift) are offered to visitors. There are two mirrors to reflect performative people and the canvas. Using VR viewers participants may see a 360o performance where performers/actors reproduce gestures taken from the canvas (i.e. from the characters represented on canvas) (pp. 182 – 183).
Such embodied high-interactivity with AR/VR suggests only limited direct experience for a few for the multi-layered and complex display, but perhaps there are indirect experiences, such as by readers and those consuming this second-hand through media. The author conducted research to capture information about the viewers’ experiences with this multimodal approach. As a side note, the author cites a work observing potential risks arising in the uses of VR and the need for “new ethical recommendations” (Madary & Metzinger, 2016, as cited in Guazzaroni, 2020, p. 186).
Democratizing Cultural Heritage at the Urbino’s Ducal Palace
The preservation of cultural heritage into future generations is an important part of European identity, purpose, and practice. One of the main draws of Europe for travel are the many historical sites, for learning and for pleasure. Paolo Clini, et al’s “Digit(al)isation in Museums: Civitas Project – AR, VR, Multisensorial and Multiuser Experiences at the Urbino’s Ducal Palace” (Ch. 11) focuses on the role of museums to help people reflect on the past and consider what it means for today’s identities and practices and meaning-making. Museums are the “custodians of heritage and culture” and educators of the public (Clini, et al., 2020, p. 199). New digital enablements support human experiencing of various contents through visual, haptic, auditory, and combined sensory means. CIVITAS, defined as ChaIn for excellence of reflective Societies” (p. 196), has set for itself five challenges regarding digital cultural heritage:
• “Digitisation of Cultural Heritage at different scales (CH1),• Digital Content Management for 3D/4D semantic-aware models (CH2),• Enhancement of Visitor Experience and Social Inclusion (CH3),• Fruition by multisensory (visual, haptic, sound) Interaction paradigms (CH4),• Business models based on Digital Heritage for Culture, Research, Tourism, Reflective Society promotion (CH5)” (Clini, et al., 2020, p. 201).
The digitization of various artworks, monuments, buildings, and other architectures (p. 195) are captured with attention to details, so citizens and others may engage with the digital objects in multisensory ways (p. 196). The co-authors describe the 3D modeling of a historical building (by capturing various point clouds) requiring “fine-grained segmentation of possibly complex architectural elements” (Clini, et al., 2020, p. 197).
An important challenge is to make such digital renderings available across the entire populations of the world, regardless of varying abilities (given the context of longer human lives). They write:
The evoked reactions and positive impacts of art programs are influenced by different conditions, such as the severity of dementia, the specific cognitive domain impairments and the type of artwork shown. Accordingly, different types of visual art, such as representational, abstract, or conceptual art, may impact the effectiveness of a cultural program. Many studies, concerning the reproduction of artworks and art labs, have shown that creativity remains preserved or can emerge during the evolution of cognitive impairment… Recently, some studies have described different neural systems that are thought to contribute to the aesthetic experience: the sensory-motor system, the emotional valuation system, and the knowledge meaning system. It can be speculated that some attributes of art, such as color, shapes, natural elements, or complexity of the artwork, may impact the reactivity for particular types or severity of dementia. Moreover, even the type of the prevalent compromised cognitive function can require different access to cultural heritage and consequently a different response to stimulation… In this respect, Alzheimer’s Disease patients are characterized by a prevalent memory impairment while in Vascular Dementia attention and executive functions are specifically compromised. (Clini, et al., 2020, p. 198) (Note: Some citations of research sources were removed where the ellipses are placed for improved readability. This is not to create the impression that the research team did not do their due diligence in researching the relevant literature.)
What follows are detailed descriptions of lessons learned from their digitizing cultural heritage (DCH) work, while trying to advance the five challenges simultaneously. They describe the challenges of engaging the artifacts at varying levels of details, and also depicting changes over time (4D). They describe how they enriched digital explorations of buildings with “domain knowledge and links to the Web of Data” (Clini, et al., 2020, p. 205). They worked to understand how museum visitors experienced the various installations, such as monitoring their affect (p. 214). They used eye-tracking software to understand some interactions with digital visuals. They managed interrelated data and applied various statistical and machine learning to the data for analysis. They worked to develop new human-machine interfaces that may be built to heighten user engagement with multisensory digital-based objects (p. 211). For the fifth challenge, they integrated cultural tourism and other information to align the DCH efforts with market needs (p. 215). These efforts involved the whole of organization efforts and also meant that staff had to retrain and add more skills to their repertoires. They had to work in an interdisciplinary way.
Practically, the team observes that the effectiveness of a museums “digitalization path depends on the pursuit and constant monitoring of two levels of coherence: on the one hand that between the choices of technology adoption and the acquisition of objectives of emerging and growing tourism segments; on the other, between the digital skills possessed by (or accessible to) the museum and the functionalities of digital devices in use” (Clini, et al., 2020, p. 220).
Promoting Deep Learning with AR/VR
Figure 7: Head-Mounted Display (by Ricinator on Pixabay)
Roberto Pierdicca, et al.,’s “Evaluating Augmented and Virtual Reality in Education through a User-Centered Comparative Study: SmartMarca Project” (Ch. 12) describes a multi-use AR application around shared art with one user group as tourists and the other as learners. This work is also based on cultural heritage:
The app permits the augmentation of the artwork through tags that, deepening the critical contents of the paintings, propose a more attractive view both the details and the complete work. This project is mainly targeted for tourists, but with the aim to also use for didactic use that allows starting a first suggestion of knowledge of artworks in its fundamental contexts. These tools have been employed to undertake an educational authentication path of AR (Pierdicca, Frontoni, Puggioni, Malinverni, & Paolanti, 2020, p. 237).
Many of the AR annotations seem to be informational labels, so additive information to the artwork or the depicted building. The student learning from the viewable informational overlays based on particular associated markers was validated through a multiple-choice questions application.
Figure 8: Augmented Reality and Smart Phone (by geralt on Pixabay)
The researchers also launch their own additional research on didactics, such as by comparing the learning of one group in AR vs. another in VR (using head-mounted displays). They summarize their early and provisional findings as follows:
• “The AR system:
o has greater ability to transmit information and content and their relative acquisition;o determines less sensation of detachment from reality;o needs less concentration during use;o it is easier to use and less tiring to look at;o is recommended for use in other areas and not only for teaching purposes.
• The VR system:
o Less fatigue in the hand / arm system;o Creates more fun;o Can be proposed for further educational uses.” (Pierdicca, Frontoni, Puggioni, Malinverni, & Paolanti, 2020, p. 250)
More research in this area of educational AR and VR and MR and XR would be helpful.
Virtual Immersing in the Lost Artifacts of the Museum of WWII Stolen Artworks
The devastation of human warfare is ever a part of the mass human psyche. For generations after mass-scale conflict, humanity is still working to address some of that devastation. That truism comes to mind at the mention of the “Museum of Art Taken Hostage” (or MAIO) in Cassina de’ Pecchi in Milan, Italy. During WWII, some 1,623 masterpieces were stolen in Italy and considered “prisoners of war” (POWs) (p. 263) and never recovered. These included pieces by Michelangelo, Tiziano, Raffaello, and Canaletto. Is it possible to use digital technologies to remember some of the lost works? Perhaps so, according to Giuliana Geronimo and Salvatore Giannella’s “Employing Real-Time Game Technology for Immersive Experience (VR and Videogames) for all at MAIO Museum: Museum of WWII Stolen Artworks” (Ch. 13). Later in the work, there is a listing of some of the lost art, depicted in black-and-white:
Antonio Stradivari, VioletHistorical period: 1663Characteristics: bottom length 36,15Origin: Cantone (Cotignola, Ravenna), Villa Strocchi, Strocchi Collection…Stolen from Villa Strocchi in Cantone in the province of Ravenna in Emilia Romagna, at the end of January 1945 by the Nazi department marked with the initials V.E.I.T.R.Z. (Geronimo & Giannella, 2020, p. 272)
From 2016 – 2018, a team worked on this project. They began by updating the initial list drafted in the early post-war period. They reached out to the Carabinieri Headquarters for the Protection of Cultural Heritage for an updated list of found works of art. They also reached out to owners of missing artworks to verify the theft, including both museums and “ecclesiastical institutes.” The original 1,649 items from the original list was updated to 1,622. The team acquired permission from the owners of the missing masterpieces to use “high-resolution images of the works for educational purposes.” They had 27 in b/w (“masterpieces still prisoners of war”) and 5 in color that are identified as “found” masterpieces (p. 264).
Two installations were created. The MAIO Virtual Museum is based on VR in a dream-state “oniric 3D environment,” and the latter is a multiplayer video game named MAIO Play (Geronimo & Giannella, 2020, p. 263). MAIO PLAY is “a multiplayer exploratory video game (platform adventure) that can be played inside an arcade cabin in the museum before starting the following real visit to help visitors (sic) understanding and engaging the MAIO Museum mission and history” (p. 266). A video was made to showcase the masterpieces that have yet to be found, and viewers can use a touch screen station to “rotate the image, enlarge some details, to rework the image creating new, unpublished and personal fragments of memory. It is art that generates art” (p. 271). These indirect representations of the missing works bring attention to their absence and the continuing interest of authorities for their recovery. It is a sobering thought that for some of the artworks, their last existence in the world may be in virtual reality helmets.
There is another narrative here, of the power of human collective memory and the insistence on civilization and rules, even as Italy and Germany fought as Axis forces in WWII. Apparently, some 7,821 art works of Italy’s artistic heritage were saved by “seventy Superintendents” but 1,622 pieces of art “taken from museums and private collections by the Nazis never returned to Italy, including 800 paintings, dozens of sculptures, tapestries, carpets, furniture, musical instruments, including Stradivari violins and hundreds of manuscripts” (p. 267). The push to recover this lost art is not described as a sore point between present-day Italy and Germany. Included in this work are several lists of stolen artworks that have been recovered or restored and are on exhibit in Italy and abroad.
Immersing in the Mind of an Italian Luminary through VR
Virtual reality is used to laud Achille Castiglioni, the late Italian architect and designer, with a unique artistic mind. Cecilia Maria Bolognesi and Damiano Antonino Angelo Aiello’s “Through Achille Castiglioni’s Eyes: Two Immersive Virtual Experiences” (Ch. 14) describes two VR projects: the Studio Museo Achille Castiglioni depicted in 360o panoramic photographs of the actual former studio of the artist for a virtual tour of the museum on the web, and another experience enabled through a head-mounted device. The VR simulation is “based on digital surveying, digital modelling, and developing of virtual environments and interactions” (p. 283). Both works celebrate Castiglioni, his artistic vision, and his lifetime of works. The coauthors describe the firsthand steps taken to achieve both projects. (The explanatory diagrams were not easily readable because of pixellation, such as the “Flowchart of the steps for the creation of the 360 virtual tour.”) For the experience through a head-mounted display, the two conducted laser-scanning of the rooms, ran point cloud data processing, modeled the “architectural shell of the museum and of the design objects” and then created the virtual experience with a game engine (p. 290). Both projects required a Canon EOS 70D camera mounted on a tripod, a “panohead” and a “BLK 360 laser scanner” and various software (p. 290). The images were edited in Adobe Lightroom to “optimize contrasts and exposures” and then stitched on Stitcher 4. The virtual tour (using 3D Vista Virtual Tour Pro) moves the viewer fairly naturally through the depicted physical space. The explanations are methodical and precise, and the work is lushly illustrated.
Not only was the real-life studio recreated virtually, but the real physical objects in their real locations were represented in the virtualized objects (p. 300). The coauthors write:
Crossing the arches, the visitor reaches the mirror room, where he/she comes into contact with the first design objects created by Castiglioni: in the centre of the room there are the Leonardo table with the two Tric chairs and the large Sanluca armchair, flanked by the small Servomuto table, with the Noce floor lamp on it. The exhibition is completed by the Servomanto coat hanger, the Stylos floor lamp, the Mate coffee table and the iconic Arco lamp, which draws attention with its elegant curved profile. A mention should also be made to the large mirror (located behind the armchair) which is perhaps the hallmark of the room and gives it is name. During the exploration of this environment, the visitor has the opportunity to manipulate various objects… (Bolognesi & Aiello, 2020, p. 300)
The immersive experiences are a tribute to the designer, based on the physical real. Bolognesi and Aiello note that attentions to details are critical to create this type of virtual reality. They write:
The researchers are aware that it is precisely the smallest details that give credibility to a VR experience; these elements (only apparently secondary) have been inserted in order to give life to the place, to have it accepted by the visitor and to make the emotional involvement more intense and profound (p. 306)
The work is informed by a deep knowledge of Achille Castiglioni, his creative work, and his contributions in architecture and design. The various surfaces—glass, metal, leather, and others—were depicted with high visual fidelity for the material. Some scenes show a virtual camera at human height and others from near floor-level shooting up. In one room, there are “openable” drawers of objects. Even audio is harnessed for effect:
A final mention concerns some small details, which the most attentive visitor can notice during the virtual experience. It is the case, for example, of a peculiar whistle that can be heard from time to time during the visit. It is an original recording of Achille Castiglioni’s voice, which echoes through the rooms almost as if he were the soul of the museum, which continues to live through is works (Bolognesi & Aiello, 2020, p. 305)
This sounds like it offers something of telepresence, whistling as a personal touch. This reminds one of the hard work of creating a body of creative works for posterity.
Depicting Hindu Transmigration and Immersive Museums
Just the title alone of Patrizia Schettino’s “Where is Hanuman? Hindi Mythology, Transmigration, and the Design Process of Immersive Experiences in Museums” (Ch. 15) brings up questions. How would a museum curator handle representations of Hindu god (and devotee of Rama)? How can religious deities be handled to a diverse audience, some worshipers and others not read into the same understanding and the same loyalty? How can varying truths be managed without offending one or another? How can one group’s identity be affirmed without alienating another? Do deities have to be reified or debunked, or neither? In a complex world with different belief systems, how should an Indian Hindu deity be animated in 3D to an international audience, and how can panoramas of historical sites be depicted? By persons outside of the direct culture (and well outside the time period)? For the international team building PLACE Hampi, based on an archaeological site in India, the immersive virtual environment includes images of Hindu deities and the Ramayana. To consider how they might depict the deities and other aspects of the space, they draw on concepts and practices from iconology, hermeneutics, design research, and museum studies (p. 311), around which they engage in-depth reflection of the design choices and decisions. Another challenge: they had to migrate images from an Indian to a non-Indian context. There were technological challenges as well, with full audio immersion and stereoscopic vision (based on 3D glasses and 3D graphic animations).
Schettino’s work involves a multi-methods research project about how the PLACE Hampi designers evolved this project. She collected data from various sources, conducted grounded theory-based analyses, coded the data in NVivo, and came to some detailed and complex insights. This work conveys the complexity of AR / VR design decisions in a real world case.
Interpretive AR with Newly Commissioned Creative Works
Biancamaria Mori and Carlo Gioventù’s “An Augmented Reality (AR) Experience for Lorenzo Lotto” (Ch. 16) describes the uses of augmented reality to make museum experiences more inclusive to a broader audience. Those who tend to be somewhat excluded include children, the elderly, and the visually impaired (p. 326). Based on several works by a 16th century artist, Lorenzo Lotto, a team created derived (inspired) works, including choreographies to a commissioned original soundtrack and enabled on Android tablets through an AR application. One painting was about Saint Lucia over a series of events, and this is depicted in AR with the principal telling her story to the museum goer; the actor uses “a language congruent with the age of the saint: a 16-year-old girl who naturally tells her story, resolving herself into a rebellion of a teenager in the face of arrogance” (p. 327).
Another work provides “an emotional reading of the picture of the Deposition of the Lot” (La Deposizione di Cristo) based on “the first preparatory pencil” of the artist Lotto. The augmented reality experience enables the visitor to see the initial draft as compared to the final work (p. 328).
The coauthors caption a marked overlay of the work:
To realize this part of the App MenteZero wanted to design an experience based on what was assumed to have been the original narrative choice of Lotto, the pain of the dumb and terrible characters, told through their incommunicability, or the reaction to the dismay of end of the Christ with the blind desperation of taking refuge in oneself, a multimedia narration was then carried out animating the parts of the picture and creating a soundtrack made only of deaf and compound laments, in order to create an emotional state able to involve the visitor in a participatory way to the picture he is observing (Mori & Gioventù, 2020, p. 329).
Then, there is a third work, a physical triptych, depicting St. John in Patmos, the archangel Gabriel, and the virgin. (The description of this work is not sufficiently clear to visualize.) The project described in this chapter shows artists and designers inspired by historical artworks in multimodal ways to co-create new augmented reality works that bear their own artfulness.
Some Light Musings about Digital Reality (DR) Design and Enablements
Figure 9: Weave with Forms
With augmented reality, are there ways to use AR to mask (to both conceal and reveal), like a white noise application? Or is it always additive, something of an overlay? Can the physical real augmented AR in an inverse of the current state? What are some of the more complex technical aspects of the various technologies, the truly bleeding edge aspects? What are actual applications for those with varying abilities, so they can experience the education, the art, and the museum aspects thoroughly and aesthetically and creatively? What is happening with AR in other spaces that can lead to new practices in the creative space? Are there ways to capture the more ephemeral aspects of AR to make it more permanent?
Virtual realities (VRs) have been more commonplace and accessible through immersive virtual worlds and digital gameplay. What are some of the extant technical challenges here, and what are some of the more cutting edge uses? Are there new models for the deployment of VR beyond hanging out, socializing, engaging through avatars, building, and competitive first-person shooter play? (I don’t mean how snarky that prior sentence might sound.) Are there ways to more closely emulate labs (Have in-world physics and modeling of materiality improved?)?
Different generations have different ways of engaging issues in education, art, and museums, and it would be interesting to assess what these differences mean.
Conclusion
Figure 10: Residua
In this time of the COVID-19 pandemic, going virtual may seem appropriate for safe social distancing for education, art galleries, and museums. And yet, the technologies the masses have turned to are learning management systems (LMSes), social media, web conferencing tools, and not so much AR/VR/MR. In some ways, developing “digital reality” requires expertise, resources, and time…to design, to set up, to map, to draw, to render, and to deploy. The consumer technologies—the head-mounted displays, the monitors, and others—are not inexpensive for most venues. And the current (and continuing) biosafety issues make wearable devices difficult to share safely without potentially passing along pathogens to users. Such projects cannot be actualized on a dime.
For an on-campus project that this book reviewer has been working on, it is prohibitively expensive to do a photogrammetry capture of an art object—with costs in the five figures and needed to be done external to the university. The mapping of a 3D space (a building on campus) for an AR interaction via the Web cost well in the six figures on a separate other project. The technologies used to consume the AR contents are not low-cost either, with many available on a rental basis only, and many of the higher end devices running into shortages (just prior to the outbreak in early 2020).
The cost-benefit calculations for whether or how to bring on “digital reality” varies. There may be the benefit of enabling rich experiences for learners and visitors to museums and art galleries. However, here is often consideration of potential long-term benefits. There may be positive word-of-mouth, positive press, bragging rights, and mentions in academic publications, and maybe some website residua. What is taken up depends on what patrons and financial contributors will support. In some cases, grant funding from government entities and private foundations may be available. Also, coding to the various devices may be challenging. Intellectual property rights have to be negotiated when making copies of the respective objects. There are a tangle of concerns.
Giuliana Guazzaroni and Anitha Pillai’s Virtual and Augmented Reality in Education, Art, and Museums (2020) enables readers to both dream about possibilities and to start to plan to actualize the work. These chapters communicate a sense of practical aspirations and high-minded possibilities. The respective works bracket low- and high-end instantiations of digital reality…made real.
About the Author
Shalin Hai-Jew works as an instructional designer / researcher at Kansas State University. Her email is shalin@ksu.edu.
Thanks to IGI Global for a watermarked review copy of the text.
| Previous page on path | Cover, page 19 of 21 | Next page on path |
Discussion of "Book review: Digital reality in education, art galleries, and museums (with AR, VR, MR)"
Add your voice to this discussion.
Checking your signed in status ...