IDENTIFYING CUSTOMER’S EMOTIONAL RESPONSES TOWARDS GUEST-ROOM DESIGN BY USING FACIAL EXPRESSION RECOGNITION, IN HOTEL’S VIRTUAL AND REAL ENVIRONMENTS

It is well known that emotions play a key role in the customer experience in tourism. Measuring emotions can provide valuable information about customer’s perceptions regarding hotel spaces. The purpose of this study is to identify which type of hotel environment, Virtual or Real, is more effective in eliciting emotional responses from participants who are shown a scenario for the first time. Furthermore, this study aims to uncover which of the components of guestrooms, e.g., natural views to the outside or interiors are capable of triggering higher emotional responses. Two experiments were conducted to provide evidence, the first presented a Virtual Environment to participants via video, and while the second one was conducted in Real Environments of presidential suites. Emotional Iván Alvarez León, Begoña Juliá Nehme and So Yeon Yoon: Identifying Customer’s Emotional Responses Towards Guest-Room Design By Using Facial Expression Recognition, In Hotel’s Virtual And Real Environments 105 responses were analyzed using FaceReader software, a facial expression recognition system that identifies seven emotions, physiological arousal and pleasure. Results showed low overall mean intensity values of each emotion in both virtual and real environments. Nevertheless, significant differences in the maximum peak intensity values were found between virtual and real environments with intensity values being higher in the real guestroom. No significant differences were found in emotional responses to the views to the outside or views of the guestroom interiors.


INTRODUCTION
Literature in environmental design on hotel guestrooms has been generally studied based on functional and planning programming concepts. To study how customers related to the design of guestrooms, Robson (2007) analyzed pictures of guestrooms taken by customers. The premise was that customers take pictures of particular aspects of the room, and publish what they consider positive and valuable or negative (Pullman & Robson, 2007). Other authors used pictures taken by travelers as method to analyze the relation between human behavior and cultures (Collier & Collier, 1986), aspects of ethnicities in travel's context (Albers & James, 1988) or spatial and temporal dimension of a nature-based experience .
Recent studies have contributed to understand how important is emotional design for the quality of customer's experience in hotels (Lo, 2007). The concept of emotional design has been growing rapidly in different disciplines in order to improve quality of experience since researchers started to emphasize in the context of product design and compute interface design (Norman, 2004;Demirbilek & Sener, 2003).
Traditional research focused on customer satisfaction is fairly extensive. There is no evidence that emotions strongly impact on customer satisfaction, loyalty behaviors and willingness to pay more (Mattila & Enz, 2002;Barsky & Nash, 2002;Pullman & Gross, 2004). Despite of the influence emotions seem to play in fostering customer satisfaction, design decision makers in the hospitality industry have only recently become interested in the subject.
The researcher's interest is to identify aspects of environmental design that are capable of eliciting an emotional experience. How might customer's emotional responses be used to inform better hotel design? For decades, researchers have applied different methods on humans with the intention of getting an accurate understanding of people's emotions and behavior in a daily routine (i.e. eye tracking; EEG; EMG; facial expression recognition). This field of research is known as emotion reactivity (Balters & Steinert, 2015). On primary research topic areas is to determine which research methods are most in measuring emotional reaction towards space.
The facial expression recognition method has been studied and practiced for over forty years as a way to gather information about emotional responses to an event (Terzis, Moridis, & Economides, 2011). Paul Ekman, one of the pioneers in facial coding and emotions, published the Facial Affect Scoring Technique (FAST) in 1971. FAST identifies six basic emotions: Happy, Surprised, Sad, Fear, Angry and Disgusted (Ekman, Friesen, & Tomkins, 1971). According to Ekman's findings, emotional reactions in humans occur mainly in social events that are real, remembered, anticipated or imagined (Ekman, Levenson, & Friesen, 1983). When emotions are elicited because of environmental conditions, the first emotional response is considered as an innate reaction. Then, we scan and evaluate the environment consciously or unconsciously, in order to trigger an emotional response only with selected stimuli, previously scanned (Ekman, 2003;Ellsworth & Scherer, 2003;Frijda, Kuipers, & ter Schure, 1989;Roseman, 1984). On the other hand, according to Russell's Circumflex Model of Affect (Russell, 1980) emotions are organized in different affective states in a two axis circular model, in which the valence of pleasure (pleasent/unpleasent) is plotted against the arousal (active/inactive).
Based on this background, emotional analysis through facial expression recognition seems to be a proper and direct method to identify immediate emotional response towards a new stimulus. However, after a thorough review of the literature on customer emotions in Tourism did not reveal any studies applying the facial expression recognition method.
The most frequent methodologies used to study customer emotion are interviews (face to face or by phone), observations (Mattila & Enz, 2002), and surveys (selfreports in the hotel or online surveys). There is great variation in the types of emotions chosen to be studied and the literature can be unclear about what types of emotions an environment elicits in customers.
For instance, Pullman analyzed two types of emotion categories in a tourism environment: Basic and VIP emotions (Pullman & Gross, 2004). The first category measures positive reactions (pleasurearousal) based on PANAS scale (Watson et al., 1988) and Mano's domains (1991) (comfort, happy, satisfied, entertained, relaxed and amused).
The second one measures special feelings from Barksy and Nash's scale for luxury hotels (2002) and Hirschman and Holbrook's work (1982) (sophisticated, privileged, inspired, important, part of show, hip or cool and curious)..Another approach has been to study how emotional responses influence the amount of time spent in an environment. Mehrabian and Russell (1974) highlight that emotional responses in environments could be pleasurable or displeasurable.
In addition, the authors found that people want to spend time and money in those environments that elicits a high valence of pleasure. This research analyzes immediate and automatic emotional response through facial expression recognition. This method, described below, provides valuable information in understanding natural and direct reactions towards different stimuli in the environment.

LITERATURE REVIEW Emotional assessment in virtual and real environments
Many researchers firmly believe that studies on human emotion should record spontaneous facial expressions and therefore should be performed in the field rather than in laboratories (Hwang, & Matsumoto, 2015).
Virtual Environment (VE) or movie clips are unable to reproduce all the stimuli experienced in real life, like noise, temperature, crowding, and lighting, which could affect the emotional response in a particular context. However, Virtual Environments (VE) presented in labs are relatively easy to set up while having the advantage of providing many options to control and vary different aspects of the visual stimulus. Kendler (2008) showed a short movie clip as a method to understand the influence of genetic in emotional facial expressions (Kendler, et al., 2008). More recently, others studies use a short movie clips (in this case by a 3D virtual environment -V.E.-) to study the empathy of populations by analyzing automatic recognition of facial expressions of emotions (Jackson et al., 2015).
Both real and virtual environments have pros and cons, with specific advantages and potentials for research. Iván Alvarez León, Begoña Juliá Nehme and So Yeon Yoon: Identifying Customer's Emotional Responses Towards Guest-Room Design By Using Facial Expression Recognition, In Hotel's Virtual And Real Environments To date, assessments of emotional responses through automatic facial expression recognition in VE and movie clips have been applied especially in the Publicity field. In the real environment, there is evidence of one study, which analyzed spontaneous emotions by using an automatic recognition of facial expressions of emotions in participant exposed to several tasks as stimuli (Grootjen et al., 2007). Nevertheless, few studies have done in tourism to evaluate automatic emotional assessments in real physical environments.
The goal of the present study therefore, is to evaluate the effectiveness and potential of applying automatic facial expression recognition technology in both Virtual and Real Environments of hotel spaces. The information will help identify the immediate emotional responses of customers towards determined stimuli in the environment.

RESEARCH METHODS
The study was divided into two experiments, performed at different times and places. Experiment (I) was conducted in a virtual environment (a short movie clip), and experiment (II) was conducted in a real, physical environment.
Research shows that for hotels, guestrooms generate the highest potential for developing client loyalty (Dube & Renaghan, 1999).
A customer's emotional response triggered from a guestroom environment directly affects the customer's experience and emotions, and ultimately these things affect a customer's loyalty. Continuing Dube findings, the guest-rooms of hotels were the environments selected for both experiments.
In the present study researcher proposed that human emotional responses may vary between exposure to virtual environments (V.E.) and real environments (R.E.).
The study strives to uncover the impact of natural and artificial stimuli in shaping the customer experience inside a guestroom. As "natural stimulus" was represented as a view to the outside from within the built environment (guestrooms) on to the natural landscape.
The views of the natural or built landscape in touristic environments has always2014 been considered a major element in the design of hotels (Pie & Barba, 1996;Cappai, 2014;Gausa, 1996).
On the other hand, artificial stimuli refer to the complete interior design of the guestroom, such as, the furniture layout and environmental conditions. Based on the premises above, a twofold hypothesis is proposed:  H1. Humans have a higher emotional response in R.E. than in V.E.  H2. In R.E. the views to the outside trigger a higher emotional response than the guestroom interiors.
The analytic method to process the video registers of participant's facial expressions was the same in both experiments, (I) and (II), using FaceReader 6 software (Fig 1).
Nevertheless, the procedure for each experiment and the environments selected were unequal. Thirty participants were recruited from a convenience sample of the researcher's personal networks. Participants were invited to participant in the study through email, phone calls in person. To assure a diverse sample of participants, people from three different countries were selected (Chile, Italy and Spain). Diversity in gender and range of age were also considered. Of the participants selected, 8 were female and 12 were male. The average age was 33.4 years old with the majority of participants (56.6%) between ages of 26-35 years old. None of the participants had seen the images showed in the video before.

Virtual Environment Stimuli
In this experiment, an emotional response was triggered in participants primarily through the visual stimulus of a "virtual tour". The environment was presented in a video for 7 minutes. All video images were dynamic and showed a human visual perspective of the space. Background music was added.
The movie showed a virtual tour of five presidential suites from five different hotels. Each space was chosen for its' specific contrasting features, e.g., colors, lighting, furniture styles, spatial perception inside (Fig 2), and views to the outside environments (Fig 3).
For each guestroom, the video contained images of areas including a bathroom, bed, desk area, living room and terrace or balcony. The video also presented images of views to the outside, for instance, a city close to the beach's hotel, a beach far away from the beach's hotel, a seabed view (aquarium), and surrounding water in a virgin environment.

. Set up
The experiment was conducted in different settings with the following three main conditions: a video display, a camera to register participant's face in a front clear view, and proper lighting conditions. The web-camera enabled computer was used to view the video display. The participant's facial expressions were recorded in mov or mp4 video format with a minimum resolution of 640x480 pixels. Lighting conditions were explained before the experiment began, highlighting the importance of having a high level of light, and avoiding shadows over the face. The video with the stimulus was installed in each computer used for the experiment. Using a flexible setting allows researcher to benefit from the advantages of the V.E. For example, participants were selected from different global locations. Bringing the experiment to the place were participants they are may be desirable for researchers, in order to achieve a wider sample of subjects with lower costs.

Experiment (II): Real physical environments 2.2.1. Participants
Thirty-one participants were recruited from the researcher's personal network plus two real customers. As well as in Experiment (I), participants were invited to collaborate through email, phone calls and personally. For the sample it was also consider a cross-cultural perspective, selecting people from seven different countries, i.e., Spain, Italy, Chile, Peru, Belgium, Canada and Argentina, considering diversity in gender and range of age. Experiment (II) counted with 15 female participants and 16 male participants. The average age was 37 years old with a majority of 60% was between the ages 26-35. None of the participants has been in the suite selected for the experiment before.

Real Environment and Stimuli
All participants walked around in a physical presidential suite, in order to appreciate the design features of the space and the views to the outside. During the experiment, participants were able to scan and evaluate the environment by the body movement and senses of touching, smelling, hearing and seeing. They were free to take their time, move around and interact with the components of the guestroom if they want to. In this moving around all the emotional responses triggered from real stimuli were recorded.
The real environment presented two types of stimulus: natural and artificial. The natural stimulus was represented by the views to the outside of each guestroom ( Fig  5). The stimulus varied according to the location of the hotel, therefore, it was possible to find different types of outside environments, including gardens and mountains, a parking area and mountains, an urban area and a beach, a pool area and the sea facing the guestroom.
Artificial stimuli are represented as the guestroom interiors (Fig 4). It considers all the elements located in different functional areas, and the environmental conditions such as colors, lighting, sound and noise, odors, temperature. The functional areas and the outside environments of each guestroom are described below (Table 1).

Set up
The experiment was performed in six presidential suites from four hotels, two of them located in Canary Islands, Spain, a third one located in Sardinia, Italy, and the last one located in Ithaca, USA. The experiment was performed in a total of six guestrooms from the hotels mentioned (two in the Canary Islands, one in Sardinia and three in Ithaca, New York). Each participant walked around in one suite in one of the selected hotels.
In order to video record the facial expressions of the participants, a specially prepared bicycle helmet with a camera pointing the participant's face was used. The camera used was GoPro Hero3 Silver. It was supported by a flexible arm, which is attached to the helmet. In addition, the helmet can be adjusted to people of different head sizes, and the angle of the camera can be adjusted as well, in order to capture the facial expressions of the participant and avoid interrupting his/her visual field at the same time. The helmet was manufactured by the researcher in a laboratory, where they studied comfort and proper angles for recording facial expressions during the experiments. The video recordings of the participants face were in mov and mp4 video formats, with a minimum resolution of 640x480 pixels, to be compatible with FaceReader software. Hypothesis 1 establishes that humans have a higher emotional response in R.E. than in V.E. Two types of analysis were performed with the purpose of proving this hypothesis.

First Analysis
Repeated measures ANOVA were used to compare mean values of R.E. and V.E. groups, with the objective of comparing results from the overall emotional experience during the experiment. The dependent variables were the intensities of the six emotions (happy, surprised, sad, angry, scared, disgusted and contempt), arousal level and valence. Emotional intensities and arousal have values between 0 and 1. Valence can be positive or negative, from -1 to +1. Gender was considered as an independent variable.
According to the values of mean summarized in Table 3, the intensities of 'happy', 'angry', 'surprised' and 'disgusted' in R.E. are higher than V.E. For this analysis it is also relevant to notice that intensity values are low in both environments, which mean that participants were mainly showing a neutral expression during the experiment, and maximum peak values were diluted between the lower intensities (Table 2).
In the real environment, the gender variable only had significant difference for sadness (F (1,29) = 13.713, p < 0.05) where females elicited a higher emotional response than male. In the virtual environment, anger is the only emotion with a significant difference in gender (F (1,28) = 5.414, p < 0.05), where the emotional intensity triggered in males is higher than in females (Fig 6).

Second Analysis
Complementary, for the second analysis the maximum peak intensity of each emotion during the experiment (happy, surprised, sad, angry, scared, disgusted and contempt) was selected, and the data was compared between participants in RE and VE. The purpose of this test is to provide evidence about the capability of each environment, virtual and real, to elicit emotional responses.
For all the emotions mentioned, the highest value of maximum level of intensity was found in Real Environment. Furthermore, the high values reached by positive and negative emotions (happy, surprised and anger) demonstrate that people do emotionally react to the components of a real environment, and the methodology presented is effective.

Hypothesis 2
Hypothesis 2 proposed that views to the outside would trigger a higher emotional response in participants than would the guestroom interiors.
To prove this hypothesis, only actual environmental data was included. The output of each participant was divided in two groups, according to the type of stimulus that the participant was observing: views to the outside and guestroom interiors.
In order to obtain evidences in Hypothesis 2, a repeated One-Way ANOVA analysis was conducted between both groups, considering the intensities of each emotion (happy, surprised, sad, angry, scared, disgusted and contempt), arousal level and valence as dependent variables. Emotional intensities and arousal have values between 0 and 1. Valence can be positive or negative, from -1 to +1.
Gender and number of people, whether the participant was alone or accompanied during the experiment, were considered independent variables.
The ANOVA analysis of emotion intensities showed that only 'Scared' presents significant differences (F (1,60) = 4.29, p < 0.05) between both groups. Despite of the significance found, it is important to notice the intensity values, which are very low in both cases. The higher and minor values correspond to views to the outside with 0.12 'Happiness' and 0.01 'Scare', respectively (Table 5). Also, in both cases standard deviation (SD) is higher than the intensity value, therefore, results in Scared could be due to a subtle facial bias.
The gender analysis showed that in the interior of the guestroom, the only emotion that presented significant differences were 'Sadness' (F (1,29) = 12.49, p < 0.01) and also for the stimulus "views to outside" (F (1,29) = 5.98, p < 0.05). In both cases females triggered a higher emotional intensity than males.
ANOVA analysis for arousal did not show any significant difference between the stimuli "views to the outside" and the interior of the guestroom. Nevertheless, researcher consider relevant for the study to know that participants experienced similar level of arousal in both conditions, being 0.40 for the views to the outside and 0.40 for the guestroom interiors. Arousal is expressed through the face level of facial activation, therefore, it can be said that participants were equally motivated by both natural and artificial stimuli. This fact could be a key guidance for Architects and Interior Designers, in order to create hotel spaces giving same importance to the access to the view and interior design. Valence values didn't show statistical significance between "views to the outside" and the guestroom interiors.
The number of people participating in the study was also analyzed in the R.E. to find out how that variable could impact participants' emotional responses. Participants were divided into two groups: walking alone and walking with a partner. The results showed that there was not a significant difference between emotional intensities, arousal and valence values. Means of emotional intensities and arousal levels between the factors of 'alone' and 'in a couple' were primarily equal but the intensity in some emotions appeared to be slightly higher in 'couple' (Table 6). This result is highly valued by the researcher, in order to refine and improve the methodology used in the present study, which is first of its kind.   Table 6. Intensity of facial expressions and level of arousal alone and in couple (R.E.)

Discussion
The study shows that humans do emotionally respond towards spatial conditions when they see them for the first time, and this response can be measured by automatic facial expression recognition.
Nevertheless, emotional responses are not shown all the time and participants perceive the environment mostly with neutral expressions. Consequently, overall mean values of emotional intensities during the complete experiment in V.E. and R.E. were very low.
Analysis of maximum peak intensities provided better quality information about participant's experience. Values demonstrate that participants are stimulated by certain aspects of the environment triggering punctual high intensity emotions, especially in R.E where participants interact directly with stimuli. This occurs for positive and negative emotions: Happy, Surprised, Angry and Disgusted presented statistical significant difference between R.E. and V.E. Higher values are found in R.E. with maximum intensity levels over 0.8.
The variety of emotions that showed high intensity values indicate that the results were not due to facial bias, and that both positive and negative stimuli can be evaluated by facial expression response. Furthermore, significant differences were found in arousal values, showing that participants are more in R.E.
Maximum emotion intensity was considered the most valuable output for the objectives of the study. Meanwhile, valence was not a meaningful measurement when emotional intensities are too low.
About emotional responses towards natural or artificial stimuli, it is known that nature has restorative properties, which help people to recover from direct attention fatigue (Pals, Steg, Siero, van der Zee, 2009), and views to outside are highly valued in hospitality. On the other hand, the design of hotel guestroom interiors conveys the quality of the hotel and its' style. The elements within guestroom interiors need to fulfill customer's expectations, while also providing comfort and satisfaction. Despite being different in nature, results showed no significant difference between them, neither in emotion intensities, valence or arousal. Because this study did not analyze the components of the individual guestroom interiors, future research may evaluate emotional responses towards each environment. Additionally, a second camera recording the visual field of the participant could provide complementary data about the specific stimulus that is triggering a certain emotional response. The methodology could be complemented by an eye-tracker system. Results of further studies would provide architects and interior designers with more detailed evidence about customer reactions towards spatial specific elements.
There was no main effect of gender on the level of facial expressions recorded in V.E. or R.E. This study concluded that gender did not influence emotional responses to guestroom interior or views to the outside.
Participants from a total of seven countries took part in the study. Due to the drive to find a universal pattern in the expression of human emotions, crosscultural emotional variables necessitate further exploration and understanding. Since 1969, Ekman's findings have defended the existence of six basic universal emotions, and the ability to interpret these emotions through universal facial expressions (Ekman, Sorenson & Friesen, 1969;Ekman, 1994). Many researchers have applied Ekman's principles in their studies since then, with the purpose of collecting cross-cultural evidence in different contexts. In general, the hospitality industry strives to bring memorable customer experience beyond satisfaction from all over the world, therefore, studies including cross-cultural factors may become increasingly important for future studies in the field. There is a heightened awareness among researchers that future studies should consider expanding their research to engage participants from other cultural backgrounds for greater scope of knowledge.
There was no significant difference between the recorded emotional intensities of participants who fulfilled the experiment alone or were accompanied by another. Nevertheless, it is necessary to conduct more tests to evaluate this particular issue. In future research, asking Participants who are in couples to restraint verbal communication while doing the experiment in couples, may be a way to find out whether familiar company can help in the elicitation of facial expressions.
It is possible that different expectations also influence emotional responses. In this study, the participants recruited were not on vacation nor were they real customers of the hotels. Being careful to maintaining the novelty of the experience (participants must not have been in the space tested before), it would be useful in future research to recruit real customers, even if it takes longer time to collect the data. This approach could lead to collecting more accurate and spontaneous emotional responses.
Future studies could consider designed interventions in hotel spaces as a novel stimulus for customers. The goal would be to quantitatively analyze how adding new features enriches the customer experience. Interventions could include technology, interactive platforms, natural interior spots, art, music, and lighting, among others.

Limitations
It is important to consider FaceReader 6 limitations. The pose, movement and rotation of the participant are limited. The participant needs to look straight into the camera (at an angle <40°). The face must not be partially hidden by hair, facial hair or glasses.
Lighting conditions must also be appropriate. A medium to high level and uniform lighting setting is essential in obtaining a high quality image. Diffuse frontal lighting is most desirable as it is optimal to avoid strong shadows or reflections over the face. While it is simple to control lighting conditions in V.E. tests, controlling these aspects of the R. E. presents greater difficulties. All curtains in the guestrooms were opened widely before the experiment. However, areas far from the windows were darker, causing obscure video images and thus, losing face recognition in FaceReader. All videos from the R.E. lost a percentage of data due to this condition. A complementary lighting system should be implemented in the helmet to illuminate the face, taking special care in not to create glare effect on the participant eyes. Besides generating visual discomfort, glare can trigger a reactive facial expression that does not express any emotion, which can influence results.
Valence value was calculated by subtracting the highest negative emotion (sad, angry, scared, disgusted) to the only positive emotion, Happy. According to FaceReader, Surprised can be positive or negative depending on the context, therefore, it is not considered in the equation. According to studies in tourism, 'surprised' is related to the verbal expression 'wow' (Lo, 2007), thus, positive. While this study used valence value calculated according to the FaceReader equation, it may be interesting to personally customize the calculus of Valence to evaluate the highest value between the positive emotions, 'happy' and 'surprised'. This configuration could be used for studies in which there is evidence of the positive condition, 'surprised'.

CONCLUSIONS
Currently there are several methodologies applied in hospitality industry to collect information about a customer's preferences and satisfaction during their stay. However, there are few studies developed to evidence of studies developed to understand customer's immediate response towards spatial environments in hotels. The present study provides evidence of the effectiveness of applying automatic facial expression recognition methodology by FaceReader 6, in both real and virtual hotel environments. The R.E. proved to be the best context for conducting evaluations of hotel environments, eliciting higher emotional responses in participants. The methodology proposed provides a direct way to identify and analyze the immediate emotional response triggered by the stimuli present in a particular environment, in order to understand a customer's reaction towards specific hotel environments.
In the R.E, guestroom interiors and the views to the outside landscapes elicited the same level of emotional response in participants. Design decision makers should take this fact into consideration, and implement design components to provide comfort when customers interact with both stimuli.
Involving emotions in the design of hotel spaces may enrich the experience of customers, and consequently, increase satisfaction and loyalty. Further studies will aim to provide more detail about the emotional response elicited by different stimuli. The methodology has the potential of being applied in other contexts including hospital, airports and museums.