Leveraging Multimodal Interaction and Adaptive Interfaces for Location-based Augmented Reality Islamic Tourism Application

— A Location-based Augmented Reality (LBAR) application leveraging multimodal interaction and adaptive interface based on Islamic tourism information was proposed to enhance user experience while travelling. LBAR has the potential to improve tourist experience and help tourists to access relevant information, thus improving their knowledge regarding touristic destination while increasing levels of their entertainment throughout the process. In LBAR application, Point of Interest (POI) displayed are exposed to the “occlusion problem” where the AR contents are visually redundant and overlapping with one another causing the users to loose valuable information. Previous research have suggested the design of AR POI which help user to see the augmented POI clearly. The user can click on the desired POI but it still displays a large amount of POI. From our best study, there is limitation of research studying on how to minimize the amount of displayed POI based on user’s current needs. Therefore, in this paper we suggest to use an adaptive user interface and multimodal interaction to solve this problem. We discussed the process of analysing and designing the user interfaces of previous studies. The proposed mobile solution was presented by explaining the application contents, the combination of adaptive multimodal inputs, system’s flow chart and multimodal task definition. Then the user evaluation was conducted to measure the level of satisfaction in terms of the usability of the application. A total of 24 Islamic tourists have participated in this study. The findings revealed that the average SUS score of 75.83 of respondents agree in terms of satisfaction of the LBAR application to be utilized while traveling. Finally, we conclude this paper by providing the suggestion of future works


I. INTRODUCTION
Tourists need meaningful experience and want to find something different that can help them to expand their knowledge Tourism information usually contains brochures, manuals, maps, interpretations etc. [1]. However, this kind of information exposure has been regarded as traditional information exposure. The method of delivering the traditional information is now inadequate for tourists. Augmented reality (AR) has the potential to improve the tourist experience and help tourists to access relevant information, thus improving their knowledge regarding their touristic destination, while increasing levels of user's entertainment throughout the process [2]. AR content presentation is important in enhancing tourist experience [1], [3]. Location-based Augmented Reality (LBAR) is a technology that can augments virtual object such as GPS location data (e.g. point of interest), text, audio and 3D model on top of the real world. User position relative to the real world will be located and digital information will be overlaid to provide information on the user's current location and surroundings [4]. However, there are some limitations on how the point-of-interests (POI) are displayed in LBAR. LBAR is exposed to the "occlusion problem" where the overlaid AR contents may be visually redundant and overlapping with one another causing users to loose valuable information [5]. AR content should be displayed based on tourism context because tourists sometimes are struggling to explore their current environment. Obviously, they do not have any idea of which information is more important to be explored. Focusing on the Islamic tourism sector, an adaptive location-based AR application with multimodal interaction can be used to help and guide Muslim tourists to find locations that are close to their current location. Therefore, in this paper, we proposed a mobile location based augmented reality tourism application by leveraging multimodal and adaptive interaction to cater the limitations. It is because interaction using multiple modes can improve the effectiveness in delivering information to the system.
The paper started with the discussion an overview of location-based augmented reality in tourism and explaining the concept of adaptive and multimodal interaction. The paper then proposed the mobile solution and followed by system evaluation. Finally, we concluded this paper by providing idea for future works.

A. Information Technology Implementation in Tourism
Tourism is one of the largest economic contributors in Malaysia since 1960's. Therefore, it is not surprising for Malaysia to provide a great deal of effort in promoting the concept of "Malaysia, Truly Asia" around the world with interesting places in Malaysia. It shows that the role of technology in the field of tourism is paramount to provide the visitors with the opportunity to attract and concentrate on their traveling activities including applications that can help and facilitate them to plan their travels at a destination. There are various work have been done to facilitate the tourist to obtain a meaningful tourism information such as culture places [1], [5], language translator [6], halal product [7] , etc. Technology has indeed played a major role in the tourism world, making technology as one of the mediums for them to continue to grow, especially the developments in the world of economy. Most of today's ongoing tourism work has focused on the formulation of policy, though the challenge lies in the practical search of travel experts to turn theory into an action [8]. [9] has also emphasized on the sustainable tourism innovation and concentrating on the perspective of destination management and technology effectiveness, reduce or increase the impact on tourism. These efforts aims to advance research on tourism innovation and demonstrations on the use of new technologies. In Malaysia, the growth of tourism industry became faster and wider parallel with the advancement of the technologies. With that concern, Ministry of Tourism has established Islamic Tourism Centre (ITC) on February 2009 to support and focus on developing the strategies of Islamic tourism in Malaysia [10]. Based on [11], they have stated that Malaysia, Turkey and the UAE are the most visited leisure destinations of an Islamic character while other popular choices are Singapore, Russia, China, France, Thailand, and Italy. In Islamic history, to learn more about Islamic thought and civilization, Prophet Muhammad has encouraged people to travel from one place to another to spread the Islamic teaching, appreciate, and educate person [10]. ITC has defined Islamic tourism as [12]: "Any activity, event and experience undertaken in a state of travel that is in accordance with Islam".
From previous study, it shows that the information about Islamic tourism is also important to be visualized to facilitate tourist while travelling. It is because Muslim traveller are increasing in volume and value and encourage the interest of the tourism industry worldwide. Pervious works in Islamic tourism have been done which focused on halal food information, finding mosque and interesting Islamic places. For example, "Halal Trip" and "Muslim pro" applications that help Muslim tourist to find Halal-friendly hospitality products and services such as hotels, destination guides, holiday packages, airport guides and halal restaurant [13]. However, most of the research are focusing on the number of data provided instead of showing the data based on user's current needs especially using augmented reality technology. Even though [14] have produce an LBAR applications, they are not focused on user's current need. Therefore, in this paper, we focused on Islamic tourism information to be visualized using mobile LBAR application with combination of multimodal and adaptive interaction.

B. Mobile Location-Based AR for Tourism
In mobile AR history, Touring Machine are the first outdoor AR system that has been developed by Steven Feiner since 1997. Since that, mobile AR has been developed in many field such as education, architecture, tourism, marketing etc.
The emergence of mobile AR (MAR) has provided tourism organizations with the opportunity to provide a wealth of relevant travel information in various ways than just checking out online information sources or travel guides, thereby enhancing the overall travel experience [5], [15]. Existing solutions for AR applications fall into two distinct directions that is geographic and marker scale. Geographicscale AR application focus on location-based services (LBS) by adding location related information on top of the camera view. Location-based AR apps are useful as travelers are moving from one destination to one destination. According to [15], almost all applications provide an interactive view with expanded "clickable" explanations for delivering more information about the selected POI, open map view or lead to different screens with more detailed information . This shows that various ways of conveying information have been designed. Previous researchers have proposed many ideas to improve LBAR system. Beside, [16] have presented a set of design decisions based on empirical observation for LBAR. They proposed several design alternative that supports interaction between POI and user. Moreover, in [17], the authors discussed the design and implementation of such an advanced mobile application, which enables location-based mobile augmented reality on two different mobile operating systems. Table I shows the list of LBAR previous application study.

C. Multimodal and Adaptive Interaction in AR 1) Multimodal Input Interaction (MMI):
According to [18] there are several types of AR interfaces such as tangible interface, collaborative interface, hybrid interface and multimodal interface. Multimodal interfaces process two or more combined user input modes (such as speech, pen, touch, manual gesture, gaze, and head and body movements) in a coordinated manner with multimedia system output. Interaction using multiple modes can improve the effectiveness in delivering information to the system. It also provides interaction between humans and computers to be more natural and more intuitive [19]- [21]. In multimodal interaction history, Richard Bolt's "Put That There" system is the first system that brings opportunity for multimodal interfaces since 1980 [22]. This system integrates the voice and gesture inputs to enable user sitting on the chair and perform a natural and efficient interaction. QuickSet [23] was a pen/voice system running on an early tablet PC, used as a training simulator for US Marine Corp. MMI have also been implemented in AR environment, for example in 2011, [24] has suggested a combination of touch screen based interaction with two different interactions, which are finger based interaction and position and orientation of the device. These interactions were used to manipulate the AR 3D object whether by 2D clicking, 3D selection (using green marker attached to user's tip of the finger), or by pointing. Beside, CARAGS was developed using HMD and vibration device to performs a gesture sign and retrieved vibration feedback when user perform their assembly task [25]. Moreover, [26] combining visual and speech input in order to measure the accurateness of pronunciation among students. While [20] have produced an Multimodal AR system that combines speech input and free-hand gesture in a natural way to manipulate 3D AR object. Therefore, it shows that many researchers have proposed the MMI solutions to enhance the interaction between user and the system by using various kind of devices. Current devices such as wearable and smartphones, it comes with various sensors such as accelerometer, compass, camera, proximity sensor, microphone and others. The existence of these sensors can trigger interactions between humans and machines naturally [27]. In the multimodal context, as stated it combines two or more input modes such as motion and speech, touch and motion, speech and touch, and so on. For example, the user is pressing the button while speaking or the phone screen will become dark when the user puts the phone on the ear while answering phone calls are the examples of a multimodal interaction using a smartphone.

2)
Adaptive Interaction: [28] has defined adaptive interfaces as a system that adapts its display to current user needs and the ability to monitor user task, system tasks and current situation. For mobile adaptive interface, the device adapts behaviour based on the variation of the interaction context such as user, environment and device itself [28], [29]. User will notify the state of current behaviour of environment or device in some situation. For example, if the level of environment noise is high, the device will give user a notification and allow the user to perform a certain task. There are various way of adapting the display to user's current need. In [29], the authors have proposed a PC-based system called Digital Talking Book (DTB) player which is adapting the behaviour of the user. The system is adapting user information (e.g. user's impairment level), environment (e.g. ambient noise) and platform or device state (e.g. Screen resolution and the features of the speech synthesis and recognition modules) as an input to complement the user's task. Not only in pc-based, researches started to explore the use of adaptive interfaces in mobile-based such as in mobile phone and wearable devices. For example in [30], the authors use three main entities for the interaction such as user, device entity and environment. In their study, user plays an essential role for the system to reach user's personal motion, experiences, mental states and abilities. While the device is responsible to determine the modality input or output supported by the device. Then environment will sense through a device and continuously affect the interaction of a communication mode. In [31], the author applied different styles of adaptation where the system will adapt the environment context first before suggesting which kind of input modalities can be used in the current situation. For example, speech recognition is not available if current level of noise surrounding the user is high. [21] proposed an explicit interaction (e.g. touch and motion) and implicit context factor (e.g. location, ambient noise, light level etc.) which can be adapted by system to perform a task. Therefore, based on the previous research, it can be conclude the context factors are paramount to be adapted in order to allow users to perform their task based on their current needs. In our study, we categorized two different input modalities that can be performed in our system, which are multimodal interaction as explicit input and adaptive user interfaces as implicit input to the system. For example, speech, gesture and touch interaction by user are explicit input and time, level of noise and level of brightness are implicit input which adaptable by the system.

D. Proposed mobile solutions
Islamic Tourism Go, a location-based AR application that we proposed to assist tourist to find an Islamic information near their current location. It was developed using Android Studio for android operating system. This application uses several types of sensors such as proximity, acceleration, compass, gyroscope GPS, microphone and camera to detect the modalities input.
In this section, an analysis of input modalities and the flow process of Islamic Tourism Go was explained. We have also discussed the user interface design, application content and task definition in order to provide a details description about proposed application. As we want to focus on Islamic tourism relevant information to visualize in LBAR application, we choose three main categories to visualize Islamic information, which are Restaurant, Islamic Interesting Places and Prayer Places. It is because this kind of information are the most needed information by the tourist [1], [5].

1) Proposed Input Modalities:
From the previous research, it shows that multimodal and adaptive interaction are paramount input that has to be taken into account to be implemented in LBAR system especially when users are on mobile condition. Therefore, we proposed the combination of multimodal and adaptive interaction as in Table II. 2) System Flow Chart: Figure 1 shows the flow of the proposed mobile LBAR application. Tourist needs to register as a user before they can login to the system. For the first time user, they need to fill up the User Profile form, which aims to gather user's travel information. Then after the user submitted the form, an AR page will be displayed with the POI on top of camera view. The POI will be filtered or personalized based on the input modalities preferred by the users.

4) Multimodal Task Definition:
In this research, we have proposed three task for user to perform in the proposed LBAR application. User will perform these tasks one by one during the testing. Table III explaining the details of the types of modalities that explicitly and implicitly processed by the system. In order to perform the task, users are require to use three combination of modalities. The explicit modalities will be complimented by the data adapted from user profile and device state in order to display the relevant AR Output (POI) that synchronize with user's current need.

5) Application Contents:
There are three types of content that have been developed which are user profile, list of POI and location information. The content was created based on Islamic tourism context: • User Profile: In this application, multiple-choice questions form were provided in order to get user's travel information. The set of questions was based on three types of categories as the following.
i. Travelling Interest Information: User needs to choose the desired type of travel interest such as museum, historical mosque, Islamic architectural heritage and Islamic festivals.
ii. Meal Information: Meal information are the information regarding the user meal time (e.g. breakfast, lunch and dinner time) and user's preferred type of restaurant (e.g. Malay food, Chinese food, western food, Indian food and fast food).
iii. Prayer Information: User need to choose either prayer room or mosque to perform their prayer • Augmented Point of Interest (POI): The list of POI will be augmented on the screen after filtering process occurred. The POI will be filtered based on the task definition presented in Table III.
• Location Information: User will click on the specific POI to view the location information such as location image, location description and distance.

III. RESULTS AND DISCUSSION
This section explains the results obtained from the system usability for Islamic Tourism Go. The study was conducted in May 2018 in Putrajaya, Malaysia. The evaluation process was organized by asking tourists to use the Islamic Tourism Go app and distribute questionnaires subsequently. During the testing, tourists are first require to filling in the profile information provided in Islamic tourism app. After that, they will start search for AR POI by using combination of explicit modalities as stated in Table III. Fig. 3 shows the situation when the user was testing this application. The purpose of this evaluation is to test the degree of satisfaction of users as well as the usability of the application. To measure the data the System Usability Scale (SUS) was used. SUS is a Likert scale where the questionnaires consists of ten-item scale with five-value scale. Five of the question are measure positive impact of the application and the other five are to measure negative impact of the application. The results obtained from the questionnaire can be used to determine the level of user satisfaction in terms of usability. There 24 participants who participated in this survey. Note that the participant are all Muslim.    Table IV shows the demographic profile of the respondents. Eight respondents are male and sixteen of them are female. We have analysed that from the 24 respondents, 14 of the respondents are between 20-25 years old while five of them are between 26-30 years old and another five are between 31-35 years old.
As shown in Fig. 4, the majority of participants are Malaysian, which constitute 67% in total. Other 13% of participants are Indonesian, followed by those Singapore and Thailand that constitute 8% and finally 4% of the participants are from Philippine The participants have listed the preferred travel information they wanted to visualize such as restaurant, interesting place, hotel etc. Fig. 5 shows the statistics of preferred travel information. Its shows that restaurant/food are the most needed travel information by the users. Moreover, participants have also answered whether they used to download other travel app before (Fig. 6). We analysed that 14 of the participant never try to download other apps before and ten of then have experience downloading and use the travel app such as trivago, traveloka and booking.com.  Table VI, an average score of 75.8 was obtained. SUS scores have a range rate from of 0 to 100 by each of the answered sheets. In this paper, the calculation of SUS score in [32] was used. The good average System Usability Scale score is 68. The score will be under average if the score is below than 68 and vice versa. [33]. Since the average result was 75.83 (Table V), the application acceptance was above the average and it was considered good. Based on the results, it can be concluded that the application has gain user satisfaction in terms of usability to use this application. 2) User Feedback and Comments: Respondents also wrote some comments and feedback on the questionnaires. The comment is divided into two categories, which is on the improvement of application functionalities and application feedback. Table VI shows the comment provided by users. Improvement of application functionalities 1) Improve UI 2) Put more on tourism catered function e.g. foursquare 3) Use camera scan words (destination, hotels, places etc.) 4) Recommend nice, popular, best places, food etc. 5) Can share in facebook 6) Provide information rating & feedback 7) POI location of toilet information in specific tourism places. 8) Show more location pictures 9) Put the range distance at the POI instead of display at next page Application feedbacks 1) The speech recognition can detect places and give out alphabet. 2) A good application for Muslim.
3) Nice application to use when travelling but need to provide more information. 4) "The application was good. However, the information are limited. For me, it's okay to put other information like hotel or nature places even though this application are Islamic apps".

IV. CONCLUSION
We have leveraged multimodal and adaptive interaction and apply the modalities into location-based augmented reality (LBAR) application. We have reviewed the types of multimodal and adaptive interaction techniques that have been proposed by previous researchers. Therefore, in this study we have chosen speech, motion and touch as a physical interaction between user and mobile device, while the users profile, environment and device state as the context modalities to be adapted by the system. To support on developing the strategies of Islamic tourism in Malaysia, the Islamic information have been augmented as discussed in Section II and III. We categorized the Islamic information by three main categories which are Restaurant, Interesting places and Prayer places. Then we proposed a mobile location-based augmented reality to visualize the Islamic information. At the end of this study, we evaluated our application with the users and there are 24 participants who have taken part in this survey. System Usability Scale (SUS) has been used to measure the level of satisfaction in terms of application usability. The result shows that the users are satisfied with LBAR application which has leveraged multimodal and adaptive interaction with a score of 75.83.
The ideas for future work that will be considered are adding more categories to the Islamic tourism information. The POI can be minimized by filtering the information based on the recommendation (e.g. by other third parties). The multimodal and adaptive interaction can be more specific based on user mobility (e.g. users are walking on the street or in the building) that will give an effect to the displayed POI. These matter needs to be considered in order to produced good LBAR application in the future.