Footprint : Tourism Information Search based on Mixed Reality

In the quest to provide better information seeking experience during travelling, this paper is set to design, build and trial a prototype tourism information search application based on mixed reality. This paper proposes “Footprint”, an android-based tourism information search application in a mixed reality environment, whereby it overlays tourism-related information on the image that the mobile phone camera is focused at. Using Footprint, the user would only need to point the mobile phone camera on a specific spot, and Footprint will overlay all in-formation on geo-tagged tourism objects as far as 30 meters away. The user is also able to filter search information based on the data source of information (such as Google or Wiki) or based on the category (such as Food or Landmark). It is hoped that this experience would one day become a trend, as when more reality-environments will be shaped into surreal, yet virtual alternatives through Internet of Things (IoT), and there will be more ubiquitous resource access through many devices, networks, and user content. Keywords— mixed reality; information seeking; tourism; android


I. INTRODUCTION
Tourism has always played a significant role in the development of both economy and infrastructure of Malaysia.Back in the days, Malaysia was less developed in the tourism sector in comparison with Indonesia and Thailand.Yet today the tourism sector has become the country's second-largest source of foreign exchange and the third largest economic sector.Over the years, the tourism industry in Malaysia has provided a significant contribution to the growth of Gross Domestic Product (GDP) in Malaysia as well as strengthening the services account of the balance of payment.Majority of the countries have a share of services in global GDP of more than 40%, of which further proven that tourism could provide a great impact on the economy and an overall development of a particular country.Do note that tourism industries are not a standalone industry, but a sector that has many indirect relations with another primary sector which includes education, agricultural, industrial, environment, and much more.A mere growth of Tourism sector could benefit many, and the most important resources come from the foreign exchanges.
The domestic tourism survey in 2015 by the Department of Statistics Malaysia reported that the receipts from tourism had shown a proper and steady growth.The receipts have a good increment from RM53,367 million in the year of 2009 to RM60.5 billion in 2015.In the year of 2015, there has been a report of 14.4% increment from the previous year.To emphasize on the impact of Tourism in Malaysia, the Tenth Malaysia Plan 2011-2016 were raised, of which tourismrelated activities provided 492,000 jobs for employees in 2006, representing 4.4% of total workforce in Malaysia.This is a huge 26% increment as compared to 390,600 employments since the year of 2000 that accounted for 4.2% of total workforce.The tourism industry impacted more Malaysian residents in urban and rural areas.A total of 176.9 million visitors were recorded in 2016 with a growth 4.5% compared to previous year.The report indicated an increase of 8.1% from 217.5 million trips in 2014 to 235.2 million trips in 2015 in terms of the number of trips.In total, the year 2015 generated an income RM67,842 million from domestic tourism alone.
Meanwhile, the international tourist arrivals and receipts are the main popular methods or indicators to use in measuring tourism performance.Based on the report from 2016 United Nations World Tourism Organization (UNWTO), Asia and the Pacific will achieve 17 million additional international tourist arrivals on average a year of 43 million worldwide towards 2030.International tourist arrivals in Asia were 279 million in 2015 with 24% growth.Population in 2015 was 56%, GDP of 33%, International tourist arrival was 24%, and international receipts were 33%.Asian tourism is also maturing and moving towards more demand in innovation, diversification of product market and segment, product development, ICT and technology in general.These performances were realized by the governments' tourism training, incentives, and promotional programs, which subsequently include tourism infrastructure fund and special tourism fund to support tourism-related development and projects.
With that said, tourism proves to be a vital requirement for the development of Malaysia, both in the sense of economy and infrastructures.However, tourists still have a hard time acquiring satisfying tourism-related information.In 2008, a survey was carried to rank a list of 17 problems that human face when performing information search on tourism or traveling-related activities.Based on a five-point Likert scale, the highest problem identified in tourism information seeking was "Lack of required materials" with the mean of 3.71 and the second highest ranked problem was "Source and contents are too scattered" with a mean of 3.60 [1].When there is too many information everywhere, the accuracy of information and sources will be questioned because much sorting and research are due.
Information searching via the Internet is not always a straightforward task, as search engine also provide irrelevant results due to search indexes [2].It is a game of probability, where the more users with the same search term have looked for a certain site, the higher the chances that it would be on the top list.Thus in a sense, millions of info-gathering habits would impact the outcome [3], and it might not be the correct information one is looking for.Collaborative filtering might be efficient in information seeking, but the user would pay the price of reducing search accuracy in exchange for speedy information seeking.
Nonetheless, the most threatening obstacle to getting a satisfying result when performing an in-formation search is no other than the natural human's behaviour of gathering and acquiring information.Human information behaviour is rather passive, where most look for information only when he or she has information needs [3].Information need, in the tourism industry, should not yield one specific result but rather a collection of recommendations for the user to choose the one that satisfies his or her needs.Such behaviour also dictates which information is selected and which being neglected.This will not only reduce the information perceived by people but would also receive a false information due to selective in-formation acquiring methods.
To manage the shear tons of tourism or traveling-related information freely available on the World Wide Web, this project proposes for an android-based information search application specific for tourism but capitalizing on the concept of mixed reality.The android platform is chosen because tourist or travellers are always on-the-go, but they are usually equipped with camera-based smartphones and built-in GPS tracker.Meanwhile, the mixed reality environment will enable the application to overlay the search information on images captured by the camera phone, giving a new information seeking experience to the users.
The fact that current technologies are rather advanced and with the aids of the World Wide Web, information search, particularly in tourism, has been much easier.Information search is not a straightforward activity that will yield a definite result given a specific query.The nature of human query itself is most of the time ambiguous and intuitive.This results in a communication gap or a translation gap between the users and intermediaries such as the Internet, or devices like books and GPS locator [4].When human perceive information through intermediaries such as searching for terms via a search engine or acquiring information through a book, the outcome depends on how information is provided from these intermediaries, and how an individual human naturally process the information.Often miscommunication occurred between the intermediaries and users, and it would lead to confusion, thus perceiving and understanding the wrong information.
Mobile tourism applications can be roughly classified into two main categories; tour guide companions and recommendation systems [5].A tour guide companion such as the GoGoBot (https://www.gogobot.com/) is capable of providing basic yet important information about a place of interest.A tourist could enjoy interactive and personalized tours that match the users' interests, and therefore able to acquire a better knowledge of the attraction places.Another example of tour guide companion is the well-known Singapore Heritage Walking Tours called the Waalkz, which can provide a pre-defined walking city guide and allow the user to learn about four popular pre-defined destinations including Chinatown, Boat Quay, Little India, and Arab Quarters.Fig. 1 shows the screenshot of the Waalkz application (http://download.cnet.com/Waalkz-Singapore-Heritage-Walking-Tours/3000-20428_4-75308230.html).Recommendation system, on the other hand, is used for tourists to plan and select suitable service option for their trip.The biggest difference between the recommendation systems and the aforementioned tour guide companion is that the application is more on a trip calculator rather than a trip guide.TripAdvisor (https://www.tripadvisor.com.my/) is one of the leading commercial recommendation systems that allow tourists to discover a wide variety of selections on nearby lodgings, restaurants, or local attractions.The user can use the sort feature to filter the recommended listing by certain attributes, such as price and distances.Fig. 2 shows the screenshot for the TripAdvisor application.Information seeking is human behaviour that refers to the manner people looking for a specific set of information depending on a specific set of goal [6].For the case of travellers, the information seeking behaviour would be related to looking for places of interest depending on their current location.Many of the travellers do not have prior information to start with (e.g., the name of the place) except for real-time location and surroundings.This is where the concept of mixed reality delivers the users with more accurate information and better experience.
One application that has proven an efficient mixed reality experience to end users is the Wikitude (https://www.wikitude.com).Wikitude applies the concept of meaningware -an intermediate medium between software, hardware, and the user [7].It is all-in-one augmented reality SDK combines 3D tracking technology, image recognition, and tracking, as well as geo-location augmented reality (AR) for mobile, tablets and smart glasses.With regards to the tourism domain, the fundamental idea is that the traveler uses the mobile phone camera to capture a real-time image, the software renders the information, and the meaningware will translate and overlay real-time information on the mobile screen in conjunction with the real-time image captured for the traveller to interact with.The advent of meaningware would mean that interaction with information is forever changed, where information is no longer perceived and looked for when a user has a specific need but rather dynamic whenever the user interacts with the real world.

II. MATERIAL AND METHOD
Mixed reality or hybrid reality is an environment where real and virtual objects are presented together within a single display and interact in real time [8].According to [9], digital media is conceived of as a three-level dependent hierarchy of hardware, software, and meaningware.Meaningware is the domain of individual messages and texts and the genre systems they belong to.The proposed Footprint application is designed and developed based on the mixed reality concept, whereby there are three interconnecting mediums needed -hardware, software, and meaningware as shown in Fig. 3. Based on Fig. 3, the application Footprint is operated through the software layer by taking input from a mobile phone camera (hardware layer) and producing overlay information on based on tourism-related objects back to the users (meaningware layer).This concept is similar to the concept of Content Discovery [10] where users receive and gather information from time to time no matter where they are.This means rather than purposely looking for and expecting the outcome; the user would instead be discovering and learning more new information.
The basic mechanism for Footprint is as follows.Users point the camera from a smartphone with a built-in global positioning system (GPS) and a gyroscope sensor.Information from any tagged tourism-related object within the range of 30 meters will be overlaid on the image on the screen.GPS is important to provide location and give directions, while the gyroscope sensor is required to measure the rate of rotation in a radius around the mobile phone's x, y, and z axis.Next, navigating through the respective icons of the tagged tourism-related object will prompt more information, directly fetched from its information sources such as website or search engine results.Fig. 4 shows the information overlay on camera image in Footprint.

A. Adjust Scan Radius
Footprint allows users to adjust the scan radius by using the radius gauge at the right side of the application screen to adjust the distance for scanning.The radius is displayed at the radar section as shown in Fig. 5.

C. Filter Scan Category
In order to maintain a readable viewing window on the mobile phone, Footprint also includes a function to filter scan category.This is to reduce the pack of overlay information, hence increasing readability.In the selection window, there are several options to choose from, which are: Food, Transport, Landmark, Entertainment, Medical or Emergency, Other, and Highlighted Places of Interest (POI).Selecting any of the options will omit out all other categories except the selected category.As a side note, another option refers to POI that does not fit any of the categories, and Highlighted POI option refers to selective elements from all the categories.
For example, if the user selected Food category, only overlay information that is concerned with foods and dining will be displayed as shown in Fig. 8(a).Footprint's capabilities to filter data source and scan categories are imperative because, without the information categorization, the users may experience information overflow [11].

III. RESULTS AND DISCUSSION
The footprint is evaluated using Visual Verification, which is a type of application verification method that uses distinctive visual approaches such as photos and videos to record down application behaviours during typical usage, as well as any certainty and uncertainty that the application has undergone during product usage.Unlike traditional testing and verification processes where it emphasizes on written materials and standards to follow, this type of verification emphasizes on realistic usage scenarios and record down the process for product feature and function verification.
Verification in Footprint focused on recording real-time and in realistic usage scenarios in user's environment.A video recording of the product has been conducted.Fig. 9 shows the display of additional information pertaining to the respective Places of Interest (POI) based on Google Place API.The user only needs to click on any markers (overlay information) that are displayed on the screen.To acquire the knowledge on how to reach to the POI, users can first click any POI marker that is desired.Once additional information is displayed, users can then click the "Take Me There" button.Fig. 11 shows the display on the mobile display.Fig. 11 Mobile phone user interface on POI: Take Me There Fig. 12 shows that once the button is clicked, a list of options or intents will be displayed.Users can open up any navigator application for navigation purpose, but it is highly encouraged to use Maps since the intent is being declared with Google Maps in mind.The next figure shows that once the map application is being loaded, the application will have the destination coordinates.This is because Footprint will send device coordinates as a source, and POI coordinates as a destination for any navigator intent that will be launched.For Maps, simply click the Start button on the bottom right, and the application will navigate for the user.This paper presented the design and development of Footprint, an android-based tourism information search application that capitalized on the mixed reality methodology.The benefit of Footprint is two-fold.First is cost elimination on physical information materials such as tourism brochures, maps, and pamphlets.Second is timesaving for searching travelling-related information, such as restaurant, hotel, and landmark.Footprint also presents added values such as a free promotion for merchants because Footprint is capable of presenting overlay information on geo-tagged tourism objects as far as 30 meters away.In addition, Footprint provides an entirely new information seeking experience to the users, whereby overlay information is displayed on the phone screen by just pointing the mobile phone camera to the direction of the tourism object.
However, there are only two limitations that would critically affect the performance of Foot-print.The first limitation is the constant requirement of Internet connection.Unlike other similar applications (there are actually less than 3 applications with the same functionality), Foot-print does not store geographical data locally for the sake of minimizing application size and reduce device workload.This means Footprint will have to acquire geographical data through online method, as it has shown on Data Retrieving Logic chapter.Footprint requires sending HTTP GET request to geographical web services, only then it could retrieve JSON that carries geographical information.Without an Internet connection, Footprint will not be able to display information overlay on the screen, because Footprint will inevitably fail to retrieve JSON text from web services.
Secondly, Footprint requires mobile phones with a back camera.Realistically, smart devices with no back camera are almost non-existence, but nothing is sure in this world.Not too long ago, a Singapore mobile device company has launched a camera-less android smartphone that comes with v4.0.3 Ice Cream Sandwich version.Even though the application should work within an environment that has no camera, because the main concept of retrieving and displaying result is through sending HTTP GET request, acquire the information, and paint the icon on screen whereas camera view is just for aesthetic purposes, the application behaviour is still unknown in the camera-less phone.
If device requirements and platform can be considered as a system limitation, then it is an arguably acceptable limitation that has been set by the markets.Footprint strictly requires devices that run on an android platform with the minimum of Gingerbread version 2.3.3.Realistically, application with platform restriction and platform versioning is hardly a distinct limitation, because it suffers by almost all programs and applications, especially those applications that are not mainstream and designed to be platform-specific.Ultimately, Footprint does have its system limitation on what environment it could work in, but the same limitation could be said to apply to almost every application in the market.The requirement of needing an Internet connection is the only turning point here, where it would transform Footprint from a mixed reality tourism application into a useless camera-view application.
In the future, Footprint is hoped to capitalize on crowdsourcing information to provide a sense of community-based, tourism-adventuring and to make recommendations based on user ratings.This will encourage a highly personalized information seeking activity based on real environment.This experience would one day become a trend, as when more reality-environments will be shaped into surreal, yet virtual alternatives through Internet of Things (IoT), where it aims to offer ubiquitous resource access through multiple devices, networks, and user content [12], [13].

Fig. 1
Fig. 1 Screenshot of the Waalkz application

Fig. 2
Fig. 2 Screenshot of the TripAdvisor application However, both tour guide companions and recommendation systems are slowly changing human behaviour during the information seeking process.Information seeking is human behaviour that refers to the manner people looking for a specific set of information depending on a specific set of goal[6].For the case of travellers, the information seeking behaviour would be related to looking for places of interest depending on their current location.Many of the travellers do not have prior information to start with (e.g., the name of the place) except for real-time location and surroundings.This is where the concept of mixed reality delivers the users with more accurate information and better experience.One application that has proven an efficient mixed reality experience to end users is the Wikitude (https://www.wikitude.com).Wikitude applies the concept of meaningware -an intermediate medium between software, hardware, and the user[7].It is all-in-one augmented reality SDK combines 3D tracking technology, image recognition, and tracking, as well as geo-location augmented reality (AR) for mobile, tablets and smart glasses.With regards to the tourism domain, the fundamental idea is that the traveler uses the mobile phone camera to capture a real-time image, the software renders the information, and the meaningware will translate and overlay real-time information on the mobile screen in conjunction with the real-time image captured for the traveller to interact with.The advent of meaningware would mean that interaction with information is forever changed, where information is no longer perceived and looked for when a user has a specific need but rather dynamic whenever the user interacts with the real world.

Fig. 4
Fig. 4 Information overlay on camera image in footprint Footprint has three main system functionalities, which are to (1) adjust scan radius, (2) filter data source, and (3) filter scan category.

Fig. 5
Fig. 5 Adjust scan radius functionalityB.Filter Data SourceFig.6showsthe filter data source functionality, where the user can choose to display data from Google only, Foursquare only, Wikipedia only or display all.The choice is flexible to cater user preferences and to avoid overcrowding of overlay information on the screen.

Fig. 6
Fig. 6 Filter data source functionality Fig. 7(a) to Fig. 7(d) shows the interface of each filter category; Google, Foursquare, Wikipedia, and Display All.

Fig. 7 (
Fig. 7(a) Filter based on Google only

Fig. 8 (
Fig. 8(a) Filter based food category If users selected Medical or Emergency, only overlay information that is concerned with medical and emergency, such as hospital and police station will be displayed as shown in Fig. 8(b).

Fig. 8 (
Fig. 8(b) Filter based on medical or emergency category

Fig. 9
Fig. 9 Display all additional informationNext, Fig.10shows an example of additional information from the Geonames-Wikipedia web service when Wikipedia markers are being clicked.Note that Wikipedia provides a better description of the vicinity and other details.Standard information such as the latitude and longitude of the location and distance to reach location is being provided by all APIs.