Evaluating and Redesigning the Self-Monitoring Tool

— In order to facilitate self-monitoring interventions designed by the Faculty of Computer Science Universitas Indonesia a web-based self-monitoring tool was created. This paper aims to evaluate the tool regarding its usability and user-experiences prior to its wide adoption. The System Usability Scale (SUS) and the User-Experience Questionnaire (UEQ) were used to evaluate the self-monitoring tool. The tool was implemented in a Human-Computer Interaction course during odd semesters, and the evaluations were taken by the participants of the course. The evaluation results were analysed to help determine whether the tool needs to be enhanced or improved. Based on the results, a new design of self-monitoring tool was developed using the interface design principles.


I. INTRODUCTION
In higher education, it is advantageous for students to become independent and reflective learners in which they engage in a reflective review of their learning with colleagues as part of a self-regulating learning process.Selfregulated learning is an intervention that encompasses calling, monitoring, and evaluating their learning process.To enhance these processes, self-monitoring intervention can be used as a strategy to analyse how students' progress through their learning process [1].
Self-monitoring is a component of self-management intervention [2] and the self-regulated learning process [3].Self-monitoring can be defined as an ability to observe, investigate, evaluate, and criticize the cognitive quality of the knowledge being gained [4].This ability is an effective cognitive process that has significant influence in enhancing learning strategy.Self-monitoring interventions facilitate students ability to conduct self-observation and selfrecording by providing a stepping stone to create such a focus.
In addition, as a component of self-regulated learning, self-monitoring is an essential independent process in the learning process [3].There are three steps of self-regulated learning, i.e. planning, monitoring, and evaluation.A thorough investigation of self-monitoring reveals its reliance upon performance monitoring and control process.In this process, the student must focus on both their learning activity and learning process so they can evaluate their progress and initiate any improvement (if needed) in the process.To conduct a self-monitoring intervention five steps should be executed [5], (1) identify the student behaviour; (2) choose and design the self-monitoring system; (3) choose the supporting criteria; (4) train the students to use the system, and (5) decrease the instructor role in the intervention.
There are two purposes for self-monitoring intervention; self-observation and self-recording [2].Self-observation is an activity conducted by students to observe their learning process and self-recording is an activity focused on recording instances of behavior that indicate hindrances or helps in the learning process.To facilitate an intervention using these techniques students are given a series of sequential prompts that f o c u s a n d f o r c e t h e m to review their knowledge development.Although the effort can be made traditionally, the use of computerbased tools can enhance the effectiveness and efficiency of reviewing learning progress.In this case, development of the tool and evaluation of its usability should involve learners who take control in their learning process.
Usability is one of the existing software quality criteria that situates itself as a non-functional requirement [6].The website usability.govdefines usability as how users learn and use a product to fulfil their objective including their satisfaction in using the system by following certain processes [7].With this consideration in mind, the usability becomes an important indicator to measure the interactivity of an IT system or product [6].In general, there are five components of usability [8], including learnability, efficiency, memorability, errors, and satisfaction.
To ensure a high level of usability for a system, usability testing is one of the techniques that can be used.There are some basic educational methods that may be used to conduct usability testing focusing on qualitative-and quantitativebased evaluation.One of the qualitative-based evaluations involves conducting interviews of the users [9].A quantitative-based evaluation can be conducted using a System Usability Scale (SUS).
Usability is just one aspect of many that impact userexperience.User-experience may be described as the quality of interaction a participant has when interacting with a computer system [10].Thus the experience of a person interacting with a computer system is called userexperience.
One of the most familiar principles in interface design is seen demonstrated in Google Scholar following the Eight Golden Rules set forth by Ben Shneiderman [11], [12] .The high citation references on Google Scholar indicate that the principle has significant effect on interface design.The Eight Golden Rules consist of eight guidelines [12] The Faculty of Computer Science Universitas Indonesia is conducting research on a self-monitoring intervention situated around the process described above.To support this intervention, a web-based self-monitoring tool has been developed using a usability approach [13].The lecturers, as facilitators, provide several sequential prompts (questions) during the learning process in a computer science c o u r s e .Students are asked to answer the prompts based on their current knowledge levels.Prompts and answers are displayed in sequential order so the students can analyse their knowledge as it develops.
There is significant need to evaluate such a web-based self-monitoring tool to validate its use.This paper presents work conducted to evaluate a tools usability and userexperiences.A research protocol was implemented to discover if the students can successfully interface with the tool to investigate its use as an intervention.The research questions were created to investigate participants' views on (1) the usability of the web-based self-monitoring tool; (2) the quality of user-experiences found within the web-based self-monitoring tool, and (3) possible improvements forming a strategy to enhance better user-experience and usability for the user.Results indicate whether the tool needs improvement or redesign before dissemination.

II. MATERIAL AND METHOD
To answer the research questions proposed in the introduction, a usability evaluation was conducted by implementing a self-monitoring intervention in a computer science course.A Human-Computer Interaction (HCI) course was chosen as a case study of this research.In the HCI course, the lecturers and facilitators conducted selfmonitoring interventions using a self-monitoring tool [3].
Imbedded within the lecturers where sequential prompts to certain topics that needed to be answered by the students.The prompted questions were given during individual semesters in every section of the HCI course.For example, in the first week, the lecturers covered material about cognition, so the prompted question was addressed towards the same topic.This cycle of curriculum delivery followed by prompted questions continued for every topic in HCI course.
At the end of the semester, two surveys were given to the students.The SUS survey was used to assess the usability of the tool itself [14] while the User-Experience Questionnaire (UEQ) survey was used to assess the userexperience of the tool [15].In every survey, the students of the HCI course had to answer several questions based on their experience using the tool.To also discover the personal experience of the students, we also completed a qualitative evaluation focusing on students opinions about their use of the self-monitoring tool.
The result of the usability and user-experience evaluation were considered as the fundamental evidence towards a decision whether or not we should improve and enhance the self-monitoring tool.If the results were positive, we would enhance current self-monitoring tool features and keep using it.On the other hand, if the result of this evaluation was poor, we determined to re-engineer the tool into a new version of the existing self-monitoring tool.
The objective of this study is to answer the research question by conducting usability and user-experience evaluation to assess the self-monitoring tool.The evaluations were conducted using a case study as follows.

A. Characteristic of the Course
This study was conducted with students enrolled in an HCI course taught at the Faculty of Computer Science Universitas Indonesia during odd semesters in 2015.HCI is a course that has a curriculum situated around human and computer interaction.There were two class sessions every week each operating within 100 minute duration each.In this course, the typical class sessions were delivered using seminars and enhanced with group discussions.Students who join this course are registered for four SCU (Semester Credit Units).

B. Self-Monitoring Intervention Strategy
The self-monitoring intervention was given to the students of HCI course during a single semester.The selfmonitoring intervention was conducted following these steps [5]:

1) Identify the Student Behaviour
This step identifies behaviour, isolates a needed action, and facilitates the planning of the self-monitoring intervention.This strategy provides a set of self-monitoring plans that are appropriate with the student's behaviour.

2) Choose and Design the Self-Monitoring System
In this step, an appropriate self-monitoring approach is chosen, designed, and then prepared.The result of this strategy is a web-based self-monitoring tool that facilitates the intervention.

3) Choose the Supporting Criteria
Within this step, we create the prompts (questions) for certain topics.There were 6 prompts that were used in the HCI course during the semester.The prompts covered the following topics: cognition, user interface, data collection technique, requirements, design and constructions, and application evaluation.A set of prompts established in this strategy were given when the topic was taught in the classroom.

4) Train the Students to Use the System
In this step training for the students using the system is initiated by asking them to answer the prompt within a certain time period and which was based on the topic taught in the classroom.In this strategy, the lecturer asks students to use the self-monitoring tool, so the section also incorporates training on its use.

5) Decrease the Instructor Role in the Intervention
The last step is to decrease the role of the instructor within the intervention process by attempting to make the students become more independent.By the end of this intervention, we expected that students would need less intervention from the instructor.
Based on the self-monitoring strategy, a self-monitoring tool was developed [3].The students who used this tool were given a sequence of prompts.They had to answer the prompts, and the respective responses are shown in their course dashboard.The lecturers can see the students' responses and develop a sense of how they develop their knowledge [3].
Fig. 1 shows the user interface of the student's course dashboard.The prompts were shown in an accordion format with the answer as hidden content.

C. Participants
The participants of this study are students that enrolled in an HCI course.This course consists of 66 students including 3 rd year students and 4 th year students.The numbers of male and female students are 46 and 20, respectively.

D. Instrumentation
The purpose of this study is to evaluate the usability and user-experience of a developed self-monitoring tool.The instrument that was used to conduct the usability evaluation is SUS.The instrument that was used to conduct the user-experience evaluation is UEQ.

1) System Usability Scale (SUS)
The SUS has been shown to be a reliable approach for usability measurement [14].It was originally created by John Brooke in 1986.The purpose of SUS development is to create an independent tool for evaluating hardware, consumer software, website, cell phone, etc. SUS has 10 questions that are shown in Table 1.Each question comes with 5 possible responses that must be chosen.Each question covers a variety of aspects that may impact system usability, such as the requirement of support or training allowing a high level of face validity for measuring usability [14].I think that I would like to use this website frequently.
2. I found this website unnecessarily complex.

3.
I thought this website was easy to use.

4.
I think I would need assistance to be able to use this website.

5.
I found the various functions in this website were well integrated.

6.
I thought there was too inconsistency in this website.

7.
I would imagine that most people would learn to use this website very quickly.

8.
I found this website very cumbersome/awkward to use.

9.
I felt very confident using this website.
10.I needed to learn a lot of things before I could get going with this website.
The SUS instrument provides response options that fall between "strongly disagree" to "strongly agree".The "strongly disagree" option records a score of 1, while the "strongly agree" option records a score of 5. Intermediate options have scores falling somewhere between these two.The calculation of a SUS Score can be more thoroughly investigated in Brooke's work [14].The calculation treats odd numbered questions by subtracting 1 from the score and treats for even numbered questions by subtracting the score from 5. It then sums up the new value and multiply it by 2.5.The final result of this calculation is out of 100.

2) User-Experience Questionnaire (UEQ)
The UEQ implements a data analytical approach that evaluates the interaction of a product [15] with a user.The purpose of this approach is to ensure a practical relevance of the distinct qualitative analysis of an interactive product (scale).As they are different from the SUS surveys, every UEQ question belongs to a particular evaluation measurement called a scale.There are six scales with 26 items of product evaluation measurement: • Attractiveness: how attractive the product in overall impression.• Perspicuity: how easy is the product is to use.
• Efficiency: how fast the user solves the tasks.
• Dependability: is the interaction u n d e r control of the user?• Stimulation: how well the product motivates the user.
• Novelty: how innovative the product is.
The mapping framework for 26 items of product evaluation measurement is shown in Fig. 2.
The UEQ survey is used to measure the user-experience of interactive products, such as statistics software packages, cell-phone address books, online-collaboration software or business software.The user-experience not only evaluates as an independent application but can also subsequently be compared with evaluations of other products.The comparison of the products can be seen through the benchmark of the UEQ survey.Finally, some UEQ typical application scenarios are [15]: • Compare user-experience of two products • Test a product user-experience • Determine areas of enhancement Fig. 2 The Distribution of 26 UEQ questions into 6 UEQ scales [15] III.RESULT AND DISCUSSIONS The evaluation of the self-monitoring tool was conducted by involving 66 students in the HCI course at the Faculty of Computer Science in Universitas Indonesia.Participants were asked to answer both the evaluation surveys, the SUS and the UEQ, based on their experience using the tool during the semester.

A. System Usability Scale (SUS)
As describe above, system SUS was used to evaluate the usability of the self-monitoring tool.The evaluation and calculation were conducted based on SUS guidelines [14].
The result of the survey's questions was computed using the calculation rule of SUS.
Fig. 3 shows us the result of the SUS questionnaire calculation.The x-axis shows the SUS scores and the y-axis show the frequency of every SUS scores.The value of the SUS score is distributed between 40 and 90 with the largest value falling in the 60's.However, recognizing that within the SUS evaluation an individual items/score has no meaning; we point out that only the average of the SUS scores is pertinent to the usability of a system.
The average of the SUS scores for the self-monitoring tool was 68.1 out of 100.Considering a benchmark of 68 as defining a categorization of average, the result of this study obtains a usability rating of average.Recognizing that a threshold of 80 is required for a good usability rating [14], it was determined that the tool needs to be improved and enhanced before it is used widely.

B. User-Experience Questionnaire (UEQ)
In the UEQ approach, the students have to complete 26 items for the product evaluation measurement.Based on the result of the questionnaire a UEQ calculation of a Cronbach-Alpha coefficient for the six scales of the UEQ follows the following model ( 1): Where r is the mean correlation of the items in a scale and n is the number of items in a scale.Based on t h e UEQ Handbook, every UEQ scale belongs to one of three kinds of UEQ result."Negative evaluation" is considered to have a mean value <-0.8, "neutral evaluation" has a mean value between -0.8 and +0.8, then "positive evaluation" has a mean value > +0.8.The minimum and maximum result of the mean value is between -3 and +3.
The calculation of the UEQ Cronbach-Alpha coefficients results in yields value per item as shown in Fig. 4. It tells us that there is no negative evaluation regarding the product since it has no negative mean value per questionnaire item [15].In addition, the result of the evaluation based on 6 UEQ scales is shown in Fig. 5. Based on the figure, we conclude that attractiveness, perspicuity, efficiency, and dependability scales have positive evaluations, but stimulation and novelty have neutral evaluations.
Evaluation using the UEQ survey also gives us a benchmark or an approach to compare our product with 163 other product investigations [15].Fig. 6 shows the benchmark of our self-monitoring tool evaluation.Based on this benchmark, our tools perspicuity is good while the efficiency and novelty are above average.However, the attractiveness, dependability, and stimulation are below average.
From the result of UEQ approach evaluation, the selfmonitoring tool has neutral and positive evaluation based on 6 UEQ scales.However, based on comparison with other products (benchmark), the self-monitoring tool has 3 positive evaluations (perspicuity, efficiency, and novelty) and 3 negative evaluations (attractiveness, dependability, and stimulation).As a result, the self-monitoring tool has average user experience based on UEQ survey evaluation.

C. Qualitative Evaluation
Once the usability evaluation was completed, we asked the students to fill the question about their opinion related to the self-monitoring tool.The question asked was given below the SUS questions as follows: "please provide any comment about this website".The result of this question provided 34 responses there are various responses to the self-monitoring tool including positive, negative, and neutral comments.Research not only asked for comments about the self-monitoring tool but also asked for a recommendation for the further development.The students provided a recommendation based on their experience using the selfmonitoring tool during the semester.Table 2 shows the students recommendations for the self-monitoring tool.Some bugs needed to be fixed 1 2.
Variance of alternative answer need to be propagated There is more information about HCI The user needs to be activated 1

D. Analysis
The result of this study focused on determining the need for self-monitoring improvement within the educational software.From the usability and user-experience evaluation above, we can see the web-based self-monitoring tool was of average quality, neither falling in a good or bad classification.Results from the SUS score and UEQ regarding the quality of the self-monitoring tool confirms this average assessment [16].As a result, the authors deem it is essential to enhance and improve the usability and userexperience of the tool.
The tool received 33% negative comments and 67% positive comments with regards to the qualitative feedback.As such it is not felt that the results were adequate enough to conclude that the self-monitoring tool provided a good experience.This is because one-third of the responses indicate that they perceived quite poor experience.
The improvement of the self-monitoring tool can be accomplished by improving distinct features within the tool.This strategy was expected to improve the function of the self-monitoring tool to enhance the learning environment [17].In this version of the self-monitoring tool, there are features designed for the student, including viewing the prompts, response/answer to the prompts, viewing feedback from the lecturer, and answer the feedback (if necessary) [13].Based on the UEQ results, we need to improve the attractiveness, dependability, and stimulation of the product.The attractiveness can be enhanced by modification of the user-interface and user-experience of the tool.This should facilitate student enjoyment in using it.In addition, we consider enhancing the tool's security, consistency, and navigation in order to improve the dependability [15].We also recommend that the tool should motivate and become more inventive so it can stimulate students to follow the selfmonitoring intervention [15].

E. Improvement Strategy
With regards to future work, and based on the usability and user experience evaluation, the research team has determined to re-engineer the interface design of the selfmonitoring tool.The newer version of the self-monitoring tool was developed based on the previous self-monitoring tool features that will be enhanced with the recommendations from the students.Finally, the improvement strategy for the self-monitoring tool was conducted by following these steps: 1) Analyze the result of the usability and useexperience evaluation of the self-monitoring tool.2) Evaluate the recommendations that were provided by the students.3) Analyze and design the newest version of selfmonitoring tool.
Based on the strategy above, the new self-monitoring tool was developed in the server environment.The use case and basic flow of the newer version of the self-monitoring tool followed the previous version.However, some features, such as security, prompt, and answer interface design, were improved and enhanced.This strategy is enhancing students' experiences with the tool.It is also expected that they are more motivated to conduct self-monitoring process.
Work has also focused on redesigning the interactivity of the self-monitoring tool.This work implemented the Shneiderman's Eight Golden Rules as interface design principles.Table 3 shows the realization of Eight Golden Rules Principle that was applied in designing selfmonitoring interaction.The results of the re-engineering process are shown in Fig. 7.The figure shows the sequence of prompts in the new version of self-monitoring tool.When compared with the Fig. 1, the newer version is seen to provide a clearer presentation of prompts.In addition, the newer version of the selfmonitoring tool also creates a newer presentation of the prompt and their feedback.Fig. 8 shows the user interface of a prompt with a detailed answer.The prompt question delivered by the lecturer is followed by the student's answer.If the lecturer wishes to give feedback related to the student's answer, he/she can choose the blue-button.We expected that students would gain better experience in selfmonitoring activities due to this enhanced version of selfmonitoring tool.The result from both SUS and UEQ surveys rate the tool as average regarding usability but also indicate some areas for improvement.Specifically, based on a user experience evaluation, the results of the UEQ survey places this tool a little above average.With reflection upon these results, it is seen that the tool needs to be enhanced and improved from both a usability and user-experience perspective prior to its wider adoption.We consider re-engineering the tool from both a usability and user-experience perspective in order to deliver a higher quality self-monitoring tool and experience.
Based on the usability and user-experience evaluation, the team will redesign the self-monitoring tool.We intend to apply the same features as the previous self-monitoring tool, but additionally, we will also enhance features based on the students' recommendations.We expected that the new version of this self-monitoring tool would provide better experiences to the students.
There is much future work still need to be conducted related to self-monitoring research.This study is recognized as still being somewhat limited, and the research team intends to continue its research by implementing the new version of self-monitoring tool within a self-monitoring intervention strategy for several courses.

Fig. 1 A
Fig. 1 A Sample of the sequence of prompts visible to students through the course dashboard (in Bahasa Indonesia)

Fig. 4
Fig. 4 Mean value per item

Fig. 7
Fig. 7 The sample of sequence of prompt shown on the lecturer's dashboard