Evaluation of Malaysian Universities Websites based on Quality in Use Evaluation Model

— Quality-in-Use (QinU) is one of the critical quality factors in measuring website quality. Most existing studies on measuring website quality only focuses on evaluating quality from the user point of view but not on the similarities and the differences between the users and decision-makers perspective. Different stakeholders have a different preference in term of quality aspects that are important. The objective of this study is to analyze the quality aspects of the websites from different stakeholders’ perspectives and rank the websites based on the results. In this study, we develop a Quality-in-Use Evaluation Model (QinUEM) to identify the quality aspects' priorities. Two quantitative approaches were used for this purpose. The first was a Multi-Criteria Decision Making (MCDM) approach using the Fuzzy Analytic Hierarchy Process (FAHP) method to determine the priority and the weight of each quality aspect from the users’ viewpoint. Then the statistical analysis was used to determine the priority of the same quality aspect from the developers’ perspective. To evaluate the model, we conducted a survey. The respondents of the survey were the students (users) and developers (decision-makers) from six Malaysian universities with 486 numbers of questionnaires been distributed. Based on the results, it shows users (students) prefer Functional Quality rather than Content and Appearance Qualities while the decision makers (developers) favor on Content rather than Appearance and Functional Qualities. These results show different viewpoint and priority in quality aspects needed for users and decision-makers. Based on the results we then used the QinUEM to rank the universities websites according to the defined QinU.


I. INTRODUCTION
Websites are playing a primary role in diverse application domains such as business, education, entertainment, and industry in order to promote and share information to viewers. As a result, the quality of these websites is of high importance, since the end users' demands are also increased in parallel. Evaluating these sites have become more complex and multi-dimensional. Therefore, it is important to understand the quality dimensions of websites [1]. Recently most of the researches on web quality stated that Quality-in-Use (QinU) is considered to be the most important factor that has been accepted in software application [2]. According to ISO/IEC 25010 standards, QinU represents the users' viewpoint of software quality [3]. The literature reveals that numerous studies have identified various methods to evaluate the QinU. Although these studies assessed the quality from the users' perspective, they did not mention which quality aspect was more important to endusers, and whether the decision-maker shared the end-users' view or if they have their views on quality aspects. Studies, such as [4]- [12] evaluated educational website quality using different approaches, without identifying the similarities and the differences between students and decision-makers' perspectives on educational website quality aspects. In this study, QinUEM represented three quality aspects as demonstrated in Figure 1. There are three (3) categories of QinU as follows: Functional quality (FQ): contains attributes, which related to web application functionality such as navigational links, search, security, and others.
Contents Quality (CQ): contains attributes, which related to the content of web application such as information quality, accuracy, information updating, and others. Appearance Quality (AQ): contains attributes, which related to the user interface design.
In this work, the definition of QinUEM attributes follows the ISO/IEC and IEEE organization [3], [14]. This study used the quality aspects as defined in QinUEM to evaluate the QinU for six Malaysian universities. The objectives of this study are; first, to highlight the similarities and differences between users (students) and decision-makers (developers) views regarding the quality aspects priorities. The second objective is to investigate whether the postgraduate and undergraduate students had the same view of the importance of quality aspects. The third objective is to rank six Malaysian universities based on QinUEM. To achieve these objectives, two quantitative approaches were used. The first approach is by using the Multi-Criteria Decision Making (MCDM) that refers to find the best alternative from all of the feasible alternatives in the presence of multiple decision criteria [15]. A Fuzzy Analytic Hierarchy Process (FAHP), which is one of the MCDM tools, was used to rank and identify which of the quality aspects were more important to the students and to measure the weight of each aspect. The second approach is a statistical analysis method to test the QinUEM reliability and to determine the perceived value for each quality aspect of the university website, which reflected the decision-makers' belief on the importance quality aspects of the university website. Here, these two approaches were used to measure the QinU of six Malaysian universities websites.
This paper is organized as follows. QinUEM reliability and validity using statistical analysis methods and quality aspects priorities from the decision-makers' perspective is discussed in section two. A brief background to the FAHP method is in section three. Section four explained the quality aspects priority and weights from student's viewpoint. Section five revealed the difference between the users and decision-makers' quality aspects priorities. Integrating FAHP and statistical analysis to measure QinU of universities websites is discussed in section six. The conclusion is in the last section.

II. MATERIAL AND METHOD
QinUEM attributes are analyzed using various statistical methods to evaluate its reliability. Next subsections will describe the evaluation environment.

A. Identifying the case study
QinUEM was implemented on six (6) Malaysian universities in order to rank the QinU of their websites. Three (3) of the universities are the public universities and another three (3) are the private universities [16]. The total population of these six universities was around 100,000; the required sample size was 384 according to [17].

B. Developing the questionnaire
A subjective instrument was employed to evaluate the universities' QinU from the students' perspective. A questionnaire was developed to measure the attributes of QinUEM as demonstrated in Appendix A. Stages of developing the questionnaire are shown in Fig. 2. C. Collecting data 500 questionnaires were distributed to six selected Malaysian universities namely USIM, UKM, UPM, MSU, MMU, and IUKL. 486 questionnaires were returned, where 409 were accepted, and the rest were rejected due to incomplete or completed unsatisfactory. The questionnaires were distributed through WhatsApp groups, Viber groups, emails, and Facebook. Table 1 shows the total number of respondents of the survey.

D. Statistical analysis
The questionnaire had 48 questions divided into five sections. One section measured the overall satisfaction of the university website (not included in this paper), the other three sections measured the students' satisfaction on FQ, CQ, and AQ and the last section was related to FAHP to rank the quality aspects according to students' preference. The QinUEM model was analyzed using several statistical approaches such as detecting outlier, normality, reliability, and correlation using SPSS Statistics 22 Application. Outliers test defined as the observations that have unique characteristics and differ distinctly from others [18]. According to [19], outliers are detected when the standardized residual is more than +3.3 and less than -3.3. For the current study, all the standardized residuals were in the accepted range. Therefore, the 12 attributes of the model were free of outliers. Reliability test involves the extent to which the instrument measures some attributes in a systematic and therefore in a repeatable way [18], [20]. The reliability of all constructs was investigated through the Cronbach's alpha technique that accepted a reliability score of above 0. 70 [18], [21]. Correlation is a statistical technique that is used to measure and describe the strength of the relationship between two variables. The descriptive statistics showed that the variables had a significant correlation at 0.01 significant levels as demonstrated in Table 3.
A statistical mean measures the students' satisfaction with each quality aspect in QinU. At the same time, it reflected the quality aspect value of the decision-maker (see Table 4). Table 4 shows that the mean value for each quality aspect been evaluated. Based on the results it shows that the students were more satisfied with the content of the university website followed by the appearance quality and the functional quality.
Analytic Hierarchy Process (AHP) has been introduced by Saaty [22] to solve the identification of multiple attributes decision-making priorities. Saaty assumed that the decision maker could provide accurate assessments when comparing attributes. Human judgment on the importance of alternatives or attributes is always subjective and imprecise [4]. To overcome this issue, several researchers integrated fuzzy theory [23] with AHP to determine the attribute weights from the subjective judgments of decision makers, such as [4], [12], [15], [24], [25]. Consequently, this study applied the fuzzy AHP method to determine the priority and weight of each quality aspect.

E. FAHP implementation
In order to use the FAHP, the students were asked to compare the relative importance of two given quality aspects to build the pairwise comparison matrix using Saaty's scale. Also, fuzzy triangular numbers (TFN) of [26] was adopted to represent the students' judgments. A TFN converted the crisp value of the pairwise comparison matrix into three numbers expressed as a triple (l, m, u), where l ≤ m ≤ u (see Table 5).  [24] Some main operation of TFN can be expressed on T1= (l1, m1, u1) and T2= (l2, m2, u2) as follows: According to [29]  Very Strongly important 7 (5,7,9) Important 9 (7,9,9) Where is the fuzzy comparison matrix of the student k. is the fuzzy relative importance between two quality aspects from student's k view.
is in TFN form. The pairwise comparison matrix will be subjected to a consistency ratio (CR) test to assure the quality level of the matrix before going through calculating the weight of the quality aspects. According to [4] [28], the first step to obtain CR is to find the weight vector by: (4) Where n is some attributes. In this study, the attributes are 3 (F.Q, C.Q, and A.Q).
The second step is to calculate the maximum eigenvalue for each comparison matrix by: The third step is to find consistency index (CI) by: (6) Consistency ratio obtained by: Where random index (RI) for n=3 is 0.58 according to Saaty [22]. The accepted comparison matrix should have CR less than 0.1.

The fuzzy weight
of QinU aspects is calculated as follow: First, the geometric mean for each row is determined as: (8) Second, the fuzzy weight is given as: (9) can be defuzzified to a crisp value by the formula: The final weight for every quality aspect can be obtained with:

F. Example: Implementation of FAHP
A random participant was chosen in this experiment. The dimension of the comparison matrix is 3 (F.Q, C.Q, and A.Q). The required entry is 3 obtained by the formula: No. of entry = (n -1) * n /2 (12) The pairwise comparison matrix is shown in Fig. 3. Fig. 3 The pairwise comparison matrix A weight vector can be calculated by using formula (4) in order to check the consistency of the previous pairwise matrix as shown in Fig. 4.  The pairwise comparison matrix is consistent since CR is less than 0.1, to show that the fuzzy comparison matrix is consistence [4]. The accepted pairwise comparison matrix was converted to a fuzzy comparison matrix using the TFN as shown in Figure 6.  The fuzzy weights of FQ, CQ, and AQ are calculated using formula (9); the results are shown in Fig. 8. The fuzzy initial weights are defuzzified to a crisp value using formula (10). The final weights of the quality aspects are obtained by formula (11); as Fig. 9 shows. According to the final weights, students are concerned with Content Quality over Functional Quality and Appearance Quality.

G. Implementation of FAHP in Java
A Java program was developed to calculate the weight value of each quality aspect based on FAHP from 409 respondents. The weight value will determine the priority of the quality aspect. The evaluation excluded 51 respondents due to the inconsistency in their answers. The result of the FAHP evaluation is shown in Table 6.
Based on the FAHP results, it shows that the Functional Quality (FQ) has the highest value with a value of 0.5358, followed by the Content Quality (CQ) with a weight value of 0.2717 and the Appearance Quality (AQ) with a weight value of 0.1925.  Table 7 shows the weight value of FAHP between two categories of students, i.e., postgraduate and undergraduate. Based on the results, it can be concluded that the sequence importance of quality aspects is similar except the value is slightly different. The weight value is slightly lower for FQ and CQ and slightly higher for AQ in the comparison between postgraduate and undergraduate. This may occur due to the age factor, since the average age for undergraduate students were around 21 years old, while the average age of the postgraduate students was around 33 years old. This may imply that younger students are more interested in AQ than the matured students.

A. Users and decision-makers' quality aspects priorities
Based on the statistical analysis performed to the decision makers, i.e., the developers, it shows the different viewpoint and priority in quality aspects needed. The decision makers concerned more on the Contents Quality (CQ) rather than the Appearance Quality (AQ) and considered the least important as the Functional Quality (FQ) (see Table 8). In contrast, students prefer the Functional Quality as the most important quality for the university website followed by Content Quality and Appearance Quality.

B. Raking of six Universities based on QinU
QinU can be evaluated by the accumulation of the statistical mean of each quality aspect (as shown in Fig 1) multiplied by the FAHP weight for the same aspect. Therefore, QinU achieved by: This formula can be used to rank of the university websites' based on Quality in Use (QinU). Table 9 shows the weight value based on the QinU formula as shown above based on the students' viewpoint of the quality aspects needed.

IV. CONCLUSION
Although there are several studies on evaluating the quality aspects of university websites, most of this evaluation did not compare the results between different viewpoints, i.e., the users and the decision makers. This study implemented QinUEM to detect the similarities and differences between two different viewpoints, i.e., users (students) and decision-makers (developers) of the university websites, using statistical analysis and the FAHP method. The results of the FAHP analysis show that the students have different needs in term of the quality aspects compare to the developers. The students, i.e., undergraduate and postgraduate students prefer functional quality as the important quality aspects rather than contents quality and appearance quality. In contrast, the developers considered the content quality as the most important quality rather than the appearance quality and functional quality. As a conclusion, it shows that the QinUEM can be used to rank university websites based on a QinU evaluation. For future work, we will investigate whether Quality in Use (QinU) influences satisfaction.