The Impact of Information Systems from the Perspective of IS Stakeholders in Malaysia

— The importance of canvassing multiple levels of stakeholders (i.e. Strategic, Managerial, Operational and Technical) for measuring information system (IS) success has been discussed by academics for several decades. From the literature, it is apparent that IS success researchers have measured IS success from different groups of stakeholders. This make results difficult to compare. It becomes more complicated when there are many classifications of IS stakeholders suggested in the literature. With different level of users, there are many arguments on whose perspective is important when measuring IS. Some IS success researchers argue that IS Success can be measured from only selected level of stakeholders, thus different measures are needed for different level of stakeholders. Others suggest that canvassing all level of stakeholders would be beneficial to arrive at a complete and meaningful IS success score. In this paper, we present findings from a survey that indicates that different groups of stakeholders have differing views when evaluating the success of an IS. However, we disagree that different set of measures should be used for different level of stakeholders. Based on the empirical evidence that we collected at four state governments in Malaysia, we recommend that IS should be evaluated across all level of stakeholders in order to arrive at a holistic impact score.


I. INTRODUCTION
A "Stakeholder is a person or group in whose interest the evaluation of IS success is being performed" [1, p. 5]. The importance of evaluating IS success from different group of IS stakeholders has been discussed by academics for several decades [2]. However, evidence from the IS literature indicates that there is no consensus on what stakeholder groups should be canvassed when measuring the success of IS. Many researchers are found to have measured IS success or impact from different groups of stakeholder, thus making results difficult to compare [2] and seldom provide rationale on the selection of the stakeholders. Besides that, researchers have different classification of IS stakeholders, for example Rai and friends [3] have classify IS stakeholders as Operational, Tactical and Strategic; Shang and Seddon [4] identify IS stakeholders as Strategic, Management and Operational; Sedera and friends [2] have measured IS success from the Strategic, User and Technical perspectives and Wu and Wang [5] have include the perception of external users thus have classify IS stakeholders in this context as Internal Project Team, External Contractor and System Users [5]. Furthermore, a few IS success researchers argue that IS Success can be measured from only selected level of stakeholders [1], thus different measures are needed for different level of stakeholders. Others suggest that canvassing all level of stakeholders would be beneficial to arrive at a complete and meaningful IS success score [2].
From the review of literature on IT/IS evaluation in Malaysia, it is seen that only two studies mentioned different type of IS stakeholders in their studies. These studies, however, did not provide any description of the stakeholder. Furthermore, both of these studies reported different classifications. The first study collected evidence from the perspective of Strategic, Operational, Technical, and Professional stakeholders, while the second study discusses how IS affects the work and organisation structure from the perspective of the Middle-level Managers, the Top-level Managers and Support Staff [6]- [8]. Thus, further investigation of IS stakeholders in Malaysia and whether different type of stakeholders have differing experience with IS will be of benefit for cumulative knowledge in IS success or impact, specifically to IS research in Malaysia and generally to the whole IS society .  T  stat  an  Gov  (SP  curr  (Ma  terr  200  elev  dep  eac  use  fun  dev  yea  cop  Dep   T  mea  intr  coll  ben  anti  The  dim  two  Org  date  Info  syst  tim  pred  mod  gen  acro  syst  per  ans  ena

V. RESPONDENTS CLASSIFICATION
A total of 415 questionnaires were distributed to SPEKS users at 26 departments across four state governments. The survey received a total of 310 questionnaires, thus indicate 75% response rate. From the data cleaning process, 56 respondents were removed from the data due to perceived frivolity leaving 254 valid respondents to be used in the analysis. Fig.3 presents the number of respondents according to the state government. Using the classification provided by Anthony [14], and Sedera and friends [2], the respondents were classified into three employment cohorts (Managerial, Operational and Technical), hence the IS stakeholders in this study context, based on the respondents job title and their job descriptions pertaining to SPEKS uses provided by the respondents in the survey. The classification process revealed that 17 (7%) of the respondents are from the Managerial level, 222 (87%) of the respondents are from the Operational level and 5 (2%) from the Technical staff (see table I).

VI. FINDINGS AND DISCUSSIONS
In this section, we will present the findings from the statistical analysis. The analysis addresses the following objectives: (1) to assess whether different cohorts have different views on the impact of SPEKS to their organisation, (2) to assess whether these cohorts able to assess the impact of SPEKS using all IS-Impact dimension, and (3) to identify whether different cohorts have different emphasis of the IS-Impact dimension. First, we will find out if different cohorts have differing views on the impact of SPEKS to their organisation. Fig.4 presents the mean scores for each dimension given by the cohorts. All mean scores are above the scale midpoint (3.5). This indicates that all cohorts demonstrate higher agreement with the impact and quality aspects of SPEKS. From the bar charts we can see that a certain cohort has scored higher on certain dimension (as indicated by the arrows). The Managerial cohort has scored Organizational Impact higher and System Quality lower as compare to the Technical and Operational cohorts. On the other hand, the Operational and Technical cohorts have scored the Individual Impact slightly higher than the Managerial cohort. The bar charts clearly demonstrate that a certain cohort may have different opinions on the impact of SPEKS based on their experience with the system. It may also suggest that a certain cohort may have a closer experience with the measures in the dimension, for example, the Managerial cohort may have more knowledge on the Organizational Impact measures, and therefore, the respondents in this group are experiencing the impact of SPEKS to the state government more than the rest of the cohorts. The mean scores have also suggested that the Operational and Technical cohorts are experiencing the benefits that SPEKS has provided to them as the user of the system, therefore scores for the Individual Impact dimension given by these two cohorts is higher compare to the Managerial group.
Next, we would like to see if these three cohorts are able to assess the impact of SPEKS using all IS-Impact dimensions. We assume that these three cohorts can evaluate the impact of SPEKS using the same measures and there is no significant difference between these three cohorts. Due to small sample size for the Managerial and Technical cohorts, a non-parametric alternative to ANOVA, the Kruskal-Wallis test, was conducted. The result from this test is presented in  The result shows that there is no significant difference (p>0.05) in the mean scores for Individual Impact, Organizational Impact and Information Quality as given by the three cohorts. However, there is a significant difference of mean score for the System Quality (p<0.05) as given by the three cohorts. The mean rank table (table III) indicates that the Technical cohort has the highest mean score compare to the Managerial and Operational cohorts. Therefore, this demonstrates that the Technical cohort has provided a higher score for the System Quality measures when evaluating SPEKS. This result may suggest that, when measuring the impact of SPEKS, all cohorts demonstrate similar perception on the Individual Impact, Organizational Impact and Information Quality. However, the Technical cohort has different perception on the System Quality aspect from the Managerial and Operational cohort. Lastly, we try to find out if different cohorts have different emphasis of the IS-Impact dimensions. In order to address this issue, we will investigate the relative strength given by each cohort on the dimensions in the model. We assume that if a cohort has a particular interest in one of the dimensions in the IS-Impact model, a strong correlation between that particular dimension and a criterion measure will be demonstrated. To test the claim, the strength of the correlation coefficients of the three employment cohorts (Managerial, Operational and Technical) with each of the dimensions is investigated. This is done by correlating individual dimension scores with a criterion measure for each dimension, for three separate groups of employment cohorts. Results from this analysis are presented in table IV.
Results from the correlation analysis depict significant relationships between two of the three employment cohorts (the Managerial and Operational cohorts) with all of the dimensions of the IS-Impact model. However, it is observed that the correlations between the criterion measures with the dimension's items are stronger for certain cohorts at certain dimension. Referring to table IV, the Managerial cohort demonstrates strongest and significant relationship with the Organizational Impact dimension. This indicates that respondents in the Managerial group almost have similar perception on the impact of SPEKS to the state government, thus, demonstrates a stronger emphasis on the Organizational Impact measures than the rest of the dimensions.
Meanwhile, the Operational cohort demonstrates strongest and significant relationship with System Quality dimension. The Technical cohort, on the other hand, demonstrates strong and significant relationship with Organizational Impact. Although there are small numbers of respondents from the Technical cohort, these respondents are observed to have similar perception when evaluating SPEKS in relation to the impact that the system has given to the organisation (Organizational Impact) based on a very strong and significant coefficient.

VII. CONCLUSIONS
This paper addresses several key questions that relate to the perspective of IS stakeholders (referred in this paper as 'employment cohort') on the impact of IS. Previous studies have measured IS Success or Impact from different groups of stakeholders. However, we argued that being selective on a certain group or groups of stakeholders may result in partial evaluation of the IS. With the main objective of understanding the perception of different group of stakeholders on the IS, this study analysed data from 254 users of a financial systems at four state governments in Malaysia using the extended IS-Impact model as the measuring instrument.
We found that, different stakeholders may have different opinions on how much SPEKS have affecting them and their organisations. It is also an indication that a certain stakeholder group is experiencing a certain aspect more (reflected by a higher dimension score) than the other dimensions. For example, the Managerial group of users may have experienced the impact of SPEKS to the state governments more than the other stakeholders, hence a higher mean score of Organizational Impact is observed from the Managerial group compare to the rest of the dimensions. Findings from this study show that it is important when measuring an IS, all views from all groups of stakeholders (those who are affected by the system directly or indirectly) should be accounted. Collecting data from only a certain group of stakeholders may have resulted in partial evaluation of the system, thus biasness towards the system may be introduced.
We further observed that; (i) the Managerial and Operational cohorts have significant and strong relationships with all dimensions in the IS-Impact, (ii) the Managerial cohort demonstrates stronger emphasis on the Organizational Impact, (iii) the Operational cohort place more emphasis on the System Quality, (iii) although there are evidences that the Technical cohort demonstrate strong relationships with three of four dimensions in the model, however, because of small number of Technical respondents the correlation are non-significant, and (iii) all employment cohorts demonstrate strong (r>0.5) correlations with almost all dimensions in the IS-Impact model (although coefficients are non-significant for the Technical cohort but still the coefficients demonstrated high relationship with all dimensions except System Quality). With these findings, we recommend that IS should be evaluated across all groups of stakeholders of the system being evaluated in order to arrive at a holistic impact score.