Evaluating Services Computing Systems Engineering Framework using An Acceptance Model

The gap between business services and IT services becomes a major concern in services computing. As an approach for service-based IT solution, services computing systems are promised to be able to bridge the gap between these services. The implementation will require an engineering framework as a guide to building the systems. The framework needs to be evaluated to provide important feedback to the framework development. This paper outlines the evaluation of SCSE framework through an acceptance model. The study develops an acceptance model based on the experiences of a group of engineers after using the framework to build smart campus services systems. A survey involving 54 systems engineers with various engineering backgrounds was conducted to assess the experiences of the engineers in using the framework. The results o f the acceptance model show that both perceived ease of use, represented by the level of agreement ( υ1) and perceived usefulness, represented by the level of importance (υ 2) deliver good results almost for the entire stages of the proposed framework. In addition, the user experiences of using the proposed framework are in the acceptable levels. The contribution of this paper is an enrichment of the engineering methodologies for the service-oriented system from the perspective of services computing. Keywords— acceptance model; framework; services computing systems; services computing systems engineering; smart campus.


I. INTRODUCTION
Service innovations are required by each organization to improve its business services [1], [2]. Service innovations can be carried out by proposing new services or improving existing services. New services can be provided by enhancing existing services [3]. Better services can be provided to its users by improving their service experiences [4]. This can be achieved through the service technology and features improvements. Service technology and architecture, such as web services, cloud computing, mobile computing, and service-oriented architecture (SOA) have evolved in providing opportunities for realizing IT services systems that enable the service innovations [5], [6]. This circumstance will trigger the ability to present the systems that support business services innovations.
One of the challenges in services computing research is the service design process. This challenge is also a fundamental research problem in services computing [7]. Service design comprehension is essential for building and developing a service computing-based system [7]- [9]. So far, a service design has not been based on a formal model of services computing systems [7], [10]- [12]. Service design in a large complex system often disregards the necessity for unifying systems engineering framework [13]- [16]. Highly complex services systems drive the need for an engineering framework. The required framework must support the interaction among services systems components with diverse functionalities. Thus, an engineering framework to build the systems is urgently required in this field. An engineering framework for services computing systems has been proposed in previous studies [17], [18]. The framework is expected as a structured guide in developing the systems. However, the framework needs to be implemented and evaluated in the real systems environment [18].
From a service domain, studies on service engineering have been accompanied by various forms of research streams [19]- [21]. Some of these streams are commonly based on the SOA concept. From the more comprehensive services systems domain, the studies on services systems engineering have also been done extensively [15], [22]- [25]. The studies combine service engineering and systems engineering methodologies. These works offer a framework to be used as a process guide in engineering services and services systems from various perspectives.
Two perspectives underlie the concepts of services computing systems. First, from the view of systems characteristic, services computing systems are the main research subject that collaborates IT-enabled services systems, SOA, and services computing technology. The systems cover IT services, business services, and service values. So instead of focusing only on IT services, the systems must consider and meet the needs of business services [26]. The systems are built based on an alignment consideration between business and IT services [27].
Furthermore, the implementation of the systems must be able to provide service values to the organization [28]. The second perspective takes the view of systems engineering. Services computing systems engineering is considered a collaboration of multi-disciplines engineering methodology, i.e., services systems engineering, services engineering, and systems engineering. Services systems engineering combines SOA-based service engineering and systems engineering. From the SOA-based service engineering approach, the methods implement SOA principles to meet business service needs. From a systems engineering approach, principles and life cycle of systems engineering are adopted.
There are limitations to the previous studies. A service engineering methodology in general emphasizes the SOA approach to design IT services that support the business services. Some popular SOA based service engineering methodologies, i.e., SOMA, SOAF, MSOAM, and SOAD [21], [29] focus solely on IT services design based on the given business services needs [25] [17]. The methodologies commonly shield the design of individual IT services without considering the characteristic of systems. This circumstance may risk the lack of the systems integration of the services. In practice, the analysis of business needs domains are typically required, such as in term of business strategy, business service model, business service requirement, business service innovation, service technology adoption, and service performance. The activities shall be fully considered to guarantee the alignment between IT services and business services [30].
The methodology of services systems engineering emphasis is on the development of services systems to yield business services based on the systems perspective [31]. However, a gap still exists since the methodology does not cover the design and analysis of IT services. Some undertakings that emphasis on the analysis of existing IT systems and implementation strategy for technology adoption is a lack in the methodology. The concern for IT services operationalization process in enabling the service's systems is also neglected.
A systems engineering methodology should be a key consideration in building the services computing systems that align both IT and business services [17], [31]. The methodology elaborates system design and development process comprehensively and systematically. It should also consider a multi-discipline engineering approach in designing and realizing the systems [32]. From the engineering approach, the services computing systems engineering practices a similar approach with the services systems engineering. It should also consolidate more extensive engineering knowledge to promote collaborative value creation of the systems [31], [32].
In this study, services computing systems engineering (SCSE) framework is defined as a methodology for developing the services computing systems. It covers the design, development, deployment, and evaluation of the systems. The methodology covers a complete engineering life cycle that enables the design and the implementation of the systems in a systematic manner. The methodology is visualized as stages and phases (sub-stages) in developing the systems. Fig. 1 shows the SCSE framework that enhances the previous studies [17], [31]. The framework is constructed from the meta model and lifecycle of services computing systems [31]. The SCSE framework covers two types of services, i.e., business services and IT services. In the area of services computing, IT services represent software services. The framework contains five stages, i.e., objectives and requirements, modeling, development, deployment, and evaluation. The first stage focuses on the identification of service strategy, objective, requirements, and innovation. The stage shall be based on the business needs with main consideration to improve the business service of the organization. There are three phases covered in the stage, i.e., service strategy and objectives, service requirements analysis, and service innovation. The second stage focuses on service modeling and system design. There are three phases covered in the stage, i.e. business service modeling, IT service modeling, and service design and architecture. This is the essential stage of the framework. The third stage comprises three phases: service development, service integration, and testing, and service implementation. The stage requires the engineers to implement the design and architecture of the service from the second stage. The implementation of the stage relevant to the software development activities, but with a service-oriented approach. [33]. The fourth stage covers three phases: service migration and roll-out, service operation and maintenance, and service monitoring. The stage focuses on the deployment and operation of the systems. Finally, the fifth stage focuses on the evaluation process of the systems. There are three phases included in the stage, i.e., service performance measurement, service analysis, and optimization, and service improvement. Services dependability can be used to measure and evaluate the system's performance from the internal systems perspective [34], while services quality can be used to measure the performance from the external perspective [32], [35].
The purpose of this paper is to evaluate the framework using an acceptance model. The motivation of this research is to provide a guide for building the systems through a validated engineering framework. The evaluation delivers important feedback to the improvement of the framework. Furthermore, the evaluation of the framework can maximize the utilization of the framework. The remainder of the paper is ordered as follows. Section II describes the materials and methods for evaluating the framework. Section III discusses the results, and section IV provides the research conclusion.

II. MATERIALS AND METHODS
In assessing a newly designed artifact, evaluation is an important step to be made [36]- [38]. The artefact produced in this work is the engineering framework of services computing systems. The evaluation provides important feedback to the framework development [28]. A practical evaluation methodology is used to evaluate the proposed framework based on the experiences of the systems engineers after using the framework. This leads to the construction of the acceptance model. The evaluation methodology used in the study involves the implementation of both qualitative and quantitative techniques.
The study develops an acceptance model to evaluate the framework based on the experiences of the engineers in using the framework. Fig. 2 shows the methodology for evaluating the framework. First, the group of potential systems is identified to support the evaluation in a real environment. Second, an acceptance model is built to define user experiences. Third, survey data is collected from the engineers involved in the systems development. Fourth, the acceptance model is run using the data that have been collected. Finally, acceptance results are obtained and presented.  Fig. 3 shows the acceptance model of the framework evaluation. The model adopts the technology acceptance model [40]. Four external variables are used in the acceptance model, i.e. Z 1 : level of the clarity of the framework (systematic stages, phases and steps), Z 2 : level of the clarity of the artefacts in every stage, Z 3 : level of the easiness for documenting the artefacts in every stage, and Z 4 : level of the easiness for a trace-back mechanism if error occurs. These variables will construct the perception of the users. There are two internal variables used in the acceptance model that construct the perception results of the systems engineers, i.e., perceived ease of use (υ 1 ) and perceived usefulness (υ 2 ). In this study, υ 1 is represented in the form of a level of the agreement while υ 2 is represented in the form of level of importance. Level of the agreement states the level of understanding, clarity, and ease of each stage. The variable represents the answers to individual perceptions in understanding each stage, phase and steps proposed in the framework. On the other hand, the level of importance states how important the role, contribution, and evaluation of each phase of the entire stages. This variable represents the individual's perception of the importance of each stage, the phases, and steps proposed in the framework. The acceptance model (υ) for both υ 1 and υ 2 are defined as follows:

A. Acceptance Model
Where μ(Zυ 1 ) and μ(Zυ 2 ) are the average value of Z for υ 1 and υ 2 respectively, Z i is the value of Z for both υ 1 and υ 2 where i : [1..4] and Z 1 , Z 2 , Z 3 , Z 4 ∈ Z. Sυ 1 and Sυ 2 are the stage value for each υ 1 and υ 2 . The model of Z i for υ 1 and υ 2 are defined as follows: Where ω s (Cυ 1 ) and ω s (Cυ 2 ) are the weighted factors for the criteria used in υ 1 and υ 2, respectively, with the range value of 20 ≤ ω ≤ 100 by 1 to 5 scales. Rυ 1 and Rυ 2 are the response value for both υ 1 and υ 2 for every Z i. σ ij : Binary variable with value 1 when the υ 1 response of engineer i is valid on stage j, 0 otherwise. τ ij : Binary variable with value 1 when the υ 2 response of engineer i is valid on stage j, 0 otherwise. Np is the number of participants involved, and M is the number of stages.

B. Data Collection
This research conducts a case study to evaluate the proposed framework. For these purposes, a case study for smart campus services systems development is fully conducted. This study collected data and information through a case study of smart campus services systems development. SCSE framework was tested on the development of smart campus as a representation of service computing systems. The case study produced a smart campus design and application.
Furthermore, the development of smart campus services systems represents the engineering process using the SCSE framework. The practical experiences of the system engineers as participants in using the framework were evaluated using the acceptance model. There are 18 services systems developed in this case study that involve a group of systems engineers (See Table I). Each service system is assigned to different systems engineers. Thus, for each service system i selected, the engineer is assigned to work on the service system i.
There are 54 engineers with various engineering backgrounds involved in the framework evaluation, i.e., Information Technology (IT), Information Systems (IS), Information Technology Services (ITS), and Software Engineering (SE) (See Fig. 4). A training and knowledge sharing is delivered to the systems engineers, both from the theoretical and technical concepts that are relevant to the application of the proposed framework. Each engineer is also equipped with a technical briefing regarding the service systems characteristics and the technique to develop the systems using the proposed framework. The acceptance model in this study is implemented through a survey. Respondents of the survey are the systems engineers that involved in the case study. A survey was conducted to assess the experiences of the systems engineers after using the framework. The objective of the survey is to observe the feasibility of the proposed framework, constructed from four variables [21], i.e. Z 1 , Z 2 , Z 3 , and Z 4 (See Section IV.B for the details). The questionnaire is based on two acceptance models, i.e., level of agreement (υ 1 ) and level of importance (υ 1 ). The questionnaire uses a Likert Scale with 5 value for each of the acceptance model based on level of agreement (1: strongly disagree, 2: disagree, 3: neutral, 4: agree, and 5: strongly agree) and level of importance (1: not important, 2: slightly important, 3: moderately important, 4: important, and 5: very important). Table I shows the list of service systems that are used to evaluate the framework. The systems are used as a medium to implement the framework. Group of engineers is assigned to build the systems using the proposed SCSE framework.

A. Systems Identification
The set function of the services system (SS) is formulated as follows:

SS SI S O SS O S S SS
Where SS: service system, SI: services interface, SS i ←SI 1j , SI 2j , SI 3j , …, SI ij , S: service, SS i ←S 1j , S 2j ,.. ,Sij, O: operation of service, S j ←O j1 , O j2 ,..,O ij , m: number of service in service system i, n: number of service system. An operation O pertains to service in its relevant service system, and a service interface SI belongs to its respective service system SS.  [41]. Each service system contains a various number of services interfaces, composite services, and service operations. The numbers are obtained from the design results produced by the engineers when building the system. The differences in the number of SI, S*, and O show that the systems are varied as well as the size of the systems. Smart learning management system and teaching management system are two service systems that have the most number of service interfaces. Although the size of the service systems varies, in principle, the systems engineers will get the same environment in using the SCSE framework. The practical experiences of the engineers in building the systems using the proposed framework are the main data for the evaluation. Table II and Table III show the results of data processing for both levels of the acceptance model υ 1 and υ 2 for every Z value based on stages (Stg). Each Z value for both υ 1 and υ 2 shows the average perceptions of variable Z for each stage. The perceptions of the engineers are assessed based on the criteria of Cυ 1 and Cυ 2 which have been corrected by the weighted factor ω Z . It can be seen from the Table II that    The result of the acceptance model for level of importance υ2 shows the results that are slightly different from the level of agreement (See Table III). The results come with a relatively higher average value. The highest average value for the level of importance υ 2 is in the variable Z 4 which reaches 84.6, then followed by the variable Z 1 which reaches 84.1, variable Z 3 which reaches 82.6, and the variable Z 2 with the lowest average value reaching 82.5. The results indicate that the engineers' perceptions of the variable Z 1 and Z 4 are likely to be better than two other variables, although the difference is relatively small.

A. Acceptance Results
Meanwhile, if the results are breakdown per stage, it can be seen that the average value of the importance level shows the results that are relatively higher than the level of agreement. The average value of the engineers' perceptions of stage 1 (86.4) and stage 2 (85.9) occupies the two highest positions, followed by stage 3 and stage 5 which have the same value (82.9). This indicates that these stages play a very important role in implementing the framework. Showing from the value of R 2 , the variable Z 4 gives the highest value of contribution in the acceptance model for the level of importance, which reaches 93.10 percent. This is slightly different from the results in the previous model. From a level of the important point of view, the easiness for a trace-back mechanism, if an error occurs is a top priority for the engineers.  variables. This is indicated by the number of percentages that agree to the four Z variables above 70 percent (agree and strongly agree). The total percentage of neutral is below 30 percent. Meanwhile, the percentage for disagree and strongly disagree criteria are zero. The results indicate that the proposed framework can be well received by the engineers. The engineers agree that the framework can be easily understood and implemented, the artefacts can be easily evaluated, the output can be clearly documented, and an error trace-back mechanism can be done simply.
The following are the detailed results of the level of agreement for all Z variables. For the variable Z 1 , there are 72 percent of the engineers who agree that the stages, phases, and steps in this framework are clear, systematic and easy to understand, while 18 percent of the engineers are neutral, and none of them state disagree. For the variable Z 2 , there are also 72 percent of the engineers who agree that the steps of each stage and phase are clear, can be well implemented, and easy to be evaluated, while 28 percent of the engineers are neutral, and none of them state disagree. For the variable Z 3 , it represents 73 percent of the engineers who agree that the results of each stage and phase are easily documented, while 27 percent of the engineers are neutral, and none of them state disagrees. For the variable Z 4 , there are 71 percent of the engineers who agree that when an error occurs in a particular stage and phase, it is easy to do a trace-back mechanism, while 29 percent are neutral, and there are no engineers who respond did not agree.   6 shows the assessment result of the acceptance model for the level of importance υ 2 . It can be seen generally from the Figure, that the responses to the proposed framework for the level of importance are good for all Z variables. This is indicated by the number of percentages that agree to the four Z variables above 75 percent (very important and important). This percentage value tends to be better compared to the results of the level of agreement υ 1 which is only above 70 percent. Even for Z 1 and Z 2 , the percentage value is above 80 percent, while for Z 3 and Z 4 , the percentage value is close to 80 percent. The total percentage of neutral is below 25 percent. In addition, the percentage of slightly important and not important criteria are zero. The results indicate that the proposed framework has a good level of importance in almost all Z records. The engineers consider that it is very important to understand and implement each stage, phase, and step easily, it is very important to easily evaluate the artefacts of each stage, it is important to easily document the results, and it is important to do an easy trace-back mechanism.
The following are the detailed results of the level of importance for all Z variables. As for the variable Z 1 , it represents 83 percent of the engineers who feel the importance of a systematic framework, while 17 percent of respondents feel moderately important, and none of them states not important. For the variable Z 2 , there are 81 percent of the engineers who feel the importance of step clarity and the results of each stage and phase so that it can be easily evaluated, while 19 percent of the engineers feel moderately important, and no engineers stated not important. For the variable Z 3 , there are 78 percent of the engineers who feel the importance of ease in documenting the results, while 22 percent are neutral, and none of them stated not important. For variables Z 4 , there are also 78 percent of the engineers who feel the importance of ease in documenting the results, while 22 percent of the engineers are moderately important and no engineers who respond not important.
Based on the results of the acceptance models above, both perceived ease of use, represented by the level of agreement υ 1 and perceived usefulness, represented by the level of importance υ 2 , show good results almost for all stages of the proposed framework. Overall, each stage of the framework shows the good performance of ease and clarity in building the systems. The acceptance results of the Z values are promising especially on stages 1, 2 and 5. Each stage, phase, and step can be easily understood and well implemented. Outputs and artefacts of the stages can be properly evaluated. The outputs of the stages can be well documented. If something goes wrong, a good evaluation and trace-back mechanism can be systematically carried out.

IV. CONCLUSIONS
This paper describes the evaluation of the SCSE framework. The framework is implemented through a smart campus services systems development and involving systems engineers with various engineering backgrounds. The evaluation of the framework is based on the acceptance model that delivers the experiences of the engineers of using the framework for building the systems. The framework evaluation is successfully conducted with promising results. The results of both acceptance models, in general, show a good acceptability level. The perceived ease of use, represented by the level of the agreement and the perceived usefulness, represented by the level of importance, show good results almost for the entire stages of the framework. According to the acceptance results, it can be seen that the user experiences of using the proposed framework are in the acceptable levels. Our future research aims to elaborate the SOA principles with the proposed framework by SoaML to build the services computing systems. Thus, several key principles of service-oriented architecture design will be fully covered in future efforts.