Indonesian online learning system evaluation framework based on UTAUT 2.0

known as PPG


Introduction
Currently, the role of information and communication technology (ICT) cannot be separated from any activities and works. Especially in education, the use of ICT is to communicate, create, disseminate, store, and manage information [1]- [3]. Moreover, a learning process with face-to-face method has been long time combined with technology assistance. In its application this, method commonly known as online learning. This method is able to conduct a flexible learning process that can be done anywhere and anytime as long as user connected to the internet. Moreover, it can provide a sufficient learning quality in a wider scope [4], [5].
In Indonesia, one of ICT utilization in online learning is SPADA referred as Sistem Pembelajaran Daring Indonesia. It is a program from Indonesian Ministry of Research, Technology, and Higher Education to increase equitable access of quality learning in higher education. SPADA is developed to answer several education challenges [6], which are (a) limited capacity of university, (b) limited scope of university due to uneven distribution, (c) there are still many universities that do not have adequate and quality educational resources, (d) higher quality universities are still concentrated in Java, and (e) a lack of equal and quality higher education services. Therefore, the use of ICT is expected to be a solution with a concept of networked higher education and online learning.
One of SPADA application is to provide and accommodate a teacher profession education program which known as PPG. This program is actualization from law about teacher profession which state that teacher should have academic and competency qualification, teaching certificate, physically and mentally healthy, also have an ability to realizing the goals of national education. This regulation become a basic Indonesian teacher reformation that teacher should prepare through PPG after completed a bachelor degree. In other words, this program will prepare a graduates in either education or non-education majors who have an interest to become a professional teacher.
The implementation of PPG is consists of three main stages which are online learning, workshop, and teaching practice at school. Fig. 1 show the implementation flow of PPG. Based on the flow, it can be seen that the early 12 weeks of the program is done by completed the course of pedagogic competency through an online learning. The online platform used is Brightspace web-based learning management system. A several features in this platform are discussion forum and virtual classroom. The advantage of this platform is instructor can monitoring an improvement of learning participant such as user activity, visited content, learning achievement, final exam, and quizzes. Based on the program phase, it can be seen that the most proportion required is the first phase which are online course. From that phase, an instructor will evaluate the participants' learning achievement and filter participants who deserve and succeed to continue to the next phase. In this case, the use of web-based platform has an important role in a learning process. Therefore, it is required to evaluate the system quality, also the user intentions to use a system. An evaluation is used to examine the effectiveness of content and features that provided in the system. Also, it will improve the system quality based on users' technology acceptance.
One of evaluation model used to test the system is unified theory of acceptance and use of technology (UTAUT). This model has four main factors in acceptance and use of technology, which are (a) performance expectancy, (b) effort expectancy, (c) social influence, and (d) facilitating conditions [7]. That four factors will affecting technology behavioral intention and use behavior. Nowadays, the use of UTAUT has been improved become UTAUT2. This model extends the previous model of UTAUT to study acceptance and use of technology [8].
Previous studies shows that UTAUT describes an acceptance and use of technology in the educational fields [9]- [14]. In this case, an evaluation of the SPADA system needs to be done because the learning implementation of PPG students are new and different. So, acceptance, use and adaptation will be different for each user. This evaluation will be useful for explaining the differences in use and acceptance so that it will produce a feedback that can be used as a reference for improving the system according to user needs.

Unified Theory of Acceptence and Use of Technology 2.0 (UTAUT2.0)
Generally, UTAUT is a technology acceptance model. This model is a development and synthesis of several existing technology application models including theory of reasoned action (TRA), technology acceptance model (TAM), motivation model (MM), theory of planned behavior (TPB), combined TAM and TPB, model of PC utilization (MPTU), innovation diffusion theory (IDT) and social cognitive theory (SCT) [7], [15], [16].
Along with the development of information technology and communication, UTAUT also developed into new version known as UTAUT2. This developed model added three main factors in its implementation which are (a) hedonic motivation, (b) price value, and (c) habit [8]. It's produced a variation of substantial improvement in behavioral intention and technology use. Fig. 2 shows the initial model of UTAUT2.
Based on the initial model of UTAUT2, it consists of several constructs or variable. Those variables are independent, dependent, and moderating variable. Independent variables are seven main factors that affecting dependent variables (behavioral intention and use behavior). While, the three factors which are age, gender, and experience acts as moderating variable. The first variable is a performance expectancy (PE). This is a level of an individual's belief that using the system will help him get benefits at work [10], [17]. The higher level of trust will make him to use the technology longer. In its application, there are five constructs related to performance expectations. The five constructs are perceived usefulness, extrinsic motivation, job fit, relative advantage, and outcome expectations.
The first construct, perceived usefulness, is related to the benefits when using a certain technology, which will facilitate his job. The second construct, extrinsic motivation is the perception of doing something to get results or rewards from the work such as wages, bonuses, and others [18]. Furthermore, job fit is the suitability of the system functions that will help someone's personal work. The fourth construct is the relative advantage associated with the extent to which the system development will be better than the previous one. The final construct is the outcome expectations associated with the expected results achieved by using a system.
The second variable is effort expectancy (EE). This construct defined as the level of ease associated with using the system [9], [19]. In other words, it refers to the expectation of ease of doing work when using a particular system. This variable has three basic constructs, which are perceived ease of use, complexity, and ease of use.
The perceived ease of use is related to the level of user confidence in the use of a system [20]. In other words, the use of a system can ease the work effort. The second construct is the complexity associated with the level of ease and use of a system. This is because the level of convenience varies between users. Each user will feel the difference in manual work with system assistance which impacts on their level of understanding of a system. The final construct is ease of use associated with innovation and development of a technology or system. This causes a different experience when a system is experiencing development or innovation. Furthermore, users will feel whether the developed system is better used or not.
Social influence (SI) as a third variable is defined as how other people influence the use of a system [21]. This shows that the role of other people influences the user to use a system or not. This variable has three main constructs which are subjective norms, social factors, and views.
Subjective norms mean the user's perception of someone who is important to him will think that the user must use a certain system. The second construct is social factors related to the social environment. This means that the social environment will influence someone to use a certain system. The last construct is the view or impact of using the system on one's views. This is related to the use of a system that can increase the view of a person's social status.
Next, the fourth variable is a facilitating condition (FC). It is defined as organizational and technical infrastructure will affecting and supporting a system [22]. This variable has three main construct [7]. The first construct is the perception of behavioral control that reflects internal and external perceptions including the condition of facilities, resources, and technological facilities. The second construct is a facilitating condition related to the environment that supports the use of a system both from the system itself and from others. The final construct is compatibility which is related to the consistency of innovation against the value, needs and user experience.
In the fifth variable there is a hedonic motivation (HM). Hedonic motivation is a pleasure thing from using technology or systems. This will be an experience and an important factor for users in using this technology [23]. If the user experience is pleasant, then users will tend to use the technology. This will affect the acceptance level and use of a technology or system.
The sixth variable is a price value (PV). From the consumer side, the price value is related to the costs required to use a system or technology [24]. This is related to the feasibility of a system for the costs incurred to access it. But, this does not apply if the system or technology used is free or open source. This factor also affects the user in the intention to use a system.
In the seventh variable, there is a habit (H). This is related to the pattern of using the system in their daily life. In other words, this familiarity is a factor in using a system [25]. The more accustomed to users using a system, the impact on their daily work. Interest or behavioral intention (BI) as an eighth variable can be interpreted as a person's level of exploiting a system [7], [26]. The higher the interest and desire in using a system will have an impact on the intensity of its use in that system. This is the dependent variable which is influenced by the previous variables. And the last variable is a use behavior (UB). This is the way and intensity of a person using a system [10]. This is influenced by the factors or independent variables previously described.

Evaluation of Indonesian Online Learning System
The implementation of ICT are related to the user acceptance and understanding of technology [27]. It becomes an important factors that affecting the success of technology implementation [27], [28]. The user acceptance is determined by two factors which are (a) user's perception about technology that will assist their works, and (b) the level of difficulty in using technology. It means that the use of technology should be easy to operate.
In a technology usability, there are five aspects to evaluate the system or technology. First aspect is a learnability related to the ease in learning a system. Second aspect is a rememberability related to the ease of remember to operate the system. Third aspect is an efficiency of use the system. Fourth aspect is a reability in use means that system should reduce the human error in their works. And the fifth aspect is a user satisfaction when using a system.
As we mention before, this study proposes an evaluation for Indonesian online learning system (SPADA) for teacher profession education program (PPG). In its application, SPADA has a several features to support an online learning. These features such as learning content, discussions, course tools and so on. Fig. 3 show the preview of course overview in SPADA web page. Moreover, the next sub-section will discussed an evaluation design for SPADA.

Evaluation Model
This study uses a quantitative approach with the UTAUT2 model for evaluating the SPADA. In its implementation, there are several variables tested according to the UTAUT2 model. However, not all variables are tested because they will be adjusted to the needs and conditions of system use. Fig. 4 shows the model used in this study and adopted from the UTAUT2 model. Fig. 4, there are several differences from the UTAUT2 model. The first difference is that from the independent variable there is no price value variable. This is because in using the SPADA users do not need to pay extra for accessing it. The use of this learning system is an integral part of the PPG program. Then the moderating variable such as age, gender, and experience also not measured because it does not directly affect the dependent variable. Therefore, further research can be done to discuss a moderating variables in UTAUT2.

Evaluation Design
A quantitative approach is used to analyze data using numerical data and processed by statistical analysis methods. After the data is processed, the results will be described and conclusions based on the statistical analysis. The data collection method used in this study is to collect information from the sample using a questionnaire. Meanwhile, the data analysis used is path analysis [29], [30]. Path analysis is a form of multiple regression statistical analysis used to evaluate causal models by examining the relationship between the dependent variable and two or more independent variables [31]. Moreover, Fig. 5 illustrates the evaluation design in this study.

Fig. 5. Evaluation Design
The population is selected for participants who finished the online learning stage through the system to be tested. Then, a purposive sampling technique is used with the selection of a group of subjects based on certain considerations. The number of samples with an error rate of 5% by looking at the table for determining the number of samples.
The next important thing is the evaluation instrument to measure and analyze a data. A questionnaire is used as an evaluation instrument that would be distributed to the research sample. The questionnaire contains questions arranged based on the adoption of the UTAUT2 model. The questions are structured by adjusting the variables used only in this study. Table 1 describes the indicators of the variables in this research model. Furthermore, the weighting uses a Likert scale in the questionnaire using five answer choices consisting of strongly disagree (1), disagree (2), neutral (3), agree (4) and strongly agree (5). Based on Table 1, it can be seen that there are several indicators from each variable. These indicators are the basis for making the questionnaire questions to be tested. Each indicator can be broken down into several questions that refer to that topic. An example of the question such as "using the system in my job would enable me to accomplish tasks more quickly" for PE1; "it is easy for me to learn operating the system" for EE1; and "I use the system because of the coworkers who use the system" for SI1. The question is adjusted to the intent of each indicator. After collecting the data, in the last stage the data will be processed using statistical techniques. The analysis used is path analysis, wherein the results of the analysis will be described. Then, based on the results, a conclusion and suggestions can be used as a reference for system development.

Conclusion
This paper proposed an evaluation model for Indonesian online learning system (SPADA). It is used for accommodate online learning of teacher profession education program (PPG). The evaluation is to discover user acceptance and understanding of technology. It becomes an important factors that affecting the success of technology implementation. The user acceptance is determined by two factors which are (a) user's perception about technology that will assist their works, and (b) the level of difficulty in using technology.