The Analysis of Rubric Feasibility Using Video Snippets of Learning Process

Multiple interpretations can cause a decrease in the quality of rubrics. This research aims at developing an appropriate rubrics description for the measurement of students' scientific attitudes. The process of developing the rubrics was done by using the video snippets of learning process. A total of 23 observers whose final year students were involved in analyzing the video snippets of learning process. They assessed eight indicators of the scientific attitude of ten students through the video snippets. The feasibility of the rubric is considered based on the responses given by 85% of the observers who have the similar answer to the scientific attitude of students in the video snippets of learning process. The description test of the rubrics was analyzed descriptively, which then obtained two rubrics that obtained answer differences less than 85% of the observers. Therefore, this research focuses on investigating the two rubrics. This research implies that there needs to be a clear description of the rubrics in relations to the time of observation of the number of students and behaviors.


INTRODUCTION~
. A rubric is a descriptive scoring developed by teachers or other evaluators to guide the analysis of the product or process of students' effort (Moskal, 2000, p. 1).Scoring rubrics are often used to measure students' performance when assessing complex performance-based tasks (Becker, 2016, p. 22).Scoring rubrics can be powerful tools for assessment and learning, if it is designed with the purpose and type of learning goal to be assessed in mind.
Rubrics usually include a hierarchical score with a numerical scale, descriptive, or both of them, and each level has its own description (Goldberg, 2014, p. 1).Scoring rubric can be said to be good if the description of the scoring criteria is clear.
Scoring criteria as an assessment tool must be valid and reliable.In fact, many teachers have known the terms 'validity' and 'reliability' in statistical studies, but they do not often discuss how these

RESULTS AND DISCUSSIONS
The Analyzing the observer assessment results

Revising rubric
Feasible Rubric   results are presented in Table 3.
Table 3 shows that there were two rubrics (2a and 3a) that were not feasible because the average percentage of observers' assessments was not yet suitable, which was below 85% of the total observers.Thus, those rubrics needed to be revised and re-tested.

Aspect Indicator Before Revision After Revision
2 a 4 = The student answers all the questions asked by the teacher.Table 5. Rubric Revision on the Aspect 3 Indicator a (Open-minded Attitude with the Indicator of Respecting Other People's Opinions or Findings).

Aspect Indicator Before Revision After Revision
3 a 4 = The student listens to their friends' opinions and does not make noise when their friends express their opinions.
3 = The student listens to their friends' opinions, but occasionally makes noise when their friends express their opinions.

Field Test 2
The second field test was conducted on the revised rubric, especially on aspects 2a and 3a.There were 20 video snippets observed (consisting of each of 10 videos for each observed student with 2 observed aspects).The current observers were different from the observers in Field Test 1, but the criteria used were the same.This was done so that the results obtained were more authentic by assuming this rubric could be used by anyone later.The results of the second test showed the overall assessment of the observers to the 10 students on aspects 2a and 3a.In general, the observer's assessment of each student in the video snippets showed a great similarity, namely above 85%.The average results of the second test also showed that aspects 2a and 3a had a mean of 100% and 96.4% respectively, so that the rubric could be said to be feasible.

Implications of developing
Doing assessment through direct observation leads to subjective results.The subjective assessment results can be resolved if the indicators in the assessment are clear, and do not cause many behavioral interpretations of the object being assessed.One way to minimize subjective assessment is by developing a rubric as scoring reference to assess the achievement (Dornisch &

Figure 1 .
Figure 1.Flow of Feasibility Test through Analysis of the Learning Process Videos.
feasibility test of the rubric focused on several aspects of scientific attitudes in during the learning process done by individual student, as follows: 1.a: Caring for the environment attitude, with the indicator of maintaining the cleanliness of the school environment; 2.a: Curiosity attitude, with the indicator of enthusiastically answering teacher questions; 2.b: Curiosity attitude, with the indicator of paying attention to the observed object; 3.a: Open-minded attitude, with the indicator of respecting other people's opinions or findings; 4.a: Critical thinking attitude, with the indicator of asking about any new changes or things; 4.b: Critical thinking attitude, with the indicator of repeating activities carried out; 5.a: Diligent attitude, with the indicator of working diligently on instructed tasks; 6.a: Careful attitude, with the indicator of re-checking answers Stage 3: Testing Rubric Feasibility Observing of video snippets of learning process Scoring of the scientific attitude of students through rubrics (by video observers))

Figure 1 .
Figure 1.Scene of Learning Video Snippets from the Attitude of 1.a (Caring for the Environment Attitude).
assessment of students' scientific attitudes in the video snippets of the learning process.The results were obtained from the observation sheet in the rubric based on the observed aspects.The observers' the number of observers who had chosen 1, 2, 3, or 4 of the assessments per aspect for each student in the video.After calculating the average percentage of the 10 students, the average similarity score of responses from each aspect was obtained.The results of the average percentage for each aspect and the feasibility of the rubric based on Test 1 observers provided a diverse rubric assessment of students' behavior in the video.It was suspected that the video did not show the number of students answering questions.However, the actual learning process was difficult to observe, since the observers took longer to observe the behavior of 10 students in the video snippets, and to calculate the number of answers each student in learning.Furthermore, it was difficult to get a question and answer between the teacher and students with a certain amount (in the rubric of more than three answers) in one learning frame with limited observation time.Further analysis, if the rubric mentioned number of students answering teacher questions, it was necessary to was not done due to considering the time, boredom, and focus of the observers on observing a number of video snippets with the duration of time and the number of videos.Therefore, in order to overcome this problem, the description of the rubric was changed

Figure 2 .
Figure 2. Scenes of Learning Video Snippets from Attitude 2.a (Curiosity Attitude with the Indicator of Answering Teacher Questions Enthusiastically).

2=
The student listens to the opinions of his friends, but laughs 4 = The student pays attention to the opinions of their friends from beginning to end without any disturbing behavior 3 = The student is not paying attention to the opinions of their friends, occasionally chatting or playing objects around.2=The student pays less attention to the opinions of their when the opinions of their friends are wrong.1 = The student does not listen to their friends' opinions.friends, frequently chatting or playing objects that are around.1 = The student does not pay attention to the opinions of his friends from beginning to end.The statements that contained dualism in different circumstances were changed.For example, "listens to opinions, but..." (In rubric 2 and 3 before revision).The statement was seen uncertain and can be used in different situations.It was considered that the use of language with positive intentions followed by negative statements could lead to multiple interpretations.Then, there was a need for consistency in the use of words student did not pay attention to the opinions of their friends, occasionally chatting or playing objects around.2 = the student paid less attention to the opinions of their friends, frequently chatting or playing objects around.The analysis of changes in the rubric was taken into account since stating clear situation mutually reinforcing between the statement and the rubric was important.The developed rubric must consider the use of methods that was easy to understand so that they could be used to specifically assess the students' behavior through observation.Similar to the revised form of rubric 2a, the frequency or number of these behaviors in the revision of section 3a could be observed directly at one time.The observers paid attention to one frame of the video that contained the students' behavior when expressing their opinion.This situation in the video could certainly be well observed without taking a long time.

Figure 2 .
Figure 2. Scenes of Learning Video Snippets from Attitude 3.a (Open-minded Attitude with the Indicator of Respecting Other People's Opinions or Findings).
directly at that time, while the measurement by of written test results could be done at certain times by checking the results of the answers.Assessment of students' behavior with direct observation requires clear and directed instruments, and also the preparation on what aspects of behavior were observed at the time of learning.The more important thing was whether the learning carried out by the teacher brought changes in students' behavior.Therefore, the intention of the assessments was to improve the quality of learning.Assessments using an observation tool must clearly state the measured aspects and indicators.Rubrics to describe the achievement of competence must also be clear in size.A rubric had not been considered good if it was still interpreted differently by some assessors/observers (Wulan, 2018, p. 52).In the development of the rubric, negotiation to equalize perception was one of the effective methods to guide scoring and reduce the tendency of assessors to be subjective.Negotiation was a good way for assessors to build together interpretations of the construction of the language being measured (Trace et al., 2016, p. 42).The development of scoring rubric descriptions with the help of video snippets of the learning process had provided an alternative of the feasibility test for the rubric.There had been many research examining the use of videos in learning, but not focusing more on helping students to understand learning (Most, Academy, Selin, & Education, 2015, p. 3).Therefore, this research seeks to examine the use of video as an important instrument in helping to validate the development of a scoring rubric.The video in question came from the recording of the learning that had been cropped into shorter videos.The videocropping process was adjusted to the description of the scoring rubric being tested.By using the video snippers, the observers were not required to be present directly in the learning.The observers could simply watch a certain part of learning.This method was very practical because it could involve more observers in obtaining the better validity and feasibility of using the scoring rubric.The main requirement that must be fulfilled by scoring developer is to record the provided a way to test the feasibility of a scoring rubric.Tests were carried out by analyzing the results of the scores given by the observers to the students' behavior displayed on the video snippets of the learning process.The feasibility of a rubric was determined by the level of similarity of the assessment results in each observer rubric.Clarity in describing the level of students' behavior was important to be considered because it was related to the observers' perception of what they had observed.Descriptions of behavior attainment had to be specific essence of this rubric test was to analyze the similarity of the scoring given by the observer to the students' behavior through the description of the assessment in the rubric.The more similarities in the scores given by the observer to the students' behavior mean the better the description of the behavior in the rubric.The video snippets of the learning process were only used as a tool to ease rubric developers to test the feasibility of scoring rubric descriptions.However, the results of this research revealed that the use of scoring descriptions based on the frequency of emergent behavior could be incorrectly used when observing the videos based on the frequency of the desired behavior, which was difficult to calculate.In addition, the wording was also important to avoid multiple interpretations.The use of words or sentences in a category (score) in description had to support each other and be clear.

Table 1 andTable 2 .Table 1 .
Results of the Assessment from the Observers to the Students 1 to 5 in Test 1.

Table 2 .
Results of the Assessment from the Observers to the Students 6 to 10 in Test 1.

Table 1 and
Table 2 show the observers'

Table 3 .
Average Percentage for Each Aspect and the Feasibility of the Rubric in Test 1.

Table 4 .
Revision of Rubric in Aspect 2a (Curiosity Attitude with the Indicator of Enthusiastically Answering Teacher Questions).

Table 6 .
Assessment Results from the Observers to the Students 1 to 5 in Test 2.

Table 7 .
Assessment Results from the Observers to the Students 6 to 10 in Test 2.

Table 8 .
Average Percentage for Aspects 2a and 3a and the Feasibility of the Rubric in Test 2.