ANALYZING VARIANCE SOURCES IN INDONESIAN MICROTEACHING ASSESSMENT THROUGH GENERALIZABILITY THEORY
Abstract
Objective This study aims to examine the reliability of student performance assessments in microteaching using G-Theory, and to identify the primary sources of score variance that may affect the accuracy of evaluation outcomes.
Methods A fully crossed p × r × i design was employed, involving 32 students, 2 raters, and 124 observation items representing ten core teaching skills. Secondary data were analyzed using EduG 6.1e software. The analysis included both a G-Study, to estimate variance components, and a D-Study, to simulate changes in reliability based on different rater configurations.
Results The results of the G-Study revealed that person variance (24.9%) and person × rater interaction (23.5%) were the dominant sources of score variability, while item and rater variances were negligible. The residual three-way interaction (person × item × rater) accounted for 50.9% of the total variance, indicating substantial unexplained error. The initial generalizability coefficient was 0.68, categorized as moderate reliability. The D-Study demonstrated that increasing the number of raters to three or more significantly improved reliability, with the coefficient reaching 0.76-0.82, and reduced the standard error of measurement.
Conclusion The study concludes that microteaching assessments are susceptible to reliability issues due to rater inconsistency. Applying G-Theory provides empirical justification for improving rater calibration and increasing the
number of rater.
Keywords
Full Text:
PDFReferences
Altammar, J., & Aljassar, S. (2021). The effect of microteaching on the enhancement of female mathematics and social studies teachers’ skills in the college of education at Kuwait university. International Journal of Knowledge and Learning, 14(4), 324–344. https://doi.org/10.1504/ijkl.2021.118558
Andersen, S. A. W., Nayahangan, L. J., Park, Y. S., & Konge, L. (2021). Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis. Academic Medicine, 96(11), 1609–1619. https://doi.org/10.1097/ACM.0000000000004150
Aruğaslan, E. (2025). Self-, Peer, and Tutor Assessment in Online Microteaching Practice and Doctoral Students’ Opinions. International Review of Research in Open and Distributed Learning, 26(1), 99–117. https://doi.org/10.19173/irrodl.v26i1.7970
Atilgan, H. (2019). Reliability of essay ratings: A study on generalizability theory. Eurasian Journal of Educational Research, 2019(80), 133–150. https://doi.org/10.14689/ejer.2019.80.7
Becker, M. R. (1999). Rater bias in observational data: A generalizability analysis.
Bracken, B. A. (1987). Limitations of preschool instruments for identifying children with high intellectual ability: A critical review and recommendations. Gifted Child Quarterl.
Brennan, R. L. (2001). Generalizability theory: Statistics for social science and public policy. In New York: Springer-Verlag. (Vol. 30).
Brennan, R. L. (2009). Generalizability Theory. In International Encyclopedia of Education, Third Edition (pp. 61–68). https://doi.org/10.1016/B978-0-08-044894-7.00246-3
Brennan, R. L. (2010). Generalizability theory and classical test theory. Applied Measurement in Education, 24(1), 1–21. https://doi.org/10.1080/08957347.2011.532417
Briesch, A. M., Chafouleas, S. M., & Johnson, A. (2016). Use of Generalizability Theory Within K–12 School-Based Assessment: A Critical Review and Analysis of the Empirical Literature. Applied Measurement in Education, 29(2), 83–107. https://doi.org/10.1080/08957347.2016.1138955
Briesch, A. M., Swaminathan, H., Welsh, M., & Chafouleas, S. M. (2014). Generalizability theory: A practical guide to study design, implementation, and interpretation. Journal of School Psychology, 52(1), 13–35. https://doi.org/10.1016/j.jsp.2013.11.008
Cardinet, J., Sandra Johnson, & Pini, G. (2010). Applying Generalizability Theory using EduG Quantitative Methodology Series. Routledge.
Cicchett, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment.
Clayson, P. E., & Miller, G. A. (2017). ERP Reliability Analysis (ERA) Toolbox: An open-source toolbox for analyzing the reliability of event-related brain potentials. International Journal of Psychophysiology, 111, 68–79. https://doi.org/10.1016/j.ijpsycho.2016.10.012
Crawford, A. R., Johnson, E. S., Moylan, L. A., & Zheng, Y. (2019). Variance and Reliability in Special Educator Observation Rubrics. Assessment for Effective Intervention, 45(1), 27–37. https://doi.org/10.1177/1534508418781010
Deshpande, S., & Shastri, S. (2020). A cross-sectional study to evaluate teaching skills of postgraduate medical students using component skill approach in microteaching. Journal of Education and Health Promotion, 9(1). https://doi.org/10.4103/jehp.jehp_743_19
Dzakadzie, Y., & Quansah, F. (2023). Modeling unit non-response and validity of online teaching evaluation in higher education using generalizability theory approach. Frontiers in Psychology, 14(September), 1–14. https://doi.org/10.3389/fpsyg.2023.1202896
González-Mélendez, R. C., Sánchez-Rodríguez, M. A., & Robles-López, F. (2023). Validity and reliability of an instrument for assessing microteaching in chemical biological sciences. Revista Digital de Investigacion En Docencia Universitaria, 17(2). https://doi.org/10.19083/ridu.2023.1581
Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education, 18(3), 375–396. https://doi.org/10.1007/s10459-012-9376-x
Gudiato, C., Cahyaningtyas, C., & P., N. (2024). Alat Pendeteksi Kebocoran Gas LPG Pada Resto Ayam Bakar dan Goreng Kremes Tata Berbasis Internet Of Things. G-Tech : Jurnal Teknologi Terapan, 8(1), 186–195.
Hendrickson, A., & Yin, P. (2018). Generalizability Theory. In The Reviewer’s Guide to Quantitative Methods in the Social Sciences: Second Edition (pp. 123–131). https://doi.org/10.4324/9781315755649-9
Hong, S.-J. (2008). Applications of generalizability theory to estimate the variance components in basic nursing technical tests. Journal of Nursing, 55(4), 41–52. https://www.scopus.com/inward/record.uri?eid=2-s2.0-49749131173&partnerID=40&md5=aed8a59f1bf1e930b4a9c9162da6ab45
Jones, E., & Bergin, C. (2019). Evaluating Teacher Effectiveness Using Classroom Observations: A Rasch Analysis of the Rater Effects of Principals. Educational Assessment, 24(2), 91–118. https://doi.org/10.1080/10627197.2018.1564272
Koech, H. C., & Mwei, P. K. (2019). How secondary school mathematics teachers perceive the effectiveness of microteaching and teaching practice in their preservice education. Humanities and Social Sciences Letters, 7(1), 46–55. https://doi.org/10.18488/journal.73.2019.71.46.55
Komolafe, B. F., Ogunniran, M. O., Zhang, F. Y., & Qian, X. S. (2020). A comparative perspective of teaching skill acquisition in pre-service physics teacher (Pspt) training program in china and nigeria. Journal of Baltic Science Education, 19(3), 356–373. https://doi.org/10.33225/jbse/20.19.356
Matt, G. E., & Sklar, M. (2015). Generalizability Theory. In International Encyclopedia of the Social & Behavioral Sciences: Second Edition (pp. 834–838). https://doi.org/10.1016/B978-0-08-097086-8.44027-4
Mishra, R. (2024). Utilizing Online Micro Teaching as the Main Technique in Education Practice. Proceedings of International Conference on Sustainable Computing and Integrated Communication in Changing Landscape of AI, ICSCAI 2024. https://doi.org/10.1109/ICSCAI61790.2024.10866844
Monteiro, S., Sullivan, G. M., & Chan, T. M. (2019). Generalizability Theory Made Simple(r): An Introductory Primer to G-Studies. Journal of Graduate Medical Education, 11(4), 365–370. https://doi.org/10.4300/JGME-D-19-00464.1
Nalbantoǧlu Yilmaz, F., & Gelbal, S. (2011). Comparison of different designs in accordance with the generalizability theory in communication skills example. Hacettepe Egitim Dergisi, 41, 509–518. https://www.scopus.com/inward/record.uri?eid=2-s2.0-84857386442&partnerID=40&md5=884440963ee930f6c0028796cfd3709f
Özcan, Ö., & Gerçek, C. (2019). Multidimensional analyzing of the microteaching applications in teacher education via videograph. European Journal of Teacher Education, 42(1), 82–97. https://doi.org/10.1080/02619768.2018.1546285
Padmadewi, N. N., & Artini, L. P. (2019). Assessment instruments for improving English teaching skills through microteaching in Indonesia. Asian EFL Journal, 21(2), 49–77. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85063669842&partnerID=40&md5=a0086fa8150e880afe126455909c69e9
Parriott, D. (2016). Using Generalizability Theory to investigate sources of variance of the Autism Diagnostic Observation Schedule-2 with trainees. UNLV Theses, Dissertations, Professional Papers, and Capstones, 2721. https://digitalscholarship.unlv.edu/thesesdissertations/2721
Raharjo, H. P., Kusuma, D. W. Y., Mohamed, A. M. D., Rahayu, T., Annas, M., Putra, R. B. A., Suripto, A. W., & Kurniawan, W. R. (2025). Analysis of the online microteaching practice of undergraduate physical education students. Cakrawala Pendidikan, 44(1), 63–71. https://doi.org/10.21831/cp.v44i1.61743
Şen, A. I. (2009). A study on the effectiveness of peer microteaching in a teacher education program. Egitim ve Bilim, 34(151), 165–174. https://www.scopus.com/inward/record.uri?eid=2-s2.0-84886292464&partnerID=40&md5=cd2527f8998e9e397f7aa086fdddffb8
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage Publications.
Sudrajat, A. K., Ibrohim, I., & Susilo, H. (2024). Preservice teachers’ reflections on lesson study integration into a microteaching course. Social Sciences and Humanities Open, 10. https://doi.org/10.1016/j.ssaho.2024.101140
Sung, Y. T., Chang, K. E., Chang, T. H., & Yu, W. C. (2010). How many heads are better than one? The reliability and validity of teenagers’ self- and peer assessments. Journal of Adolescence, 33(1), 135–145. https://doi.org/10.1016/j.adolescence.2009.04.004
Teker, G. T., Guler, N., & Uyanik, G. K. (2015). Comparing the effectiveness of spss and edug using different designs for generalizability theory. Kuram ve Uygulamada Egitim Bilimleri, 15(3), 635–645. https://doi.org/10.12738/estp.2015.3.2278
DOI: https://doi.org/10.17509/e.v24i3.86980
Refbacks
- There are currently no refbacks.
Copyright (c) 2025 EDUTECH
Ciptaan disebarluaskan di bawah Lisensi Creative Commons Atribusi-BerbagiSerupa 4.0 Internasional.