Analysis of artificial intelligence (AI)-assisted high-order thinking skills (HOTS) test items for motorcycle chassis maintenance and repair learning in vocational high schools

Abstract

This study aims to analyze the quality of High Order Thinking Skills (HOTS)-based test items generated by Artificial Intelligence (AI), specifically ChatGPT, in the subject of Motorcycle Chassis Maintenance and Repair in vocational high schools (SMK). The analysis includes validity, reliability, difficulty level, discriminating power, and distractor effectiveness. The research method used is a survey with a descriptive approach, with item analysis techniques utilizing Anates V4 software. The research subjects consisted of 153 twelfth-grade students majoring in Motorcycle Engineering at SMKN 8 Bandung and SMKN 7 Baleendah. Data were collected through a written test comprising 60 HOTS-based questions generated by ChatGPT. The results show that the tested questions have good validity and high reliability. In terms of difficulty level, 13.33% of the questions were classified as easy, 58.33% as moderate, and 28.33% as difficult. The discriminating power analysis indicates that most questions effectively differentiate between students with high and low abilities. The distractor effectiveness was also optimal in guiding students' answer choices. This study concludes that AIgenerated HOTS-based questions have the potential to enhance the quality of learning evaluation in vocational high schools, with good validity and reliability, AI can serve as an innovative solution for educators in developing more efficient and high-quality assessment instruments.