Interrater Reliability Checklist Osce Kateterisasi Urin Di Program Studi Ilmu Keperawatan Fakultas Kedokteran Universitas Gadjah Mada

Abstract

Background: Objective Structured Clinical Examination (OSCE) is one of summative test method for performance-based assessment. One of component that make up an OSCE is assessment instrumen. Whereas checklist is one of OSCE’s component that affect OSCE’s reliability. As long as this checklist was implemented in Nursing Science Program, Faculty of Medicine, UGM, the reliability of urinary catheterization checklist hasn’t been tested
Objective: This study aims to assess interrater reliability of OSCE checklist instruments for urinary catheterization in Nursing Science Program, Faculty of Medicine, Universitas Gadjah Mada.
Methods: This study is a psychometric testing study. Two rater consisted of a fourth-year student and a lecturer who performed measurement on 93 second-year students who was taking the OSCE examination. The measurement result were analyzed using kappa test and percent agreement (PA). Whereas the item’s reliability were analyzed using weighted kappa dan some items which is paradox can be count with Prevalence and Bias Adjusted Kappa-Ordinal Scale (PABAK-OS) to separate the bias and prevalence effect.
Results: The results of measurement of the reliability was 0,57, which indicated that the checklist was in the moderate category, and the PA was 78,49%. According to Osborne (2008) and Stemler and Tsai (2008), this checklist reliability considered as acceptable. Meanwhile, the result of measurement of each item indicated various reliabilities. Reliability value on this checklist’s item was around 0,24-0,96. Meanwhile, some factors that affect OSCE’s rating categorized as item and rater.
Conclusion: The checklist of urinary catheterization has moderate reliability value and can be used as an instrument for the OSCE assessment. However, there were 9 items that weren’t reliable and must be improved.

ABSTRAK

Latar belakang: Objective Structured Clinical Examination (OSCE) merupakan salah satu metode penilaian sumatif dalam penilaian berbasis performa. Salah satu komponen yang menyusun OSCE adalah instrumen penilaian. Instrumen checklist merupakan komponen OSCE yang mempengaruhi reliabilitas penilaian tersebut. Selama diterapkan di PSIK FK UGM, reliabilitas checklist kateterisasi urin belum pernah diuji.
Tujuan: Penelitian ini bertujuan untuk mengetahui interrater reliability checklist OSCE kateterisasi urin di Program Studi Ilmu Keperawatan Fakultas Kedokteran Universitas Gadjah Mada.
Metode: Penelitian ini merupakan penelitian psikometri. Dua rater yang terdiri dari mahasiswa tingkat  4 dan seorang dosen menilai performa 93 mahasiswa tahun kedua dalam stase kateterisasi urin saat OSCE. Hasil pengukuran akan dihitung dan diuji menggunakan uji kappa dan Percent Agreement (PA). Sedangkan reliabilitas tiap item kateterisasi urin akan dihitung dengan menggunakan weighted kappa, dan beberapa item yang mengalami paradoks akan dihitung menggunakan Prevalence And Bias Adjusted Kappa-Ordinal Scale (PABAK-OS) untuk menghilangkan efek bias dan prevalensi.
Hasil: Hasil dari penghitungan nilai kappa menunjukkan bahwa checklist kateterisasi urin memiliki nilai kappa sebesar 0,57, dan PA sebesar 78,49%. Sedangkan pengukuran item menunjukkan hasil yang bervariasi. Nilai kappa item berada pada kisaran 0,24-0,96. Adapun faktor yang mempengaruhi penilaian OSCE dapat dilihat dari sudut pandang item maupun rater.
Kesimpulan: Checklist kateterisasi urin merupakan checklist dengan kategori reliabilitas sedang dan merupakan ceklis yang reliabel. Namun, terdapat 9 item pada checklist tersebut yang memerlukan perbaikan karena reliabilitasnya tidak dapat diterima.