6533b82bfe1ef96bd128cd59

RESEARCH PRODUCT

Threats to validity when using open-ended items in international achievement studies : Coding responses to the PISA 2012 problem-solving test in Finland

Inga Arffman

subject

Semi-structured interviewavoimet tehtävätvalidityAcademic achievementTest validityoppimistuloksetEducationarvioinnitinternational achievement studiesMathematics educationAchievement testta516open-ended items060201 languages & linguisticscodingItem analysis05 social sciences050301 educationRubric06 humanities and the artskansainväliset oppimistuloksetpisteytysContent analysisvaliditeetti0602 languages and literaturePsychology0503 educationCoding (social sciences)

description

Open-ended (OE) items are widely used to gather data on student performance in international achievement studies. However, several factors may threaten validity when using such items. This study examined Finnish coders’ opinions about threats to validity when coding responses to OE items in the PISA 2012 problem-solving test. A total of 6 discussions during 6 coder practice sessions (on 6 OE items) and an interview between 5 coders were audiorecorded and analyzed by means of content analysis, and 3 main threats to validity were found: (1) unclear and complex questions; (2) arbitrary and illogical coding rubrics; and (3) unclear and ambiguous responses. Suggestions are given as to how to respond to these threats in order to improve the validity of international achievement studies. peerReviewed

yearjournalcountryeditionlanguage
2015-09-14
http://urn.fi/URN:NBN:fi:jyu-201610114336