In search of equivalence - translation problems in international literacy studies

Threats to validity when using open-ended items in international achievement studies : Coding responses to the PISA 2012 problem-solving test in Finland

Open-ended (OE) items are widely used to gather data on student performance in international achievement studies. However, several factors may threaten validity when using such items. This study examined Finnish coders’ opinions about threats to validity when coding responses to OE items in the PISA 2012 problem-solving test. A total of 6 discussions during 6 coder practice sessions (on 6 OE items) and an interview between 5 coders were audiorecorded and analyzed by means of content analysis, and 3 main threats to validity were found: (1) unclear and complex questions; (2) arbitrary and illogical coding rubrics; and (3) unclear and ambiguous responses. Suggestions are given as to how to res…

The problem of equivalence in translating texts in international reading literacy studies : a text analytic study of three English and Finnish texts used in the PISA 2000 reading texts

Tutkimuksessa tarkasteltiin ongelmia, joita kansainvälisissä lukukokeissa on käännettäessä lukukoetekstejä osallistujamaiden kielille ja pyrittäessä samalla pitämään tekstien vaikeustaso muuttumattomana – sillä elleivät koetekstit ole vaikeustasoltaan ekvivalentteja, vertailukelpoisia, ei lukukoekaan ole validi. Tieto ongelmista on tarpeen, koska se auttaa tekemään käännöksistä laadukkaampia ja vertailukelpoisempia, mikä vuorostaan johtaa luotettavampiin, pätevämpiin ja tasa-arvoisempiin lukukokeisiin. Tutkimuksen aineistona käytettiin kolmea PISA 2000 -lukukokeeseen sisältynyttä englanninkielistä lähtötekstiä ja niiden suomennosta. Tekstit edustivat eri tekstityyppejä: yksi oli ekspositori…

International Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews

In international education studies, the different-language test versions need to be equally difficult to read and answer for the test to be valid. To ensure comparability, several quality control procedures have been developed. Among these, surprisingly little attention has been paid to judgmental reviews and their ability to identify language-related sources of bias. Also, the reviews have often failed in identifying biases. This paper explored whether it is possible to improve the ability of judgmental reviews to identify language-related sources of bias. A new review was made of two Finnish items which in the PISA (Programme for International Student Assessment) 2000 reading test showed …

Translating international achievement tests : translators’ view

Kansainväliset arviointitutkimukset edellyttävät laajaa käännöstyötä, koska kaikki koemateriaalit on käännettävä kunkin osallistujamaan kielille. Kaikkien erikielisten koeversioiden on lisäksi oltava keskenään vertailukelpoisia, jotta tutkimustulokset olisivat valideja. Arviointitutkimuksissa onkin kehitetty nimenomaisia käännösprosesseja ja -käytänteitä varmistamaan koeversioiden vertailukelpoisuus ja tutkimustulosten validius. Tässä tutkimuksessa tarkasteltiin kääntäjien kokemuksia näistä prosesseista ja käytänteistä ja niissä ilmenneistä ongelmista. Tavoitteena oli kehittää kansainvälisten arviointitutkimusten käännösprosesseja ja -käytänteitä ja lisätä näin näiden tutkimusten tulosten v…

Equivalence of Translations in International Reading Literacy Studies

Recent years have witnessed a significant increase in the interest in international assessments of student performance. In such assessments it is mandatory that all the different‐language texts be equivalent to each other, that is, equally difficult to understand. The article summarizes a study made on the topic, examining the problems of equivalence encountered when translating texts in international reading literacy assessments. In the study, three English and Finnish texts used in the PISA 2000 reading test were compared text‐analytically. The analysis revealed six different types of problem, which, moreover, differed considerably between the three texts. As a result of the problems, non…

Unwanted Literal Translation: An Underdiscussed Problem in International Achievement Studies

In international achievement studies, a common test is typically used which is translated into the languages of the participating countries. For the test to be valid, all the translations and different-language test versions need to be equally difficult to read and answer. An underestimated and underdiscussed threat to this validity is unwanted literal translation. This paper discusses the problem of unwanted literal translation in international achievement studies. It defines what is meant by unwanted literal translation and explains why it is a threat to the validity of international achievement studies and why it is so difficult to avoid. It also discusses problems there have been when t…

Problems and Issues in Translating International Educational Achievement Tests

The article reviews research and findings on problems and issues faced when translating international academic achievement tests. The purpose is to draw attention to the problems, to help to develop the procedures followed when translating the tests, and to provide suggestions for further research. The problems concentrate on the following: the unique and demanding purpose of the translation task, the partly contradictory task specifications and translation instructions, the indecision as to whether to produce one or two target versions, the indecision as to whether to use one or two source versions, inadequate revision and verification, deficient translator competences, and a lack of time.…

Pisa 2000

The Finnish success in Pisa - and some reasons behind it : 2 Pisa 2003

The success of Finnish students in PISA has been a great joy but at the same time a somewhat puzzling experience to all those responsible for and making decisions about education in Finland. At a single stroke, PISA has transformed our conceptions of the quality of the work done at our comprehensive school and of the foundations it has laid for Finland's future education and development of knowledge. Thanks to PISA, Finnish schools and school practices have suddenly been brought into the focus of international attention. Why are Finnish students performing so well? What is the secret behind the Finnish success? In this publication, we as the researchers responsible for the implementation of…