CEFLING: Combining Second Language Acquisition and Testing Approaches to Writing

Investigating Syntactic Complexity in EFL Learners' Writing across Common European Framework of Reference Levels A1, A2, and B1

Abstract The study investigates the linguistic basis of Common European Framework of Reference (CEFR) levels in English as a foreign language (EFL) learners’ writing. Specifically, it examines whether CEFR levels can be distinguished with reference to syntactic complexity (SC) and whether the results differ between two groups of EFL learners with different first languages (Sindhi and Finnish). This sheds light on the linguistic comparability of the CEFR levels across L1 groups. Informants were teenagers from Pakistan (N = 868) and Finland (N = 287) who wrote the same argumentative essay that was rated on a CEFR-based scale. The essays were analysed for 28 SC indices with the L2 Syntactic Co…

L2 English vocabulary breadth and knowledge of derivational morphology : One or two constructs?

Derivational morphology (DM) and how it can be assessed have been investigated relatively rarely in language learning and testing research. The goal of this study is to add to the understanding of the nature of DM knowledge, exploring whether and how it is separable from vocabulary breadth. Eight L2 (second or foreign language) English DM knowledge measures and three measures of the size of the English vocabulary were administered to 120 learners. We conducted two confirmatory factor analyses, one with one underlying factor and the other treating vocabulary breadth and DM as separate. As neither model had a satisfactory fit without introducing a residual covariance to the two-factor model,…

Kieltenopetuksen varhentamisen kärkihankkeen seurantapilotti : loppuraportti

Ensikielen tunnistamisen merkityksestä suullisen kielitaidon arvioinnissa Yleisissä kielitutkinnoissa

In this paper, we present a multidisciplinary study addressing fairness in the speaking test in a high-stakes language proficiency test in Finnish, National Certificates of Language Proficiency. The background of the research lies in studies on language assessment and (reversal) linguistic stereotyping and language attitudes. The focus L1 groups were Thai, Estonian, Finland Swedish, Arabic and Russian. Altogether 49 speech samples of test takers of these L1s were rated on a digital platform by 44 raters of the test system. The current paper reports on the sub-study that investigated whether the raters’ recognition of the test takers’ L1 affected their ratings and whether the effect differed…

Arvosanan ansaitsemme : asteikolla vai ilman?

Kielitutkinnot ja muu laajamittainen kielitaidon arviointi Suomessa

This paper describes the development, current state and expected future development of the assessment systems that most of the papers in the present volume cover. Our focus is on the systems serving general education, particularly the Matriculation Examination and national evaluations of educational achievement in languages, as well as the National Certificates of Proficiency aimed at adult language learners. We describe how changes in our understanding of what language proficiency is have affected these assessment systems and discuss the impact that some of them have been expected to have and what is known about that impact. We also discuss how the Common European Framework of Reference ha…

Characteristics of Weak and Strong Readers in a Foreign Language

This study investigated the cognitive (first language [L1] and foreign language [FL]), linguistic (L1 and FL), and motivational characteristics of weak FL readers in contrast to strong readers in 3 groups of L1 Finnish-speaking learners of English, aged 10, 14, and 17 years. This cross-sectional study covered a wide range of potential correlates, and therefore predictors, of FL reading based on previous research on reading in first, second (L2), and foreign languages. The weakest and strongest FL readers (1 standard deviation below or above the mean reading score) in each age group were selected for the comparisons reported in this article. The FL (English) skills other than reading were fo…

Selvitys aikuisten maahanmuuttajien suomen/ruotsin kielen taidon arvioinnista : suunnitelma kotoutumiskoulutuksen lähtötason arviointimallin valtakunnallistamisesta : loppuraportti 30.6.2015

Towards the phonetic basis of spoken second language assessment : temporal features as indicators of perceived proficiency level

Maailma muuttuu - mitä tekee koulu? : äidinkielen ja vieraiden kielten tekstikäytänteet koulussa ja vapaa-ajalla

Assessing learners’ writing skills in a SLA study: Validating the rating process across tasks, scales and languages

There is still relatively little research on how well the CEFR and similar holistic scales work when they are used to rate L2 texts. Using both multifaceted Rasch analyses and qualitative data from rater comments and interviews, the ratings obtained by using a CEFR-based writing scale and the Finnish National Core Curriculum scale for L2 writing were examined to validate the rating process used in the study of the linguistic basis of the CEFR in L2 Finnish and English. More specifically, we explored the quality of the ratings and the rating scales across different tasks and across the two languages. As the task is an integral part of the data-gathering procedure, the relationship of task p…

Lärarnas uppfattningar av att inleda undervisningen i A1-språk i årskurs 1

Since 2020, the teaching of the A1 language starts in grade 1. In order to study the shift to an earlier start to the teaching of the A1 language, a survey of the teaching of A1 languages in grade 1 was carried out with the focus on the most common A1 languages: Finnish in the Swedish-medium schools and English in the Finnish-medium schools. The article examines the teachers’ perceptions of starting the teaching of A1 languages in grade 1. The data consists of 644 answers to an online questionnaire, of which 94 apply to A1 Finnish and 550 to A1 English. The results show that the teachers generally have a positive perception of early language teaching and consider starting the language in gr…

Discursive construction of a high-stakes test: the many faces of a test-taker

As part of a larger project, we studied how a foreign language test got discursively constructed in the talk of upper-secondary-school leavers. A group of students were asked to keep an oral diary to record their ideas, feelings and experiences of preparing for and taking the test over the last spring term of school, as part of a high-stakes national examination. In addition, they took part in discussions either in pairs or groups of three after having learned about the final test results. After transcribing the data, drawing on a form of discourse analysis originally launched by a group of social psychologists, we identified (at least) four interpretative repertoires in the students’ acco…

Laajan koululaisaineiston tutkimuseettiset haasteet

The article focuses on the ethical practices involved in collecting and organizing large datasets, which can also impact the quality of the data and the study as a whole. We report our experiences in the DIALUKI project (Diagnosing reading and writing in a second or foreign language), in which we visited over a hundred schools to obtain a dataset of over a thousand variables from nearly a thousand pupils. Such a project is interesting from the ethical perspective, as data collection for such a large project requires special attention to ethical standards and issues throughout the project, from planning and recruiting the participants to organizing, storing, analyzing and interpreting the da…

Mitä oppilaat pitävät varhaisesta A1-kielen opetuksesta ja millainen käsitys heillä on omasta osaamisestaan?

Vuoden 2020 alusta kieltenopiskelu on aloitettu Suomessa vuosiluokalla 1. Opetussuunnitelman perusteissa korostetaan, että opetuksen tulee olla ikätasolle sopivaa ja herättää myönteinen asenne kielenoppimiseen sekä vahvistaa oppilaan pystyvyyttä. Artikkelissa tarkastellaan suomen- ja ruotsinkielisten koulujen oppilaiden asennoitumista ja osaamiskäsityksiä. Tutkimuksen kohteen on englannin kielen (suomenkieliset koulut, n=106) ja suomi toisena kotimaisena kielenä -oppiaineen (ruotsinkieliset koulut, n=86) opiskelu. Tutkimus osoitti, että oppilaat pitivät pääosin niin opiskelemastaan kielestä kuin sen opiskelusta. He kokivat osaavansa tervehtimiseen, numeroihin ja väreihin liittyviä ilmaisuja…

Predicting Placement Accuracy and Language Outcomes in Immigrants’ L2 Finnish Education

Kaksikieliset oppilaat suomea ja venäjää kirjoittamassa : minun rakkaus väri - valeasininen ja violetti

The present study focuses on Russian-Finnish bilingual pupils’ writing performances which were evaluated on the Common European Framework of Reference scale by three raters. The aim is to investigate how the pupils differ in their two languages in writing. In addition to CEFR-ratings we attempt to characterize the pupils’ performances on a more qualitative level as well as to describe their background in order to understand the reasons behind the level of their performances. Especially the pupils performing poorly in both languages are discussed. The results show that growing to become bilingual is complex: in addition to speaking Russian at home, the immigrant pupils should also read and w…

sj-docx-1-ltj-10.1177_02655322221114015 – Supplemental material for L2 English vocabulary breadth and knowledge of derivational morphology: One or two constructs?

Supplemental material, sj-docx-1-ltj-10.1177_02655322221114015 for L2 English vocabulary breadth and knowledge of derivational morphology: One or two constructs? by Dmitri Leontjev, Ari Huhta and Asko Tolvanen in Language Testing

An analysis of the quality of english testing for Aviation purposes in Finland

This article describes and analyses the development of a new test of aviation English by the Finnish Civil Aviation Authority (FCAA), as well as the overall situation in Finland as regards the testing of aviation English. The article describes the FCAA development project and evaluates the strengths and weaknesses of the new test and the whole testing system, often with reference to the framework of test usefulness proposed by Bachman and Palmer (1996). The quality of the overall system in Finland appears to be quite variable as it is based on the principle of decentralization, in which the FCAA evaluates and approves different tests to be used for certifying the English language skills of …

Kieltenopetus vuosiluokilla 1–3: Miten oppilaat siihen suhtautuvat ja mitä tuloksia sillä saavutetaan puolen vuoden opiskelun aikana?


 
 
 As of January 2020, all Finnish pupils are to start their foreign language studies in the first grade. This study examined the effects of early language teaching by comparing pupils in grades 1, 2 and 3. The pupils had started studying English in August 2017 and had all had roughly the same number of English lessons by the time of the study in early 2018. The study examined pupils’ attitudes towards English and studying English as well as how pupils use English in their free time. Pupils also self-assessed their English skills and completed vocabulary and text segmentation tasks. The results show that the pupils had an overall positive attitude towards English and langu…

Understanding self-assessment - what factors might underlie learners' views of their foreign language skills?

Word derivational knowledge and writing proficiency: How do they link?

Abstract Although word derivational (WD) knowledge, i.e., how new words are formed from existing words with help of derivational affixes, is considered important for learners of second or foreign languages (L2), there is still no clear answer as to what aspects comprise the construct of L2 English word derivational knowledge and how it develops. The present study adds to our knowledge on how the ability to derive English words develops among L2 English learners. More specifically, it sheds light on how word derivational knowledge relates to communicatively defined Common European Framework of Reference (CEFR) language proficiency levels regarding learners' writing skills. In the study, 117 …

Interaction of Language Policy and Assessment in Finland

This paper reviews developments and future challenges in language policy, planning and assessment in Finland, where several important changes in legislation, curricula and assessment systems have recently taken place. Language proficiency requirement of immigrants and civil servants have been redefined, school curricula have been revised and new language examinations have been developed. The Common European Framework has been particularly influential. The paper also explores the tensions caused by changes, such as uncertainties in implementing new curricula in teaching and assessment, and differences between curricula and national examinations. The gate-keeping function of examinations also…

Designing and Assessing L2 Writing Tasks Across CEFR Proficiency Levels

Can research into the diagnostic testing of reading in a second or foreign language contribute to SLA research?

Language testing researchers have recently shown interest in diagnostic testing. However, diagnostic testing requires a better understanding of language abilities at a less general level than has been normal in language testing, and this has posed challenges to testers to define their diagnostic constructs both theoretically and operationally. In theory, second language acquisition (SLA) should be able to offer insights into the construct of reading in a second or foreign language (S/FL), and testing ought to be able to base diagnostic tests of S/FL reading on theoretical insights from SLA. It is, however, unclear whether this happens in practice. In this paper we examine the potential syne…

Developing Automated Feedback on Spoken Performance : Exploring the Functioning of Five Analytic Rating Scales Using Many-facet Rasch Measurement

In this study, we used the Many-facet Rasch measurement (MFRM) to explore the quality of ratings as well as the functioning of five analytic rating scales developed for automated assessment of L2 speech. This study is part of a multidisciplinary research project that develops automatic speech recognition (ASR), automated scoring and automated feedback for L2 Finnish and Swedish. The data include the analytic ratings (task completion, fluency, pronunciation, range, accuracy) gathered from human raters (n=14) who assessed L2 Finnish learners’ (n=64) speech samples using Moodle. The four-facet Rasch analysis showed that the raters performed and the rating scales functioned well, although task …

Syntactic complexity in Finnish-background EFL learners’ writing at CEFR levels A1–B2

Eurooppalaisen viitekehyksen (EVK) merkitys kielikoulutukselle on lisännyt tutkimusta sen taitotasojen kielellisistä piirteistä; tarkempi tieto näistä piirteistä auttaisi EVK:n soveltamista opetusmateriaalien, kurssien ja arviointin laatimiseen. Tutkimuksessa selvitettiin eroavatko EVK:n tasot toisistaan syntaksin kompleksisuuden perusteella. Suomalaiset 14- ja 17-vuotiaat englannin oppijat (N=379) kirjoittivat kolme kirjoitelmaa, jotka arvioitiin EVK:n taitotasoille. Arviointiaineisto tutkittiin monitahoisella Rasch-analyysillä ja tekstien piirteet selvitettiin automaattisilla analyysiohjelmilla. Tuloksien perusteella alimpia EVK-tasoja (A1–A2) erotti selvimmin toisistaan lauseiden ja T-yk…

Ensikielen tunnistamisen merkityksestä suullisen kielitaidon arvioinnissa Yleisissä kielitutkinnoissa

In this paper, we present a multidisciplinary study addressing fairness in the speaking test in a high-stakes language proficiency test in Finnish, National Certificates of Language Proficiency. The background of the research lies in studies on language assessment and (reversal) linguistic stereotyping and language attitudes. The focus L1 groups were Thai, Estonian, Finland Swedish, Arabic and Russian. Altogether 49 speech samples of test takers of these L1s were rated on a digital platform by 44 raters of the test system. The current paper reports on the sub-study that investigated whether the raters’ recognition of the test takers’ L1 affected their ratings and whether the effect differed…

Administration, Scoring, and Reporting Scores

This chapter covers administration, scoring, and reporting scores from language tests and examinations. Procedures typically used in major language examinations and small-scale classroom testing and assessment are both covered. It is argued that the administration, scoring, and reporting procedures are highly dependent on the purpose and stakes of the assessment. Selected national and international guidelines of good practice are reviewed to see what they have to say about these phases of the assessment process, and examples are given on how, for example, high stakes certification and achievement tests differ from teacher-based formative and diagnostic tests. The review considers how the sk…

Toimijuus ja asiantuntijaksi kasvaminen monimediaisessa kielenopettamisessa

This case study investigates the role of agency in the development of two students’ expertise during their participation in the year-long KOO-KIT program (Language Technology for Language Teachers) for pre-service language teachers at the Centre for Applied Language Studies of the University of Jyväskylä in the 2000s. The curriculum of the program was organized around project-based learning, with theoretical and pedagogical lessons and hands-on technology courses alternating with practical team work in web-based design. The data consist of the teachers’ observations and narrative assessments of the students’ performance, and the students’ self-reports during the one-year course. The finding…

Varhennettua englannin opiskelua Suomessa : tutkimustuloksia sukupuolten eroista ja yhtäläisyyksistä

Suomessa yhä useampi opetuksen järjestäjä on aikaistanut kieltenopetusta alkamaan jo alkuopetuksessa, esiopetuksessa tai varhaiskasvatuksessa. Tammikuusta 2020 lähtien jokainen aloittaa kielten opiskelun viimeistään 1. luokan kevätlukukaudella. Alkuvuodesta 2018 selvitimme, miten varhennettu kieltenopetus vaikuttaa oppilaiden suhtautumiseen vieraan kielen opiskeluun, miten oppilaat käyttävät koulussa opetettua kieltä vapaa-ajallaan ja minkälaista kielitaitoa he oppivat ensimmäisen puolen vuoden opiskelun aikana. Tässä artikkelissa kerromme selvityksen alustavia tuloksia sukupuolivaikutuksen näkökulmasta ja avaamme muutamia tulokulmia mahdolliselle jatkotutkimukselle. nonPeerReviewed

9. Fluency in Language Assessment

Changes in Language Assessment Through the Lens of New Materialism

AbstractIn this chapter, we analyze English tests that are part of two computerised assessment systems, the Finnish Matriculation Examination and the Danish National Tests. Language assessment is a fruitful field to explore from the perspective of materiality, to better understand what materialities exist in modern language tests and how students interact with such systems. Within the assessment and test-taking space, material objects exist that are imbued with political values and force test-takers to perform in specific ways. We explore what new materialism has to offer for interpreting current trends in language assessment and to what extent these perspectives allow for new insights to e…

Comparing the outcomes of two different approaches to CEFR-based rating of students’ writing performances across two European countries

This study investigated to what extent two teams of experienced raters from different European countries (Finland and Austria), using their own CEFR-based rating scale (one holistic and one analytic), agreed on the CEFR level of students’ writing performances. Both teams rated one hundred performances written by Austrian secondary school students based on two tasks. The Finnish raters (N = 3) applied a holistic CEFR-linked rating scale consisting of verbatim CEFR descriptors developed in Finland, while the Austrian team (N = 6) used an analytic CEFR-linked rating scale consisting of four criteria developed in Austria. The ratings were analysed using the Rasch model. Although there were indi…

Maailma muuttuu ja niin myös vieraiden kielten ja toisen kotimaisen kielen ylioppilaskokeet

Ylioppilastutkinnon suorittaa hyväksytysti vuosittain noin 30 000 opiskelijaa, ja heistä valtaosa sisällyttää kokeeseen myös vieraan kielen ja/tai toisen kotimaisen kielen kokeen. Kokeiden painoarvo korkeakouluvalinnoissa on viime vuosina kasvanut. Tässä artikkelissa tarkastelemme, miten tutkinnon kielikokeet ovat viime vuosina kehittyneet ja mitä muutoksia on näköpiirissä. nonPeerReviewed

The Ideal Learner as Envisioned by Can Do Statements and Grammar Revisions: How Textbook Agency Is Constructed

AbstractIn this chapter, we analyse the features of textbooks that enable and facilitate their role as material agents in the classroom. Rather than analyse the ways textbooks are used in interaction with humans, we analyse the elements in the textbook itself that facilitate intra-action and the ensuing material agency. Based on a discursive analysis of self-assessment in one textbook and discussing that construct against the Finnish national core curriculum and previous research, we present an ‘ideal imaginary’ of classroom activities as construed in the textbook. This helps us understand the textbooks in their pedagogical ergonomics; i.e. as socio-material in the classroom. We conclude by…

Miten tutkia englannin sananmuodostusta? : tutkimusvälineiden vertailua

Even though word-formation skills may greatly enhance vocabulary acquisition, they are not often emphasized in language teaching or research. Since there are no validated methods to assess learners’ derivational skills, three different test types were developed and their applicability is discussed. This study also investigates the relationship between the written performances of Finnish learners of English and their derivative skills. The results showed quite expectedly that there is a fairly strong relationship between written skills and word formation knowledge. However, the fi ndings suggest that learners need to have reached level A2 in English before they can start to make use of word …

Aikuisten maahanmuuttajien kotoutumiskoulutuksen tulosten ennustaminen lähtötason arviointien perusteella

Artikkelissa selvitetään aikuisten maahanmuuttajien kotoutumiskoulutuksessa saavuttamaa suomen kielen taitoa ja sitä, ennustaako koulutuksen alussa tehtävä lähtötason arviointi saavutettua kielitaitoa. Tutkimukseen osallistuneet maahanmuuttajat (N = 286) osallistuivat lähtötason arviointiin Testipisteessä. Sen jälkeen he suorittivat kotoutumiskoulutuksen opinnot peruspolun tai hitaan opintopolun mukaisesti muutamassa pääkaupunkiseudun oppilaitoksessa. Testipisteen mallin mukaisessa lähtötason arvioinnissa selvitetään maahanmuuttajan suomen kielen taitoa ja opiskeluvalmiuksia testeillä ja taustatietoja keräämällä. Niiden pohjalta maahanmuuttaja ohjataan sopivimmalle opintopolulle ja kielitai…

Oppimista tukeva arviointi kieltenopetuksessa

Monipuolisen arvioinnin merkitys korostuu opetussuunnitelman perusteissa (Opetushallitus 2014, 2019) sekä perusopetus- (1998/628) ja lukiolaissa (2018/714). Toisin sanoen arvioinnin ei tule koostua pelkästään kokeista ja muista summatiivisen arvioinnin muodoista, vaan sen rinnalle tarvitaan oppimista tukevaa, formatiivista arviointia. Oppimista voidaan tukea esimerkiksi palautteen, itsearvioinnin ja vertaispalautteen avulla. Nämä tyypilliset formatiivisen arvioinnin välineet ja niiden implementointi opetukseen mainitaankin opetussuunnitelman perusteissa (Opetushallitus 2014, 2019). Formatiivisen arvioinnin avulla voidaan lisäksi tukea oppilaiden oppimaan oppimista ja reflektointia, mikä on …

Taitotasojen hyödyntämisestä opetuksessa ja arvioinnissa

Taitotasot ovat määritelmiä suoriutumisen tai osaamisen (taidon) määrästä ja laadusta. Sanan taso käyttö osana tätä termiä viittaakin jo siihen, että osaamisessa nähdään olevan useita eri vaiheita, portaita tai tasoja. Kielitaidon kuvaamisessa taitotasoajattelu tarkoittaa, että kielen hallinta hahmotetaan jatkumona, joka kuvaa kielitaidon edistymistä aivan alkeista kohti kielen yhä parempaa (tarkempaa, monipuolisempaa, jne.) hallintaa. Tässä kielen oppimisen jatkumossa on lukematon määrä toisiaan seuraavia ’tasoja’ ja piirteitä, jotka usein limittyvät toisiinsa saumattomasti. Käytännön syistä tämä jatkumo halutaan kuitenkin usein jakaa vain muutamaan vaiheeseen eli (taito)tasoon, jotta void…

Assessment in educational settings

Kielitaito ja psykolingvistiset taidot suomi toisena kielenä -lukemisen ja -kirjoittamisen selittäjinä

The study investigated young learners of Finnish as L2 from Russian-speaking families mostly in grades 3–6 in Finnish-medium schools. 169 pupils completed language tests in Finnish and Russian, and psycholinguistic measures of basic linguistic processes related to phonological awareness, retrieval of words from memory, and working memory. The study examined to what extent linguistic and psycholinguistic skills predict learners’ Finnish L2 reading and writing. Regression analyses indicated that psycholinguistic factors (particularly processing Finnish phonemes) accounted for about 34% of variance in L2 reading and 38% in writing, whereas measures of language skills, mostly L2 skills (e.g., d…

Ihmisarvioijien näkemyksiä suullisen kielitaidon automaattisesta arvioinnista, digitaalisesta arviointiprosessista sekä puhesuorituksista arvioitavista ulottuvuuksista

Digitaaliset kielikokeet ovat nykypäivää muun muassa ylioppilastutkinnossa, ja lähes jokaisessa oppilaitoksessa on jonkinlainen digitaalinen oppimis- ja opetusalusta. Tekniikka on mullistanut kielitaidon testaamisen ja harjoittelun, mutta täysin automaattiset suullisen kielitaidon kokeet ovat vielä harvinaisia kansainvälisestikin. Tekoäly on kuitenkin arkipäiväistymässä monilla elämänalueilla, ja se herättää monenlaisia mielikuvia ja pohdintaa sen vaikutuksista ja eettisyydestä. Artikkelissa kartoitamme ja analysoimme ihmisarvioijien näkemyksiä digitaalisesti toteutetusta arviointiprosessista, jossa keräsimme ihmisarvioijilta arvioita suomen ja ruotsin puhenäytteisiin. Tutkimme myös arvioin…

The development of a suite of computer-based diagnostic tests based on the Common European Framework

DIALANG is an on-line language assessment system, which contains tests in 14 European languages and is based on the Common European Framework of Reference (CEFR). It is the first major testing system that is oriented towards diagnosing language skills and providing feedback to users rather than certifying their proficiency. This article describes the contents of Version 1 of DIALANG and the way in which the system works. This is followed by an account of the development of DIALANG tests and of the pilot testing and standard setting procedures. The results of the first analyses of items and self-assessment statements, and of the standard setting procedures, are reported. The article focuses…

Instrument to study rater behaviour: Rater uncertainty and its impact on the quality of assessment in the official language examination, the National Certificates of Language Proficiency (YKI) project

This research instrument was developed for a study on the uncertainty experienced by language raters and the impact of uncertainty on the quality of assessment in the official language examination, the National Certificates of Language Proficiency. The instrument consists of three online questionnaires using Webropol software and a note-taking sheet using Excel spreadsheet software. The survey focused on the assessment of speaking and writing in the intermediate-level Finnish language examination. The questions in the survey include both closed and open questions. Two almost identical versions of all the questionnaires and the Excel sheet were created for the collection of data for the diff…