Competency Research in Higher Education: Conceptual and Methodological Challenges and Perspectives for Future Interdisciplinary Research

In recent years, an increasing number of research projects have been dedicated to competency assessment not only in school and vocational education, but also in higher education (Blomeke, Zlatkin-T...

Editorial – About a ‚PLATO‘

While learning has constantly been an object of research in manifold disciplines and fields, it has generally been understood in a positive sense. In the Age of Information, we are witnessing an increasing number of phenomena in the context of knowledge construction and accumulation that we describe as “negative learning”. This includes, for example, the deliberate circulation of counterfactual knowledge leading to negative learning outcomes, i.e., deficient decision-making and acting, like medical errors.

Performance Assessment of Generic and Domain-Specific Skills in Higher Education Economics

Following criticisms by employers about academic graduates’ lack of 21st century skills, students need to develop skills such as professional knowledge, critical thinking and problem solving. Accordingly, there is a demand for suitable assessments of these skills. One approach is to develop a performance assessment using tasks adapted from real-world decision-making and judgment situations that students and graduates have to face in academic and professional domains. Such tasks employ real-life scenarios and require generic and domain-specific skills in different facets to handle a given problem adequately. In this paper, we present a newly developed performance assessment that aims to meas…

Valid Competency Assessment in Higher Education

The aim of the 15 collaborative projects conducted during the new funding phase of the German research program Modeling and Measuring Competencies in Higher Education—Validation and Methodological Innovations (KoKoHs) is to make a significant contribution to advancing the field of modeling and valid measurement of competencies acquired in higher education. The KoKoHs research teams assess generic competencies and domain-specific competencies in teacher education, social and economic sciences, and medicine based on findings from and using competency models and assessment instruments developed during the first KoKoHs funding phase. Further, they enhance, validate, and test measurement approa…

Study Success and Opportunities for Refugees in Higher Education – an Effectiveness Analysis

Constantly high numbers of refugees, particularly among young individuals, require effective approaches to providing this group with access to higher education. Findings from studies with refugees and comparison groups, such as migrants or international students, indicate that refugees are characterized by specific factors that distinguish them and their educational challenges from other student groups. In the tertiary sector, attempts are being made to support refugees’ access to higher education through online study platforms. However, there are hardly any findings on how effective these platforms are, nor what requirements they have to meet to effectively address the needs of the target …

Validation and Applications of KoKoHs Assessments

Modeling and Measuring Competencies in Germany

In order to meet the challenges facing higher education policy and practice, and initiate the necessary fundamental assessment-research, the German Federal Ministry of Education and Research (BMBF) established the Modeling and Measuring Competencies in Higher Education (KoKoHs) research program in 2011. The program was part of the BMBF funding priority Research on Science and Higher Education aimed at building a capacity for empirical research in higher education.

The Relationship between General Intelligence and Media Use among University Students

Students’ information selection process might be influenced by their choice of media sources, their learning contexts and motivation to use certain media as well as their general intelligence, which is crucial for information processing. This study examines the relationship between the general fluid intelligence and the media use of 709 first-year business & economics students from 44 universities in Germany for two different learning purposes: informing oneself about B&E topics and preparing for lectures and exams. Accordingly, the motivator information seeking is divided into curiosity driven and goal driven information seeking. Three types of media sources were included: common news sour…

Validating Test Score Interpretations by Cross-National Comparison

Cross-national assessment of students’ competences in higher education is becoming increasingly important in many disciplines including economics but there are few available instruments that meet psychological standards for assessing students’ economic competence in higher education (HE). One of them is the internationally valid Test of Understanding in College Economics (TUCE), which has been adapted and employed successfully in HE systems in various countries, but the test results have seldom been used for international comparisons of students’ Economic Content Knowledge (ECK). Here, we compare the German and the Japanese test adaptations of the TUCE with reference to the American origin…

International Vocational Education and Training Research: An Introduction to the Special Issue

The seven articles in this special issue represent a wide range of international comparative and review studies by international research teams from China, Germany, India, Russia, Switzerland and Mexico. The presented projects are part of the national program "Research on the Internationalisation of Vocational Education and Training", funded by the German Federal Ministry of Education and Research (BMBF). An adapted version of Urie Bronfenbrenner���s ecological systems theory forms the conceptual framework of the special issue. The four system levels (micro, meso, exo and macro) are addressed by one article each. The article on the microsystem level focuses on the intended and implemented c…

Entry Assessment of Student Learning Preconditions in Higher Education: Implications for the Transition from Secondary to Tertiary Education in Germany

The acquisition of domain-specific knowledge and interdisciplinary skills such as critical thinking is increasingly gaining significance as key learning outcomes in higher education that are crucial for all professionals and engaged citizens and that enable lifelong learning. Despite this socio-political consensus, up until the last decade there have only been a few evidence-based insights into the competencies of higher education students. Therefore, the Germany-wide research program Modelling and Measuring Competencies in Higher Education (KoKoHs) was established in 2011 by the Federal Ministry of Education and Research. In the 85 projects, theoretical-conceptual competence models and cor…

Macroeconomic knowledge of higher education students in Germany and Japan – a multilevel analysis of contextual and personal effects

Recent trends towards harmonising and internationalising business and economics studies in higher education are affecting the structure and content of programmes and courses, and necessitate more transparent and comparable information on students’ economic knowledge and skills. In this study, we examine by linear multilevel regression modelling the current state of macroeconomic knowledge of higher education students in Germany and Japan, while controlling for the effects of key study-related aspects such as study progress and completion of economics courses. We assess macroeconomic knowledge using the internationally established Test of Understanding in College Economics, which has been ad…

Diagnostik von studentischen Kompetenzen im Hochschulsektor

Integrating Computational Linguistic Analysis of Multilingual Learning Data and Educational Measurement Approaches to Explore Learning in Higher Education

This chapter develops a computational linguistic model for analyzing and comparing multilingual data as well as its application to a large body of standardized assessment data from higher education. The approach employs both an automatic and a manual annotation of the data on several linguistic layers (including parts of speech, text structure and content). Quantitative features of the textual data are explored that are related to both the students’ (domain-specific knowledge) test results and their level of academic experience. The respective analysis involves statistics of distance correlation, text categorization with respect to text types (questions and response options) as well as lang…

Implementierung und Wirksamkeit der erweiterten Autonomie im öffentlichen Schulwesen – Eine Mehrebenenbetrachtung

Das Konzept der erweiterten Schulautonomie hat als eine umfassende Innovation in der letzten Dekade zunehmende Beachtung erfahren. Spatestens nach der Veroffentlichung der ersten PISA-Ergebnisse in 2000 gehort „Schulautonomie“ zweifelsohne zu einem der Schlagworter der bildungspolitischen Diskussion und auch mehrere Forschungsdisziplinen und -richtungen (wie Educational Governance- Forschung, padagogische Implementationsforschung, Innovations- und Schulentwicklungsforschung etc.) haben sich mit diesem Konzept aus ihrer jeweiligen Perspektiven intensiv auseinandergesetzt. Je nach Annahme bzw. Fokus solcher theoretischen Sichtweisen (z. B. die juristische bzw. verwaltungsrechtliche Sicht, ste…

An Overview of Assessment Practices

On a global scale, attempts to specify conceptually academic learning outcomes have been made mostly in OECD countries. Definitions of learning outcomes can be used not only as an orientation for accrediting degree courses and universities; in some countries, they are used also as a basis for the development of assessment methods. The following will provide an overview of various key approaches and projects in this field.

Advances and perspectives of competence research in higher education – Report on the German KoKoHs program

Higher education in Germany is characterized by specific historicalcultural traditions stemming from von Humboldt’s humanistic understanding of education. In this context, higher education focuses on a variety of multi-perspective goals. Accordingly, higher education focuses on both the acquisition of knowledge and skills in specific domains (e.g., medicine, economics) and on the development and promotion of generic (interdisciplinary) skills (e.g., problem solving, critical thinking), whereby the latter aspect is becoming increasingly more important in the 21st century. This paper analyses ten years of competence research in higher education in germany, and considers advances and perspect…

Relationships between Domain-Specific Knowledge, Generic Attributes, and Instructional Skills

We introduce a theoretical framework on teachers’ instructional skills to describe how they can be modeled across different domains. This framework conceptualizes teachers’ instructional skills as action-related skills (during instruction) and reflective skills (before and after instruction), which are considered crucial for coping with the practical demands of everyday teaching in a specific subject. The theoretical framework assumes that both skill facets are influenced by the teacher’s professional knowledge, generic attributes such as general cognitive abilities or ambiguity tolerance as well as affective and motivational factors. To investigate the relationships between teachers’ instr…

Überblick zu nationalen und internationalen Assessments

Zur begrifflichen und konzeptionellen Prazisierung dessen, was unter akademischen Lernergebnissen zu verstehen sei, finden sich – weltweit betrachtet – die meisten Initiativen in OECD-Landern. Solche Definitionen werden nicht nur als Orientierungsrahmen bei Akkreditierungen von Studiengangen und Hochschulen genutzt, sondern bilden in einigen Landern auch eine Basis bei der Entwicklung von Assessmentverfahren. Im vorliegenden Kontext sollen einige zentrale Entwicklungen und Projekte aus diesem Bereich skizziert werden.

Adapting and Validating the Collegiate Learning Assessment to Measure Generic Academic Skills of Students in Germany: Implications for International Assessment Studies in Higher Education

Starting in 2015, a German research team from the program Modeling and Measuring Competencies in Higher Education (KoKoHs), in collaboration with the US Council for Aid to Education (CAE), adapted and validated the Collegiate Learning Assessment (CLA+) for the German language and cultural context to measure generic higher-order cognitive skills of university students and graduates in Germany. In this chapter, the conceptual and methodological background, the framework of the adaptation and validation study, as well as preliminary results are presented. Finally, findings are discussed critically, and future challenges and perspectives are explored.

Correction to: Measuring Scientific Reasoning Competencies

Conceptual Fundamentals for a Theoretical and Empirical Framework of Positive Learning

We present a short overview of the idea, genesis and developing process of PLATO (Positive Learning in the Age of InformaTiOn) as a new, complex, international, interdisciplinary program aimed at investigating the phenomena of positive and negative learning in the digital age. While the basic idea of PLATO originated in empirical educational research, many other very different disciplines have consequently been incorporated into this program, in order to expand learning research and to offer a comprehensive and multi-perspective explanation of learning in 21st century higher education. We draw upon and critically discuss previous research, highlighting implications for developing a theoreti…

Influences on Master’s Degree Students’ Economic Knowledge

Despite the growing body of research, little is known about students’ economic knowledge at the beginning of, or at various points throughout, their master’s degree programs. In this study different impact factors on students’ microeconomic knowledge are tested. Based on the “utilization of learning opportunities model” by Helmke we focus on 1) individual and sociodemographic characteristics of the students, 2) their learning potential and study-related characteristics and 3) the characteristics and structure of their learning environment. For each of the three influencing factors we used different indicators that were expected to correlate with students’ knowledge. We tested the assumption…

The Impact of Entry Preconditions on Student Dropout and Subject Change in Business and Economics

The proportion of students dropping out of higher education economics is remarkably high at approximately 25 % and has been constant for years. The results on the entry preconditions of dropping out and changing subjects in Bachelor’s degree programs in business and economics presented in this paper are based on a representative longitudinal study in Germany. In two survey waves, the cognitive entry preconditions and characteristics of business and economics students related to their study and learning processes were tested. Dropouts have lower levels of previous knowledge than students changing subjects and students who continue to study business and economics. The students’ characteristic…

Gender Effects in Assessment of Economic Knowledge and Understanding: Differences Among Undergraduate Business and Economics Students in Germany, Japan, and the United States

Gender effects in large-scale assessments have become an increasingly important research area within and across countries. Yet few studies have linked differences in assessment results of male and female students in higher education to construct-relevant features of the target construct. This paper examines gender effects on students’ economic content knowledge with a focus on construct-relevant explanations. Moreover, we compare gender effects cross-nationally between Germany, Japan, and the United States. To assess economic content knowledge of higher education students, we used translated, adapted, and validated versions of the Test of Understanding in College Economics (TUCE, 4th ed.), …

Assessing the previous economic knowledge of beginning students in Germany: Implications for teaching economics in basic courses

Study-related prior knowledge plays a decisive role in business and economics degree courses. Prior knowledge has a significant influence on knowledge acquisition in higher education, and teachers need information on it to plan their introductory courses accordingly. Very few studies have been conducted of first-year students’ prior economic knowledge or differences among students in such knowledge. In this article, the prior economic knowledge and the influence of personal factors on first-year students’ prior economic knowledge are examined. For this purpose, an adaptation of the fourth, revised edition of the American Test of Economic Literacy was administered, which was adapted in 2014…

Influences on the Development of Economic Knowledge over the First Academic Year

Despite significant research, it remains unclear whether the goal of developing domain-specific knowledge in higher education is actually being achieved. This is also true for the internationally most popular study domain of business and economics. In Germany, a test for measuring economic knowledge was validated, enabling the analysis of change in knowledge over the course of studies. Business and economics students from across Germany were surveyed over the course of one study year: 7,111 beginning students in the winter term of 2016/2017, and 1,705 third semester students in the winter term of 2017/2018. Investigating the longitudinal matched sample of 734 students who took part at both …

Integrating refugees into higher education – the impact of a new online education program for policies and practices

Gaining access to higher education (HE) is a serious concern for refugees. Although studies on the integration and success of refugees in HE are scarce, the challenges associated with this issue as...

The relationship between medical students’ media use and learning progress

The acquisition of warranted domain-specific knowledge is essential for practical work in medicine. The medical field, however, suffers from 'information overload', and students and physicians rely...

Students’ online information use and learning progress in higher education – A critical literature review

Learning using the Internet has become a vital factor for academic success in higher education. Students increasingly rely on the Internet as their main information source. However, related researc...

IRT Modeling of Decomposed Student Learning Patterns in Higher Education Economics

Researchers have spent decades arguing how to measure improvements in learning within formal settings, where achievements are intended and presupposed, in a reliable and valid way. Repeated measures are required to investigate improvements of any kind. Students usually take a multiple-choice test at least twice with the difference between the two measurements indicating how much they have learned. Walstad and Wagner (J Econ Educ 47:121–131, 2016) presented a new approach to gathering more information about different learning patterns by decomposing these difference measures. They describe the patterns of positive learning (PL) and negative learning (NL), i.e., the development from not knowi…

A Concept for Quantitative Comparison of Mathematical and Natural Language and its possible Effect on Learning

Starting with the question whether there is a connection between the mathematical capabilities of a person and his or her mother tongue, we introduce a new modeling approach to quantitatively compare natural languages with mathematical language. The question arises from educational assessment studies that indicate such a relation. Texts written in natural languages can be deconstructed into a dependence graph, in simple cases a dependence tree. The same kind of deconstruction is also possible for mathematical texts. This gives an idea of how to quantitatively compare mathematical and natural language. To that end, we develop algorithms to define the distance between graphs. In this paper, w…

Die Signalwirkung von Hochschulrankings - eine empirische Studie unter Studienanfängerinnen/-anfängern der Wirtschaftswissenschaften

Hochschulrankings sind in den letzten Jahren auch in Deutschland gerade in den Wirtschaftswissenschaften sehr popular und damit zunehmend zum Gegenstand der Forschung geworden. Allerdings erweisen sich die Erkenntnisse sowohl zur generellen Bedeutung von Rankings bei der Hochschulwahl als auch zur Wirkung der Rankings in Abhangigkeit bestimmter personenbezogener Eigenschaften der Studieninteressierten bislang als begrenzt. An diesem Forschungsdefizit setzt der vorliegende Beitrag an, indem zunachst die Wirkungsweise von Rankings theoretisch betrachtet wird, bevor Befragungsergebnisse von 1.314 Studienanfangerinnen und -anfangern der Wirtschaftswissenschaften prasentiert werden. 19.12.2014 |…

Evaluating a Technology-Based Assessment (TBA) to Measure Teachers’ Action-Related and Reflective Skills

Teaching performance can be assessed validly only if the assessment involves an appropriate, authentic representation of real-life teaching practices. Different skills interact in coordinating teac...

On the complementarity of holistic and analytic approaches to performance assessment scoring.

BACKGROUND A holistic approach to performance assessment recognizes the theoretical complexity of multifaceted critical thinking (CT), a key objective of higher education. However, issues related to reliability, interpretation, and use arise with this approach. AIMS AND METHOD Therefore, we take an analytic approach to scoring students' written responses on a performance assessment. We focus on the complementarity of holistic and analytic approaches and on whether theoretically developed analytical scoring rubrics can produce sub-scores that may measure the 'whole' performance in a holistic assessment. SAMPLE We use data from the Wind Turbines performance assessment, developed in the iPAL p…

Effects of prior economic education, native language, and gender on economic knowledge of first-year students in higher education. A comparative study between Germany and the USA

The assessment of university students’ economic knowledge has become an increasingly important research area within and across countries. Particularly, the different influences of prior education, native language, and gender as some of the main prerequisites on students’ economic knowledge have been highlighted since long. However, the findings are often only available within countries and focus on students who are at different levels of their studies or graduates. To remedy this research deficit, the goal of our article is to analyze the status of economic knowledge of students at the beginning of their course of studies and compare the effects of prior economic education, gender, and nati…

Pretest-Posttest-Posttest Multilevel IRT Modeling of Competence Growth of Students in Higher Education in Germany

Longitudinal research in higher education faces several challenges. Appropriate methods of analyzing competence growth of students are needed to deal with those challenges and thereby obtain valid results. In this article, a pretest-posttest-posttest multivariate multilevel IRT model for repeated measures is introduced which is designed to address educational research questions according to a German research project. In this model, dependencies between repeated observations of the same students are considered not, as usual, by clustering observations within participants but rather by clustering observations within semesters. Estimation of the model is conducted within a Bayesian framework. …

An Overview of KoKoHs Projects, Models, and Instruments

Demands in Higher Education

The field testing of the models and instruments at higher education institutions throughout Germany and the analyses of the data gathered on students’ competencies and their determinants pointed to a number of common challenges that institutions, policy-makers, practitioners, and students face with regard to different study phases of higher education today.

Assessing the Quality of the Bachelor Degree Courses: An Analysis of their Effects on Students' Acquisition of Economic Content Knowledge

Qualitat der Bachelorstudiengange auf dem Prufstand: Analyse der Effekte auf den Erwerb des okonomischen Fachwissens Die Erfassung, Bewertung und Optimierung der Leistungen im Bereich der Lehre sind entscheidende Bestandteile einer Qualitatsentwicklung im Hochschulbereich. Fur die neuen BA- und MA-Studiengange liegen jedoch Ergebnisse objektiver Messungen zu zentralen Leistungsindikatoren wie Fachwissenserwerb der Studierenden bislang nur vereinzelt vor. Um diesem Erkenntnisdefizit zu begegnen, wurde in einer Langsschnittstudie das betriebs- und volkswirtschaftliche Wissen von Studierenden des BA/MA-Modells im Vergleich zum „alten“ Diplomstudienmodell erfasst und unter Kontrolle einer Vielz…

Validating a Test for Measuring Knowledge and Understanding of Economics Among University Students

Abstract. University students' knowledge and understanding of economics have mostly been investigated cross-sectionally; however, longitudinal analyses are needed to determine which factors influence knowledge development and to draw valid conclusions based on test results. In the WiWiKom II project, a quasi-experimental longitudinal study was conducted assessing bachelor students of business and economics over the course of their studies in Germany (N = 39 universities). In this project, running from 2016 – 2019, the test-takers complete the WiWiKom II-test of economic knowledge (adapted TUCE4G and TEL4G items) and a general cognitive ability test (BEFKI 11) at four measurement points, ea…

Experiences of Refugees When Entering Higher Education and Over the Course of Their Studies: A Literature Review of Structural and Personal Challenges

In view of the significantly increasing number of individuals around the world who are forced to flee their country of origin, research discussions about integrating refugees into education are gaining relevance. A central research area is their entry into and study success in higher education, focusing on the associated challenges this learner group faces. Though the results of studies dealing with this topic often overlap, they usually focus on individual sub-areas and set specific thematic priorities. As a result, certain sub-areas such as administrative challenges are examined far more frequently than others. In this literature review, data from 14 English-language studies published bet…

Teachers’ Evidence-Based Actions

Currently, there is a dynamic development in the German school system. It is a new governance model which has been designed to have measurable effects on the political regulation of education.

Modeling and Measuring University Students’ Subject-Specific Competencies in the Domain of Business and Economics – The ILLEV Project

Current research provides very little empirical evidence regarding the influence of academic higher education on the development of professionalism among students. Over the course of the reform of the higher education systems in Europe (Bologna declaration) this issue has become increasingly important.

Measurement Advances and Challenges in Competency Assessment in Higher Education

Domain-specific and generic competencies have become target outcomes of students' learning in higher education, and competency assessment plays an increasingly important role in higher education on an international scale. Recent reviews of the literature show that there is still a substantial lack of research on assessment practices in higher education, especially on measurement models and valid instruments for competency assessment. This special issue provides an overview of current international research and innovative developments in this research area, covering a broad range of assessment perspectives, addressing prevalent shortcomings of educational assessment practices, and presenting…

Modeling and Measuring Competencies in Higher Education

Measuring competencies acquired in higher education has to be regarded as a widely neglected research field. The progress made in empirical research on the school system since the 1990s – for example, through large-scale assessments such as the Trends in International Mathematics and Science Study (TIMSS) and the Programme for International Student Assessment (PISA) and through a massive expansion of instructional research in general – has revealed that nothing comparable exists at the higher education level.

Konzeptioneller und methodischer Rahmen zur Analyse des Forschungsstandes

Die vorliegende Forschungssynthese dokumentiert den Stand der nationalen und internationalen Forschung zur Erfassung von akademisch vermittelten Kompetenzen bei Studierenden und Hochschulabsolventen fur die Jahre 2010 bis 2014. Zu seiner Erstellung wurde im Sommer/Herbst 2014 – Winter 2015 eine breit angelegte Dokumentenanalyse durch Literatur- und Datenbankrecherchen (inkl. qualitativen Inhaltsanalysen) durchgefuhrt. Um eine systematische Vorgehensweise zu gewahrleisten, wurde der Recherche eine Auswahl von national und international relevanten hochschul- und bildungsforschungsspezifischen sowie disziplinubergreifenden Literatur- und Projektdatenbanken zugrunde gelegt; auf nationaler Ebene…

How do university students’ web search behavior, website characteristics, and the interaction of both influence students’ critical online reasoning?

The Internet has become one of the main sources of information for university students’ learning. Since anyone can disseminate content online, however, the Internet is full of irrelevant, biased, or even false information. Thus, students’ ability to use online information in a critical-reflective manner is of crucial importance. In our study, we used a framework for the assessment of students’ critical online reasoning (COR) to measure university students’ ability to critically use information from online sources and to reason on contentious issues based on online information. In addition to analyzing students’ COR by evaluating their open-ended short answers, we also investigated the stude…

Automated scoring of teachers’ pedagogical content knowledge : a comparison between human and machine scoring

To validly assess teachers’ pedagogical content knowledge (PCK), performance-based tasks with open-response formats are required. Automated scoring is considered an appropriate approach to reduce the resource-intensity of human scoring and to achieve more consistent scoring results than human raters. The focus is on the comparability of human and automated scoring of PCK for economics teachers. The answers of (prospective) teachers (N = 852) to six open-response tasks from a standardized and validated test were scored by two trained human raters and the engine “Educational SCoRIng Toolkit” (ESCRITO). The average agreement between human and computer ratings, κw = 0.66, suggests a convergent …

The international state of research on measurement of competency in higher education

With the Program for International Student Assessment and Trends in International Mathematics and Science Study surveys, competency assessment became an important policy instrument in the school sector; only recently has international competency measurement gained attention in higher education with the Assessment of Higher Education Learning Outcomes (AHELO) feasibility study. As AHELO showed, measurement of competencies in higher education is a complex and multidimensional task, which poses great methodological challenges. These challenges arise out of the high diversity of degree courses, study programs, and institutions of higher education. Nevertheless, we need to address these challeng…

Development Perspectives for Higher Education Research and Practice

In the 24 collaborative projects of the KoKoHs program, comprising more than 70 individual research projects, objective test instruments were developed for assessing the competencies of students and graduates and tested in higher education practice (see table in section 8). The KoKoHs assessments provide empirically substantiated information on students’ competency levels and insight into the effectiveness of teaching-and-learning opportunities in higher education. KoKoHs has offered means to address major challenges of higher education in a more effective manner and, in the long term, contributed to improving structural, organizational, and individual conditions for teaching and learning a…

Identification of Vulnerable Groups in Terms of Language Proficiency in Higher Education

The language proficiency of university students is considered a key factor for knowledge acquisition and success in higher education. Recent research has revealed a connection between language proficiency and academic performance. Some studies identified certain groups of learners who show deficits in language proficiency when entering higher education and who can be considered to be vulnerable groups who need particular support. The present study analyzes the level of proficiency in the language of instruction among two vulnerable groups of students at the beginning of their university studies: (i) students with a refugee background and a mother tongue other than English enrolled in Englis…

Student understanding of graph slope and area under a curve: A replication study comparing first-year physics and economics students

The understanding of graphs and extraction of relevant information from graphs plays a major role in physics education and is also important in several related fields. Recently, Susac et al. [Phys. Rev. Phys. Educ. Res. 14, 020109 (2018)10.1103/PhysRevPhysEducRes.14.020109] compared physics and psychology students’ understanding of graphs in the contexts of physics and finance. They showed that physicists scored significantly higher in both domains and that all students solved the slope problems better than the area problems. Moreover, eye-tracking data revealed that physics students spent more time on problems associated with the area under the graph and focused longer on the axis labels o…

Publication of KoKoHs Projects

Positive and Negative Media Effects on University Students’ Learning: Preliminary Findings and a Research Program

Research in communication science highlights positive as well as negative effects of news and social media on learning but focuses predominantly on the largely unintended knowledge acquisition of the overall population. Research in educational science deals with students’ knowledge acquisition but is largely limited to formal learning such as in university courses. In this paper, we report findings of a pilot study combining both approaches by dealing with mass and social media effects on university students’ learning. While this study reveals several effects, their influences and causality remain largely unclear. Therefore, we propose a research program to explain positive and negative med…

Changes in students’ understanding of and visual attention on digitally represented graphs across two domains in higher education : a postreplication study

Domain-specific understanding of digitally represented graphs is necessary for successful learning within and across domains in higher education. Two recent studies conducted a cross-sectional analysis of graph understanding in different contexts (physics and finance), task concepts, and question types among students of physics, psychology, and economics. However, neither changes in graph processing nor changes in test scores over the course of one semester have been sufficiently researched so far. This eye-tracking replication study with a pretest–posttest design examines and contrasts changes in physics and economics students’ understanding of linear physics and finance graphs. It analyze…

Gender gap in Swiss vocational education and training teachers’ economics content knowledge and the role of teaching experience

In the commercial sector, which is of crucial importance to the Swiss economy among other countries, a large number of apprentices are trained on a vocational education and training programme every year. Besides other subjects, the subject Economics and Society forms an integral part of the vocational education and training curriculum and serves to prepare apprentices for professional, economic and civic participation. Although content knowledge is widely considered necessary to both teaching quality and student achievement, little is known about the subject-specific content knowledge of Swiss Economics and Society teachers. As previous research has shown a gender gap in the content knowled…

Digital Approaches to Integrating Refugees into Higher Education—Frontiers and Advances

This SpringerBriefs volume illustrates the potentials of digital learning opportunities in higher education for refugees with different educational, social, cultural, and linguistic backgrounds from various perspectives and based on research conducted in several countries. Global challenges for academic success that refugees face when seeking to enter higher education are outlined. One possibility to reduce the obstacles for refugees entering higher education is to offer digital learning platforms. A selection of international digital education platforms for refugees are presented, which are based on a variety of approaches to accommodate the needs of prospective refugee students. Based on …

An analysis of economic learning among undergraduates in introductory economics courses in Germany

In this article, the authors present the findings of a pretest-posttest measurement of the economic knowledge of students in introductory economics courses in undergraduate study programs in Germany. The responses of 403 students to 14 items selected from the Test of Economic Literacy (Soper and Walstad 1987) were analyzed to identify four types of economic learning: positive, retained, negative, and zero learning. In addition, a survey was conducted to gather data on the students' personal characteristics to determine their effect on the learning process. Retained learning prevailed for most items, followed by zero learning and positive learning. To determine which factors influence beginn…

Competency Orientation in Higher Education

In recent years, higher education, particularly in Germany, has been marked by fundamental structural and programmatic changes after the implementation of the Bologna reform and the orientation towards competencies in the national standards for education, for instance, set forth by the Standing Conference of the Ministers of Education and Cultural Affairs in Germany. The quality of academic education and its social returns have received increasing attention, and endeavors to develop and implement fair assessments of students’ learning outcomes have revealed diverse challenges. In Germany and internationally, it is becoming increasingly important to define and assess in an objective, reliabl…

Validity and fairness of a new entry diagnostics test in higher education economics

Abstract Research has been focusing increasingly on measuring students’ prior economic knowledge in higher education. However, in German-speaking countries, valid instruments are rare. A diagnostic study investigated the validity and fairness of an internationally established economic knowledge test (adapted German version of TUCE IV and TEL IV) with a representative sample of 7664 students from 46 universities across Germany. Previous findings with the dataset ( Zlatkin-Troitschanskaia et al., 2019a ) suggest that native language significantly predicts missing answers in the knowledge test. Similar effects can be found for gender. After coding missing answers as incorrect, students with a …

Ein videobasiertes Lehr-Lernformat als innovativer hochschuldidaktischer Ansatz in der wirtschaftspädagogischen Lehramtsausbildung

Wir prasentieren einen hochschuldidaktischen Ansatz, bei dem sich Lehramtsstudierende mittels eines videobasierten Lehr-Lernformats fachspezifisch mit ihren fur das unterrichtliche Handeln erforderlichen Kompetenzen auseinandersetzen. Die im Lehr-Lernformat fokussierten handlungsnahen Unterrichtskompetenzen werden dabei in eine aktionsbezogene und reflexive Kompetenzkomponente differenziert. Beide Komponenten kommen, unter Ruckbezug auf das fachliche und fachdidaktische Wissen, bei der praktischen Bewaltigung komplexer Anforderungssituationen im Fachunterricht zum Einsatz. Eine Interventionsstudie (N = 27 Studierende der Wirtschaftspadagogik) mit Vergleichsgruppendesign, kontrollierter Vari…

Implikationen für die weitere Forschung

Der hier vorgelegte Uberblick zum nationalen und internationalen Forschungsstand im Bereich der Messung akademisch vermittelter Kompetenzen von Studierenden und Hochschulabsolventen belegt, dass Bedarf und Bedeutung dieses Forschungsfeldes vielerorts inzwischen erkannt wurden. Die bisher in Deutschland vor allem im Rahmen des KoKoHs-Programms (s. Tabelle 2c) entwickelten und erprobten Kompetenzmodelle und Assessmentdesigns sowie die zugehorigen Messinstrumente liefern eine Grundlage, um dieses Feld auf nationaler Ebene systematisch weiter auszubauen. Es wird zukunftig darum gehen, insbesondere langsschnittlich angelegte, mehrere Ebenen umfassende Ansatze zu verfolgen, die durch (feld-)exper…

A new video-based tool to enhance teaching economics

Abstract We present a new video-based tool developed in Germany to enhance the teaching of economics. The tool consists of authentic teacher-student interactions in economics classes to which instructors need to react. With 18 videos, the tool provides an easily accessible opportunity to assess and enhance teaching performance in economics. The tool was rated by 41 highly experienced secondary school economics teachers according to different criteria relevant for curriculum design and instructional practice, including authenticity of the simulated classroom situations, and typical difficulties and misconceptions students have in learning economics. The teachers confirm the curricular and in…

Epistemic network analyses of economics students' graph understanding. An eye-tracking study

Learning to solve graph tasks is one of the key prerequisites of acquiring domain-specific knowledge in most study domains. Analyses of graph understanding often use eye-tracking and focus on analyzing how much time students spend gazing at particular areas of a graph&mdash

Patterns of domain-specific learning among medical undergraduate students in relation to confidence in their physiology knowledge : insights from a pre–post study

Research FocusThe promotion of domain-specific knowledge is a central goal of higher education and, in the field of medicine, it is particularly essential to promote global health. Domain-specific knowledge on its own is not exhaustive; confidence regarding the factual truth of this knowledge content is also required. An increase in both knowledge and confidence is considered a necessary prerequisite for making professional decisions in the clinical context. Especially the knowledge of human physiology is fundamental and simultaneously critical to medical decision-making. However, numerous studies have shown difficulties in understanding and misconceptions in this area of knowledge. Therefo…

MAPPING COMPETENCY ACQUISITION IN HIGHER EDUCATION AND ITS INFLUENCES: A META-STUDY

In the 10-year research program "Modelling and Measuring Competencies in Higher Education (KoKoHs)" that ran from 2011 until 2020, more than 100 theoretically sound models and corresponding tests were developed to analyze the level and development of students’ competencies. Using these newly developed assessments, generic and domain-specific competencies of more than 75,000 students at more than 350 German universities in numerous disciplines (e.g., economics, teacher education) were measured in 39 interdisciplinary collaborative KoKoHs projects. Method: In these measurements, a nationally and internationally unique database was created that contains information on the level, acquisition, a…

The gap between knowledge and belief: narrative, affect and students’ deeper learning in higher education

One of the central challenges educators face today, especially in higher education, is the gap between warranted (domain-specific) knowledge and the prior beliefs students hold about certain concep...

What Can We Learn from Theoretical Considerations and Empirical Evidence on Learning in Higher Education? Implications for an Interdisciplinary Research Framework

Since the development of the hypertext markup language (HTML), a myriad of data from different sources has become increasingly cross-linked, resulting in entire information landscapes; more than 21 billion electronic devices (Web 4.0) will be in use by 2020. Access to and processing of multimedia content and unverified and algorithmically preselected information are constitutive elements of learning in the Internet Age. As learning occurs in- and outside of formal higher education institutions and cannot be explained by educational or technological factors alone, it is a crucial cross-disciplinary topic, equally important for all areas of science and education. Yet, research on learning in …

International performance assessment of critical thinking: framework for translation and adaptation

Journal of supernational policies of education, Nr, 13 (2021), S. 62–87

Assessing student learning outcomes in higher education: challenges and international perspectives

Profound structural and programmatic changes in higher education, due in part to globalisation and the harmonisation of educational structures and study models (Lucena et al. 2008; Brennan, Patel, ...

Progressions in Learning in the Age of (Mis)Information (PLATO) – Advances in Higher Education Learning Research

In the 21st century, using the Internet for learning has become one of the most crucial factors of students’ study success in higher education (HE). Students use formally approved HE learning mater...

Performance Assessment of Critical Thinking: Conceptualization, Design, and Implementation

Enhancing students’ critical thinking (CT) skills is an essential goal of higher education. This article presents a systematic approach to conceptualizing and measuring CT. CT generally comprises the following mental processes: identifying, evaluating, and analyzing a problem; interpreting information; synthesizing evidence; and reporting a conclusion. We further posit that CT also involves dealing with dilemmas involving ambiguity or conflicts among principles and contradictory information. We argue that performance assessment provides the most realistic—and most credible—approach to measuring CT. From this conceptualization and construct definition, we describe one possible framework for …

Threshold Concepts for Modeling and Assessing Higher Education Students’ Understanding and Learning in Economics

In the last decade, the research carried out on threshold concepts as a content-based way to model students’ understanding and learning in several domains has increased. However, empirical evidence on this approach is still scarce. In this chapter, the authors investigate the adequacy of the threshold concepts approach in the domain of business and economics in higher education following an established differentiation between basic, discipline, and modeling thresholds. After conducting a cognitive interview study using verbal reports, a self-assessment questionnaire was used to assess the respondents’ familiarity with the content and their security to solve the tasks. Results indicate that …

Standards für pädagogisches Testen

Die „Standards for Educational and Psychological Testing“ bieten eine umfangreiche Darstellung von uber 240 Standards, die zur Entwicklung, Durchfuhrung und Evaluation padagogischer und psychologischer Tests praktische Handlungsempfehlungen geben. Zur Berucksichtigung der Besonderheiten der Zielsetzung und Entwicklung padagogischer Tests konnen die Standards insbesondere zu Fragen der Validitat einen Beitrag leisten. Weitere Implikationen aus den Standards fur Anforderungen padagogischen Testens lassen sich u. a. zum Standardsetting, zur Fairness, zur Transparenz des Untersuchungsgegenstands und Interpretation, zu Formen der Diagnostik, zum Feedback sowie zum Datenmanagement finden. Zusatzl…

Assessment of University Students’ Critical Thinking: Next Generation Performance Assessment

Following employers’ criticisms and recent societal developments, policymakers and educators have called for students to develop a range of generic skills such as critical thinking (“twenty-first c...

Kompetenzen von Studierenden – Ergebnisse des „KoKoHs“-Programms im Kontext der nationalen und internationalen Assessmentpraxis

Ausgehend von aktuellen Entwicklungen im Hochschulbereich und daraus resultierenden Anforderungen wird in diesem Beitrag die nationale und internationale Assessmentpraxis skizziert und vor diesem Hintergrund zentrale Ergebnisse des „KoKoHs“-Programms dargestellt. Abschliesend werden weitere theoretische, methodologische und empirische Herausforderungen sowie Entwicklungsperspektiven fur Hochschulpolitik und -praxis aufgezeigt.

Positive Learning in the Internet Age: Developments and Perspectives in the PLATO Program

The Internet has become the main informational entity, i.e., a public source of information. The Internet offers many new benefits and opportunities for human learning, teaching, and research. However, by providing a vast amount of information from innumerable sources, it also enables the manipulation of information; there are countless examples of disseminated misinformation and false data in mass and social media. Much of the information presented online is conflicting, preselected, or algorithmically obscure, often colliding with fundamental humanistic values and posing moral or ethical problems.

Konstruktion, Implementation und Evaluation eines multimedialen Assessmenttools in der wirtschaftspädagogischen Ausbildung

Im Kontext der Outputorientierung im Bildungssystem gerat die Forderung handlungsbezogener Kompetenzen zunehmend in den Fokus der Lehrerausbildung und Forschung. Kompetenzorientierung verlangt nach einer Konvergenz von Curriculum, Instruktion und Assessment. Der vorliegende Beitrag analysiert die curriculare Verankerung der Forderung handlungsbezogener Kompetenzen innerhalb der wirtschaftspadagogischen Studiengange und stellt ein innovatives Lehr- und Assessmentkonzept vor, das im instruktionalen Kontext auch als komplexes Assessmenttool zur gezielten Forderung solcher (bisher vernachlassigten) Kompetenzfacetten in der wirtschaftspadagogischen Ausbildung implementiert und evaluiert werden s…

International Performance Assessment of Learning in Higher Education (iPAL): Research and Development

Educators and policy makers now recognize how important and necessary it is to assess student learning outcomes (SLOs) in higher education. The question has shifted from whether such outcomes should be measured to how they should be measured. Today SLOs are typically assessed by student self-reports of learning or with multiple-choice and short-answer tests. Each of these methods has its strengths and limitations; each one provides insights into the nature of teaching and learning. An alternative approach is the assessment of performance using “criterion” tasks that are drawn from real-world situations in which students are being educated, both within and across academic or professional dom…

How migration background affects master degree students’ knowledge of business and economics

ABSTRACTIn this study, we examine the knowledge of students in a master program in business and economics and explore whether migration-specific influences can be determined. We tested the knowledg...

A Multilevel Analysis of Economic Literacy Among International Students

International student mobility has increased in the past years. The inclusion of a highly heterogeneous group of students requires updated recruitment and admission strategies. A particularly vulnerable group of international students are refugees, who have an exceptionally high risk of dropping out of their studies. We present an entrance assessment of incoming international students from 77 countries who are at the beginning of their studies in Germany. Based on this unique sample, we examine (i) whether there are systematic country-specific effects on the economic literacy of beginning students of business and economics and (ii) whether refugee students differ from the group of internati…