Validation and Applications of KoKoHs Assessments

Modeling and Measuring Competencies in Germany

In order to meet the challenges facing higher education policy and practice, and initiate the necessary fundamental assessment-research, the German Federal Ministry of Education and Research (BMBF) established the Modeling and Measuring Competencies in Higher Education (KoKoHs) research program in 2011. The program was part of the BMBF funding priority Research on Science and Higher Education aimed at building a capacity for empirical research in higher education.

Validating Test Score Interpretations by Cross-National Comparison

Cross-national assessment of students’ competences in higher education is becoming increasingly important in many disciplines including economics but there are few available instruments that meet psychological standards for assessing students’ economic competence in higher education (HE). One of them is the internationally valid Test of Understanding in College Economics (TUCE), which has been adapted and employed successfully in HE systems in various countries, but the test results have seldom been used for international comparisons of students’ Economic Content Knowledge (ECK). Here, we compare the German and the Japanese test adaptations of the TUCE with reference to the American origin…

Macroeconomic knowledge of higher education students in Germany and Japan – a multilevel analysis of contextual and personal effects

Recent trends towards harmonising and internationalising business and economics studies in higher education are affecting the structure and content of programmes and courses, and necessitate more transparent and comparable information on students’ economic knowledge and skills. In this study, we examine by linear multilevel regression modelling the current state of macroeconomic knowledge of higher education students in Germany and Japan, while controlling for the effects of key study-related aspects such as study progress and completion of economics courses. We assess macroeconomic knowledge using the internationally established Test of Understanding in College Economics, which has been ad…

An Overview of Assessment Practices

On a global scale, attempts to specify conceptually academic learning outcomes have been made mostly in OECD countries. Definitions of learning outcomes can be used not only as an orientation for accrediting degree courses and universities; in some countries, they are used also as a basis for the development of assessment methods. The following will provide an overview of various key approaches and projects in this field.

Adapting and Validating the Collegiate Learning Assessment to Measure Generic Academic Skills of Students in Germany: Implications for International Assessment Studies in Higher Education

Starting in 2015, a German research team from the program Modeling and Measuring Competencies in Higher Education (KoKoHs), in collaboration with the US Council for Aid to Education (CAE), adapted and validated the Collegiate Learning Assessment (CLA+) for the German language and cultural context to measure generic higher-order cognitive skills of university students and graduates in Germany. In this chapter, the conceptual and methodological background, the framework of the adaptation and validation study, as well as preliminary results are presented. Finally, findings are discussed critically, and future challenges and perspectives are explored.

Gender Effects in Assessment of Economic Knowledge and Understanding: Differences Among Undergraduate Business and Economics Students in Germany, Japan, and the United States

Gender effects in large-scale assessments have become an increasingly important research area within and across countries. Yet few studies have linked differences in assessment results of male and female students in higher education to construct-relevant features of the target construct. This paper examines gender effects on students’ economic content knowledge with a focus on construct-relevant explanations. Moreover, we compare gender effects cross-nationally between Germany, Japan, and the United States. To assess economic content knowledge of higher education students, we used translated, adapted, and validated versions of the Test of Understanding in College Economics (TUCE, 4th ed.), …

Influences on the Development of Economic Knowledge over the First Academic Year

Despite significant research, it remains unclear whether the goal of developing domain-specific knowledge in higher education is actually being achieved. This is also true for the internationally most popular study domain of business and economics. In Germany, a test for measuring economic knowledge was validated, enabling the analysis of change in knowledge over the course of studies. Business and economics students from across Germany were surveyed over the course of one study year: 7,111 beginning students in the winter term of 2016/2017, and 1,705 third semester students in the winter term of 2017/2018. Investigating the longitudinal matched sample of 734 students who took part at both …

Evaluating a Technology-Based Assessment (TBA) to Measure Teachers’ Action-Related and Reflective Skills

Teaching performance can be assessed validly only if the assessment involves an appropriate, authentic representation of real-life teaching practices. Different skills interact in coordinating teac...

Effects of prior economic education, native language, and gender on economic knowledge of first-year students in higher education. A comparative study between Germany and the USA

The assessment of university students’ economic knowledge has become an increasingly important research area within and across countries. Particularly, the different influences of prior education, native language, and gender as some of the main prerequisites on students’ economic knowledge have been highlighted since long. However, the findings are often only available within countries and focus on students who are at different levels of their studies or graduates. To remedy this research deficit, the goal of our article is to analyze the status of economic knowledge of students at the beginning of their course of studies and compare the effects of prior economic education, gender, and nati…

An Overview of KoKoHs Projects, Models, and Instruments

Demands in Higher Education

The field testing of the models and instruments at higher education institutions throughout Germany and the analyses of the data gathered on students’ competencies and their determinants pointed to a number of common challenges that institutions, policy-makers, practitioners, and students face with regard to different study phases of higher education today.

Validating a Test for Measuring Knowledge and Understanding of Economics Among University Students

Abstract. University students' knowledge and understanding of economics have mostly been investigated cross-sectionally; however, longitudinal analyses are needed to determine which factors influence knowledge development and to draw valid conclusions based on test results. In the WiWiKom II project, a quasi-experimental longitudinal study was conducted assessing bachelor students of business and economics over the course of their studies in Germany (N = 39 universities). In this project, running from 2016 – 2019, the test-takers complete the WiWiKom II-test of economic knowledge (adapted TUCE4G and TEL4G items) and a general cognitive ability test (BEFKI 11) at four measurement points, ea…

Contributions of Response Processes Analysis to the Validation of an Assessment of Higher Education Students’ Competence in Business and Economics

Higher education domains such as Business and Economics (B&E) generally lack evidence that standardized tests of competency actually assess those aspects of knowledge and reasoning that constitute important targets for learning and instruction. The construction of such a validity argument benefits from a framework to guide the collection of relevant evidence regarding the interpretive meaning of tests results, especially with respect to instruction and learning in B&E. Response processes analysis provides much needed evidence within such a validity argument. The present paper illustrates application of such a framework and the use of response processes analysis for assessment in the B&E hig…

How do university students’ web search behavior, website characteristics, and the interaction of both influence students’ critical online reasoning?

The Internet has become one of the main sources of information for university students’ learning. Since anyone can disseminate content online, however, the Internet is full of irrelevant, biased, or even false information. Thus, students’ ability to use online information in a critical-reflective manner is of crucial importance. In our study, we used a framework for the assessment of students’ critical online reasoning (COR) to measure university students’ ability to critically use information from online sources and to reason on contentious issues based on online information. In addition to analyzing students’ COR by evaluating their open-ended short answers, we also investigated the stude…

Development Perspectives for Higher Education Research and Practice

In the 24 collaborative projects of the KoKoHs program, comprising more than 70 individual research projects, objective test instruments were developed for assessing the competencies of students and graduates and tested in higher education practice (see table in section 8). The KoKoHs assessments provide empirically substantiated information on students’ competency levels and insight into the effectiveness of teaching-and-learning opportunities in higher education. KoKoHs has offered means to address major challenges of higher education in a more effective manner and, in the long term, contributed to improving structural, organizational, and individual conditions for teaching and learning a…

Integrating the Analysis of Mental Operations Into Multilevel Models to Validate an Assessment of Higher Education Students’ Competency in Business and Economics

The Standards for Educational and Psychological Testing indicate that validation of assessments should include analyses of participants’ response processes. However, such analyses typically are conducted only to supplement quantitative field studies with qualitative data, and seldom are such data connected to quantitative data on student or item performance. This paper presents an example of how data from an analysis of mental operations collected using a sociocognitive approach can be quantitatively integrated with other data on student and item performance to validate in part an assessment of higher education students’ competency in business and economics. Evidence of forward reasoning an…

Student understanding of graph slope and area under a curve: A replication study comparing first-year physics and economics students

The understanding of graphs and extraction of relevant information from graphs plays a major role in physics education and is also important in several related fields. Recently, Susac et al. [Phys. Rev. Phys. Educ. Res. 14, 020109 (2018)10.1103/PhysRevPhysEducRes.14.020109] compared physics and psychology students’ understanding of graphs in the contexts of physics and finance. They showed that physicists scored significantly higher in both domains and that all students solved the slope problems better than the area problems. Moreover, eye-tracking data revealed that physics students spent more time on problems associated with the area under the graph and focused longer on the axis labels o…

Publication of KoKoHs Projects

Changes in students’ understanding of and visual attention on digitally represented graphs across two domains in higher education : a postreplication study

Domain-specific understanding of digitally represented graphs is necessary for successful learning within and across domains in higher education. Two recent studies conducted a cross-sectional analysis of graph understanding in different contexts (physics and finance), task concepts, and question types among students of physics, psychology, and economics. However, neither changes in graph processing nor changes in test scores over the course of one semester have been sufficiently researched so far. This eye-tracking replication study with a pretest–posttest design examines and contrasts changes in physics and economics students’ understanding of linear physics and finance graphs. It analyze…

Gender gap in Swiss vocational education and training teachers’ economics content knowledge and the role of teaching experience

In the commercial sector, which is of crucial importance to the Swiss economy among other countries, a large number of apprentices are trained on a vocational education and training programme every year. Besides other subjects, the subject Economics and Society forms an integral part of the vocational education and training curriculum and serves to prepare apprentices for professional, economic and civic participation. Although content knowledge is widely considered necessary to both teaching quality and student achievement, little is known about the subject-specific content knowledge of Swiss Economics and Society teachers. As previous research has shown a gender gap in the content knowled…

Competency Orientation in Higher Education

In recent years, higher education, particularly in Germany, has been marked by fundamental structural and programmatic changes after the implementation of the Bologna reform and the orientation towards competencies in the national standards for education, for instance, set forth by the Standing Conference of the Ministers of Education and Cultural Affairs in Germany. The quality of academic education and its social returns have received increasing attention, and endeavors to develop and implement fair assessments of students’ learning outcomes have revealed diverse challenges. In Germany and internationally, it is becoming increasingly important to define and assess in an objective, reliabl…

Validity and fairness of a new entry diagnostics test in higher education economics

Abstract Research has been focusing increasingly on measuring students’ prior economic knowledge in higher education. However, in German-speaking countries, valid instruments are rare. A diagnostic study investigated the validity and fairness of an internationally established economic knowledge test (adapted German version of TUCE IV and TEL IV) with a representative sample of 7664 students from 46 universities across Germany. Previous findings with the dataset ( Zlatkin-Troitschanskaia et al., 2019a ) suggest that native language significantly predicts missing answers in the knowledge test. Similar effects can be found for gender. After coding missing answers as incorrect, students with a …

A new video-based tool to enhance teaching economics

Abstract We present a new video-based tool developed in Germany to enhance the teaching of economics. The tool consists of authentic teacher-student interactions in economics classes to which instructors need to react. With 18 videos, the tool provides an easily accessible opportunity to assess and enhance teaching performance in economics. The tool was rated by 41 highly experienced secondary school economics teachers according to different criteria relevant for curriculum design and instructional practice, including authenticity of the simulated classroom situations, and typical difficulties and misconceptions students have in learning economics. The teachers confirm the curricular and in…

Epistemic network analyses of economics students' graph understanding. An eye-tracking study

Learning to solve graph tasks is one of the key prerequisites of acquiring domain-specific knowledge in most study domains. Analyses of graph understanding often use eye-tracking and focus on analyzing how much time students spend gazing at particular areas of a graph&mdash

What Can We Learn from Theoretical Considerations and Empirical Evidence on Learning in Higher Education? Implications for an Interdisciplinary Research Framework

Since the development of the hypertext markup language (HTML), a myriad of data from different sources has become increasingly cross-linked, resulting in entire information landscapes; more than 21 billion electronic devices (Web 4.0) will be in use by 2020. Access to and processing of multimedia content and unverified and algorithmically preselected information are constitutive elements of learning in the Internet Age. As learning occurs in- and outside of formal higher education institutions and cannot be explained by educational or technological factors alone, it is a crucial cross-disciplinary topic, equally important for all areas of science and education. Yet, research on learning in …

Threshold Concepts for Modeling and Assessing Higher Education Students’ Understanding and Learning in Economics

In the last decade, the research carried out on threshold concepts as a content-based way to model students’ understanding and learning in several domains has increased. However, empirical evidence on this approach is still scarce. In this chapter, the authors investigate the adequacy of the threshold concepts approach in the domain of business and economics in higher education following an established differentiation between basic, discipline, and modeling thresholds. After conducting a cognitive interview study using verbal reports, a self-assessment questionnaire was used to assess the respondents’ familiarity with the content and their security to solve the tasks. Results indicate that …

Standards für pädagogisches Testen

Die „Standards for Educational and Psychological Testing“ bieten eine umfangreiche Darstellung von uber 240 Standards, die zur Entwicklung, Durchfuhrung und Evaluation padagogischer und psychologischer Tests praktische Handlungsempfehlungen geben. Zur Berucksichtigung der Besonderheiten der Zielsetzung und Entwicklung padagogischer Tests konnen die Standards insbesondere zu Fragen der Validitat einen Beitrag leisten. Weitere Implikationen aus den Standards fur Anforderungen padagogischen Testens lassen sich u. a. zum Standardsetting, zur Fairness, zur Transparenz des Untersuchungsgegenstands und Interpretation, zu Formen der Diagnostik, zum Feedback sowie zum Datenmanagement finden. Zusatzl…

Konstruktion, Implementation und Evaluation eines multimedialen Assessmenttools in der wirtschaftspädagogischen Ausbildung

Im Kontext der Outputorientierung im Bildungssystem gerat die Forderung handlungsbezogener Kompetenzen zunehmend in den Fokus der Lehrerausbildung und Forschung. Kompetenzorientierung verlangt nach einer Konvergenz von Curriculum, Instruktion und Assessment. Der vorliegende Beitrag analysiert die curriculare Verankerung der Forderung handlungsbezogener Kompetenzen innerhalb der wirtschaftspadagogischen Studiengange und stellt ein innovatives Lehr- und Assessmentkonzept vor, das im instruktionalen Kontext auch als komplexes Assessmenttool zur gezielten Forderung solcher (bisher vernachlassigten) Kompetenzfacetten in der wirtschaftspadagogischen Ausbildung implementiert und evaluiert werden s…