Designing a Model for International
Learning Assessment
By Lee Sternberger, executive director of the
Office of International Programs at James Madison University;
Dawn Thorndike Pysarchik, professor and associate dean
of International Studies and Programs at Michigan State
University; Zee-Sun Yun, assistant professor, Department
of Family and Consumer Sciences at Western Michigan
University; and Darla Deardorff, executive director
of the Association of International Education Administrators
at Duke University.
Calls to internationalize higher education have grown more urgent over the
past decade as universities endeavor to meet the shifting
social, political, and economic exigencies of our interconnected
societies. Pressing issues such as global warming, the
yawning chasm between rich and poor, and international
violence compel educators to equip students with the
global competencies necessary to address complex challenges
in both local and international contexts. At the heart
of this process is the need to evaluate global education’s
impact on the institution and its students.
Assessment can provide a range of benefits: meaningful
information to ensure effective programs; a rationale
for advocacy of international education; and knowledge
of the learning and developmental processes that underpin
transformation, to name just a few. In the context of
international education, assessment can examine factors
such as language acquisition, content knowledge (geography,
history, cultural customs and practices), and intercultural
competence, communication, and sensitivity. When planned
and implemented effectively, assessment moves beyond
traditional notions of inputs and outputs to capture
the complex impact of international learning. This article
invites you to set your own agenda for assessing international
learning outcomes at your institution.
Assessment Basics
| Table
1: Assessment Team’s Checklist |
__ Aligned and
Articulated: Are goals, objectives, and assessment
measures aligned and articulated?
__ Intentional: Is assessment intentionally addressed?
__ Developed: Have assessment issues been carefully
analyzed before a plan is implemented?
__ Integrated: Is assessment integrated throughout
the program and not viewed as an “add-on”
(implemented only as a pre-post phenomenon)?
__ Focused: Is the assessment scope realistic,
with two to three outcomes assessed per program
per year?
__ Shared: Is assessment shared with others on
campus through partnerships?
__ Supported: Is the senior leadership supportive
of assessment efforts?
__ Resourced: Is there adequate time and funding
for assessment efforts, and have administrators
received sufficient training in assessment, with
ongoing professional development?
__ Analyzed: Have the assessment tools, results,
and process been analyzed and evaluated?
__ Communicated: Have the results been communicated
to all stakeholders?
__ Used: Have the results been used for program
improvement as well as for learner feedback?
__ Reviewed: Has the assessment process and strategy
been reviewed on a regular basis and improved
upon?
Developed by Darla Deardorff, 2008
|
Assessment should begin with a review of the institution’s
mission statement and overall goals. What are the institution’s
priorities, and what evidence can demonstrate their
achievement? How can the institution best combine summative
assessment (which focuses on learners’ development
at a particular time) and formative assessment (a more
holistic approach that uses multiple points of measurement
to provide continuous feedback and improve educational
opportunities) to meet its goals? By answering these
and related questions, administrators can determine
which assessment methods and tools to use.
The next step in developing an assessment protocol is
to explore the institutional context and available resources,
including others who are already engaged in assessment.
This process should lead to the creation of a multiunit
assessment team, as the undertaking is too complex for
any one office to implement. The team should develop
an assessment plan and review it regularly for refinement.
Table 1 provides guidance in this process.
A good assessment plan should include multiple tools
and methods integrated throughout the program. Within
coursework, for example, assessments can evaluate both
direct evidence of student learning (tests, papers,
capstones, portfolios) and indirect evidence, meaning
student perceptions of their learning (self-report instruments,
focus groups, interviews). Questions to consider when
selecting tools include: What does the tool measure,
and how does it support stated goals? Is the tool valid,
reliable, and based on a theoretical framework? What
are the tool’s limitations and biases? Are the
tool’s logistics manageable, and is the tool affordable?
Once the team has developed a strategy, assessment can
begin. The assessment cycle consists of several steps:
1) Define outcomes (based on mission/goals) and establish
measurable criteria 2) identify appropriate assessment
methods 3) collect data 4) analyze data 5) use data:
design and apply changes to the curricular and noncurricular
program components 6) communicate results to all relevant
stakeholders and 7) evaluate the assessment process
and plan. (For further discussion of assessment in international
education, see Deardorff 2007.)
Case Study: MSU
Michigan State University (MSU) is currently implementing
a formative assessment project with promising results.
The MSU study aims to determine the efficacy of using
a mixed-methods approach to examine the outcomes of
students’ international learning relevant to global
and domestic issues, to investigate the influence of
key antecedent factors, and to determine how findings
can influence curricular and noncurricular enrichment
decisions. To address these goals, MSU developed a pioneering
conceptual model to assess students’ progress.
The qualitative (e-portfolio) phase of the project originated
with five other institutions in a project funded by
the Fund for Improvement of Postsecondary Education
(FIPSE) and coordinated by the American Council on Education
(ACE 2008). The quantitative phase (administration of
the Beliefs, Events, and Values Inventory, or BEVI)
originated with an international learning assessment
project coordinated by the Forum on Education Abroad
(2008). To our knowledge, MSU is the only university
using a mixed-methods approach that combines the BEVI
and e-portfolio.
Theoretical Background
To address the complexity of international learning,
the MSU team analyzed different theoretical perspectives
for insights into antecedent factors that influence
student learning, different dimensions of international
learning, and ways to measure these dimensions. The
team drew from three theoretical frameworks (Equilintegration
Theory, Attribution Theory, and Learning Theory) to
structure a conceptual model.
Equilintegration (EI) Theory attempts to explain the
processes by which beliefs, values, and worldviews are
acquired and maintained, why students typically resist
their alteration, and how and under what circumstances
their modification occurs (Shealy 2004, forthcoming).
EI Theory recognizes that values and beliefs are not
easily modified: they represent the unique culmination
of affective and attributional processes that arise
from life experiences.
Attribution Theory (AT) focuses on three dimensions
that influence an individual’s motivation: locus
of control, stability, and controllability (Weiner 1974).
Locus of control refers to the underlying causes of
life events. An individual whose locus of control is
internally oriented believes that controllable decisions
and efforts guide behavior, while an external orientation
suggests that behavior is guided by fate, luck, or other
uncontrollable external factors (including race, gender,
and socioeconomic level). Students are most motivated
when they believe that success or failure results from
their own (controllable) behavior rather than external
(uncontrollable) circumstances.
Finally, Learning Theory (LT) explores how complex processes
and environments affect international learning. According
to Rogers (2003), learning can be examined as a product
(change in behavior) and as a process (how and why behavior
changes). Bloom (1956) classified learning products
as cognitive (knowledge and intellectual skills), psychomotor
(physical movement, coordination, and the use of motor
skills), and affective (feelings, values, motivations,
and attitudes). These three domains are also known as
KSAs (knowledge, skills, and attitudes). Assessments
can apply these theories to determine the impact of
different learning processes in a range of learning
environments—curricular, cocurricular, and extracurricular—all
of which should be included in the assessment process
(Rubin, Bommer, and Baldwin 2002; Mahoney, Cairns, and
Farmer 2003).
Project Design
Guided by the missions and goals of their universities,
the six teams involved in ACE’s early qualitative
project developed nine common international learning
outcomes (and associated performance indicators and
scoring rubrics)—three outcomes each for the three
domains of knowledge, skills, and attitudes (ACE 2008).
(These KSAs are described in detail on the ACE Web site.)
To address the complexity of learning outcomes and yield
more accurate and comprehensive results, MSU chose to
adopt a mixed-methods research design (Greene, Caracelli,
and Graham 1989).
The quantitative instrument, the BEVI, is predicated
on EI Theory and “designed to understand whether,
how, and to what degree people are (or are likely to
be) ‘open’ to various transformational experiences”
(Shealy 2005, 99). Selected students take the pre-BEVI
(as freshmen) and post-BEVI (ideally as seniors) to
detect changes in international learning. The instrument
contains three validity scales (to ensure that the respondent
is answering in a consistent fashion across items) and
ten “process scales” (assessing, for example,
basic openness, receptivity to different cultures, tendency
to stereotype, self/emotional awareness), as well as
sixty-five demographic, situational, and background
variables (Shealy 2005, forthcoming). The instrument
is innovative, accessible (Web-based), affordable, and
has demonstrated reliability and stability in previous
applications. For more information on the BEVI, see
www.forumea.org/research-bevi.htm.
The qualitative assessment incorporates AT and LT theories
and relies on electronic portfolios to which students
submit at least five work pieces over a period of time.
Ideally, these artifacts represent a broad range of
work from different class levels and international learning
environments (curricular, cocurricular and extracurricular).
Examples include course papers or other written work
(in English or another language), photographs and digital
images with commentary, course presentations, audio
files containing music or recorded language skill demonstrations,
and reflective essays. These artifacts provide direct
evidence of changes in student attitudes and insight
into values, affective development, and students’
potential for growth. Collecting them compels the student
to select, interpret, and reflect on interconnected
global experiences (Cambridge 2001, 15; Palomba and
Banta 1999, 80-81). The e-portfolio provides an easily
accessible mechanism to gather and represent international
learning from a range of disciplines, learning environments,
and class levels (Palomba and Banta 1999, 96, 26). For
more information on the e-portfolio project, see www.acenet.edu.
Thus MSU’s conceptual model applies LT, AT,
and EI Theory across a variety of processes and environments
(see fig. 1). Preliminary data analysis suggests that
this mixed-methods approach provides a more holistic
and in-depth view of students’ development of
multicultural competency. We believe that the project’s
findings will inform MSU’s liberal learning outcomes,
help coordinate effective classroom approaches with
co- and extracurricular activities, and suggest other
events and activities that might assist in internationalizing
the student experience at MSU.
Figure
1: Conceptual Model |
|
Developed by Pysarchik & Yun, 2008 |
Conclusion
The MSU project is illustrating that a well-implemented
assessment protocol—linked to theory and using
both qualitative and quantitative methods—is a
powerful tool. While assessment requires resources and
careful planning, it yields information that is highly
beneficial to the institution and to the wider international
education community. Its thoughtful implementation allows
educators to help students develop the competencies
that are critical to their success in our complex and
interconnected world.
References
American Council on Education. 2008. Lessons learned
in assessment. www.acenet.edu/Content/NavigationMenu/ProgramsServices/cii/res/assess
/intl_learn_Outcomes.htm#lessons
Banta, T.W., J. P. Lund, K. E. Black, and F.W. Oblander.
1996. Assessment in practice: Putting principles
to work on college campuses. San Francisco: Jossey-Bass.
Bloom, B. S. 1956. Taxonomy of educational objectives,
Handbook I: The cognitive domain. New York: David
McKay Co. Inc.
Cambridge, B. L., ed. 2001. Electronic portfolios
as knowledge builders. Electronic Portfolios: Emerging
practices in student, faculty, and institutional learning.
Washington, DC: American Association for Higher Education,
1-11.
Deardorff, D.K. 2007. Principles of international
education assessment. IIENetworker (Spring
2007): 51-52.
Forum on Education Abroad. 2008. www.forumea.org/research-bevi.htm
Greene, J. C., V. J. Caracelli, and W. F. Graham.
1989. Toward a conceptual framework for mixed-method
evaluation design. Educational Evaluation and Policy
Analysis 11 (3): 255-274.
Mahoney, J. L., B. D. Cairns, and T. W. Farmer. 2003.
Promoting interpersonal competence and educational success
through extracurricular activity participation. Journal
of Educational Psychology 95 (2): 409-418.
Palomba, C. A. and T. W. Banta. 1999. Assessment
essentials: Planning, implementing, and improving assessment
in higher education. San Francisco: Jossey-Bass.
Rogers, A. 2003. What is the difference?A new
critique of adult learning and teaching. Leicester:
NIACE.
Rubin, R. S., W. H. Bommer, and T. T. Baldwin. 2002.
Using extracurricular activity as an indicator of interpersonal
skill: Prudent evaluation or recruiting malpractice?
Human Resource Management 41 (4): 441-454.
Shealy, C. N. 2004. A model and method for “making”
a combined-integrated psychologist: Equilintegration
(EI) Theory, and the Beliefs, Events, and Values Inventory
(BEVI). Journal of Clinical Psychology 60 (10):
1065-1090.
———. 2005. Justifying the justification
hypothesis: Scientific-humanism, Equilintegration (EI)
Theory, and the Beliefs, Events, and Values Inventory
(BEVI). Journal of Clinical Psychology 61 (1):
81-106.
———. Forthcoming. Making sense
of beliefs and values. New York: Springer Publishing.
Weiner, B. 1974. Achievement motivation and attribution
theory. Morristown, NJ: General Learning Press.
