Savvy Assessment
and Evaluation Skills
Hazel Symonette, Ph. D
University of Wisconsin System Administration
Office of Multicultural Affairs
Seize assessment and evaluation as
your opportunity to speak for yourself-to
specify your vision and interpretation
of the "appropriate" facts:
- What services were delivered?
- To whom?
- What difference did it make with
regard to desired educational outcomes?
This is your opportunity to communicate
your data-grounded understandings of
how and why desired educational outcomes
do or, do not, occur for program participants.
As both a technical and a political
process, quality program evaluations
contribute to strategic image management.
They are worth the sacrifice of time
and effort for many reasons but especially
because poorly crafted evaluations are
typically presumed to represent poor/ineffective
programs. Strategic image management
is not a case for a public relations
propaganda document, which would simply
strain credibility. Rather, it is a
case for critically reflective, evidence-based
analyses of program strengths and weaknesses.
Effective assessment is an iterative
self-diagnostic process. It ideally
involves continuous cycles of program
improvement and refinement. So, each
program should map out the shape and
pacing of its own developmental trajectory--given
its configuration of resources, needs
and short-/long-term priorities. Over
time, that trajectory will probably
change as resources and priorities change.
Savvy program staff/administrators
need to be the first to know their assessment
and program development blind spots
and vulnerabilities, whatever the reasons--especially
before some outsider identifies and
capitalizes on them. They can, then,
more self-consciously and strategically
make decisions regarding improvement
initiatives, future priorities and tradeoffs.
Savvy program administrators and staff
should also constantly monitor the extent
to which their services are perceived
as important, if not indispensable,
resources by their clients and other
stakeholders. The bridge must reflect
a solid articulation between the target
population's agenda and the program's
agenda.
1. INCREASE SKILLS IN CONCRETELY
SPECIFYING YOUR PROGRAM MISSION AND
IMPLEMENTATION MODEL.
Program Goals and Objectives. What
student needs are you addressing and
what do you expect students to "look
like"/to learn as a result of going
through your program?
- Given your program expectations,
and thus implicit (if not explicit)
claims, what evidence would convince
a reasonable person (among key stakeholders)
that these claims are accurate?
- What are the appropriate outcome
indicators that best characterize
the level of progress towards your
program objectives?
Program Intervention Process. Through
what organized actions and practices-implementation
mechanism-did your program seek to
effect the expected changes?
- Are your program activities, and
the implicit causal model that they
represent, consistent with current
realities and the actual causal
model?
- To what extent has your program
served the total population in need
of its services? Eligible population
versus actual program participants
- How completely has your program
served the students receiving its
services? Services requested (needed)
versus Services received
Without some reasonably accurate
and coherent definition of the program's
operational identity, one does not
know to what to attribute the observed
outcomes or lack thereof .
2. INCREASE SKILLS IN CONCRETELY
SPECIFYING APPROPRIATE PROGRESS BENCHMARKS
AND IN COMPILING COMPELLING EVIDENCE.
Program-Relevant Outcomes. How do you
know if students have changed/are changing
in the ways that you intended and expected:
what tangible evidence exists that program
goals and objectives-the desired state
of affairs-have been attained?
- Specify appropriate comparison
baselines. To assess what program-engineered
changes occurred requires either
a) a within-group design, before
versus after intervention, or b)
a between-group design, e.g., exposed
to program (levels of exposure)
versus not exposed.
- Ensure that program data are compelling
given the expectations of the program's
primary stakeholders.
The level of implementation for individual
students (service delivery) and the
quality of measurement of that implementation
are crucial prerequisites for any
meaningful program monitoring and
outcomes assessment evaluation. One
cannot occur without the other. In
fact, one should not expect a program
to have its intended effect if it
has not been fully implemented for
the population of students eligible
for its services (target group). Furthermore,
one should not expect to identify
a program effect if program efforts
have not been reliably and validly
measured
CAUTION
A program or program component
may be erroneously rejected as not working
when, in fact,
3. CULTIVATE A CONTINUOUS QUALITY
IMPROVEMENT MINDSET.
Effective assessment is an iterative
self-diagnostic process. It ideally
involves continuous cycles of program
improvement and refinement. More specifically,
effective program development incorporates
a feedback loop for ongoing program
refinement. In Total Quality Management
terms, this process parallels Deming's
Plan-Do-Check-Act model and the continuous
quality improvement cycle. Reasoned
analyses and plans are followed by "experimental"
trials with continuous testing, learning
and program refinement from those planned
trials.
IMPROVEMENT INITIATIVES
AGENDA. Using assessment
as a critically-reflective, self-diagnostic
resource requires continuous cycles
of program refinement and improvement
to maximize educational benefits.
- Actual Program Improvement
Efforts. Identify changes
already made in the program to improve
the quality and effectiveness of
its operations, e.g., modes of service
delivery, staffing practices, policies,
procedures, etcetera.
- Future Improvement Plans.
Recommend and explain what needs
to be done to strengthen individual
campus programs as well as the overall
array of, for example, precollege,
recruitment and academic support
programs. Identify any anticipated
changes in program resources vis-a-vis
the current or projected needs of
the target population.
Embracing the spirit of iteration
and planned "experimentation"
is essential because there are so
many unknowns and uncertainties. There
is no "one-size-fits-all"
ideal academic support strategy that
works on all campuses for all students
of color or disadvantaged students.
The particulars of what works varies
over time, place and circumstances.
Without an iterative spirit and an
experimenting mindset, unknowns and
uncertainties may inhibit progressive
program developments. At worst, they
may paralyze efforts.
4. INCREASE SKILLS IN ESTABLISHING
THE PROGRAM'S LINKAGE TO THE GENERAL
CAMPUS MISSION.
To ensure the viability of its place,
each program/unit needs to link its
mission and agenda to the fundamental
values and principles of their university
community: most notably, excellence
in teaching and learning outcomes,
progress towards degree attainment,
career development progress and mastery
of competencies for ultimate success
in the "real world," etcetera.
In the absence of a program making
a case for the centrality of its activities
to the educational enterprise, program
efforts may be viewed as peripheral
or nonessential amenities that are
easily dispensable.
Some view academic support programs
and other student services activities
as making only a marginal contribution
to the university mission. Such views
are vocalized with ever greater frequency
and intensity during "budget-crunch"
times like today. Student precollege,
recruitment and retention/support
programs cultivate the college pipeline
and complement the university's basic
"teaching and learning"
mission. Without adequate numbers
of students entering and being effectively
prepared to live, work, develop and
thrive in the 21st century, the educational
enterprise would collapse.
5. INCREASE SKILLS IN SPECIFYING
THE ORGANIZATIONAL CONTENT WITHIN WHICH
PROGRAMS OPERATE.
In postsecondary institutions, few
if any programs/units can fully accomplish
their missions in isolation. Map out
the organizational context within which
your program operates--the network of
"suppliers" and others you
depend upon to effectively deliver your
services and products. Identify, in
a politically savvy way, those elements
and forces that support/reinforce your
program's efforts versus those that
undermine your efforts. Within your
university community, how would you
characterize the teaching/learning enterprise
and the student services network?
- Student-Centered versus Not?
- Coherent Student Services Network
versus Not?
- Collaborative Partnerships versus
Not?
6. INCREASE SKILLS IN MAKING
HARD CHOICES TO ACCOMMODATE SHRINKING
RESOURCES.
When absorbing budget and other resource
cutbacks, use great care in determining
and packaging the tangible consequences--notably,
the final configuration of services
and modes of delivery. Use special caution
to avoid sending the wrong message:
that the cutbacks were easily absorbed.
Skillfully document and highlight the
special efforts and accommodations made,
especially if services appear to remain
intact. More specifically, seize every
opportunity to highlight the consequences
of inadequate resources in order to
minimize programs getting "punished"
for their commitment to students and
their willingness to sacrifice much.
Otherwise, difficult consequences and
operating conditions will be discounted
and ultimately ignored.
Selective self-serving amnesia and
blindness may lead powers-that-be
to, at best, continue the resource
drought or, at worst, to further erode
already inadequate resources. Expose
even unintentional tendencies to institutionalize
hard times and untenable working conditions:
"You've done so much for so
long with so little, you're now expected
to do everything with nothing."
STRATEGICALLY BALANCE
- Commitment to serve immediate needs
- Commitment to send the right message:
one which increases future resource
prospects for initiatives whose institutional
value waxes and wanes.
May be reproduced or cited with proper
acknowledgement.
© Hazel Symonette, Ph.D. |
