WPC4 2BPZCourier 10cpi#|xmNx6X@8+X@IBM Proprinter X24EIBPROX24.PRSx  @ !+X@#|x2BVP ZF`Courier 10cpi?xxxxx6X@8+X@IBM Proprinter X24EIBPROX24.PRSx  @ !+X@22   ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  < ` `  hh#(-pp27  <1 "ASSESSMENT, OUTCOMES MEASUREMENT AND ATTRITION" {Reflections, Definitions and Delineations} One need not immerse oneself in the literature on college student assessment, outcomes studies and attrition research to any great degree to realize that these areas of inquiry are still very much in the process of being defined as legitimate avenues of educational research. Attrition research began in earnest about a quarter of a century ago, Astin (1975), Cope and Hannah (1975), Tinto (1975), Pascarella and Terenzini (1978), Johnson (1980), while efforts to develop models for assessment and student outcomes studies are more recent, Astin (1991 & 1993), Light (1992), Angelo and Cross (1993), Banta and Associates (1993), Ewell (1985 and 1988), and many others. Despite this volume of literature, there is still a good deal of ambiguity in the way the terms assessment, outcomes measurement, and attrition are defined for research purposes. For example, the National Association of Student Personnel Administrators (1995), recently published a booklet titled "Successful Student Outcomes Assessment." In this short work, NASPA included a partial inventory of available instruments currently in use for assessment and outcomes research. The title of NASPA's publication implies that the terms assessment and outcomes are, if not interchangeable, then at least h)0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <2 similar enough so as to require no further delineation regarding the precise meaning of these terms. This notion is further reinforced when one examines carefully the inventory of assessment/outcomes instruments NASPA selected for its partial inventory. That list included more than 27 instruments which addressed topics as diverse as scholastic aptitude, vocational interests, career paths, concepts of work, capacity to realize opportunities, extracurricular activities, critical thinking, judgement, confidence, sexual identity and a host of other subjects. The diversity of subject matter which these inventories purport to measure implies that regardless of its emphasis, all student research is, in effect, defacto assessment research. And yet not all researchers would agree. For example, Mentkowski (1994) recently observed that educators must somehow "connect assessment with teaching and learning." She views the assessment process as an altogether different exercise from outcomes research. Her view, which is shared by others, is that outcomes studies strictly speaking should address academic and intellectual growth, an emphasis manifestly different from assessment which is often used to imply data gathering. Despite the ambiguity in how these research areas are presently defined, there is little doubt that educational (0*0*0*Ԍ` `  hh#(-Assessment, Outcomes ` `  hh#(-pp273 administrators recognize the value of the assessment process largely because this research enables them to better understand the students they seek to educate. They also understand that data gathered under the rubric of student assessment and outcomes studies have the potential to mitigate student attrition by anticipating which students apriori have a demonstrable likelihood of failing to complete an academic program. In a larger context, when institutions establish carefully defined assessment programs, they are in effect demonstrating their interest in learning as much as they possibly can about their students and their students' needs. This in turn reassures legislatures which allocate funds in support of public higher education, governing boards which approve budgets for private institutions, and an increasingly skeptical public that institutions of higher learning are in fact trying to maximize the likelihood of graduation for each student. At a time when an investment in higher education is so closely correlated in the public's mind with the value of higher education, institutions are seeking to do everything possible to maintain and increase graduation rates thereby justifying the very considerable investment which the cost of a college education represents. Toward that end, the purpose of this reflection is three (0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <4 fold. First, observations are offered on how assessment, student outcomes studies and attrition research terminology might be more clearly defined and made more useful to consumers of this research. Second, the suggestion is made that these three areas of inquiry, each distinctive in and of itself, are nonetheless, highly interrelated phenomena. Finally, in operationalyzing definitions and outlining a recursive research model, this paper highlights how one small liberal arts university is currently thinking through the process of implementing a longitudinal assessment program including student outcomes studies and student attrition research. When administrators plan assessment programs, they must first distill the term assessment into its component parts. Broadly defined, assessment consists of three highly interrelated phenomena each of which has its own distinctive characteristics. The first of these components is assessment itself. According to the (1995) New American Heritage Dictionary of the English Language, the word assessment means to determine the value of something or to determine its significance or its extent. In higher education, this implies data gathering; specifically, gathering various kinds of descriptive data on students who comprise the institution's clientele. The term assessment therefore should be carefully (0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <5 defined to mean the collection or assembly of data, demographic, vocational, personal, for example. To employ the term in this way enables researchers to differentiate  assessment from other measurement activities. These data can then be used in a variety of ways, perhaps none more important than assisting institutions in learning who their students are as well as what their students' needs are. The first stage of the assessment exercise is to establish a baseline of data. These data sets should include information on 1) incoming freshmen, 2) sophomores and juniors who can be operationally defined as continuing students, 3) graduating seniors, 4) graduate students, 5) alumni whose graduation date was within 5 years of the date of the survey and finally, 6) alumni whose date of graduation was older than five years. To accomplish this phase of the research, one approach, (there are many others) is to use the National Center of Higher Education Management Systems (NCHEMS) Student Opinion and Information Survey (SOIS) as a survey vehicle. These instruments are specifically designed to collect information, not to measure academic outcomes. In addition, these surveys are easy to administer, clearly understood by the students who complete them and easily tabulated for analysis. "NCHEMS" survey instruments enables the researcher to establish comprehensive group profiles which reflect, among(0*0*0* ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <6 other things, students' experiences, expectations, aspirations, specific goals, and their evaluation of the institution and its services. In addition to telling institutions just who their students are, these surveys also summarize additional information having to do with the overall quality of the association between the students and the institution, something all academic and student affairs administrators want to know. In addition, these group profiles can also serve as a valuable foundation upon which strategic planning might be based. In short, the initial phase of the assessment process should be a relatively easy exercise of collecting extensive descriptive data on the institution's audiences. All other things being equal, it seems a perfectly reasonable way to begin the assessment process. In defining assessment in this way, institutions are able to differentiate this activity from student outcomes measurement and attrition research which constitutes separate but obviously related, collateral activities. To begin the second phase of research, institutions should identify a group of students on whom baseline data reflecting rudimentary academic skills can be established. Those students should be asked to complete a set of examinations which measures verbal and quantitative abilities. The collection of such information is of course commonplace at(0*0*0* ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <7 many institutions. As elsewhere, these baseline data enhance academic advising and program planning. But these data have an additional purpose. Students who participated in this phase of the study can then be tracked to determine whether or not they acquired the skills in English and mathematics that institutions seek to instill in their students. Pre and post designs are particularly appropriate for this purpose. During their first two years, these students then take a variety of typical freshman and sophomore English and mathematics courses. Pre and post test analysis, can determine whether a statistically significant difference could be identified between students' initial scores and their subsequent scores on alternate versions of, for example, the ASSET examination published by the American College Testing Service. If statistically significant differences for these dependent groups are found, institutions may conclude that the intervening course work did, in part, contribute to those differences. Such results however speak only to elementary English and mathematics skills, not to the broader issue of higher order, relational and abstract thinking. As long as the variables used in outcomes measurement are very narrowly defined and easily quantified, it is not a particularly difficult matter to test and retest groups of students to determine whether or not academic objectives are being met. (0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <8 Scores on standardized math and English tests are a case in point. More substantial outcomes measures however, specifically those which test for the acquisition of critical thinking skills and complex problem solving, are much more difficult to identify. In order to determine whether or not certain higher order academic skills have been acquired by students, institutions must first determine exactly what they are attempting to impart to students through the process of education. Most faculty would agree that basic skills in English and math are desirable outcomes of the education process. But beyond this basic agreement, faculty are often quite polarized regarding their opinions on what characterizes an educated person. Variations of the often repeated themes of thinking with clarity, achieving depth and breadth in a field of knowledge, acquiring the capacity to grapple with moral and ethical problems, distilling complex problems into their component parts or learning to appreciate the process whereby knowledge itself is acquired continue to be discussed as desirable traits for educated persons to possess. And herein lies one of the central problems with academic outcomes studies. Not only must each institution decide what it wishes to impart to its students, it must also devise ways to determine whether those objectives have been met. It is a widely accepted truism that the more sophisticated (0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <9 intellectual behavior becomes, the more difficult it is to quantify and measure. It is equally true that the primary reason higher order reasoning skills are often not part of student outcomes research is that university faculties simply cannot agree on what should be taught and how such skills, whatever they are perceived to be, should be measured. In discussions at the University of Great Falls which focused on how one could determine whether or not graduates were reflections of what faculty wanted them to be, it became clear that our perceptions of what constituted an educated person were quite disparate. We also agreed that if left unaltered, our various notions of what an educated person should be would preclude any long term meaningful student outcomes research. If we could not even agree on what belongs in the curriculum, how then could we begin to think through an outcomes process? Responding to this impasse, we began a twoyear long dialogue on reconfiguring the core components of our undergraduate curriculum to bring about some common agreement on what comprised a relevant and rigorous course of undergraduate study. Only by establishing common goals were we able to reach some consensus on what our graduates should know. And only then could we undertake a meaningful dialogue on how to begin to structure measurement instruments to tell us if the ( 0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <10 goals we had set for students were being realized. Student outcomes research therefore has the potential to tell an institution not only a great deal about its students' intellectual achievements but also a great deal about the faculty who educate them. This then, is the real value of student outcomes research and what sets it apart from simple assessment. Outcomes measurement which gauges real intellectual depth and breadth is only possible when faculty share a common vision regarding what intellectual characteristics are desirable and how those skills might be measured appropriately. The third component of the assessment triptych addresses student attrition research, the goal of which is twofold. The first task is to develop predictor equations which identify students who fit a "high risk" profile. The second goal is dictated by the results of the first. It entails the design and implementation of programs to decrease the likelihood that a particular student will become an attrition statistic. How is the first goal to be accomplished? Each institution should develop its own set of predictor equations using assessment and outcomes data on students who have graduated compared to those who have not. Multivariate statistics, especially multiple regression and discriminant analysis, are praticularly useful in this area of research. ( 0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <11xxA F` `  hh#(- Once predictor models have been developed, assessment and outcomes data can then be used to predict into which category, (persisters, voluntary withdrawers, or academic dismissals) any given student's profile suggests he or she is likely to fall. Demographic variables, prematriculation academic variables and other measures can be used to isolate the best predictors of subsequent academic standing. Once students are identified as high risk, institutions can intervene with tutorial programs, counseling, remedial work and the like to attempt to mitigate what initially might seem to be an inevitable outcome. In summary, the three activities commonly referred to as assessment, outcomes measurement and attrition prediction are in reality separate but highly interrelated phenomena. None of the three can be meaningfully understood in isolation or without reference to the other two. As administrators and researchers develop and continue to develop assessment programs, it will be helpful to differentiate these research activities into three specific areas so that each component is clearly defined and its relationship to the other two can be made clear. The assessment process as defined here should be the initial exercise. Periodic outcomes measurement can then be used in conjunction with assessment data to develop and refine predictor equations which can identify at risk ( 0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <12 students. However, in order for the process to serve the institution and its students, institutions must plan their research so that it is consistent with the overall educational goals as determined by its faculty. Outcomes measurement research methodology must be drawn directly from those goals and aspirations. Models for assessment, for outcomes measurement and for attrition prediction must be developed by individual campuses for localized use. These models must be constructed using clearly defined, operational definitions of assessment and related terms. Failures to provide such definitions almost certainly guarantees continued obfuscation and ambiguity in this area of research which is so important to institutions of higher education. ` `  hh#(-pp2 ( 0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <13 h# References  American College Testing Program. (1994). Asset Technical Manual, Iowa City: ACT Publications. American Heritage Dictionary of the English Language. (1992). Boston: Houghton Mifflin, Co. Angelo, Thomas and Cross, K. Patricia. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. 2nd Ed. San Francisco: JosseyBass. Astin, Alexander W. (1974). Measuring the Outcomes of Higher Education. In H. Brown (ed.), Evaluating Institutions for Accountability. San Francisco: JosseyBass. _____________ (1991). Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education. New York: MacMillan. _____________ (1993). What Matters In College? San Francisco: JosseyBass. ( 0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <14 Banta, T. and Associates. (1993). Making A Difference. San Francisco: JosseyBass. Cope, Robert and Hannah, William. (1975). Revolving College Doors: The Causes and Consequences of Dropping Out and Transferring. New York: John Wiley and Sons. Ewell, Peter T., Ed. (1985). Assessing Educational Outcomes. New Directions for Educational Research. No. 47. San Francisco: JosseyBass. ____________ (1988). Outcomes assessment and academic improvements: In search of usable knowledge. In J. Smart (ed.) Higher Education: Handbook of Theory and Research. Vol. 4. New York: Agathon Press. Johnson, A. E., Jr. and Hutchinson, J. (1980). Identifying persisters, voluntary withdrawers and academic dropouts at a liberal arts college. Journal of the National Association of Student Personnel Administrators, 18. 4145. Light, R. J. (1992). The Harvard Assessment Seminars. First Report. Cambridge: Harvard University Press.(0*0*0*Ԍ` `  hh#(-pp2Assessment, Outcomes ` `  hh#(-pp27  <15 Light, R. J. (1992). The Harvard Seminars. Second Report. Cambridge: Harvard University Press. Mentkowski, Marcia. (1994). How assessment practitioners who are educational researchers can contribute to assessment in higher education. Assessment Update 6 No. 1. Pascarella, E.T. and Terenzini, P.T. (1978). The relations of students' precollege characteristics and freshman year experiences to voluntary attrition. Research in Higher Education. 9. 347366. Successful Student Outcomes Studies. (1995). National Association of Student Personal Administrators. Washington, D.C. Tinto, Vincent. (1995). Dropouts from higher education: a theoretical synthesis of recent research. Review of Educational Research. 45. 89125.