Quiet Please: Indicators at Work


Association for Institutional Research

Orlando, Florida

May, 1997


R. Bowman, Co-ordinator, Government Relations
K. Snowdon, Associate Vice-principal (planning)

Queen's University 
Kingston, Ontario, Canada





Contents

Introduction		p. 1

Purpose		p. 1

Factors Influencing the 
Development of Indicators		p. 2

The Role of Indicators in the 
Planning and Policy Process		p. 4

The Role of Indicators in 
Developing an Accountability Framework		p. 6

The Role of Indicators in Government Relations		p. 7
	Use of indicators for government briefings
	Use of indicator material to inform political staff
	Distribution of indicator material to elected members

Implications for Institutional Research		p. 9

Summary		p.10

References		p.11




Quiet Please: Indicators at Work1

Increasing interest in measures of accountability must be seen as an opportunity for post-secondary institutions to improve and strengthen the relationship with public sector and private sector constituents.  This paper focuses on the policy and planning framework for the use of such indicators in a specific institution and points toward the use of indicators as one of the vehicles to address the issues surrounding accountability.  The use of indicators to 'tell the story' to government is one aspect of a broader initiative to improve government relations.  Implications for institutional research are explored with an emphasis on improving organizational effectiveness. 

Introduction
The term "performance indicators" is now an integral part of the higher education lexicon.  Yet within the higher education sector there are divergent views about the utility of and rationale for such indicators.  As a potentially blunt resource allocation instrument of government, indicators are viewed as an external threat impinging on institutional autonomy and regarded, at best, as superficial information and, at worst, dangerously misleading.  Viewed in another light, however, the interest in performance indicators provides an opportunity for institutions to strengthen existing management processes, deal directly with increasing calls for public accountability, and broaden the understanding about the work of the university.  

Purpose
The purpose of this paper is fourfold: 
	¡	to highlight some of the common factors influencing the development 			of indicators;
	¡	to focus attention on the role of indicators in a particular institution;
	¡	to highlight the use of indicators to address accountability concerns with 			a special emphasis on government relations; and  
	¡	to explore the changing role of institutional research with particular 			emphasis on improving organizational effectiveness.

Driven primarily by external pressure for indicators - as part of a cry for greater accountability - universities have responded in a variety of primarily defensive ways, appearing to have lost sight of the fact that indicators are a normal part of the evaluation process and are already evident in many of the key evaluation processes within many post-secondary institutions.  What has been lacking is an articulation of how indicators 'fit' into the management of institutions through the policy and planning process, their place in addressing accountability issues, and their role in the necessary communication strategy to 'tell the institutional story.'  Institutional researchers and planners have a key role to play in the development and utilization of indicators to accomplish those goals. 

But, it is not easy.  Using indicators to help evaluate institutional policies or practices requires significant effort and commitment and a willingness to probe into many aspects of a particular issue.  Using indicators to help address accountability issues requires a concerted effort to address accountability from an institutional perspective and will require the co-ordination and efforts of individuals from various parts of the institution.

Factors Influencing the Development of Indicators
Over the past decade, interest in performance indicators has exploded.  Various state reviews in the United States and provincial reviews of higher education in Canada have called for the implementation of indicators and magazine "rankings" of institutions have simply heightened interest in the area.  In Canada, proposals to use indicators in some fashion to allocate resources have been discussed in at least two provinces (Alberta and Nova Scotia) and Alberta's Key Performance Indicator (KPI) initiative is well underway.2

Indicators have been the subject of intense debate in a host of countries and, in some cases, have been incorporated into funding mechanisms as governments strive for greater return on the public investment.  In other jurisdictions, indicators are increasingly tied directly to accountability concerns.  There are a number of common threads in various jurisdictions that tie the experiences together.  Interest in greater accountability is one.  Improving outcomes is another.  Nadeau (1992) outlined twenty-eight potential uses of indicators.  Overviews of experiences with indicators in a number of countries provide interesting and useful perspectives on the major forces influencing the development of indicators and also point towards the differing emphases for the use of such indicators (Kells 1990; Cave et al 1991; Nedwek and Neal 1993; Davis 1996).  

Ultimately, however, one of the key factors influencing public interest in indicators is the university community's apparent inability, or unwillingness, to develop and communicate what are regarded as effective accountability mechanisms of its own.

************************************************************************
The painful truth for the university world is that we have invited any inappropriate and harmful use of performance indicators and any damage so inflicted.  We have failed to build, individually or collectively, multi-faceted systems which would enable us both to give information to others about our effectiveness and our efficiency and to improve our programmes and services. 
Kells, H.R., The Inadequacy of Performance Indicators for Higher Education-The Need For A More Comprehensive and Development Construct, Higher Education Management, November 1990, Vol.2 No.3, p.259.
************************************************************************

In a report on accountability in Ontario universities, a government Task Force "broadly representative of the university community"3 concluded that:

	although the present system (of accountability) is basically sound, 	improvements are required.  In addition, there is a major perceptual 	problem, primarily the result of significant deficiencies in 	communications.4 (emphasis added)

Having stated the system is basically sound, but noting there exists a 'major perceptual problem,' the task force makes 47 recommendations aimed at addressing, for the most part, a problem that may well be more perception than reality - largely because Ontario's universities (and higher education generally) have not done a very good job of keeping the 'public' informed about developments in higher education. 

In the most recent Ontario budget (May, 1997) reference is made to the planned introduction of the Public Sector Accountability Act and explicit reference has been made to a requirement to have publicly funded organizations "develop and communicate measurable performance indicators" as part of an effort to ensure that 

	all money being spent by publicly funded organizations is being wisely, 	prudently and efficiently spent for the intended purpose regardless of the 	source of the money.5

Increasingly, it appears that students are also becoming interested in accountability issues and they too argue for greater attention to performance indicators as part of an accountability framework.6  One would expect such interest to continue as students are required to pay an increasing proportion of higher education costs.

The shortcomings of indicators are well-known and criticism of indicators in higher education has become a discipline unto itself.  The challenge, however, is to harness the interest in indicators in a manner that will both strengthen the policy and planning process and address external accountability concerns at the institutional level.  The real value of 'indicators' is their use within an institution and given differences in mandate, mission, and resources, it is imperative that each institution develop its own set of indicators.

The Role of Indicators in the Planning and Policy Process
Interest in indicators can sharpen debate and discussion about institutional goals and may help spark a more comprehensive approach to university planning, budgeting, and human resource management.  Ultimately, performance indicators are but one part of what might be called a Planning and Policy Framework - a framework that must involve: 
	¡ 	the development of a mission statement; 
	¡ 	setting institutional goals;
	¡  	recognizing the key inputs (Faculty, Students, Facilities, 'Learning Tools')
	¡ 	developing policies/practices and strategies to 	
		encourage the achievement of goals (process); and 
	¡ 	the use of indicators and other measures for evaluation. 

There are several key ingredients that influence the quality of any institution.  The faculty, students, facilities, 'tools', and learning environment are all important factors.7  This "input" approach has long been a traditional way of developing notions of quality - with increasing attention now being paid to outputs.  However, while those factors play a key role in the development of an institution, by themselves they are not enough.  The policies and practices that affect those resource inputs are key parts of the quality and excellence process.  Indicators can play a role in evaluating the success of those policies or practices, with a view to improving performance.  Kells (1992) outlined a process for improving the use of indicators through the development of a more comprehensive evaluative model involving self-study and review and the use of indicators.8  

In the past decade or so, both the private and public sector have been taken with the goals of what is referred to as Total Quality Management or TQM.  One of the key principles in TQM is a commitment to Continual Improvement Process, that is recognizing there is always room for improvement and working towards continually improving the process or outcome.  That concept - and all that it embodies - is, in fact, alive and well in many universities - although it may not be recognized by the TQM jargon of Continual Improvement Process.  

If we focus on faculty for a moment, one can think of a number of measures that would provide some indication of faculty quality.  Publication record, highest degree, source of highest degree, teaching evaluations,9 rank progression, and scholarly awards are but a few of the indicators one might review to garner some notion of faculty quality.  However, those indicators, by themselves, are not sufficient to improve the quality of faculty.  If improving the quality of faculty is deemed an institutional priority (the goal), then there should be some definable way to measure progress.  The identification of the goal is the first step in the process for it leads to the examination of the factors influencing faculty quality (inputs, process).  Ultimately, in this example, that will lead to an examination or review of recruitment practices and reward systems with recommended changes if required.  The point is indicators (evaluation and feedback) are not an end in themselves, but simply act to encourage questioning about existing policies and practices - with a view to understanding the factors affecting changes in performance and consequently improving performance. 

One should not underestimate the amount of time and effort required to embark on a review of policies or practices that affect specific indicators.  Faced with daily crises, countless meetings, and basic operational issues, attempting to launch a review initiative can and will be a daunting task.  It requires executive support and a continual reinforcement of the importance of striving towards a particular goal. 

There is, of course, another important use of indicators that extends well beyond simple evaluation and feedback; that is improving the perception of the university.  Part of an accountability framework is only effective if it is satisfying the 'funders' and that requires efforts to ensure that government officials and elected representatives are apprised of the work of the university.

The Role of Indicators in Developing an Accountability Framework

	accountability can be a very strong mechanism for enhancing the position of post-secondary education in Canada as it competes with other public sector services for funds.
	To accomplish this goal, however, accountability must be viewed as an argument for, rather than a defense of, higher education: accountability must be thought of as a tool as opposed to a requirement. 10
  
How does an institution address external accountability concerns? What is the role of indicators in the process?  At Queen's University, both questions are being addressed through the development of an Accountability Framework with several components. 



The development of the framework involves the consideration of common formats, common presentation graphics, and reference to institutional goals and objectives.  The annual report provides a review of major activities, with an emphasis on people and programs and institutional mission.  This document is a key component in 'telling the Queen's story'.  The Financial Statements are accompanied by a Financial Review with key financial problems, pressure points, and trends highlighted.  The Report of Cyclical Reviews11 will present, in summary form, the results of departmental reviews and will include departmental indicators.  The Indicators Report provides a set of key indicators to help assess the progress towards key institutional goals while the Report on the Annual Budget emphasizes key planning assumptions and strategic decisions in the context of the annual budget.  Taken as a whole, the components of the Accountability Framework will provide the Board of Trustees with the requisite accountability information and provide an integrated approach to addressing accountability concerns.

The Role of Indicators in Government Relations
Since universities spend huge amounts of other people's money, donors and other friends as well as levels of local, provincial, and federal governments in Canada have become more interested in accountability issues.  They want to know where institutions are doing well in meeting their objectives and where they are not, and how they intend to move forward.

Universities have tended to view accountability as a willingness to assess the extent to which the job they have set for themselves has been achieved and the extent to which it has not.  And they have reported on these matters in various formal ways to all levels of government.  Taxpayers are interested in these inputs to inform them of the dividends of their investment in higher education and research.

While the more formal and required report information is critical to government bureaucracies in answering these questions, the more formal report formats are not normally convenient to elected members and political staff and to a wider public readership.

The indicators work quietly in Queen's accountability framework to meet this need in many ways, offering background and understanding to various governance groups inside the institution, to external government audiences, and groups and individuals in the wider public.  It is particularly helpful for introducing newly-elected members to university issues, activities, and goals.

Use of indicators for government briefings
Performance indicator information is used regularly to brief and update elected officials, alumni in government, members of political research staff, and public servants.  The information may be used as a starting point in group briefing sessions or one-to-one meetings with alumni who serve as elected members of government.  The materials are useful in updating alumni and benchmarking institutional activities broadly and comparatively.  In addition, aspects of the reports may be used to respond to a specific area of interest in the memory bank of graduates.

Especially in this latter area, indicator materials often lead to requests for further information and responses to the requests offer opportunities to improve and extend relationships.  And the information is helpful to use as a way of beginning relationships.  In one example, senior officials interested in accountability issues, were unaware the university was prepared to publish measurements of its performance in reader-friendly formats.  They were interested in the information and its circulation to members of their office staff and wished to be "put on the list" for updates.

Use of indicator material to inform political staff
Members of political staff are frequently called upon to deal with requests for information from constituents and to answer questions required in legislative debates.  Briefing them and offering comparative performance indicator materials for their use gives them an official source to help them set a context for responses on higher education issues.  Materials have been used in letter responses directly and quoted from in response to questions in debates.  Political party researchers have found the indicator material useful in providing a comparative context for the wider university system and the position of Queen's University within it.  And the indicators are forwarded to speech writers in their organizations, again as part of "informed" official background information.

Distribution of indicator material to elected members
Planned annual publication cycles make possible the flexible distribution of accountability framework material for government relations.  Materials not personally delivered are distributed by mail at quarterly intervals during the year, usually with a tabbed message from the Principal of the university.  A monthly one page update on university activities is faxed to key government people and random items such as the "Community Report" and the "Graduating Student Survey" are target mailed as they become available.

Implications for Institutional Research
The interest in indicators provides a challenge to institutional researchers and not simply because of the related data issues.  The emphasis here is on improving organizational effectiveness.  The development of indicators in any institution requires the effort of a number of individuals - generally from differing parts of the organization.  While institutional researchers have always played a 'data' role, the level of skills required to successfully integrate indicators into an overall framework for improving quality is increasing and changing.  Terenzini (1993) argued that there are basically three tiers of what he termed "organizational intelligence" that characterize institutional research: 

	1) technical and analytical intelligence, foundational;
	2) issues intelligence, focusing on organizational problems and decisions;
	3) contextual intelligence, an understanding of the organization and its 		culture.

As the complexity of the evaluative process increases, institutional researchers will find that their role will shift from simply suppliers or interpreters of data.  Co-ordination, tieing the pieces of the evaluative process together, will become an increasingly important part of the job.  In fact, in light of the increasing availability of data, it is important that institutional researchers recognize that to increase organizational effectiveness, the involvement of individuals in the indicators process must be broadened.  Building a sense of institutional ownership is a key part of the evaluative process and thus the more individuals with a vested interest in the success of the venture, the more likely the chance of success.  At the same time the broadening of involvement will improve the contextual intelligence of institutional research personnel.  The development of an integrated Accountability Framework simply reinforces the need for close working relationships with various parts of the organization including Finance, Public Relations, Government Relations, and Academic Units. 

The challenges facing institutional researchers have never been greater, but such a challenge presents an opportunity for institutional researchers to play a key role in helping their institutions address accountability concerns and, at the same time, improve planning efforts and strengthen working relationships with other parts of the institution.  

Summary
Interest in indicators provides an opportunity for institutions to take the initiative and address accountability issues in a manner that both reinforces internal planning (through the reinforcement of mission, goals, key inputs, and indicators) and provides an integrated approach to addressing concerns about accountability.  Members of the institutional research community have a key role to play in using this relatively new found interest to improve internal planning and accountability and improve organizational effectiveness through the development of better working relationships with other parts of the organization. 

Indicators have been the subject of countless discussions and reviews and this conference, Performance Indicators: Defining Measures that Matter, simply illustrates the point.  Those debates will no doubt continue and in the coming years we - as institutional researchers - will continue to refine the methodology and meaning of such indicators.  At the same time, however, it is important to put our institutional 'indicator' effort to work, quietly and effectively, to help improve institutional policy development and tell the institutional story.  


References

Bogue, E.G., and Saunders, R.L., The Evidence for Quality,  San Francisco: Jossey-Bass, 1992.

Cave, M., Hanney, S., and Kogan, M., The Use of Performance Indicators in Higher Education: A 	Critical Analysis of Developing Practice, (2nd ed.) London: Jessica Kingsley Publishers, 1991.

Davis, D., The Real World of Performance Indicators:  A review of their use in selected 	Commonwealth countries, London: Commonwealth Higher Education Management Service, 	1996. 

Elford I.C., Performance Indicators in Postsecondary Education in Alberta, Paper presentation, 	Annual Forum of the Association of Institutional Research, Albuquerque, New Mexico, 1996.

Freeland, R.M., Academia's Golden Age, New York: Oxford University Press, 1992.

Hubley L., Kipusi, P., McCulloch D., Russell A., and Ruszel. B., Performance Indicators for 	Universities in Nova Scotia, Student Report to the Nova Scotia Council on Higher Education, 	School of Public Administration, Dalhousie University, 1996. 

Kells, H.R., The Inadequacy of Performance Indicators for Higher Education - The Need for a More 	Comprehensive and Development Construct, Higher Education Management, November 1990, 	Vol. 2, No. 3.

Nadeau, G., The Use of Quality and Excellence Indicators in Post-secondary Education, CSSHE 	Professional File, Number 10, Fall 1992.

Nedwek, B.P., and Neal, J.E., Performance Indicators and Rational Management Tools: A 	Comparative Assessment of Projects in North America and Europe, Paper presentation, 33rd 	Annual Forum of the Association of Institutional Research, Chicago, May 1993.

Schmitz, C.C., Assessing the Validity of Higher Education Indicators, Journal 	of Higher Education, September/October 1993, Ohio State University Press.

Smith, S.L., Report of the Commission of Inquiry on Canadian University 	Education,  Ottawa: Association of Universities and Colleges of Canada, 1991.

Terenzini, P.T., On the Nature of Institutional Research and the Knowledge and Skills it Requires, 	Research in Higher Education, February, 1993 Volume 34, Number 1.

Whiteley, M., Porter, J., and Fenske, R., The Primer for Institutional Research, 	Tallahassee: Association for Institutional Research, 1992.

1.	Parts of this paper represent an update to The Use and Potential of Performance Indicators, presented by Ken Snowdon at the Halifax Conference of the Canadian Institutional Researchers and Planners Association, August, 1994. 

2.	Beginning in 1997/98, Alberta will incorporate performance indicators into the regular funding formula.

3.	Hon. Richard Allen, Minister of Colleges and Universities, letter to executive heads, Sept. 25, 1991.

4.	Report of the Task Force on University Accountability, University Accountability: A Strengthened Framework, May, 1993, p.20.

5.	Minister of Finance, 1997 Ontario Budget, Queen's Printer for Ontario, 1997. p. 42

6.	See Performance Indicators for Universities in Nova Scotia,  Student Report to the Nova Scotia Council on Higher Education, March 25, 1996. Also see Ontario Undergraduate Student Alliance, Improving Accountability at Canadian Universities, May 1996.

7.	For an interesting examination of how those key factors influence institutional quality, see R.M. Freeland, Academia's Golden Age, New York: Oxford University Press, 1992.

8.	See also G. Bogue and R.L. Saunders, The Evidence for Quality, Chapter nine "Promoting Campus Renewal Through Quality Assurance".

9.	Effective Fall/Winter 1994/95, implementation of Queen's Evaluation System for Teaching (QUEST) was initiated in an effort to improve existing evaluation practices. As of the end of the 1996/97 academic year, all Faculties are involved with the exception of the Faculty of Medicine. 

10.	J. Newberry, Accountability - Another Perspective, in Helping Enhance Canada's Future, a collection of papers from the First Canadian Institutional Research and Planning Conference, published by J. Lloyd-Jones, University of Ottawa, 1992. p.266.

11.	A formal process for Internal Academic Reviews has been established at Queen's with a view to having annual summaries presented to the Senate and Board.  As of the end of the 96/97 academic year, two reviews have been completed and the results will be made available in the Fall, 1997.  However, the considerable work involved has led to a re-assessment of the process and required data. 

______________________________________________________________________________________
Quiet Please: Indicators at Work