Quality in Higher Education

Most quality professionals would agree the measurement of quality begins with an understanding of the mission of an organization and is followed by an examination of specific processes that provide the results that enable the organization to realize its mission.

The field of higher education poses some interesting problems when it comes to discussing the quality of higher educational institutions and their processes and outcomes because the function of higher education responds to so many different missions. Government officials, employers, accrediting agencies, university administrators, alumni, institutional researchers, faculty, and faculty development specialists all have something to share concerning this topic.

There are at least seven potential missions that higher education institutions attempt to meet. And, depending on how you think about the education sector, you might conclude that there are even more. This section of the web site will explore questions about seven different higher education missions and will consider the implications for defining, measuring, improving, and sustaining quality based on these seven different missions.

  1. Are we providing a commodity?
    The increasing reliance on adjunct faculty, along with online education delivery leads one to suspect that academic degrees are a commodity. Each degree may be seen as a formula. X hours in science + Y hours in language + Z hours in social sciences = a college degree. The measurements of quality might relate to cycle time, throughput, and failure rates. The quality of teaching becomes a matter of conformance to the structure of a “Learning Management System.”

  2. Are we producing research and new products?
    What gets measured is what gets done, and at many universities the primary activity that gets measured is research productivity. How many scholarly articles and in what journals? How many citations? How many patents? How much money can be produced for the institution based on licenses for new technology? Are these the institutional measures of quality?

  3. Are we providing access?
    Is the outcome of higher education simply a question of admissions? Recent cheating in admissions to elite universities suggest that it is all about admissions, maintaining one’s social position or achieving upward social mobility. This may extend to being accepted into the right clubs, fraternities, and sororities that will give a student access to a network of alumni. Attending the “best” schools may be a springboard to career access, networking, and success, regardless of what a student’s grade point average might have been. Perhaps we should measure quality based on the popularity and acclaim of alumni.

  4. Are we providing entertainment?  
    While there may be a strong anti-intellectual streak among some Americans, be careful about criticizing the resources and energy devoted to college athletics. From major universities down to struggling private colleges, the students, alumni, donors, and surrounding communities are devoted to athletics. It is big business. Perhaps the quality indicators most important to higher education deal with wins and losses in the sporting arenas, and the grandeur of those arenas and locker rooms.

  5. Are we providing workers for industry and other sectors?
    Politicians and parents clammer for graduates who can get jobs to support themselves and meet the workforce needs of local and national businesses.  The healthcare industry has an ever increasing demand for qualified medical personnel, social workers, and therapists. Perhaps the most important quality indicator is the number of graduates in job-ready disciplines to meet economic demands.

  6. Are we in the business of providing certain types of values?  
    Some argue that education is the cornerstone for democracy. Some schools are promoting themselves as advocates for conservative values while others see themselves as vehicles for consciousness raising curriculum. Other institutions openly state that part of their mission is to provide education within the framework and doctrines of a specific religious group. Perhaps the quality measures should be based on the number of graduates an institution has in the government, a church, or in jail.

  7. Are we promoting a love of culture and knowledge?
    Is it all about art for arts’ sake, enjoying great literature, appreciating and performing music, dance, and theatre? Is the mission of higher education to contribute to the quality of life and to help graduates experience a life well lived? Perhaps the quality measures should be based the elegance of performance venues, the number and value of scholarships for arts and humanities.

Having served on accreditation teams for numerous universities, it can be observed that all of these missions have been used by colleges and universities to make the argument that they are institutions of high quality.  This being the case, we will explore the avenues open for quality professionals to contribute to the conversation about quality in higher education based on the quality body of knowledge as it is applied to other sectors.

What Do We Mean by Quality?

To engage in meaningful conversation, we must first recognize what we mean when referring to quality. It is common for people to speak at cross-purposes on this subject when they are using different frameworks for defining quality.  The lack of a common definition for the term “quality” in higher education summons up a suspicion that the entire higher education sector is wandering about when it comes to the most fundamental issues concerning their enterprise.

As my colleague Dr. Gregory Watson has observed, broadly speaking, there are five popular ways to frame the issue of quality in most sectors.  I would maintain that these can be applied to higher education, as follows:

•Quality as endurance. European auto makers often portray the endurance of their automobiles as the primary sign of quality. The same can be said in higher education. If an institution has been around for over a century, we might equate that endurance with quality. An institution with only a few decades under its belt may be viewed as a newcomer, and if it only has a few years to its name, some people may suspect its ability to deliver quality. Endurance is used to suggest quality by many institutions in their tag lines.

•Quality as luxury and prestige. Objects that are luxurious or that have great beauty or prestige are often viewed as being of great quality. This view of quality is certainly seen in higher education, where institutions invest in beautiful garden-like campuses, stately buildings, luxurious suites in athletic stadiums, and every convenience that students from affluent backgrounds are accustomed to have at home. Quality as luxury extends to having the most up-to-date research facilities, light teaching loads for faculty, deep pockets to support sabbaticals, and investment in scholarships to attract the most promising new students and push up rankings that imply prestige.

  • Quality as conformance to requirements.
    This approach reduces quality to a set of specified attributes or characteristics that should be met. Most approaches to accreditation are based on this framework. The accrediting body specifies a set of requirements that a college, university, or specific academic program is required to meet, and then reviews performance to see if there is conformance to the requirements. Requirements can be established for learning outcomes, support services, financial well-being, library resources, and even for demonstrating effective planning, assessment, and improvement. In authorizing accrediting agencies, the U.S. Department of Education establishes quality requirements that the accrediting agencies are likewise expected to meet in terms of how they accredit educational institutions.

  • Quality as continuous improvement
    Although American and Japanese quality leaders accepted the need for conformance to requirements, they also broadened their framework by focusing on the reduction of variation in repeatable processes that would lead to continuous improvement and encouraged innovation through applications of new technology. This concept has found its way into higher education among those who think that defined requirements can never keep pace with organizational learning and technology, so that quality should mean achieving the fastest rate of innovation and improvement in all aspects of an institution. From this perspective, conformance to requirements means that an institution passes muster based on quality expectations that may already be out of date.

  • Quality as value-added
    As service organizations began contemplating quality in the 1980s, a perspective emerged that a process, such as education, should add value to the consumer or society. In education this perspective suggests that students should know more after they complete an academic program than before they started. Completion of a college degree should mean some measurable improvement in student learning, social skills, social contacts, writing skills, reading skills, critical thinking, or other attributes that might be consistent with the mission of an institution, such as the ability to dance, speak another language, or plan how to construct a building.

It is easy to see how conversations about quality in higher education can become nonproductive quickly if people approach the topic from fundamentally different frameworks. It is like having a conversation about the quality of a bridge, where one person thinks it has quality because it has stood for 200 years; another thinks it has quality because of the way it graces the skyline; another defines its quality based on the conformance to requirements for the grade of steel that was used; another thinks its quality is defined by the improved construction methods used when it was built; and another perceives its quality based on how it has improved overall flow of traffic in the region. If a participant in a conversation about quality in higher education refuses to consider all of these perspectives, then the conversation becomes an argument about positions rather than an opportunity for dialogue and learning.

Dangerous Measures

Issues surrounding measurement further complicate the concept of quality in higher education. A general truism, “what gets measured is what gets done,” demonstrates that measures play an important role in the world of quality. However, what people propose to measure in higher education is grounded in their definition of quality, so there are widely divergent views about measurement, and how organizations propose to conduct measurement activities can create many difficulties. It is not hard to measure the enduring age of an institution. Beauty contests are also possible—using professional evaluators and/or the satisfaction ratings of students, faculty, alumni, politicians, and others. Accrediting bodies have demonstrated that it is possible to establish all manner of measures related to the conformance to quality requirements. It is likewise possible to develop measurement systems to assess improvement and innovation, and, if society is willing to invest the resources, to assess value added.

Is it possible, however, to establish a uniform set of quality measures that will work throughout the entire higher education community? At one level, the answer is probably yes, and at another level, the answer is probably no.

At the degree program level, it is possible to establish quality measures for use in any higher education setting, and that is exactly what specialized accreditation is all about. The very definition of a profession centers around a definable body of knowledge, making it possible for any profession to establish a set of common requirements that all higher educational institutions will meet in terms of student learning outcomes, methods of assessment and improvement, faculty credentials, and resources to support this specific aspect of higher education.1 Professions, such as engineering, nursing, and education, are increasingly relying on common examinations that students in any type of higher education institution should have the ability to pass to demonstrate his or her qualifications to practice in a profession, regardless of the broader mission of the institution.

It should be noted that it is the profession overall, that determines the body of knowledge. Faculty members teaching in professional degree programs are not independently deciding what they should teach, but are aligning their teaching with the body of knowledge to ensure their students are competent in their field. Degree programs are the common denominator in higher education, where it’s possible to generate and assess comparative data, even though the mission of institutions that offer these degrees may vary significantly.

At any level broader than degree programs, however, the attempt to establish a common set of quality indicators across institutions immediately runs into the problem when institutions have very different missions—both stated and unstated. Some exist to cater to the developmental needs of young people in the “traditional” 18 to 22 year-old age group from families in their region or in the nation. Some exist to meet the needs of young people whose economic situation required them to immediately enter the workforce from high school. Others focus on the educational needs of adult students. Some exist to prepare people to function in the world from the perspective of a specific religious point of view, while others exist to focus on the preparation of scientists, performing artists, or warriors. Some reject the notion that they are part of a supply chain preparing graduates for professions, but see the institution as a means to cultivate critical thinkers and life-long learners. This variety of missions among higher educational institutions is widely considered to be a strength of American higher education.

To further compound the issue, even higher educational institutions with similar missions within the same state may experience significant inequities in terms of funding formulas, policies regarding which academic programs will be offered where, and investments in facilities and faculty. Politics, prejudice, and the influence of alumni can certainly influence the higher education landscape and impact quality.

 When selecting measures great care is necessary. Some suggest using graduation rates as a comparative measure. Are graduation rates a valid measure of quality when well-funded institutions that select mainly students whose parents are college graduates and who come from well-funded high schools (who are most likely to achieve success in college) compete with institutions that are marginally funded and that serve students who come predominately from poorly-funded school systems, whose parents never attended college, and who must maintain a full-time job to stay in school?

Qualitative measures based on student’s opinions and self-reported behaviors that are thought to infer something about student learning may be useful internal measures, but are problematic as meaningful comparative measures due to the differing missions of institutions. Even questions about student satisfaction can be problematic. Students whose educational experience included free time to socialize, private living accommodations, unlimited meal allowances, and every possible opportunity for tutoring and academic support may be expected to indicate they would be more likely to select their institutions all over again, compared to those who have worked their way through school, taken night classes, and pinched pennies or gone in debt to earn their degree.

Nevertheless, while many measures are problematic for comparative purposes, they may still have great value for understanding longitudinal performance within a single institution and should by no means be discounted.

Measurements are like engines on a train. Whatever measurement engine is hooked up to an institution, it is going to start pulling that institution in a certain direction, so leaders should ensure it is a direction in which they really want to go. If external bodies, such as state legislatures, begin to dictate comparative measures, it is certain that they will set changes in motion that will have unanticipated and probably unwelcome adverse consequences.

Arguments Over Assessment

Further confusion awaits those who think quality in higher education should focus solely on student learning outcomes. Probably the best approach to take will combine components of the three competing perspectives listed below:

  • Summative assessment
    Measures taken at the end of the educational experience enable educators to have a global perspective of what the educational process produced. This approach uses direct measures of student learning through national examinations in professions, major field tests in disciplines, and tests such as the Measure of Academic Proficiency and Progress and the College Assessment of Academic Proficiency to provide summative data concerning student learning. These measures can provide useful understanding of student learning in both mastery of content (through major field tests) and performance in terms of skills such as reading and critical thinking. This approach also includes the use of some indirect measures, such as surveying employers to understand their perceptions about the new graduates’ abilities to function effectively in the workplace. Faculty can use these data, often through the auspices of a curriculum committee, to redesign an academic program or to identify specific deficiencies in how a subject within a professional discipline is being taught. An advantage of this approach to assessment is that it does produce data that can be used for comparative purposes across institutions, which may also be viewed by some as a potential disadvantage. And, if an institution had the resources and was willing to invest in longitudinal data collection among first-year students and seniors, it would be possible to provide data regarding the value added in terms of student learning.

  • Formative assessments
    In this approach, direct measures of student learning in specific classrooms are used. This approach requires faculty to not just teach a subject, but to engage in critical reflection on how they are teaching their subject by observing how well their students are learning.2 This view extends beyond the routine use of tests and exams, requiring faculty to develop rubrics to explicitly show how their formerly implicit evaluation of student performance happens. One advantage of this approach to assessment is that it enables faculty to quickly identify problems and make more immediate improvements. A disadvantage is that it does not provide data that can be used comparatively, since the observations are wedded to the individual faculty member. The formative assessment focus is faculty-centered, requiring faculty to invest in learning how to assess what they have decided to teach and can be used in settings where curriculum is designed to align with an established body of knowledge (such as in professional programs) and where faculty have complete autonomy to teach whatever they wish to teach in their class.

  • Assessments of activities that imply student learning
    These activities include the amount of time spent doing homework, the amount of time spent in service learning projects, the amount of time attending performing art events, and a host of other activities.3 While proponents of direct measures of student learning outcomes may be somewhat skeptical that time invested in these activities necessarily translates into student learning that can be measured, others make the case that research suggests that there can be a cause and effect relationship between student behaviors and learning outcomes.

Universities offering programs in the professions, such as business, engineering, nursing, and education, are finding that specialized accreditation reviews are increasingly advocating a combination of all three approaches to assessment. The emerging view is that a robust academic program includes summative assessment (using both direct and indirect measurement), active faculty engagement in formative assessment, as well as assessment of behaviors that are thought to promote student learning.

Common Quality Principles

The discussion of quality in any field, whether it is education, healthcare, manufacturing, government, or service industry, is both challenging and routine. Regardless of the field, there are some common principles identified in the past 60 years that are applicable to our understanding of quality and that can inform our decisions about measurement and action.

Systems Comprised of Interacting Parts

All organizations (including universities) are systems, made up of interacting parts. This realization led Kaoru Ishikawa, the Japanese quality expert, to develop the famous fishbone diagram that illustrates how parts of a system affect quality and also leads to the use of interrelationship diagrams to understand how one component of the system affects other components.4 The quality of student learning clearly depends on activities that are within the control of a faculty member and on components of the system outside the control of faculty, as well as attributes and abilities of the student who is also part of the system. Regional accreditation programs are increasingly expanding the scope of their reviews to give consideration to institutions as entire systems with unique missions, and attempt to use quality assurance principles to ensure that all major sub-systems in a college or university are functioning at an acceptable level.

Leadership as a Vital Component

The leader’s perspective about quality and the attention that he or she gives to this issue will drive everything that happens in the organization. Change in leadership is one of the most clearly understood factors in both the decline and improvement of quality, and in how an organization decides to focus on quality in all sectors.5 W. Edwards Deming noted that the organization’s leaders are responsible for 90% of what happens within the organizational system and there is no reason to discount this observation when it comes to quality in higher education.6 Leadership, in the context of higher education, includes presidents, provosts, chancellors, the board of trustees, and in the public setting includes state legislators and governors who make critical funding decisions that will impact quality, regardless of how it is defined. Many accrediting programs are relatively silent on the issue of leadership, simply requiring that the governing board is not too involved in operational issues and that the chief executive has a periodic performance review. No one seems to be concerned if people with no relevant academic credentials or experience are appointed to leadership roles.

A Systematic Approach

Organizations must have a systemic approach to assessing their environment, developing strategic plans, taking actions, and assessing their results. W. Edwards Deming identified this need and the education community has adapted it as the Plan, Do, Study, Act (PDSA) cycle.7 Accrediting organizations have come to expect institutions to have the ability to show a macro-level approach to assessment, planning, and improvement and to show how this cycle is actualized in both academic and non-academic parts of the organization.

Focusing on Stakeholders and Processes for Listening to Them

Higher education organizations must remain mindful of the needs and opinions of their stakeholders and should have systematic processes to define those stakeholders and to engage them so that their voices are heard in the macro-level assessment and planning cycle. This includes listening and understanding the perspectives of students, faculty, staff, parents, employers, and neighbors. Accrediting bodies tend to overlook this area in preparing the quality attributes that they will require institutions to demonstrate.

Data Collection

It’s important to make an investment in the collection of data that provides faculty and administrators with the feedback they need to understand their longitudinal performance and how their performance compares to similar institutions. Along with this, an institution needs to embrace the concept of knowledge management, understanding what knowledge is necessary to effectively execute all work processes that keep the institution going. Few accrediting bodies are looking seriously at this knowledge management component of the higher education system and there is no national consensus on the minimal requirements for data collection and dissemination of performance information throughout institutions, other than the requirements to report some data to a federal information system. Data warehousing, for example, is perhaps considered a best practice, but is generally not a requirement for accreditation.

Workforce Development

Faculty qualifications have been a long-standing quality assurance component among accrediting agencies. Newer challenges relate to proficiency with information technology among faculty and staff, faculty development in terms of teaching and assessment, and inclusion of adjunct faculty as a significant part of the workforce that can impact quality, which are issues that accrediting bodies are beginning to address.

Process Improvement

All activities in higher education, whether academic or non-academic, are processes to define, study, and improve. Improvements, however, happen project by project, and in no other way, requiring a conscious effort to define, study, and improve the system.8 Whether an institution engages in academic program reviews (beautifully defined as “academic quality work” by William Massy), or organizes a task force or utilizes a standing committee structure, leadership must make a conscious effort to encourage and support improvement.9 Leadership must set the expectation that processes will be studied, must empower faculty or staff with time and resources to analyze their processes, and must work to help implement proposed improvements. The effectiveness of these efforts can be enhanced by using well-established guidelines and methods such as flow charting, basic data collection, cause-and-effect diagrams, and well-established methods that promote innovation through creative thinking, such as synectics10 and TRIZ.11

Key Indicators

Based on its mission, each higher educational institution should have a set of key performance indicators that enables leaders, faculty, staff, students, and all other stakeholders to understand what the institution is achieving. The types of indicators selected will certainly depend on how the institution defines quality and should link to the desired outcomes of the strategic plan as well as indicators of operational activities that reflect the relative health of the organization.

Anyone who has spent time studying quality as a discipline will recognize that the quality characteristics defined here are aligned with the seven categories of the Malcolm Baldrige National Quality Award, which is the gold standard for understanding quality in all sectors.12 Several specialized and regional accrediting organizations have made significant improvements to their accreditation models by studying the Baldrige criteria.13

Every System Changes

Like any other system, higher education changes in response to internal and external conditions. Changes in technology have radically altered the manner in which faculty teach, conduct research, and even enter grades. The decline of agrarian and manufacturing economies and the growth of the knowledge economy have significantly advanced the emphasis on higher education’s role in preparing people for the professions and has set the stage for rudimentary comparative analysis of institutional performance. The growing conversation and increasing awareness of how we define quality, and how we adopt quality principles and methods, will likewise affect the higher education culture as more leaders and stakeholders recognize and embrace these principles. In the words of Brent Ruben, from Rutgers University: “How we think about excellence has fundamental implications for illuminating and reconciling differences in perspective and priority with the academy.”14

(1) Peter Jarvis, Professional Education Croom Helm, Ltd., 1983.

(2) Thomas A. Angelo and Patricia Cross, Classroom Assessment Techniques, Jossey-Bass, 1993.

(3) George Kuh, Jilian Kinzie, John H. Schuh, and Elizabeth J. Whitt, Student Success in College, Jossey-Bass, 2005.

(4) Kaoru Ishikawa, Guide to Quality Control, Japan Productivity Center, 1974.

(5) Joseph Juran, Juran on Leadership for Quality, Macmillan, 1989.

(6) W. Edwards Deming, Quality, Productivity, and Competitive Position, MIT Press, 1982.

(7) W. Edwards Deming, Out of the Crisis, MIT Press, 1986.

(8) Joseph Juran, Managerial Breakthrough, McGraw Hill, 1964.

(9) William E. Massy, Academic Quality Work, Anker Publishing, 2007.

(10) William J. Gordon, The Development of Creative Capacity, Harper and Row, 1961.

(11) Geinrich Altshuller, Creativity as an Exact Science, Gordon and Breach, 1984.

(12) Charles W. Sorensen, Julie A. Furst-Bowe, and Diane M. Moen, Quality and Performance Excellence in Higher Education: Baldrige On Campus, Anker Publishing, 2005.

(13) http://www.quality.nist.gov/

(14) Brent D. Ruben, Pursuing Excellence in Higher Education Jossey-Bass, 2004.