Starting with Assessment

Over the same decades, engineers invented and assembled a powerful suite of methods to assess, control, and improve quality in industry.  Dozens of practitioners and academics combined these methods with Shewhart and Deming’s control charts and other statistical methods to establish a body of knowledge, which is the foundation of the development of a profession.

Concurrently, the early practitioners who began to orchestrate the use of these concepts and methods also recognized that these methodologies influenced, and were influenced by, the social structure within organizations.  Autocratic leadership styles inhibited open assessment and stifled efforts to control, fix, improve, and innovate within companies.  The concept of systems theory, with the work of individuals such as Kurt Lewin, began to influence the understanding of quality.(6)   The discipline of organizational psychology and the quality discipline began to become interwoven in the 1970’s and many of Deming and Juran’s observations are more about social science than statistics and engineering.  This has not changed in the ensuing decades and strongly influences the concept of radical quality.

None of the people who created this quality body-of-knowledge (drawn from statistics, engineering, and the social sciences) viewed it as being ancillary, or merely  a helpful addition to organizations.  They all saw their work as vital at the root level of organizations, as a radical imperative for all leaders at all levels of all organizations.  And, it is at this root level – a repeated and  enduring lack of understanding and acceptance of the radical meaning of quality – that organizations continue to struggle.

Conceptual Challenges

The quality discipline has always wrestled with some fundamental questions that concern our interpretations and understanding of quality.  Is the goal to simply make a process or system fit for its intended use, or do we strive for excellence?  When we assess a process or system, what are our expectations of performance?  How do we define success?   And, if all processes and systems are constantly subject to the forces of change, is there any actual point of arriving at excellence?  If so, can a process or system stay at a level of excellence?  

These questions have generated a great deal of debate and some anger over the years as different individuals have offered differing opinions and have criticized the opinions offered by others.  However, it may be that these questions are moot.  They may constitute a false choice that causes us to take our eyes off of the real situation that Whitehead framed for us.

Let’s consider the Second Law of Thermodynamics – the concept of Entropy – which aligns quite well with Whitehead’s observations.  All systems lose energy and run down.  Entropy is a force constantly at work throughout the universe, which includes everything around us.   In a very real sense entropy causes all processes and systems to change.  The leaders in an organization are constantly aging and eventually they must all be replaced.  The equipment needs repair.  The experienced and reliable operator gets sick or retires and new people need to be trained.  The piping starts to leak.  The oil needs to be changed.  The batteries run down.  Everything changes.

One of the most visionary thinkers of the 20th century, Buckminster Fuller, suggested that humanity is an anti-entropic force in the universe.  We are constantly assessing and taking actions to ensure that we do not disappear as a species, or at least to make some of us more comfortable while we are in the process of disappearing.  (7)  There is no fore-ordained future, as Whitehead noted, but there is a clear challenge to each and every organization to strive to continue to exist.

In retrospect, one of the funniest things I have heard quality practitioners say over the past 45 years is that if we do our jobs well, we will work ourselves out of a job.  There won’t be anything left to improve in our organizations.  I laugh at those assertions now because we truly did not understand the nature of entropy and the nature of change in organizations.

The great British historian, Sir Arnold Toynbee, believed that if history seems to repeat itself in some ways it is because each generation will confront some type of challenge or crisis that must be addressed and that from the time of Thucydides to today, each generation must find the internal inspiration to succeed or it will fail.(8)    The fact that the challenges from one generation to the next often seem so similar, dealing with perennial issues, gives rise to the illusion that events are repeating themselves.

The question is not one of whether it is better to make incremental improvements or to strive for excellence, whatever those things may really mean.  The question is survival in constantly changing times.  Survival can only come about through a willingness to engage in unvarnished assessment of an organization’s situation through the rigorous use of the quality thinking and methods.

If this seems melodramatic, then perhaps your experience has not included work with processes and systems in which failure entails truly significant loss.  Nowhere has the thinking and actions associated with radical quality generated more serious attention than in our nuclear industry, where there have been decades of unsung success, punctuated by terrible failures.  From the design and fabrication of naval reactors to their noteworthy operations, the leadership of the nuclear NAVY has personified the principles of radical quality.  Much of this commitment to quality has been infused into the commercial nuclear industry, as well.  While most quality practitioners may not be familiar with Nuclear Quality Assurance – 1, developed by the American Society for Mechanical Engineers, most practitioners have benefited from this foundational document upon which most quality audit programs have been founded.  The same can be said from the principles of Conduct Of Operations and Root Cause Analysis that came out of the nuclear NAVY and have had significant impact on power production, manufacturing, and health care.  When failure in the design or operation of a facility can result in numerous deaths or the contamination of hundreds of square miles, quality becomes a matter of survival.

Assessment Principles

There are some underlying principles of assessment that bear further exploration.

First, assessment starts with making a comparison of the actual characteristics of a process or system against an agreed-upon standard that describes what should be happening.  Shewhart was specifically looking at specific characteristics that were in use to define quality in processes.  Surveillance and inspections are designed to look at very specific characteristics.  Testing focuses on specific characteristics, such as temperature, chemical composition, and thousands of other characteristics, always comparing against a known standard and always employing a process to ensure the measurement equipment is calibrated.

In conducting audits, the auditor knows ahead of time what the responses and observations should be.  When deviations appear, the auditor will continue to explore the situation by using questions that define the nature and scope of the deviation between what should be occurring and what actually is happening.  An effective audit program, such as NQA -1, will include questions that will ensure the reliability of the adherence to quality at a radical level.

In one sense, assessment is like the work a coach does in reviewing the game film after a competition.  As one famous coach noted, it is enjoyable to be at home on your sofa watching a ball game, but you only learn about the game from watching the game film.  The study of the game film enables strengths and weaknesses to be revealed, along with errors that need to be avoided and opportunities for improvement.

Second, while assessments may look at very specific characteristics of a process, assessments can also  examine the entire nature of an organization through methods such as the Malcolm Baldrige Criteria for Performance Excellence and international quality standards that  provide a yardstick against which to measure the efficacy of an organization’s systems and processes.  Unfortunately, only an extremely small fraction of organizations have leaders who embrace radical quality and who will employ broad assessment methods, such as the Baldrige Program.(9)  Most leaders either do not want to know the truth about their organization, or they know the truth and do not want it revealed.  Many are required to use quality standards, which can be effective in identifying areas in processes that need to be fixed or improved, but these process-focused standards do not always drive critical thinking about achieving excellence.

Third, assessment must embrace the recognition that often we do not know what we do not know.  We know that we know some things and sometimes we know that we do not know other things and we set out with purpose to learn them.  However, when we do not know something, and we do not know that we do not know it, an organization is at great risk.  So, the purpose of assessment, in addition to understanding how closely we conform to desired levels and standards of specific characteristics, and an understanding of the overall efficacy of an organization’s systems and processes, is to discover things we do not know that we do not know.

Fourth, employing only one method of assessment may not only provide an insufficient view of what is happening in the organization, it can also skew the behaviors in an organization.  If only one form of assessment is employed, then it may not provide all of the perspectives you need.  It is important to listen to customers, but if you only assess customer satisfaction and do not assess the views of suppliers, owners, employees, and other stakeholders, you can be missing important information.  Even worse, when an organization only uses one form of assessment, then everyone in the organization will learn how to maximize performance on that one dimension and allow other important components of the work system to lag.  After all, what gets measured is what gets done. 

Assessment Methods

There are many methods of assessment from the casual to the complicated and assessment can vary greatly in scope depending on the size of an organization you are leading.  On the side of informal assessment, you might employ simple questions, such as “Tell me what is going on.”  A variety of basic questions may be enough to enable a leader to sum up the key dynamics in an organization.  On a more complex but still informal level, a leader can engage people in using metaphors to describe the status of issues in an organization to determine what is working well and what needs to be changed.  In some cases, leaders can even use an art exercise, asking people to creatively draw a picture of what things are like in the organization.

However, organizations usually need a more structured and systematic approach to assessment, drawing upon a wide variety of methods that are suitable for different types of assessment needs.  Assessments can be focused on customer or broader stakeholder perspectives, outcomes, peer comparisons, work processes, groups, leadership, or they can be institutional in nature.

Customer or stakeholder assessments seek to understand what stakeholders think about the organization, the extent to which their needs and interests are being met, what they like and do not like about their relationship with the organization, and what their aspirational thoughts that the organization might seek to fulfill in some manners.  These are typically performed with surveys, focus groups, and interviews.

Outcome assessments are examining the results of a product or service and can be quantitative and qualitative in their design.  Inspection of products, testing and grading of materials, testing and grading of students based on student learning outcomes, patient re-admissions, scores on standardized tests, and changes in physical characteristics, such as blood pressure, are all outcome assessments.  These assessments help to determine if products or services are fit for their intended use.

Comparative assessments examine how an organization performs in comparison to its peer institutions.  This could include a review of how peer institutions are performing with stakeholder assessments and outcome assessments.  Comparative assessments provide insights into areas where attention needs to be given to fix processes, make improvements, or innovate.  Common comparative assessment methods include conducting surveys of multiple, but similar, organizations, and conducting organized benchmarking exercises that focus on finding opportunities for learning from others who are doing something better in order to honorably adopt their methods.

Process assessments examine how work is actually being performed in an organization.  This can include direct observation of how classes are taught, inspection of materials at different stages in a work process, review of statistical data that informs us whether a process is operating within statistical control limits, and small or major audits that provide examine evidence regarding of how tasks are performed.

Group assessments include formal meetings where a group or team of people use a structured process to discuss their performance and/or their aspirations for the organization.  Group assessments can be structured around “team-building” activities in which people learn about their communication or personality styles and how those styles relate to everyone else in the group.  A structured process such as Appreciate Inquiry can be used to facilitate a conversation that assesses the status of the organization.(10)   The typical SWOT Analysis, used in strategic planning, is a form of group process assessment.

Leadership assessments examine the behaviors and results of leaders in an organization.  There are a wide variety of approaches to leadership assessment and most Human Resources organizations include some formal approach that is conducted on some periodic basis.  The key to the effectiveness of leadership assessments, from the perspective of radical quality, is to ensure that the yardstick against which leaders are being measured reflects the desired thinking and behaviors that support an emphasis on quality.  If the metrics for leadership assessment are wrong, then the organization is in trouble.

Institutional Assessments are structured systems for examining the entirety of an organization, including its leadership, processes, approaches to planning, outcomes, and stakeholder perspectives.  The Malcolm Baldrige Award Criteria is one example of a comprehensive system for assessing an entire organization.  Quality standards, such as the ISO 9001 standard, also provide a yardstick against which to measure a wide range of activities in an organization.  In education and health care, organizations, peer groups will develop standards for accreditation that provide assessment of multiple dimensions of an organization.

(1) Alfred North Whitehead. Process and Reality. (Corrected Edition) Free Press, 1978.

(2) Walter A. Shewhart. Economic Control of Quality of Manufactured Product. Van Nostrand, 1931.

(3) Whitehead.

(4) Joseph Juran. Juran on Quality Improvement Workbook. Juran Institute, 1982

(5) W. Edwards Deming. Quality, Productivity, and Competitive Position. MIT Press, 1982.

(6) Kurt Lewin. Field Theory In Social Sciences. Edited by Dorwin Cartwright, Harper, 1951.

(7) R. Buckminster Fuller. Utopia or Oblivion. Bantom Books, 1969.

(8) Arnold Toynbee. A Study of History. Oxford University Press, 1946.

(9) Denis Leonard and Mac McGuire. The Executive Guide to Understanding and Implementing the Baldrige Criteria. Quality Management Division, ASQ Quality Press, 2007.

(10) Jeanie Cockell and Joan McArthur-Blair. Appreciative Inquiry in Higher Education. Jossey-Bass, 2012.