Research Question

The question at the heart of my proposal flows from the concept of the problem statement. If the research problem statement, previously rendered, is that it is not known how or to what extent quality management principles impact effectiveness and efficiency when applied to instructional design efforts, the research question then becomes, "How and to what extent do quality management principles impact effectiveness and efficiency when applied to instructional design efforts?" However, let's refine this a little; it's way too broad. Instead of "quality management principles," which is all-encompassing, let's use "quality management practices," which is more precise. Furthermore, it should be a given that both quality management principles and practices are going to have a positive effect on efficiency; after all, that's what they're designed for. So, let's take out the efficiency from the equation. Thus, we are left with "How and to what extent do quality management practices impact effectiveness when applied to instructional design efforts?"

In terms of methodology, this question statement forms a mixed methodology approach. Both quantitative and qualitative measures would need to be taken. In order to separate the two methods, the question needs to be broken up as follows: 1. "How do quality management practices impact effectiveness when applied to instructional design efforts?" and 2. "To what extent do quality management practices impact effectiveness when applied to instructional design efforts?" The former requires application of qualitative research methods and the latter quantitative research methods.

Let's try to refine this even more so as to enable more precise analysis. Rephrasing the questions give us 1. "How does the application of quality management practices to instructional design efforts impact the effectiveness of the instructional outcome?" and 2. "To what extent does the application of quality management practices to instructional design efforts impact the effectiveness of the instructional outcome?" Either of these should provide sufficient data to effectively describe the relationship or to effectively quantify it.

The Problem, part 3

There is also a theoretical/philosophical element to my problem statement. There are some sources in education who believe applying quality management principles to instructional interventions is in part another symptom of the encroachment of the philosophies of scientism and naturism in the fabric of education. Loomis and Rodriguez (2009) posit this idea together with the move toward standardized testing, especially in American education. The idea flows from a belief that something is only real if it can be measured, and that measurement creates objectivity. However, the truth is that in deciding what must be measured, the foundation of the metric is built on inherent subjectivity, even if that decision is arrived at in concert by a group, since the members of the group will likely have similar if not homogeneous views.

There is always an economic element embedded in any instructional design effort. The question becomes, how important is that element compared to the other elements? Does the implementation of quality management principles focus on this element to the exclusion of other elements? Is it possible to design a process flow that increases efficiency to the point that it compromises the effectiveness of the effort?

Here's a short, humorous exchange I had with some of my supervisors regarding the issue. I commented that one could have the clearest most efficient process flows in place for production and still put poor instructional items out the door. The jocular response by the Quality Manager was "Yes, but we can do it more quickly and efficiently!" When I shared this with the Project Manager, he added "And we can do it cheaper!" While the exchange was admittedly made in jest, it points out common knowledge that the focus in quality management, much like the focus in the Project Management Book of Knowledge (PMBOK), is for production efficiency first. The product is never certified as being quality, only the process by which it is produced.

However, this aspect of the problem could end up being the Achilles' heel of this research. It cannot become such an important element that the research morphs into a position paper. It must be substantiated, ironically, with research data that will objectively analyze the "true" nature of the relationship between quality management and instructional design. That must remain my focus.

Caroline Crawford's Eternal, Synergistic Design Model

The Problem, part 2

While manufacturing production can benefit from quality management principles, there are some questions how much instructional design can benefit from the same. While it is true that production process analysis can streamline courseware development, control errors, and generally make it a more profitable enterprise, there does not appear to be a correlation with learning, at least on the surface. Two qualities are of paramount importance when building an instructional intervention: effectiveness and efficiency. Effectiveness deals with how well the student was able to learn the information and how well they were able to put it into practice. Efficiency deals primarily with how quickly the student was able to learn the information. Of course, from this perspective, we are talking about the end product. What quality management deals with, on the other hand, is the efficiency of the methodology one uses to get to the end product.


While manufacturing industries were among the first to implement quality management principles in their production, service industries are beginning to get on the bandwagon. Quality management principles and courses are even being taught in graduate business schools as part of the curriculum. Government agencies are strongly recommending that potential contractors be able to show that they can operate under quality management principles in order to substantiate their claims of a quality product. Rumors in the industry indicate this may become a mandatory requirement in the near future in order to be able to bid on a government contract.


Many proponents make claims that using quality management principles can improve instruction in many different forms, including eTraining, distance education, computer assisted instruction, and the like. However, there appears to be little of substance with respect to hard data. Mostly, one is either an avid proponent or an ardent detractor, based more on personal philosophy than metrics.


Which is another issue that goes along with the problem. Much of quality management revolves around metrics: the time it takes to build a deliverable item, the number of comments against the item from internal and external reviews, the number and types of mistakes that have to be corrected per count item, etc. All of this is done in an effort to promote objectivity, and yet whenever one deals with data, one is always constrained by the choices made concerning what is important to count and how it shall be counted. Even deciding who will decide these things will have an impact on the final product.


The most common principle in instructional design is the ADDIE principle: Analysis, Design, Development, Implementation, and Evaluation. Every instructional design model has some or all of these elements in its structure in one way or another. However, by law with regard to federal government contracts, the company that does the front end analysis for the project is prohibited from doing the rest of it. Also, typically, government entities are reluctant to spend money in the contract on evaluation. Hence, the process begins to resemble the Crawford design model, The Eternal, Synergistic Design Model (see next post). It consists of a large ball of evaluation (which is really just internal and external reviews) surrounded by a large circle of design and development in the shape of an infinity symbol. The model depicts a common scenario where the curriculum is stipulated by the customer and the contractor begins "designing" the product according to the customer's wishes, then develops it (usually in a rapid prototyping tool) and performs internal quality reviews before shipping it off to the customer. The customer in turn performs a quality review and sends the product back for correction or further development in the cycle.


In this scenario, quality management is very useful, even necessary in order to support profitability. However, there is little data to support the effectiveness of the product in terms of student pass rate, high-miss test items, low miss test items, or even if the courseware is really teaching them what they need to know to be able to do their job. So while many claim that quality management is helpful in building strong training, I question the verity of their statements. Essentially, my problem statement sounds like this: "It is not known how or to what extent quality management principles impact effectiveness and efficiency when applied to instructional design efforts."

The Problem, part 1

All research has to start with a problem. Here's how I came up with mine (near as I can remember). In my Comprehensive Exam, the first question referenced a Dr. Zane Berge of the University of Maryland, Baltimore Campus, with regard to the types of research done on instructional design. He noted that about 70% of research in ID is descriptive in nature, i.e., they send out a survey (or some other instrument), gather the data, and then do a qualitative study on the results. Less than 10% of research is done on each of the following three types: correlative, experimental, and case study.

I needed to see the articles that referenced this information, but the link on the website for the online periodical where it was located was broken, and none of my library database sources referenced it. However, I did a Google search for Dr. Berge and found his curriculum vitae, including his address, phone number, and email address. I selected email and sent him one on a Friday night, asking if he could direct me to the location of a copy of the articles. Instead, he sent me pre-publication copies that he had at home (he wasn't able to get to his office right then) which I was able to use.

The articles were from 1999, but I wanted to see if he had anything more current than that. I did a library search and found an article from the Turkish Online Journal of Distance Education (that's his primary field) in 2008 that he collaborated on with Rachel D. Echard, an employee with the US Air Force, titled "Quality Management Builds Solid eTraining." (TOJDE, 9(1), July 2008). The title seemed intriguing, especially since my employer has completed ISO 9001:2000 registration within the last two years and our partner company is currently in preparation for the ISO 9001:2008 audit. I work as an Instructional Designer for a defense contractor; we develop and implement CAI (Computer Assisted Instruction) and ICW (Interactive Courseware) lessons for the US Marine Corps for training pilots, aircrew, and mechanics on the V-22 Osprey Tiltrotor aircraft.

My concern is this: ISO 9000 standards, Six Sigma, and Deming's Total Quality Management (TQM) are all manufacturing production models designed to cut down the cost of production, improve the quality of the product (in terms of defects), and increase profitability of the business. As defense contractors, we are engaged in building and selling a product: training. However, the effect of the training is supposed to be learning. Yet, everyone who is involved in instructional design seems to be getting on the quality management bandwagon, in my opinion in rather willy-nilly fashion without stopping to analyze the true impact on instructional design principles. My next post will address this more specifically.

First Post

This is my new blog for my dissertation on my Ph.D. in Education from Capella University, specializing in Instructional Design for Online Learning. I'm going to use this as a sounding board, for brainstorming, and for note taking as I progress. At this point, my committee is in place and I have chosen a topic. I am researching the literature to develop my Pre-Proposal followed by my proposal. Comments are always welcome.