Review No. 3

Moving right along, here's a review of a doctoral dissertation by Jill W. Fresen, Ph.D. from the University of Pretoria, South Africa on quality management in instructional design in tertiary education:

In contrast to Seddon’s (2000) position that preventing non-conformance in processes does not assure quality, Fresen (2005), like many, blends the concepts of quality and assumes that quality management and quality of product are synonymous. This, in spite of an introductory statement to the contrary, pervades most of her thinking process as revealed in Fresen’s data gathering techniques and conclusions. In her research, Fresen asks the following questions:

1.  What factors promote quality web supported learning?
2. What factors contribute to client satisfaction (or frustration) with web-supported learning?
3. What lessons were learned in applying standard quality assurance theory to the instructional design process for web-supported learning?

As project manager for the department at the University of Pretoria charged with oversight for all online learning for the University, Fresen used her access to this environment to engage in an exploratory study, based on a case study of the department in which she worked. The stated unit of analysis for Fresen’s study was the instructional design process (p. 78). To gather data, Fresen used a literature survey, case analysis meetings, a student survey with 4,650 respondents, 22 lecturer interviews, expert consultation, and task teaming. Since the University of Pretoria used WebCT as their Courseware Management System, Fresen also used the Student WebCT Experience questionnaire for the student survey. In addition to personal information, the student survey asked questions in five categories: technical adequacy and technical support; educational support; affective domain (feelings and emotions of students); interactivity (use of the communication tools in WebCT); and instructional adequacy (perceived learning). Note that only perceived learning was assessed. No data was presented in the study to indicate that learning increased as a result of the use of quality management in the instructional development process. The lecturer interviews were based upon the Lecturer Experience and Satisfaction interview schedule. It included both closed questions on four categories (including overall effectiveness of the WebCT course; achievement of student learning outcomes; WebCT staff training attended; and service provided by TLEI (the instructional development department) and the Academic Information Service) and open questions on another four categories (problems experienced; benefits experienced; impact of the e-learning product; and overall evaluation). Both instruments were designed by the researcher.

The literature survey was pertinent to all three research questions; the case analysis meetings related only to research question one, factors to promote quality web-supported learning. The student survey and lecturer interviews concerned research question two, client satisfaction. Expert consultation, relating to both research question two and research question three, was “sought, with respect to the summative evaluation of products in order to provide measurements to inform the quality improvement cycle” (p. 82). Task teaming impacted research question three, process-based quality management system. Again, this involved consulting with an “expert” in quality assurance.

The findings for research question one, factors to promote quality web-supported learning, produced a taxonomy of sixty-one factors in six categories (institutional factors, technology factors, lecturer factors, student factors, instructional design factors, and pedagogical factors). This was further distilled into forty-two factors arranged in the same six categories with one category (instructional design factors) subdivided into two sub-categories: usability and learning principles.

Research question two, client satisfaction, was assessed with the student survey and the lecturer interview. Fresen notes that while the lecturers are the direct clients of the instructional design department, the ultimate clients are the students themselves. In that light, Fresen summarizes the reports from the student survey, written to five categories, in two indices, a Frustration Index and a Satisfaction Index. The Data showed that 83% of students responding experienced “moderate to high levels of frustration” (p. 165) in their online learning. Only 43% experienced “high levels of satisfaction” (p. 165). Fresen expressed disappointment at the results but recognized direction and opportunity for improvement.
Research question three, process-based quality management system, revolved around the establishment of a quality management system (QMS) to govern the instructional design process. Ironically, Fresen chose to avoid external moderation, such as that found in ISO 9000 registration, and instead focused on “the human aspects of quality assurance, which emphasized training in quality assurance theory…” (p. 169). Still, congruent to Seddon’s position, Fresen adopted a QMS as an instrument “for managing the quality of instructional design processes and procedures for web-supported learning” (p. 169). Thus, as even proponents of ISO 9000 will admit, the focus is on the quality of the development process, not the product. It is still assumed that if the process is quality, the product will be also. However, nothing in Fresen’s research bears out this assumption.

References:

Frezen, J. W. (2005). Quality assurance practice in online web supported learning in higher education: An exploratory study (Unpublished doctoral dissertation). University of Pretoria, Pretoria, South Africa. (Retrieved from http://upetd.up.ac.za/thesis/available/etd-02172005-134301/).

Seddon, J. (2000). The case against ISO 9000. Dublin, Ireland: Oak Tree Press.