SWTng 12: Learning Object Evaluation Scale

     Article number twelve by Kay and Knaack (2009) is titled "Assessing Learning, Quality and Engagement in Learning Objects: The Learning Object Evaluation Scale for Students (LOES-S)," published in Educational Technology, Research and Development. With 92 references, the keywords include assess, evaluate, learning object, middle school, quality, scale, and secondary school. Here's the abstract.

Learning objects are interactive web-based tools that support the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners. Research on the impact, effectiveness, and usefulness of learning objects is limited, partially because comprehensive, theoretically based, reliable, and valid evaluation tools are scarce, particularly in the K-12 environment. The purpose of the following study was to investigate a Learning Object Evaluation Scale for Students (LOES-S) based on three key constructs gleaned from 10 years of learning object research: learning, quality or instructional design, and engagement. Tested on over 1100 middle and secondary school students, the data generated using the LOES-S showed acceptable internal reliability, face validity, construct validity, convergent validity and predictive validity.

     Learning objects are at the heart of the Sharable Content Object Reference Model (SCORM) criterion that is required for every training course used in the Department of Defense (DoD) under Advanced Distributed Learning (ADL). Although this study deals with middle and secondary school students, the principles of learning objects remain constant. Their application to andragogy vice pedagogy may require some adjustment for learning style. As summarized in the abstract, learning objects act on the cognitive processes of learners, often through visual aids such as 2D and 3D graphics, photos, and animations or videos. This is especially important when training a technical task. Similarly, visual learning objects can have great impact on the behavioral and affective domains, such as through demonstration of a procedure or depicting the potential results of a safety violation. Significantly, learning objects offer the capability of reuse in a variety of situations, minimizing redundancy and duplication of effort.
      This research is about the evaluation of learning objects, which takes place mostly in the design and development phases in a formative analysis. However, little research can be found that incorporates the user's input as part of a  summative analysis with regard to learning objects. Thus, most often they fall into the easy to use category. There are some repositories that use content experts to evaluate the quality of the objects after they have been developed, but "the number of evaluators is usually limited, the assessors have limited background in instructional design, and the end user does not enter the feedback loop in a significant way" (p. 148). Most evaluation has been done at the level of higher education. Little has been done in the K-12 arena, and the article does not deal at all with technical education incorporate or military training.
     The definition of  what is a learning object is key to this study. Original definitions focused on characteristics such as accessibility, adaptability, use of metadata, reusability, and standardization. Contemporary definitions emphasize qualities such as interaction and the degree to which the learner actively constructs knowledge. These technically based and learning-based definitions have been replaced for this study with a pedagogically based definition that is a composite of both. It includes interactivity, accessibility, a specific conceptual focus, reusability, meaningful scaffolding, and learning.
     Three aspects of each learning object were assessed in the study through student feedback: (1) how much they learned; (2) the quality of the learning object; and (3) how much they were engaged with the learning object. Students were asked in an open-ended format to comment on what they liked and disliked about the learning object. Total response included 1,922 comments which were categorized into the 3 main constructs and analyzed using a coding scheme. Each comment was also rated on a 5-point Likert scale. Two raters were used and their ratings compared; differences between their ratings were discussed and revised as necessary.
     Next, student performance was assessed based on their exposure to the learning objects using several different tests that surveyed learning at different levels. Finally, the teachers who selected the learning objects were surveyed with a similar instrument to that used with the students to determine their perspective on (1) how much their students learned; (2) the quality of the learning object; and (3) how much their students were engaged with the learning object. The results were not as reliable statistically as those for the student evaluation. However, the focus of the research was to investigate an approach for evaluating learning objects that related most to students.
     The key features of learning objects that were most supported by the responses to the study included interactivity, clear feedback, and graphics or animations that support learning. Design qualities most supported included effective help, clear instructions, transparency of use and organization. With reference to engagement, overall theme can impact positively or negatively on learning. There was also a low but significant correlation among student evaluations of learning, quality, and engagement and learning performance. Ultimately, however, "[l]earning objects are simply tools used in a complex educational environment where decisions on how to use these tools may have considerably more import than the actual tools themselves" (p. 161).

REFERENCES

Kay, R. H. & Knaack, L. (2009). Assessing learning, quality and engagement in learning objects: The Learning Object Evaluation Scale for Students (LOES-S). Educational Technology, Research and Development, 57(2), 147-168. DOI: 10.1007/s11423-008-9094-5

SWTng 8: Quality for new learning cultures

The next article is by Ulf Daniel Ehlers of the Institute for Computer Science and Business Information Systems at the University of Duisburg-Essen in Essen, Germany. It is titled "Web 2.0--e-learning 2.0--quality 2.0? Quality for new learning cultures," published in 2009 in Quality Assurance in Education. It has 33 references and the author supplies the keywords e-learning, quality management, quality, learning processes, and learning. Following is the abstract:

Purpose - The purpose of this paper is to analyse the changes taking place when learning moves from a transmissive learning model to a collaborative and reflective learning model and proposes consequences for quality development.

Design/methodology/approach - The paper summarises relevant research in the field of e-learning to outline the differences between e-learning 1.0 and e-learning 2.0 and amalgamates it with a series of previously published works. The characteristics of quality development are analyses in a next step and suitable methodologies for developing quality for e-learning 2.0 environments are selected, proposed and explained.

Findings - Even though the question of quality is controversially discussed already when e-learning 1.0 appeared on the market, e-learning 2.0 creates even more insecurity. This paper aims at answering the following questions: what constitutes the new, innovative element, which is described by Web 2.0 and e-learning 2.0? Does this development have consequences for how it assures, manage and develop quality in e-learning? In three steps, it is described what e-learning 2.0 constitutes, which basic elements of Web 2.0 it builds on, and what has changed. In a second, step the consequences this implies for quality development in e-learning are discussed. Third, a number of methods as examples and practical advice on how to further advance quality development are described.

Originality/value - The original value of the paper is to outline the changes which have to be taken into account in new and innovative learning environment which are build on Web 2.0 technologies and to draw consequences to quality development as well as suggest methodologies for educators and learners to improve quality of such learning environments.

 The focus of this article is e-learning and quality management. While I am no longer looking at quality management and instructional design, the article also deals with constructivist issues concerning e-learning 2.0. E-learning 2.0 is characterized by the authors with terms such as learner-centered, immersive learning, connected learning, game-based learning, workflow (informal) learning, and mobile learning. (p. 297). E-learning 2.0 is further characterized as a "personalized learning environment." However, this is not individually prescribed; the nature and scope of the learning goals are identified by the learner, not the instructor. While some of that may be feasible within the scope of my topic, the goal in my situation is to make the learner more capable of doing their job. Thus, only those tasks which apply to the job will be apropos to the curriculum. Nevertheless, the point is well taken that this is different than e-learning 1.0, which resembled an online textbook. This type of e-learning must be more flexible to the wants as well as the needs of the learner. They must be able to make it their own; hence, the characterization "personalized."

From e-learning 1.0 to e-learning 2.0 - Right at the outset what needs to be stated is that e-learning 2.0 is not a scientific term. It is not about further development, a new paradigm or a replacement in the sense of a new release. Strictly speaking it is not even about a new technology, a new model of learning or a new, separate, innovative variety of e-learning. E-learning 2.0 rather refers to a number of developments, trends and points of view, which require change from teaching to learning. The new point of view essentially connects e-learning with five characteristics:

1. Learning takes place always and everywhere (i.e., it is ubiquitous) and therefore in many different contexts, not only in the classroom.
2. Learners take on the role of organizers.
3. Learning is a lifelong process, has many episodes and is not (only) linked to educational institutions.
4. Learning takes place in communities of learning (so-called communities of practice, Wenger, 1998): learners participate in formal as well as informal communities.
5. Learning is informal and non-formal, takes place at home, at the work place and during leisure time and is no longer centered on teachers or institutions. (p. 297)

 In order to provide for quality in e-learning 2.0, the following transitions need to be made from 1.0:

• From reception to participation
• From inspection to reflection
• From product orientation through process orientation to performance and competence orientation
• From planning education for the learner to planning education by the learner
• From receiver to developer of learning materials
• From the "learning island" LMS to the internet as a learning environment
• From tests to performance

Next, the authors discuss concepts and methods of quality development for e-learning 2.0. Important aspects of methods for quality assessment include the following:

• Self-evaluation
• Quality assessment with e-portfolios (web-based portfolios)
• Social recommendation and community participation
• Evaluation processes aimed at a target group

The point the authors are making is that the method of identifying quality must change because the nature of learning has changed in e-learning 2.0.

REFERENCE

Ehlers, U. F. (2009). Web 2.0 - e-learning 2.0 - quality 2.0? Quality for new learning cultures. Quality Assurance in Education, 17(3) pp. 296-314. DOI: 10.1108/s09684880910970687