Evaluation

I’ve always thought there should be 3 types of evaluation:

  1. Formative to adjust instructional materials while there’s time
  2. Summative to make certain instructional outcomes are achieved
  3. Normative to provide feedback to learners as they progress.

The division of internal and external evaluations lent a practical framework for the topic even though the specifics had been thoroughly covered in the previous course:

  • Internal
    • Errors
    • Alignment
    • Usability (learner)
  • External
    • Expert
    • One on one
    • Pilot
    • Field

The section on data analysis and reporting could have been expanded with a discussion of delivery (just as activities and delivery were earlier paired); for example, objective test construction (used in CBT environments) and the development of group-based project rubrics(used in asynchronous higher education modes) would enhance the already useful coverage.

Advertisements

Gagné’s Requirements

Gagné’s method for defining learning requirements is reminiscent of Wiggins because Gagné recommends working backward from the intended learning goal. However, Gagné’s approach of specifying internal and external conditions reflects his behavioral approach:

  • Identify the types of learning outcomes
  • Break down complex outcomes into a hierarchy of dependent outcomes and pre-requirements
  • Identify the internal conditions that have to occur to achieve the outcomes
  • Specify the external conditions that have to occur to achieve the internal conditions

Gagné’s classification scheme ties his five types of learning goals to five methods for demonstrating those learning outcomes:

  1. demonstrate intellectual skills by solving problems
  2. demonstrate cognitive strategies by using them to learn
  3. demonstrate verbal information by stating
  4. demonstrate motor skills by physical performance
  5. demonstrate attitudes by expressing preferences

While demonstration of the three latter goals is straightforward, the second goal seems intrinsically-bound to intellectual skills. Also, since the first goal is overly broad, Gagné proposed 5 types of intellectual skills:

  1. Discriminations – distinguishing
  2. Concrete concepts – identifying
  3. Defined concepts – classifying
  4. Rules – applying a single relationship to a class of problems
  5. Problem-solving – applying a new combination or rules to a complex problem

Outcomes and Assessments

While this chapter covered previous information on assessment, I appreciated the extension of Gagné’s work beyond the 9 events we covered in the previous course. The 4-step sequence on developing outcomes and assessments was particularly clear:

  1. Identify goals and outcomes
  2. Classify outcomes by type of learning
  3. Determine subskills
  4. Develop assessments for each outcome and subskill

I also appreciated the distinction between goals (the overall target) and outcomes (the details to translate those goals into measurable requirements).

The 3 methods for identifying subskills made sense, although instructional analysis seems most applicable to our work: task analysis is best-suited for procedures and motor skills (although the practical suggestion to have experts “think aloud” while you observe is applicable for any analysis); content analysis, described as best-suited for declarative knowledge, seems content-centric instead of learner-centric.

The real value of this chapter for me was in the overview of Gagné’s classification of learning outcomes:

Learning Type Knowledge Activity Assessment
Verbal Declarative Link to prior learning Objective test
Intellectual Procedural Examples & non-examples Solve problems
Motor Motor skill Practice Demonstration
Attitudes Affective Role models Frequency of new attitude
Cognitive strategies Metacognitive Coaching Reflection (community assessment?)

The fascinating (to me) aspect was the idea that in intellectual outcomes, each subcategory was a prequisite for the next. This might align Gagne with Bloom.

The deconstruction of objectives into 3 parts was helpful:

  1. Behavior to be exhibited
  2. Conditions under which the behavior must occur
  3. Criteria for acceptable performance

However, I liked the addition of Audience as a precursor.

The table of performance verbs was helpful, although I prefer the table that ties verbs to Bloom’s taxonomy (maybe because I prefer more formulaic prescriptions). What I really liked about this section was the list of how objectives can help learners:

  • Activating prior knowledge
  • Allowing learners to adopt or reject as personal goals (never thought of that aspect)
  • Set up a cognitive organizational scheme
  • Provide cues on what to pay attention to

The assessment section covered areas previously discussed; however, I wish I’d understood the distinction between “assessment as a measure of learning progress” and “assessment as a measure of instructional effectiveness” last semester. I also appreciated the concrete suggestion to develop scoring guides (checklists, rubrics, scales) by assembling a collection of actual student responses and then synthesizing the range.

The assessment tasks at each phase was helpful:

  • Define – what learners are able to do
  • Design – description of assessment plan
  • Demonstrate – assessment prototype
  • Develop -fully-developed assessment (seems to impact validity)
  • Deliver – presentation of assessment (seems to impact reliability)

I also liked the relationships to the ASC cycle:

  • Outcomes
    • Ask client to describe goals
    • Synthesize content into goals and subskills
    • Check outcomes with client
  • Assessment
    • Ask client to describe tasks that indicate learners have met goals
    • Synthesize these tasks into assessment measures
    • Confirm that these measures accurately assess the target goals