Learning Environments

Bransford, John D., Brown, Ann L., Cocking, Rodney R., (Eds.). (1999). The Design of Learning Environments in How People Learn: Brain, Mind, Experience, and School. pp. 119-142. Washington, DC: National Academy Press.

I expected more from this article by virtue of its title and its presence in a book on how people learn. While I appreciate the initial argument–that more is expected of schools now as education changes from a mass production model of sorting, assembly and quality control via standardized tests to one which emphasizes creative problem-solving–the guiding Venn diagram added no new perspective.

lenses of learning

lenses of learning

The lens of learner-centered instruction with its focus on culture is drawn from Vygotsky. The lens of knowledge-centered instruction with its focus on metacognition was more illuminating, particularly in linking this domain to the learner through each student’s preconceptions. Progressive formalization is straight out of Wiggins, although one notion resonated with the reading on networks:

a network of connections among objectives characterizes expertise

I also appreciated the argument activities must be balanced between those that promote understanding and those that promote automaticity.

The lens of assessment-centered instruction with its admonition on the value of feedback offered little new information, and in the section on alignment, the authors failed to explicitly draw the connections between the over-arching lens of community and the more focused lenses of the learner (the community defines values), the knowledge (the community defines what knowledge is valued), and the assessment (the community defines expertise). However, the passing reference to portfolios as formative assessments gave me an idea: in addition to being constructivist, portfolios could provide transparency and serve as proof for those parents and politicians who believe in the manufacturing model of standardized learning.

Advertisements

Evaluation

I’ve always thought there should be 3 types of evaluation:

  1. Formative to adjust instructional materials while there’s time
  2. Summative to make certain instructional outcomes are achieved
  3. Normative to provide feedback to learners as they progress.

The division of internal and external evaluations lent a practical framework for the topic even though the specifics had been thoroughly covered in the previous course:

  • Internal
    • Errors
    • Alignment
    • Usability (learner)
  • External
    • Expert
    • One on one
    • Pilot
    • Field

The section on data analysis and reporting could have been expanded with a discussion of delivery (just as activities and delivery were earlier paired); for example, objective test construction (used in CBT environments) and the development of group-based project rubrics(used in asynchronous higher education modes) would enhance the already useful coverage.

Outcomes and Assessments

While this chapter covered previous information on assessment, I appreciated the extension of Gagné’s work beyond the 9 events we covered in the previous course. The 4-step sequence on developing outcomes and assessments was particularly clear:

  1. Identify goals and outcomes
  2. Classify outcomes by type of learning
  3. Determine subskills
  4. Develop assessments for each outcome and subskill

I also appreciated the distinction between goals (the overall target) and outcomes (the details to translate those goals into measurable requirements).

The 3 methods for identifying subskills made sense, although instructional analysis seems most applicable to our work: task analysis is best-suited for procedures and motor skills (although the practical suggestion to have experts “think aloud” while you observe is applicable for any analysis); content analysis, described as best-suited for declarative knowledge, seems content-centric instead of learner-centric.

The real value of this chapter for me was in the overview of Gagné’s classification of learning outcomes:

Learning Type Knowledge Activity Assessment
Verbal Declarative Link to prior learning Objective test
Intellectual Procedural Examples & non-examples Solve problems
Motor Motor skill Practice Demonstration
Attitudes Affective Role models Frequency of new attitude
Cognitive strategies Metacognitive Coaching Reflection (community assessment?)

The fascinating (to me) aspect was the idea that in intellectual outcomes, each subcategory was a prequisite for the next. This might align Gagne with Bloom.

The deconstruction of objectives into 3 parts was helpful:

  1. Behavior to be exhibited
  2. Conditions under which the behavior must occur
  3. Criteria for acceptable performance

However, I liked the addition of Audience as a precursor.

The table of performance verbs was helpful, although I prefer the table that ties verbs to Bloom’s taxonomy (maybe because I prefer more formulaic prescriptions). What I really liked about this section was the list of how objectives can help learners:

  • Activating prior knowledge
  • Allowing learners to adopt or reject as personal goals (never thought of that aspect)
  • Set up a cognitive organizational scheme
  • Provide cues on what to pay attention to

The assessment section covered areas previously discussed; however, I wish I’d understood the distinction between “assessment as a measure of learning progress” and “assessment as a measure of instructional effectiveness” last semester. I also appreciated the concrete suggestion to develop scoring guides (checklists, rubrics, scales) by assembling a collection of actual student responses and then synthesizing the range.

The assessment tasks at each phase was helpful:

  • Define – what learners are able to do
  • Design – description of assessment plan
  • Demonstrate – assessment prototype
  • Develop -fully-developed assessment (seems to impact validity)
  • Deliver – presentation of assessment (seems to impact reliability)

I also liked the relationships to the ASC cycle:

  • Outcomes
    • Ask client to describe goals
    • Synthesize content into goals and subskills
    • Check outcomes with client
  • Assessment
    • Ask client to describe tasks that indicate learners have met goals
    • Synthesize these tasks into assessment measures
    • Confirm that these measures accurately assess the target goals

Peer feedback

My interest in this article was articulated in the first paragraph: a pragmatic interest in reducing faculty load while maintaining an emphasis on complex assessment. However, the pedagogical reason (peer assessment “resembles professional practice”)  is an additional benefit I had not previously considered and made me study the results in detail. I found myself particularly interested in the proposition that peer feedback may “result more often in the revision” than face to face feedback; the condition that peer assessment must be organized “in order to produce feedback of sufficient quality” may provide the bass to convince faculty of the value of this approach.

The authors mention that peer feedback provides learning in both the providing and the receiving, but focus on the receiving aspect. And while peer assessment is “in the realm of collaborative learning,” it is more limited than other forms and thus collaborative techniques are not emphasized in the article. Instead, the authors concentrate on the successful uptake of the feedback which they define as both the understanding of and the subsequent use of the feedback.

The message codification by two researchers indicated an agreement of 98.3% (80% was mentioned as a threshold, a percentage I was unaware of), indicating accurate coding. The research looked at feedback in four functional areas:

  1. analysis
  2. evaluation
  3. explanation
  4. revision

with three subject aspects:

  1. content
  2. structure
  3. style

Receptivity to feedback was measured in importance and agreement, and the use of the feedback was measured though document revision monitoring (a unique use of anti-plagiarism software).

The results from a health care education courses which used discussion boards as the feedback were useful:

  • The more that feedback included revision recommendations, the more revision occurred, especially in content and style.
  • The more that feedback was deemed important, the more revision occurred, especially in content and style.

The results from a science education courses which used an annotation tool as the feedback mechanism were even more revealing; however, the results are difficult to isolate because two variables (the course, as well as the feedback tool) were changed:

  • The more that feedback included analysis, evaluation OR revision recommendations, the more revision occurred (again in content and style).
  • The more that feedback was deemed useful, the greater the agreement; the greater the agreement, the more revision.

As a result, the research is somewhat flawed as these are essentially two separate studies; in fact, a third study is embedded: the authors contrasted the two tools and found that the annotation tool produced less evaluation suggestions but more revision suggestions.A subsequent analysis, however, revealed a potential flaw in the annotation tool: it produced a much higher incidence of comments that were solely compliments (and thus the feedback was received positively but provided little value or potential for revision); upon reflection, this makes sense because the annotation tool affords the reviewer the opportunity to comment more often as he or she proceeds through the document.Thus, annotation tools may elicit more revision but provide less criticism (and thus promote lower quality) than a more holistic discussion board tool; this suggests the need for using both tools in peer assessment.

Of particular importance to me were the findings on how the feedback was used:

  • Importance of feedback increased revision
  • Usefulness of feedback did NOT increase revision
  • Even without an easy method for doing so, students chose to interact (respond to the feedback).
  • Concrete revision suggestions increased revision.

Formative evaluation (of the design)

OK–this chapter in Dick and Carey caught me off guard. I was expecting a discussion of how to design formative assessments to provide regular feedback (self-checks) to the learner and instead discovered that formative assessment is to provide feedback to the designer for the purpose of revision. After I got over the surprise, the types of design feedback were straightforward and made a lot of practical sense (with the caveat that conducting all 5 types is impractical without a large learner population warranting a LONG development schedule ).

Here’s the complete  formative assessment plan; Dick and Carey augment this outline with a series of very useful(and generic) instruments and outlines, especially Table 10.3 (which summarizes the types) and the examples and surveys at the end of the chapter.

  1. Specialist evaluation (SME, learning specialist, learner specialist)
  2. Clinical evaluation (one-to-one)
    • Select 3 representatives of target population and at least one above average, average, and below average in task ability.
    • Clarity criteria
      • Message
      • Links (contexts, examples)
      • Procedures
    • Impact criteria
      • Attitude
      • Achievement
    • Feasibility criteria
      • Learner
      • Resources
    • Interactive nature of clinical
      • Establish rapport
      • Encourage learner to talk as she works through  material
      • Ask learner why he made a specific choice after each assessment step
  3. Small group evaluation
    • Select 8-20 representatives of target population from multiple sub-groups (ability, language, delivery familiarity, age, etc.).
    • Attitude survey
  4. Field trial evaluation
    • Select 30 representatives of target population.
    • Designer is observer only.
    • Attitude survey (of context)
  5. Performance (in-context) evaluation
    • Key questions:
      • Do learners find the the new skills appropriate in an authentic context?
      • What has been the impact on the organization?
      • What do learners (and the community) recommend for improving the instruction?
    • Allow time to pass after instruction so that new skills have a chance to be used.

The final type seems identical to Kirkpatrick’s 4th level of evaluation and the basis for (what little I know about) 3600 evaluation. At UTTC, because we typically deal with existing material (in some form), the caveat that formative evaluation of selected materials should proceed directly to field trial makes the process less daunting. I also appreciated the suggestion that delivering instruction with no formative evaluation can benefit by applying a field trial model. The best suggestion of all was almost buried in the final section on design disagreements: “Let’s have the learners tell us the answer.”

Criteria

I wish I’d read this chapter before I did my ILM conceptual segment–now I’ll have to go back and revise it again. I wasn’t expecting Wiggins to help much with “hard side” issues like test validity but this chapter proved me wrong. The idea that the criteria are the independent variables and define the task requirements (and thus the goals) clarified this relationship; the reminder that explicit goals actually define the criteria was even more useful. Seeing that an analytic rubric divides the product/performance into distinct traits led me to realize that my ILM rubric has to be organized by task, not by content (i.e., the tasks are NOT “economic/service/social” but “list/apply/personalize”). Viewing a rubric as a continuum–and perhaps even starting with samples and deriving the specific rubric entries from that body of varied work–was another practical suggestion. I think I would have benefited as much from non-examples (unsuccessful rubrics) as the anecdotes.

The attempt to tie the 6 facets with criteria was less successful for me, although the chart linking them was helpful. I wonder if every task contains every facet. I don’t think so (but does that imply a failure in the task to represent true understanding) because the Math example didn’t. I wonder if the 6 facets can be adapted to a specific task: if Accurate is the operative word instead of Explanation, is it acceptable to substitute. I think so. I loved the caution against equal weighting and averaging. The latter reminded me of my Freshman Comp class where I had a “D” going into the final paper but ended up with an “A” in the course since the instructor was clear that only your final paper demonstrated your ability, a lesson that completely ingrained in me the revision process. His grading scheme had a second effect–motivation through fear–but I’m not sure that lesson was as valuable.

The idea that validity is what we infer from test results  rather than the test itself was both puzzling and intuitive. I wonder if we can even construct a valid test because the test interpreter “interferes” with the results (sort of like the Heisenberg uncertainty principle where the mere presence of a measuring device screws up the results).  However, the two questions posed in the diorama example helped:

  1. Can a student do well on the assessment but not understand the big idea?
  2. Can a student do poorly on the assessment but understand the big idea?

The idea of using multiple tests (format as well as over time) to see a reliable pattern was much easier to comprehend and implement. [Note to self:  Sternberg’s, The Nature of Insight sounds like essential reading.] And the reminder to look at the links between Stages 1 and 2 in Chapter 7 reminded that I will need to revise my ILM from the ground up. This is a lot harder than it seemed six weeks ago.

Assessment versus activity

As Cesar (I think) mentioned in class, all the theories seem to turn behaviorist when it comes to assessment, and Wiggins is no exception. What I liked about the coverage in Wiggins was the flow: “results we desire” to “evidence we need” to “tasks we design.” The focus on desired results keeps assessment from reducing to mere activity. I also liked the analogy of assessment as scrapbook rather than snapshot although that seems to work better with summative assessment than formative.

The distinction between problems and exercises was less clear to me. While the characteristics of problems (few cues, various possible approaches, noisy environment, and search for an appropriate solution which can be justified) were clear, the parallel characteristics of exercises seemed one-dimensional. The GRASPS acronym for creating performance tasks (with problems being the goal of those tasks) was equally obvious, and the mnemonic was contrived. The connection with the six facets served as a helpful reminder (which I have listed below as a memory aid for me) but added nothing new to my knowledge:

Explanation – big idea in learner’s own words (to make connections, stimulate recall)
Interpretation – translate from one medium to another
Application – use knowledge in new (real or simulated) situations
Perspective – take a different point of view
Empathy – appreciate different points of view
Self-knowledge – self-assess

While this sounds like I didn’t gain much from this chapter, the continuum of assessments was superb. I annotated the graphic and include it here.

Wiggins' Assessment Continuum

Wiggins