Putting AAC&U Scorer Training Online

In person scoring consists of two types of experiences:

1. Overviews in a ballroom
2. Calibration discussions (after scoring)

Overviews given in a ballroom can be easily converted to videos for online training

Converting calibration discussions for online training is more challenging

AAC&U has never captured the content of calibration discussions

So calibration discussions cannot yet be converted with fidelity

One proposal is to hold calibration discussions on the phone to train new 2017 scorers remotely

and record the calls

Then videos can be created that convey the content of calibration discussions with fidelity

That way new 2017 scorers get sufficiently trained

But training materials are also created for the future of VALUE scoring so that future trainings don't require AAC&U staff time for phone calls

This proposal suggests phone calls for groups of 10 scorers

Possible New Scorer Estimates

• WC 40 new scorers
• CT 40 new scorers
• ER 20 new scorers
• CE 20 new scorers
• QL 20 new scorers
• IKC 10 new scorers

Those numbers of new scorers would require about 40 calls
across 5 time zones

This requires a phone call schedule similar to the phone call schedule for rubric development teams

The month of April allows 120 time slots
of 1.5 hours each

That gives great
flexibility for scheduling
the 40 calls

Untitled Slide

• March: inventory and assign work samples to categories (pre-scoring, new scorers, veteran scorers, post scoring, support for the uncalibrated)
• March: obtain and format any extra work samples needed
• March: Recruit scorers (AAC&U staff would do this)
• March: Create videos for overviews. etc.

Untitled Slide

• April: online training for new scorers
• April: online refresh for veteran scorers

Untitled Slide

• May: Post-scoring
• May: Individualized support for the uncalibrated
• May: Create videos for 2018 online scorer training based on calibration phone calls

Untitled Slide

• June: Finish creating videos for 2018 online scorer training based on calibration phone calls
• June: Build an online training course in Digication for each rubric and each type of scorer

Products

• 15 t0 18 short, unique videos for each rubric
• 15 to 18 videos x 6 rubrics = collection of ninety short, unique videos
• Each video focuses on one line & offers 2 to 3 faculty opinions about the line & then reaches an AAC&U conclusion about the line
• Each video refers to a unique work sample

Products

• 6 training courses for new scorers (with unique videos and unique work samples)
• 6 refresh courses for veteran scorers (with unique videos and unique work samples)

Will scorers be willing to commit to 3 phone calls that are each 90 minutes long?

3 phone calls and not 4

Will scorers be willing to commit to also watching a few videos and scoring 4 work samples?

Requirements in PDF

Feedback on Friday?

wendebmg@gmail.com