The subject matter for the
Shootout was simply to teach a learner how to add a person's
name to the address book inside Microsoft Outlook. Although
most organizations wouldn't likely create something so generic,
we felt it was a good representative sample, allowing for
a variety of simulation inputs, such as mouse clicking, pull-down
menus, typing text, key press, etc. Also, we didn't want to
require each of the 10 participating teams to have to learn
a new software application in order to understand the subject
matter.
Each team was given a very crude storyboard (if you can even
call it that). Basically, they were given a Word document
outlining the steps to be taught, including opening the address
book, creating a new record, adding the data, attaching a
picture, and saving the new contact information. The teams
were given the storyboard before the event and were allowed
to practice creating their own simulation. Even though they
were allowed to practice, the rules specifically indicated
that they were not allowed to bring any pre-created content
with them to be used in the competition.
The entire simulation had to be created from scratch, including
taking screen captures, simulating dialog boxes and menus, defining
interactions, laying out feedback, adding instruction prompts,
and even adding audio narration. Several teams added audio
to their simulation, which all had to be recorded on-site
(although it was a bit noisy in the Expo hall). The only piece
of media they were allowed to pre-create was up to a 30-second
video clip on the subject matter (to demonstrate their ability
to incorporate rich media in simulations) – nothing
else!
If you would like to download a copy of the basic simulation
storyboard to see what they had to work with, or to create
your own version as a benchmark, please click here.
(Note: This will allow you to download the file in .zip format.)
Each team had a large screen behind them so the audience
could watch their every move as they created the simulation.
There were a total of 10 teams in the competition, with rounds
occurring each of the three days of the conference –
three teams took the challenge on Monday, four teams on Tuesday,
with the final three teams taking their turn on Wednesday.
Each team had a total of 20 minutes to create their entire
simulation.
Upon completing the 20-minute production time, each team
provided a seven-minute debrief of what they were able to
produce. During the debrief, each team was asked to use the
following checklist to report on their activities:
Debrief checklist:
Were you able to finish the entire simulation during the
20 minutes? If not, what percentage was completed?
What output format did you create? (HTML/JavaScript, Flash,
proprietary, etc.)
What is the size (total of all file sizes)
for the simulation? With
media? Without
media?
Run the simulation you created in any Web
browser and demonstrate the following:
- Prompts and instructions for learner
- Navigation controls (beyond following the steps)
- Feedback for incorrect answers
- Unexpected feedback (e.g., What happens if the learner
just clicks somewhere on the screen that wasn't anticipated
by the simulation developer?)
- Remediation options when "they don't seem to get it"
- Guidance beyond feedback (e.g., hints, demo mode, etc.)
- Demonstrate that you were able to create an interaction
with multiple input possibilities (referee will verify)
- How do learners see their own scores? Is there a review
mode for remediation?
- Demonstrate or discuss the use of media in your simulation
- Show any unique and differentiating aspect of the simulation
output
Demonstrate your simulation development process, showing how
you created the target simulation. Be sure to cover the following
development aspects as part of your demonstration:
- Simulating Outlook's interface
- Creating click-even interactions
(e.g., click the "New" button)
- Creating feedback
- Creating "typed-in" interactions
- Creating the multiple-input event
(The referee will randomly choose one of the following three for you
to demonstrate: click "Save and Close" button;
click "File" menu, select "Save"; or press "CTRL
+ S" keys)
- Adding media (such as audio) to
the simulation
- Measuring/scoring performance
- Show additional innovative aspects
of your simulation development tool
- Discuss your methods for communicating
performance data to standards-conformant learning management
systems
- State the price (or discuss the
general pricing model) used when purchasing licenses to
your system
This is a lot of information to cover in seven minutes, but
the participants did an excellent job focusing on how they
approached the scenario given the allotted time.
Judging procedure and criteria:
The audience served as judges for the Shootout. They were
each given a ballot to rate the system according to what was
observed during the 20-minute production time and the seven-minute
debrief given by each team. Here is what their ballot looked
like for each product:
On a scale of 1 to 10 (with 10 being the highest rating)
how would you rate this product in the following areas?:
| Ease of use |
10 9 8 7
6 5
4
3
2 1 |
| Interactivity of the simulation output |
10 9 8 7
6 5
4
3
2 1 |
| Quality of feedback/guidance |
10 9 8 7
6 5
4
3
2 1 |
| Most innovative approach |
10 9 8 7
6 5
4
3
2 1 |
A total of 10 points was possible for each of the four categories.
The scores were calculated by averaging the scores for all
voters in each round. Winners were announced in each category
based on the highest average score. These are the scores that
are shown in the results section below.
Teams with the top three, highest-average, composite scores
were announced as overall winners for the event. We do not
list the scores for the teams that placed below third place
in any of the categories or for the overall score. The tone
of the competition was designed to be a fun, informal, educational
event with some competition added as motivation.
We'd like to thank all of the teams who joined us for this
event! We held a similar Shootout in 2001
(Los Angeles) and 2002
(London). Overall, we noticed that software simulation products
have improved dramatically since these previous events. The
level of competition was much higher as software simulation
products seem to have reached a new level of product maturity
and sophistication. |
