Most organizations, at some time or another, are faced with the challenge of creating simulations to teach employees how to use a variety of software applications such as claims processing systems, company intranets, HR systems, accounting software, CRM, ERP, and many other business-critical software systems.

We invited the best-of-the-best vendors (who offer tools you can use to create your own simulations) to create an interactive simulation in front of a live audience during the Innovations in learning Conference in Santa Clara, California. Each team was asked to create a simulation, teaching a learner how to add a name in an Outlook address book, and then to debrief the audience on their approach in areas such as rapid development, interactivity in the simulation, and creating instructional feedback based on learner performance. These are the results of the competition.

The basic idea of the Shootout is to have teams attack the same business issue using different approaches and technology. This creates an excellent learning opportunity for the audience (to observe a variety of techniques) and also provides an environment for evaluating perspective technologies. For the 2007 Software Simulation Shootout, the task was to create a short, highly interactive software simulation in front of a live audience.

The subject matter for the Shootout was simply to teach a learner how to add a person's name to the address book inside Microsoft Outlook. Although most organizations wouldn't likely create something so generic, we felt it was a good representative sample, allowing for a variety of simulation inputs, such as mouse clicking, pull-down menus, typing text, key press, etc. Also, we didn't want to require each of the five participating teams to have to learn a new software application in order to understand the subject matter.

Each team was given a very crude storyboard (if you can even call it that). Basically, they were given a Word document outlining the steps to be taught, including opening the address book, creating a new record, adding the data, attaching a picture, and saving the new contact information. The teams were given the storyboard before the event and were allowed to practice creating their own simulation. Even though they were allowed to practice, the rules specifically indicated that they were not allowed to bring any pre-created content with them to be used in the competition.

The entire simulation had to be created from scratch, including taking screen captures, simulating dialog boxes and menus, defining interactions, laying out feedback, adding instruction prompts, and even adding audio narration. Several teams added audio to their simulation, which all had to be recorded on-site (although it was a bit noisy in the Expo hall). The only piece of media they were allowed to pre-create was up to a 30-second video clip on the subject matter (to demonstrate their ability to incorporate rich media in simulations) – nothing else!

If you would like to download a copy of the basic simulation storyboard to see what they had to work with, or to create your own version as a benchmark, please click here. (Note: This will allow you to download the file in .zip format.)

Each team had a total of 15 minutes to create their entire simulation.  Upon completing the 15-minute production time, each team provided a seven-minute debrief of what they were able to produce. During the debrief, each team was asked to use the following checklist to report on their activities:

Debrief checklist:

Were you able to finish the entire simulation during the 15 minutes? If not, what percentage was completed?

What output format did you create? (HTML/JavaScript, Flash, proprietary, etc.)

What is the size (total of all file sizes) for the simulation? With media? Without media?

Run the simulation you created in any Web browser and demonstrate the following:

• Prompts and instructions for learner
• Navigation controls (beyond following the steps)
• Feedback for incorrect answers
• Unexpected feedback (e.g., What happens if the learner just clicks somewhere on the screen that wasn't anticipated by the simulation developer?)
• Remediation options when "they don't seem to get it"
• Guidance beyond feedback (e.g., hints, demo mode, etc.)
• Demonstrate that you were able to create an interaction with multiple input possibilities (referee will verify)
• How do learners see their own scores? Is there a review mode for remediation?
• Demonstrate or discuss the use of media in your simulation
• Show any unique and differentiating aspect of the simulation output

• Simulating Outlook's interface
• Creating click-even interactions (e.g., click the "New" button)
• Creating feedback
• Creating "typed-in" interactions
• Creating the multiple-input event (The referee will randomly choose one of the following three for you to demonstrate: click "Save and Close" button; click "File" menu, select "Save"; or press "CTRL + S" keys)
• Adding media (such as audio) to the simulation
• Measuring/scoring performance
• Show additional innovative aspects of your simulation development tool
• Discuss your methods for communicating performance data to standards-conformant learning management systems
• State the price (or discuss the general pricing model) used when purchasing licenses to your system

This is a lot of information to cover in seven minutes, but the participants did an excellent job focusing on how they approached the scenario given the allotted time.

Judging procedure and criteria:

The audience served as judges for the Shootout. They were each given a ballot to rate the system according to what was observed during the 15-minute production time and the seven-minute debrief given by each team. Here is what their ballot looked like for each product:

On a scale of 1 to 10 (with 10 being the highest rating) how would you rate this product in the following areas?:

A total of 10 points was possible for each of the three categories. The scores were calculated by averaging the scores for all voters in each round. Winners were announced in each category based on the highest average score. These are the scores that are shown in the results section below.

Teams with the top three, highest-average, composite scores were announced as overall winners for the event. We do not list the scores for the teams that placed below third place in any of the categories or for the overall score. The tone of the competition was designed to be a fun, informal, educational event with some competition added as motivation.

We'd like to thank all of the teams who joined us for this event! We held a similar Shootout in 2001 (Los Angeles), 2002 (London), and 2005 (New Orleans). Overall, we noticed that software simulation products have improved dramatically since these previous events. The level of competition was much higher as software simulation products seem to have reached a new level of product maturity and sophistication.

Total range of scores was 32.70 to 29.00. Top score would be 40 points.

Total range of scores was 8.50 to 6.83. Top score would be 10 points.

Total range of scores was 8.10 to 7.13. Top score would be 10 points.

Total range of scores was 7.90 to 6.82. Top score would be 10 points.