Enterprise Learning Management Systems (LMS) have moved beyond simply launching and tracking e-learning courses or managing instructor-led, classroom training events. We designed the Enterprise LMS/Performance Management Shootout to see how well these systems could manage other important aspects of performance management such as skill-gap analysis, competency management, tracking and recording on-the-job learning experiences, and, most importantly, how the collected performance data can assist a manager/supervisor to conduct annual performance reviews. These are feature sets that are often skimmed over during LMS demos. However, they are important to realizing the promise of automating learning management functionality. Many audience members commented on how great it was to see performance management functionality in a side-by-side comparison with some of the leading systems in performance management.

Our objectives in creating the Enterprise LMS/Performance Management Shootout were to (1) assess the performance tracking capabilities of LMS solutions in areas beyond e-learning and classroom instruction to include all aspects of worker learning and on-the-job performance and (2) to allow the audience to see a wide variety of creative approaches to creating a true "blended learning" environment.

To do this, we designed a performance management "use case" scenario allowing the audience to make an apples-to-apples comparison of each system. More information about this "use case" can be found below. The use case was based on several requests for proposal (RFP) collected by Brandon Hall Research over the past year. We grouped performance management requests into three main categories:

• Skill-gap analysis/competency tracking
• Tracking on-the-job (OJT) performance
• Conducting learner/employee annual performance reviews drawing on all available data sources for performance data

From this information, we:

• Created a mock use-case scenario, so that each team could creatively apply their technology solution to a realistic setting (actual company, learner, documented competencies, etc.).
• Created a checklist of tasks that each team had to address during their debrief to demonstrate how their solution could best be used to solve the problem.
• Used the checklist to create the judging criteria and a ballot for the audience to use to judge the effectiveness of each solution according to the company's needs.

We designed the scenario with specific context so that the teams could have some fun with the topic. For example, some of the teams wore white lab coats, stethoscopes, etc., to play up a medical theme.

Two months before the competition, we provided each team with the following scenario:

Andrea Johnson is an employee of Chapman Healthcare. She was hired about one year ago as a Radiology Technician. When she was hired, she was automatically enrolled in Chapman’s corporate university that uses your learning management system as the underlying infrastructure. It’s now time for Andrea’s annual performance review. Her supervisor needs to collect information to evaluate her performance according to the skills and knowledge listed on-screen (see below). As a consultant on the project, use the following checklist to help us understand how your system will help Andrea’s supervisor evaluate Andrea’s job performance according to this list. Note: The training delivery types are listed next to each item on the list.

As part of the competition, the audience was shown this list of skills and the description was given so that they could judge how well each system meets the needs.

Each team had a total of 20 minutes to walk through the "Andrea Johnson" scenario, according to the following checklist (which was also shown to the audience throughout the Shootout).

Skill-gap analysis/competency tracking

• Beyond composite test scores, how does your system automatically track skills acquired or competency-based tasks completed?
• Demonstrate how this information is made available to learners
• Demonstrate how managers/administrators interact with this data
• Demonstrate the process used by course developers to link content with skills/competencies
• What makes your system unique in terms of skill-gap analysis and/or competency management?

Tracking on-the-job training

• Most enterprise LMS solutions launch and track e-learning and offer extensive capabilities for classroom management. Beyond these capabilities, how does your system allow for the automatic creation of OJT performance records?
• Demonstrate how learners are made aware of OJT opportunities such as live demonstrations, one-on-one mentoring opportunities, continuing education credits completed outside the company, etc.
• Demonstrate how instructors/managers/administrators can enter performance records specifically for OJT.
• What makes your system unique in terms of OJT tracking and management?

Learner/employee performance reviews and feedback

• Beyond test scores and attendance, what other performance data does your system keep for individual learners? Where does this data come from?
• From a supervisor’s perspective, log-in to the system and conduct a complete performance review of a mock learner. Show us where all of the information came from, how the performance review is created, and how others in the system access and use the performance review information.
• From the learner’s perspective, demonstrate what parts of the performance review information they can see and use.
• Can you demonstrate any unique functionality in this area?

Other innovative performance management capabilities

• The categories above represent some of the basics of performance management. How have you pushed the envelope in the area of performance and human capital management?
• Demonstrate capabilities that relate specifically to performance management.

With a large, 20-minute countdown clock running, each team walked through the scenario, demonstrating and discussing how performance data was collected for Andrea Johnson over the period of one year. They showed how performance data automatically triggered Andrea's list of required skills and knowledge. They showed how gap analysis occurs in their system. Several showed tools such as 360-degree (or multi-rater) assessments, how skill levels are set in their system and how managers gain access to this information. They showed how on-the-job training is logged in the system, etc.

From the perspective of Andrea's supervisor, they showed us all how information was accessed to conduct Andrea's year-end performance review. The audience was able to simultaneously see the demonstration, the checklist showing the judging criteria, and the countdown clock to ensure a fair and accurate competition.

Several audience members commented that the use-case scenario is similar to what they use when doing due diligence in selecting learning technologies. Any organization can use Shootout-like techniques when inviting an e-learning vendor in for a demonstration. Think of writing scenarios for several types of users in the system, including the learner, instructor, LMS administrator, content developer, etc. If an organization takes the time to write prescriptive use cases it makes the technology selection process much easier.

Judging

After each team walked through the scenario, we asked the audience to rate their performance in each area. Here is a copy of what the ballot looked like:

Scores were averaged against 10 points possible for each category.

We appreciate all the teams that participated in the event and realize how much preparation it took to participate. We would also like to thank audience members for taking time during the conference to watch and evaluate each system. We could tell that the audience members take this very seriously, some taking the time to add margin notes outlining what they saw as strengths and weakness of each system. Thanks to all!