Brandon Hall Research
    Sign up for "Brandon Hall News" and get free research downloads. Your privacy is assured.
    Your Email:

Home > Community > Shootouts > 2002 Learning Management System Shootout

Brandon Hall Research: 2002 LMS Shootout
 
The Sponsors
OLL logo
BHR logo

There's lots of discussion in the industry about the "scalability" of learning management systems. However, until now, much of that discussion has been focused on how an LMS can scale to meet the needs of an organization with thousands and even tens of thousands of learners. With the extreme focus on large-scale deployments, some LMS vendors will now only engage in a project if there is a minimum of 2,000 or 2,500 learners.

This Shootout gave viewers an opportunity to see systems that can also be scaled downward to meet the needs of a relatively small number of learners (under 2,000). The event was held on September 23rd, 24th, and 25th, 2002, in Anaheim, Calif.


Gallery

Audience watches on giant screensThe competition is broadcast on giant screens in real time to an audience of judges.

View from the from back of audienceThe audience played a critical role, serving as judges for the competition, providing evaluations in each of the major categories.

Judges pose for a pictureOur proud judges.

A team explains their entry
Oracle giving a three-minute debrief, following the task checklist.

The clock ticks...In In order to be completely fair and provide a level playing field, the referees were sticklers about time limits. As soon as the countdown clock ended, the team had to immediately complete their presentation on each topic. The whole Shootout was conducted in a round-robin format.

Teams sat in front of giant screens
A view showing the placement of the four screens (four teams per day, over a three-day time period), as well as the countdown clock. The team from Websoft is walking through the scenario in this piciture.

 
The Participants
Product Company Home Page
Enterprise Knowledge Platform NetDimensions www.netdimensions.com
iLearning Oracle www.oracle.com/ilearning
KnowledgeBridge Websoft www.websoft.com
KnowledgeCentre Meridian Knowledge Solutions www.meridianksi.com
LearnCenter Learn.com www.learn.com
LIBRIX Maritz Learning www.maritzlearning.com
OnTrack Product Suite DK Systems www.dksystems.com
TrainingPartner Geometrix Data Systems www.trainingpartner2000.com
VCampus VCampus www.vcampus.com
WBT Manager Integrity e-Learning www.ielearning.com
Webmentor Avilar Technologies www.avilar.com

Background

Teams were invited to participate in the Shootout based on their ability to meet all the requirements of a scenario of a a made-up company with the following characteristics...

Scenarios:

  • The company has 250 employees, spread out in three different countries (hence the need for e-learning delivery). They would like to be able to run their LMS and e-learning courses from one central server (at the home office in the U.S.), but they are open to suggestions on the best approach.
  • They want to create about 80 percent of their own content. They aren’t necessarily looking for content authoring in the LMS, but they would at least like you to make a recommendation on what authoring products to use. They have some anxiety from reading articles about the difficulties of integrating e-learning content with an LMS. They’ve heard that “standards-compliant” doesn’t always mean plug-and-play. They are looking to you for guidance in helping them make good choices.
  • They want their internal SMEs to create much of the content.
  • They would like to entertain the idea of buying some third-party courses (sales and customer service training) to round out their curriculum. They would like to know how open your system is to launching third-party courses, especially ones that are standards-compliant (AICC/SCORM). They figure about 20 percent of their content will be off-the-shelf. If your system isn’t standards-compliant, how would you recommend they proceed?
  • They want to make sure they have full access to performance data. They’d like to do ad hoc reporting, if possible. How can they do this?
  • They also want their employees to be able to communicate with others who are enrolled in the same course. They are not looking for live, synchronous (virtual classroom) capability right now, but they would like to use threaded discussions (topic-based) and/or learner-to-learner, instructor-to-learner e-mail, or something like that. If you don’t have this capability, how would you recommend integrating it?
  • They also want some basic classroom management. They offer about 25 courses per year and want to post these courses through their learning site when they become available. They also want the system to do some basic enrollment and wait listing if possible. Other areas that would get them excited are automatic notification via e-mail for those who enroll, and they would also like to have managers grant approval before enrollment occurs.
  • They haven’t figured out yet whether they want to have you host the LMS for them or install it on their local intranet. They are looking to you for guidance in helping them figure out which would be more cost-effective and which will work best for their situation. Make your case either way….or both ways for them to choose.
  • Bottom line: How much will all of this cost? Can you give them some ballpark figures based on what they are trying to do today?
  • Finally, they think they might open up their training to end users in the next two years. Their estimate is that about 2,000 end users (in addition to internal employees) might benefit from the training. How does the system (and more importantly the pricing model) scale to meet their needs?

The format of the Shootout was to literally have each team, in a round-robin format, address the needs of our made-up company. Each team was given a list of tasks to complete in a three-minute debrief. The Shootout area was set up with four large screens, allowing the audience to view the competitors side-by-side.

5 points
Learner interface adaptation: All competitors were given the same URL of an organization and asked to match the look and feel while building a learning portal.
5 points
E-learning management: All competitors were sent a small course (packaged in either AICC or SCORM) and asked to launch the course and track performance data.
5 points
Performance reporting: All competitors were required to demonstrate their reporting function and show how data was passed from the course to the performance reports.
5 points
Classroom management: All competitors were asked to add a classroom event to a curriculum and show how the learner would enroll in that course (showing the enrollment process, wait lising, load balancing, notification, etc.).
5 points
Collaborative learning: All competitors were asked to show how they could thread collaborative learning into the environment, such as discussion groups, learner-to-learner e-mail and virtual classroom integration.
5 points
Price-to-value ratio: All competitors were asked to price implementation for a scenario with only 250 learners.
5 points
Overall impression: The audience was allowed to evaluate each of the competitors based on their overall impression

Highest Overall Score by Audience Vote

Product Company Score
1st Place iLearning Oracle 27.63
2nd Place Enterprise Knowledge Platform NetDimensions 27.32
3rd Place Knowledge Centre Meridian Knowledge Solutions 27.24

Learner Interface Adaptation

Product Company Score
1st Place LIBRIX Maritz Learning 3.88
2nd Place KnowledgeCentre Meridian Knowledge Solutions 3.79
3rd Place LearnCenter Learn.com 3.77

This category required the LMS vendors to go to the Web site www.entnet.org/index2.cfm and match their look and feel in creating an empty learning portal. We had all the vendors bring up the interface they created at the same time, then we asked the audience to vote on who had done the best job.


E-Learning Management

Product Company Score
1st Place OnTrack Product Suite DK Systems 4.25
2nd Place LIBRIX Maritz Learning 4.07
3rd Place iLearning Oracle 3.98

This category required each team to (1) attach an AICC-compliant or SCORM-conformant course to their course catalog, (2) add a new user, (3) assign the new user to the course, (4) launch the course as the learner, (5) show how performance data flowed from the course into the LMS, (6) show how a learner would see his/her own score, and (7) show how an instructor or administrator would also see this score.


Performance Reporting

Product Company Score
1st Place KnowledgeCentre Meridian Knowledge Solutions 4.00
2nd Place OnTrack Product Suite DK Systems 3.98
3rd Place iLearning Oracle 3.93

This category required each team to (1) show how a learner sees his/her own score, (2) show how an admin user would see the score, and (3) discuss how reports might be customized.


Classroom Management

Product Company Score
1st Place KnowledgeCentre Meridian Knowledge Solutions 4.27
2nd Place OnTrack Product Suite DK Systems 4.06
3rd Place Enterprise Knowledge Platform NetDimensions 3.97

This category required each team to (1) add a classroom event to the curriculum, (2) register the learner for the class, (3) show how enrollment and notification work, and (4) show how an instructor could add performance data for the classroom event.


Collaborative Learning

Product Company Score
1st Place Enterprise Knowledge Platform NetDimensions 4.24
2nd Place KnowledgeCentre Meridian Knowledge Solutions 4.21
3rd Place iLearning Oracle 4.05

This category required each team to (1) show how threaded discussion might be added, (2) show internal, learner-to-learner and learner-to-instructor e-mail, (3) show how a virtual classroom session might be added, and (4) demonstrate any other unique collaborative capabilities.


Price-to-Value Ratio

Product Company Score
1st Place Enterprise Knowledge Platform NetDimensions 4.35
2nd Place iLearning Oracle 4.33
3rd Place Webmentor Avilar Technologies 4.02

This category required each team to (1) tell how long it would take to implement their solution, given the scenario above, (2) tell how much their solution would cost for both a hosted and a locally installed solution, and (3) discuss what upgrade paths look like if our made-up company moved from 250 learners to 2,000 learners. Note: In general, the audience gave the highest scores to those with the lowest cost.


Overall Impression

Product Company Score
1st Place iLearning Oracle 3.94
2nd Place Enterprise Knowledge Platform NetDimensions 3.90
3rd Place Knowledge Centre Meridian Knowledge Solutions 3.85

Audience preference based on overall impression.

 
© 2008 Brandon Hall Research