| |
| The Sponsors |
|
|
There's lots
of discussion in the industry about the "scalability"
of learning management systems. However, until now, much of
that discussion has been focused on how an LMS can scale to
meet the needs of an organization with thousands and even
tens of thousands of learners. With the extreme focus on large-scale
deployments, some LMS vendors will now only engage in a project
if there is a minimum of 2,000 or 2,500 learners.
This Shootout gave viewers an opportunity
to see systems that can also be scaled downward to meet the
needs of a relatively small number of learners (under 2,000).
The event was held on September 23rd, 24th, and 25th, 2002,
in Anaheim, Calif. |
Gallery |
The
competition is broadcast on giant screens in real time to
an audience of judges.
The
audience played a critical role, serving as judges for the
competition, providing evaluations in each of the major categories.
Our
proud judges.

Oracle giving a three-minute debrief, following the task checklist.
In
In order to be completely fair and provide a level playing
field, the referees were sticklers about time limits. As soon
as the countdown clock ended, the team had to immediately
complete their presentation on each topic. The whole Shootout
was conducted in a round-robin format.

A view showing the placement of the four screens (four teams
per day, over a three-day time period), as well as the countdown
clock. The team from Websoft is walking through the scenario
in this piciture.
|
|
|
| The Participants |
|
Background |
Teams were invited to participate
in the Shootout based on their ability to meet all the requirements
of a scenario of a a made-up company with the following characteristics...
Scenarios:
- The company has 250 employees, spread out in three different
countries (hence the need for e-learning delivery). They
would like to be able to run their LMS and e-learning courses
from one central server (at the home office in the U.S.),
but they are open to suggestions on the best approach.
- They want to create about 80 percent of their own content.
They aren’t necessarily looking for content authoring
in the LMS, but they would at least like you to make a recommendation
on what authoring products to use. They have some anxiety
from reading articles about the difficulties of integrating
e-learning content with an LMS. They’ve heard that
“standards-compliant” doesn’t always mean
plug-and-play. They are looking to you for guidance in helping
them make good choices.
- They want their internal SMEs to create much of the content.
- They would like to entertain the idea of buying some third-party
courses (sales and customer service training) to round out
their curriculum. They would like to know how open your
system is to launching third-party courses, especially ones
that are standards-compliant (AICC/SCORM). They figure about
20 percent of their content will be off-the-shelf. If your
system isn’t standards-compliant, how would you recommend
they proceed?
- They want to make sure they have full access to performance
data. They’d like to do ad hoc reporting, if possible.
How can they do this?
- They also want their employees to be able to communicate
with others who are enrolled in the same course. They are
not looking for live, synchronous (virtual classroom) capability
right now, but they would like to use threaded discussions
(topic-based) and/or learner-to-learner, instructor-to-learner
e-mail, or something like that. If you don’t have
this capability, how would you recommend integrating it?
- They also want some basic classroom management. They offer
about 25 courses per year and want to post these courses
through their learning site when they become available.
They also want the system to do some basic enrollment and
wait listing if possible. Other areas that would get them
excited are automatic notification via e-mail for those
who enroll, and they would also like to have managers grant
approval before enrollment occurs.
- They haven’t figured out yet whether they want to
have you host the LMS for them or install it on their local
intranet. They are looking to you for guidance in helping
them figure out which would be more cost-effective and which
will work best for their situation. Make your case either
way….or both ways for them to choose.
- Bottom line: How much will all of this cost? Can you give
them some ballpark figures based on what they are trying
to do today?
- Finally, they think they might open up their training
to end users in the next two years. Their estimate is that
about 2,000 end users (in addition to internal employees)
might benefit from the training. How does the system (and
more importantly the pricing model) scale to meet their
needs?
The format of the Shootout was to literally have each team,
in a round-robin format, address the needs of our made-up
company. Each team was given a list of tasks to complete in
a three-minute debrief. The Shootout area was set up with
four large screens, allowing the audience to view the competitors
side-by-side. |
| 5 points
|
Learner interface adaptation:
All competitors were given the same URL of an organization
and asked to match the look and feel while building
a learning portal. |
| 5 points |
E-learning management:
All competitors were sent a small course (packaged in
either AICC or SCORM) and asked to launch the course
and track performance data. |
| 5 points |
Performance reporting:
All competitors were required to demonstrate their reporting
function and show how data was passed from the course
to the performance reports. |
| 5 points |
Classroom management:
All competitors were asked to add a classroom event
to a curriculum and show how the learner would enroll
in that course (showing the enrollment process, wait
lising, load balancing, notification, etc.). |
| 5 points
|
Collaborative learning:
All competitors were asked to show how they could thread
collaborative learning into the environment, such as
discussion groups, learner-to-learner e-mail and virtual
classroom integration. |
| 5 points
|
Price-to-value ratio:
All competitors were asked to price implementation for
a scenario with only 250 learners. |
| 5 points
|
Overall impression: The
audience was allowed to evaluate each of the competitors
based on their overall impression |
|
Highest Overall Score by Audience Vote |
|
|
Product |
Company |
Score |
|
1st Place |
iLearning |
Oracle |
27.63 |
|
2nd Place |
Enterprise
Knowledge Platform |
NetDimensions |
27.32 |
|
3rd Place |
Knowledge
Centre |
Meridian Knowledge Solutions |
27.24 |
|
Learner Interface Adaptation |
|
|
Product |
Company |
Score |
|
1st Place |
LIBRIX |
Maritz Learning |
3.88 |
|
2nd Place |
KnowledgeCentre |
Meridian Knowledge Solutions |
3.79 |
|
3rd Place |
LearnCenter |
Learn.com |
3.77 |
This category required the LMS vendors to go to the Web site
www.entnet.org/index2.cfm
and match their look and feel in creating an empty learning
portal. We had all the vendors bring up the interface they
created at the same time, then we asked the audience to vote
on who had done the best job. |
E-Learning Management |
|
|
Product |
Company |
Score |
|
1st Place |
OnTrack
Product Suite |
DK Systems |
4.25 |
|
2nd Place |
LIBRIX |
Maritz Learning |
4.07 |
|
3rd Place |
iLearning |
Oracle |
3.98 |
This category required each team to (1) attach an AICC-compliant
or SCORM-conformant course to their course catalog, (2) add
a new user, (3) assign the new user to the course, (4) launch
the course as the learner, (5) show how performance data flowed
from the course into the LMS, (6) show how a learner would
see his/her own score, and (7) show how an instructor or administrator
would also see this score. |
Performance Reporting |
|
|
Product |
Company |
Score |
|
1st Place |
KnowledgeCentre |
Meridian Knowledge Solutions |
4.00 |
|
2nd Place |
OnTrack
Product Suite |
DK Systems |
3.98 |
|
3rd Place |
iLearning |
Oracle |
3.93 |
This category required each team to (1) show how a learner
sees his/her own score, (2) show how an admin user would see
the score, and (3) discuss how reports might be customized. |
Classroom Management |
|
|
Product |
Company |
Score |
|
1st Place |
KnowledgeCentre |
Meridian Knowledge Solutions |
4.27 |
|
2nd Place |
OnTrack
Product Suite |
DK Systems |
4.06 |
|
3rd Place |
Enterprise
Knowledge Platform |
NetDimensions |
3.97 |
This category required each team to (1) add a classroom event
to the curriculum, (2) register the learner for the class,
(3) show how enrollment and notification work, and (4) show
how an instructor could add performance data for the classroom
event. |
Collaborative Learning |
|
|
Product |
Company |
Score |
|
1st Place |
Enterprise
Knowledge Platform |
NetDimensions |
4.24 |
|
2nd Place |
KnowledgeCentre |
Meridian Knowledge Solutions |
4.21 |
|
3rd Place |
iLearning |
Oracle |
4.05 |
This category required each team to (1) show how threaded
discussion might be added, (2) show internal, learner-to-learner
and learner-to-instructor e-mail, (3) show how a virtual classroom
session might be added, and (4) demonstrate any other unique
collaborative capabilities. |
Price-to-Value Ratio |
|
|
Product |
Company |
Score |
|
1st Place |
Enterprise
Knowledge Platform |
NetDimensions |
4.35 |
|
2nd Place |
iLearning |
Oracle |
4.33 |
|
3rd Place |
Webmentor |
Avilar Technologies |
4.02 |
This category required each team to (1) tell how long it
would take to implement their solution, given the scenario
above, (2) tell how much their solution would cost for both
a hosted and a locally installed solution, and (3) discuss
what upgrade paths look like if our made-up company moved
from 250 learners to 2,000 learners. Note: In general, the
audience gave the highest scores to those with the lowest
cost. |
Overall Impression |
|
|
Product |
Company |
Score |
|
1st Place |
iLearning |
Oracle |
3.94 |
|
2nd Place |
Enterprise
Knowledge Platform |
NetDimensions |
3.90 |
|
3rd Place |
Knowledge
Centre |
Meridian Knowledge Solutions |
3.85 |
Audience preference based on overall impression. |
|
|