How to Avoid the Pitfalls of LMS Implementations
By Chris Howard
Bersin & Associates has assisted a number of companies in dealing with the critical issue of selecting and implementing a learning management system. It also conducts an annual survey on the topic and asks respondents to share best practices. Here’s what we’ve learned.
To be sure, there are many things that can go wrong in a systems implementation. Bersin & Associates has found that organizations encounter similar problems, as well as have similar answers to the question: "What would you have done differently if you had to go through this process again?"
Our research finds that there are four common issues that companies face again and again. These pitfalls will seem obvious to those folks who have already been through a system integration project, but they may surprise some of you who are knee-deep implementing your LMS. So, let’s review them now, and discuss ways you can avoid or minimize their impact on your success.
Where is the Data? Can you access it?
Typically, a main objective for installing an LMS is to gain enterprise-wide access to data. Which employees have received training? To what degree were employees trained? What are the compliance levels of each employee? How much do programs cost? Unfortunately, the first challenge most organizations encounter is an inability to access all the information they need to answer those questions.
Indeed, Bersin & Associates's 2004 LMS market survey of some 300 organizations that have implemented LMSs revealed that reporting capabilities received the lowest marks for satisfaction (see chart below). Fewer than half the users surveyed were happy with the reporting and data analysis capabilities in their LMS. Although the LMSs were able to store a variety of information, it was irretrievable or locked to certain users.
Source: Bersin & Associates LMS Market 2004 Report, published in May 2004
To avoid this problem, you need to identify and detail every report your organization may need, as well as pinpoint who will have access. Be sure to outline data and reporting requirements during the implementation process or, better yet, during the product evaluation process. Developing specifications for reports is sometimes the last thing addressed during the implementation phase, but it’s the first problem spotted once the system is up and running.
Next, buyers need to make sure that reports (custom-built or otherwise) are developed in advance. Don’t assume that the system has built-in reports that will suit your needs. In fact, it’s a mistake to assume that all the data you need will even be collected by the system. You need to plan how you will capture the data and allocate part of your budget for custom report development.
In addition, you need to think about data quality. Once you start analyzing learning data you will realize that the way you organize employees (their reporting structure, job title, accounting codes, etc.) is critical to making your reports work correctly. Finally, if you plan to offer e-learning courseware, you need to be sure that it tracks the data you need.
Suppliers share some of the responsibility for this issue. Not all of them provide enterprise-class reporting solutions, and some fail to help customers understand the real value of the data collected and stored in their system. Fortunately, that seems to be changing as the LMS market matures. For example, SumTotal Systems (the newly merged Docent and Click2Learn) and Saba have taken a leadership role in providing their customers with advanced solutions called training analytics.
Extensive customizations
Many organizations assume that their training problems are unique and that off-the-shelf products will need to be highly customized to accommodate their particular training process. However, making extensive customizations can lead to a lengthening of the implementation by months and an increase in cost, both of which can significantly decrease the program’s worth in terms of business value. In addition, modifications typically complicate the process of upgrading the system and require a considerable amount of rework.
Bersin & Associates recognizes that training administration varies widely among programs based on audience, program type (blended versus fully online), urgency, sophistication of users, and so on. But it’s often less painful to modify business and training processes rather than make extensive changes to the application.
A recently documented Docent case study revealed the advantages of limiting system modifications. The implementation began by outlining all possible usage scenarios, which were modeled around existing training processes. This procedure revealed that the LMS would need 83 modifications to support the company’s existing training procedures.
Rather than proceed with this plan, which would have delayed the implementation by two or three months, the team reviewed existing processes to eliminate modifications that weren’t really necessary. During this review, the business reasons behind some processes were unclear. As a result, the team was able to cut the number of modifications down to 26, and they adjusted processes to conform to the workflows within the Docent system. In the spirit of compromise, Docent agreed to add one requirement that would have taken 19 man-days to complete into the next version of their system.
When Bersin & Associates asked about system modifications in the 2004 LMS Market Survey, we found that overall satisfaction is inversely proportional to the number of modifications customers make. Users who made limited customizations rated their value higher than those who made extensive customizations to an off-the-shelf system.
How would you rate the value you have received from your LMS?
(Based on the amount of modifications made)
Source: Bersin & Associates LMS Market 2004 Report, published in May 2004
Customizations can be minimized by going through a rigorous selection process. (Visit the Bersin & Associates Website for guidelines.) Selecting the right product for your needs will help reduce the need for modifications down the road. As the example above illustrates, a thorough review of your processes may reveal that they weren’t even necessary, and that you can make do with the way the supplier intended the product to work.
Worth noting, however, is the notion that most enterprise-wide applications, such as SAP, PeopleSoft, and Oracle, have built-in tools that enable upgradable customizations to the product. What this means is that these products can handle changes in business process and still accept upgrades when new versions are available. (One caveat: the upgrade may not be smooth but it can be done.) Because many of these products are now offering LMS functionality, customers can make modifications without the risk of creating a one-off system.
The reality of the market is that many LMS products are not designed to deal with extensive customizations, which is why keeping your changes to a minimum is a more desirable approach whenever possible.
Lengthy projects
Everyone agrees that most LMS implementations are complex and time-consuming. Plenty of folks have learned the hard way that a lengthy timeframe can create major risks for their organizations. The longer the project takes, the more likely the business will change during the implementation.
The length of the project is a result of many things, such as extensive customizations, which has already been discussed. But I contend that the process is made more complex—and lengthened—by project leaders and managers who try to take on too much.
The most effective way to deliver value from an LMS investment is to get the system up and running as soon as possible. You can always expand the scope later. Do you really need to begin managing skills and competencies in the first version? What value can you obtain by launching an e-learning initiative now then adding more capabilities later? Do you need an automated interface to another business application now or can you do that in a second phase? Do you need to have all departments up and running or can you add them in phases?
This is different than modifying the system to suit your business process. The thinking here is that it’s smarter to take small steps in a shorter period of time than big steps in a longer period of time. The more departments and divisions that get involved in the implementation, the more time-consuming the implementation will be.
Bersin & Associates strongly recommends that you start by implementing one manageable group. This allows you to get moving while you learn more about the system, which will better prepare you to ask the right questions when it’s time to add new users or functionality.
Conflicting requirements
Typically, when you try to please everyone you fail to please any one. This issue applies to those of you trying to install an LMS that will serve multiple departments or divisions. Different groups within an organization may have their own system that they’ve been using for some time and, therefore, try to make the case that their training needs are different from others or that switching costs are too high.
I don’t mean to imply that different businesses don’t have different needs. For example, it’s clear that the director of compliance in a financial institution will have different training needs than the director of sales training. And, given that, it may be the case that a single product cannot meet the needs of both.
But more commonly, Bersin & Associates has found that training processes were exactly the same across divisions (or should have been) and the various groups were simply vying for control. When this happens, the project leader often feels the need to try and address everyone’s requirements. In doing so, the solution may end up being cumbersome or inefficient, as well as cost too much. This problem is common in large organizations, and it takes a strong leader to make the case that the organization benefits when groups cooperate.
Without question, these four issues just scratch the surface. You’re sure to encounter more speed bumps on the road to success with your LMS selection and implementation. But, hopefully, discussing them will raise awareness and prepare buyers to meet these issues head-on. And for those of you that have already gone down the LMS path, now you know you weren’t alone.
Chris Howard is principal consultant for Bersin & Associates. He was one of the original developers of Pathware, (now part of Lotus LearningSpace), and has designed several LMSs for such industry leaders as Macromedia and DigitalThink.