How do you evaluate e-learning developers, choosing between suppliers who seem very similar? Don’t just kick their tires--check under the hood, too. Here’s what to look for.

So, your company has decided to build e-learning courses, either for internal training needs or as product offerings for clients. Reviewing the article “Buy vs. Build: A Battle of Needs” has not only convinced you that you want customized courses, but that you want to outsource their development. Now you need to choose a development firm.
But how do you evaluate suppliers?

You’ll have to do some homework, but your decision will be much easier once you realize that it’s all about risk aversion. So says Forum Corporation’s chief executive of custom solutions, David Fabianski. The risks inherent in a bad choice, he says, include being late to market, disappointing customers, and overrunning costs. To reduce your risk when evaluating suppliers, look for these crucial components.

Project and process management

In business, nobody likes surprises, especially when they come in the form of schedule and cost overruns. So an e-learning supplier’s most critical qualification lies in its development process and project management discipline. A supplier’s ability to effectively and efficiently run a project affects schedules, budgets, and ultimately your satisfaction with its work. Look for a supplier who can provide

A proven development process. Make sure the supplier’s development process is proven, stringent, demonstrable, predictable, repeatable, and documented. During a walk-through, the supplier should go over everything from analysis, design, and development to client reviews, requests for changes, and scope creep. The process should hinge on collaboration, communication, and clear expectations throughout the project, says Fabianski. And you want it to include your incremental sign-off on deliverables throughout the development cycle. Beware of suppliers who hype their custom development expertise at the expense of their development process. Custom development doesn’t imply on-the-fly or process-independent development. You should be able to have your custom courses built according to a predictable process with predictable costs.

An automated workflow. Suppliers often prove that they have a solid development process by automating process, or project workflow, in their technical development environment. Automated workflow tools route work to the right people at the right time, automate business rules, incorporate template-driven design, monitor development metrics, and generally streamline development efforts. The results are tangible: consistent processes, efficient use of time and resources, and predictable and reliable development metrics, all of which result in less risk for you.

Virtual, collaborative project management. The development environment should also allow for dispersed, collaborative development and project management. The supplier should be able to manage projects virtually, with only occasional face-to-face meetings. Expect the supplier to provide a project Website where you can review, comment, and sign off on deliverables. Virtual, dispersed development and project management allow you to overcome geographic boundaries, enable co-development between you and your supplier, and facilitate immediate deliverable reviews.

Technology and integration

The partner you choose should have both a robust development environment with state-of-the-art technologies and the ability to integrate its content with other supplier technologies. In particular, look for a supplier who is

Experienced with LMS integration. Keep in mind that e-learning content doesn’t stand alone--it’s usually integrated with a learning management system (LMS) for scheduling, tracking, and reporting. New e-learning customers typically overlook how challenging it is to integrate one company’s e-learning content into another company’s LMS, says Fabianski. If you don’t already have an LMS, you’ll have to choose one. So you might work with two suppliers, an e-learning development house and an LMS provider. Or you can find one supplier who offers both. Sometimes, however, companies that offer multiple e-learning products and services do one thing well at the expense of the other. You want a supplier who excels in course development.

If you work with separate LMS and e-learning suppliers, you’ll need to determine who’s in charge of integrating the content within the LMS, Fabianski adds. If it’s the e-learning developer, they need experience integrating their content with several third-party LMSs, both off-the-shelf and customized. A supplier experienced with several systems can anticipate challenges and apply known strategies for handling problems when they encounter a new system. If your e-learning supplier is “in bed” with just one LMS supplier--and it’s not yours--you’ll pay as the developer learns how to integrate its content with your LMS.

Standards-aware. The future of e-learning interoperability lies in the standards initiatives, which will facilitate interoperability between third-party content, LMSs, and LCMSs, as well as accessibility and reusability of the learning content. Your supplier should understand standards and how they'll affect the development environment and processes. Suppliers should be working towards standards compliance, so that the content they build will be interoperable and shareable across standards-compliant LMSs and LCMSs.

Using database storage. Industry standards assume a database-driven architecture to facilitate accessibility, maintenance, and reuse. Look for a supplier who stores content in small chunks--or as learning objects--in a database. A database structure inherently organizes content so a learner can access a discreet piece of information right when it’s needed. And learning objects simplify maintenance and reuse: You can make a change to content in the database without having to change every individual course.

Development resource pool

When evaluating a supplier, learn what you can about the resources available for your project. Examine the supplier’s

Breadth of resources. Ask a few questions about capacity and scale. How quickly can the development company ramp up and start your project? What are its back-up plans in case of resource crunches? Where is your project on its list of priorities? Does it have dedicated or shared resources on each project? If they’re sharing resources between projects, would your timeline be jeopardized by another client’s needs?

In-house resources. To avoid unexpected management headaches, ask whether the development company has its own resources or whether it’s serving as a broker for subcontractors. Common business practices today--not just in the e-learning industry--include subcontracting, often to offshore suppliers. If you know up front that you’ll be essentially managing more than one contractor, and you have the time and resources to support that model, then you can consider working with a supplier who subcontracts the development work. If you don’t have that kind of time, then it may suit you best to work with a single team, development model, culture, and architecture.

Vertical market and subject matter expertise. What do the content developers building your project know about the subject of the courses? Do they have experience in your target market? If they have little to no background in those areas, you’ll pay for one of the most common hidden costs of e-learning projects: unexpected subject matter expert time. The supplier’s subject matter and vertical market expertise directly influence the amount of SME support you’ll need to provide. The more knowledge your supplier has in your space, the more efficient the knowledge transfer from your SMEs to their content developer. If you can, find a supplier who has fundamental business acumen in your content area and market.

Instructional design expertise. In addition to having vertical market experience, the supplier should place a premium on instructional design. Some companies have a high-level instructional design guru on board. But instructional design should permeate lower levels of the company, too. Look for ID experience or training in the project management and content development teams, as well as a documented instructional philosophy or approach.

Finances

Over the past two years, the dot.com bust and general market downturn has hit the e-learning market hard. In 2001 alone, more than 100 e-learning companies went out of business, and mergers and acquisitions absorbed 26 others. Given the state of the industry and the economic market, expect the consolidation trend to continue for some time. (See “E-Learning Consolidation Update,” T+D, April 2002.) So, you’ll need to assess your supplier’s financial viability. Consider

Capital structure. To the extent that you can, try to understand the supplier’s capital structure. If its a private company, when was it founded? What round of financing is it in? The later the round, the more mature the company. Is it profitable? Ideally, yes. But profitability may be rare in the current economic climate. If it isn't profitable, then how much cash does it have? How much debt? Smaller companies should be forthright about how many months of operation they can currently and securely finance. If it's a public company, when did it go public? How has its stock performed in the last year? You can learn a lot about a public company’s finances. It’s more difficult to discern a private company’s stability, so you’ll have to be more cunning in your research. According to Julianne Frawley, vice president of alliance development at the Forum Corporation, a good rule of thumb is that the more forthright companies are about their finances, the more trustworthy and stable they likely are.

Profitability of development services. If the supplier offers other services in addition to course development, such as selling and licensing an LMS, try to learn how much of its total revenue comes from their development services. Look not only at the stability of the overall company but also at the company’s profitability in the services space, the most challenging sector of e-learning in which to be profitable. If the company’s services group makes a profit creating unique solutions for their clients, then it shows integrity in its core processes. And again, the more proven the process, the less risk for you.

Pricing model. When a supplier bids on your project, the pricing model it employs reveals confidence--or lack thereof--in its development process. According to Frawley, a fixed price bid often demonstrates that the supplier is experienced and can control scope. So if you're offered a fixed price bid, you can assume that the supplier has solid processes in place and that deliverables are fairly predictable. And for you, fixed pricing controls costs and helps avoid surprise budget overages at project’s end.

Size/experience/price trade-offs. Companies of different sizes and experience levels offer different benefits and risks. Only you can decide which benefits are more important and which risks are worth taking. In general, small companies have lower infrastructure costs than large companies, so they may offer more attractive pricing. Young companies may be also be hungry and motivated to be flexible. The trade-off, however, is that they may have less mature processes, experience, and client lists. That increases your risk and may turn out to cost more in the end, Frawley says.

Large companies may support more fixed costs, which might raise their prices. Or conversely, they may be large and resource-rich enough to have achieved economies of scale that result in more competitive pricing. Similarly, large companies may be able to offer discounts because their revenue comes from multiple sources within the organization. Frawley says that being aware of the organizational and financial variables affecting a supplier’s pricing will allow you to ask the right questions as you evaluate partners of varying sizes and experience levels. Generally it’s best to avoid either end of the size spectrum--too big or too small.

Choosing an e-learning development partner will take time and careful consideration. No matter how vigilant you are, you can’t eliminate all risks. But if you check under the hood, ask the right questions, and look for a supplier with the attributes described here, you can greatly reduce the risk of having an unpleasant development experience.