Skip to Main Content

LMS Evaluation Planning Toolkit: Initiate

This toolkit houses material created and identified by PALNI's LMS Evaluation Task Force to assist member institutions embarking upon the process of an existing Learning Management System (LMS) examination and/or change.

Overview

​

Initiating the review process can be daunting. Accept the magnitude of work that you are about to undertake while keeping in mind the plan you will follow during the process. Use the tabs below to find helpful considerations and resources for piloting systems, establishing adequate staffing, Request for Proposal (RFP) and evaluation criteria documentation, and timelines for the overall initiation process. 

Initiate

Request for Proposal Process

Narrowing down a range of LMS systems is a process in and of itself. Some institutions only take a look at a single LMS, perhaps the market leader, while others need to provide numerous options to faculty and students. 

  • Consider reviewing what other similarly situated institutions are using
  • Review the market and see what options are available
  • Investigate adoption rates

To help you along in the review process, institutions can choose to go through a Request for Proposal (RFP) process that helps the operation stay organized with structure to help determine how a system fits your needs, requirements, and budget. After sending a RFP to a vendor, expect a minimum of 3-4 weeks for a response with questions for clarification included.

 

 

Piloting Program

 

Asking the question of whether to pilot or not to pilot is important in the infancy phase of review. Some institutions opt to pilot one or more systems to compare user experience and functionalities, while others conduct the primary evaluation with a RFP (request for proposal) process (see RFP tab). A combination of these is not uncommon. As you determine which scenario best fits your needs, consider the following:

  • Consider what the institution, at large, expects and how decisions are made (piloting may be a necessary part of the process)
  • Hand-select faculty and invite them to participate in the pilot
    • Differentiate your users for diverse input and feedback
    • Select faculty from each department/college to have equal representation (this will also assist in supporting and training future users of the system)
  • Develop training materials early
  • Communicate a pilot timeline, including feedback expectations

Consult the Timelines tab for a detailed look at a sample pilot schedule.

 

Review Staffing

 

When determining what staffing is needed to successfully complete an LMS review, it's important to remember this is often a long, arduous process. The process requires a lot of hands-on work, organization, and attention to detail. Throughout, you may need to revisit these questions:

  • What resources are already available?
  • Do additional resources need to be pulled in from other departments or hired on a temporary basis?
  • Does another project need to be put on hold in order to shore up support for the review process?
  • What is the timeline for the review (see the timeline tab) and when should a final decision be made?

 

Timeline

 

The average review, from the initial question of whether or not to conduct to a final decision to adopt, takes roughly 1.5 - 2 years. Set deadlines for key steps in the review process. An example timeline may look like this:

 
 

Year 1

Year 2

Spring

Summer

Fall

Spring

Summer

Phase I: Prepare for a Formal Review

Communicate intent to review
 

Determine vendor systems to review

  

Phase II: Initiate the Formal Review

Develop Request for Proposal (RFP) if applicable

Prepare for a pilot to test systems being reviewed

Deploy pilot with training and clear review documentation expectations for user data
 

Analyze pilot data with evaluation rubric, extending to the spring semester, if applicable

Organize & invite top 1-2 vendors to campus for live presentation/demonstration

 

Complete analysis of pilot data and finalize system recommendation with formal documentation

  

Phase III: Implement Chosen System

  

Sign vendor contract

 

Communicate timeline for system access (old system expiring)

 

System administrator training

 

Plan adoption rollout (phases of rollout and users)

Early adopter training

 

Marketing system change

Consideration must be given to the method of how the review will be conducted. 

 

Literature Review

Anstey, L. & Watson, G. (2018, September 10). A rubric for evaluating e-learning tools in higher education. Retrieved from https://er.educause.edu/articles/2018/9/a-rubric-for-evaluating-e-learning-tools-in-higher-education?utm_source=Informz&utm_medium=Email&utm_campaign=ER#_zsshltf1_zl2lv85

Dahlstrom, E., Brooks, C.D., & Bichsel, J. (2014, October 30). 2014 student and faculty technology research studies. Educause. Retrieved from https://library.educause.edu/resources/2014/10/2014-student-and-faculty-technology-research-studies

5th annual LMS data update. (2017, September 17).  Edutechnica. Retrieved from https://edutechnica.com/2017/09/17/5th-annual-lms-data-update/

Roy, S. (2010, December 7). Selecting a new LMS for McGill: RFP process [PowerPoint slides]. Retrieved from https://www.mcgill.ca/science/files/science/LMS.pdf

University of Minnesota. (2017, March 3). Learning management system (LMS) review - 2017 report. Retrieved from https://it.umn.edu/sites/it.umn.edu/files/umnlmsreview.docx.pdf

Change Management & Communication Tips

Change Management

Reactions to change can range from acceptance to absolute rejection. While you may not need much change management to gain buy in with an LMS review, considerable attention and care should be paid to the average person who may not be thrilled with the idea. More important still are the faculty and staff who will be resistant to any idea of changing the LMS. Plan ahead and know how you and your team will address these issues before they arise.

Communication

In an effort to work through change resistance, it is imperative that users (faculty in particular) own the decision of change. Identifying key champions to offer sound feedback during pilot phases will prove beneficial. Consider utilizing a Task Force model approach to help establish authority and control in the process from those who will eventually be using the system.

Additionally, communication can and should start early when initiating the review process. The template offered below may be helpful along the way.