Skip to Main Content

LMS Evaluation Planning Toolkit: Prepare

This toolkit houses material created and identified by PALNI's LMS Evaluation Task Force to assist member institutions embarking upon the process of an existing Learning Management System (LMS) examination and/or change.

Overview

The prepare phase of the LMS Review Process includes questions and considerations when deciding whether or not to conduct a formal LMS review.
Use the tabs below to review institutional, support, and administrative considerations.

Preparing for an LMS Evaluation

Readiness for Change

Before making the decision on whether or not to conduct a formal evaluation, it would be helpful to test your institution's readiness for change. In addition to the technical and pedagogical aspects of an LMS, consider the socio-political aspects of:

  1. what information is needed to make a sound decision, and
  2. who should be making the decision.

Change is difficult, even if for good reason, and is often met with resistance. To help this situation, consider the following:

  • Communicate early with administrators that may have a say in the review process and final decision. Be prepared to speak to the overall Return on Investment (ROI) for the system. 
  • Communicate early with system users that preparation is taking place for a review. Be prepared to share what users can expect during the review process with explanation of why the current system is being reviewed.

 

Institutional Considerations


The decision to conduct a formal LMS review could come from a variety of factors present on campus. It is standard practice to review the LMS every five to seven years in an effort to determine what is in the marketplace and how your institutional needs are being met, or unmet, by your current system. Key questions to consider:

  • Are there aspects of your current LMS that users are unhappy with?
  • Are there aspects of your current LMS that users are happy with and should not change?
  • Do you know what your LMS can do versus what other systems in the marketplace can do?
  • Are there aspects of functionality that should not be changed?

If a decision to investigate a change is made, consider the following questions before you embark on the evaluation:

  • Is the institutional political climate in place to make a change to the LMS, if needed?
  • Who will ultimately inform or make the decision to change systems (i.e. faculty, administration, IT, combination)?

 

Support Considerations


Various support considerations may be present including, but not limited to hosted/non-hosted systems, on-site/home-grown/non-vended systems, etc. Support time, software development, and developing training materials to support students, faculty, and staff are areas to keep in mind. Here are a few questions to consider asking: 

  • Is the "cost" of your current LMS too high, in real dollars, lost functionality, and/or the burden of support?
  • Could the institution be better served by a product supported by a team of developers, trainers, and documentors?
  • Could the time spent on help-desk aspects, such as how-to's, be better spent on the pedagogical use of the LMS?
  • Is there an impending end to a current contract?
  • Are there security concerns that require a change?

Even if you choose not to conduct a full evaluation, an inventory of how your institution is using the LMS, as well as key resources such as developer time, training, documentation, and support is extremely useful information for allocating resources.

Additionally, a LMS review does not have to require a change in systems. While a bias toward change is common in LMS reviews, there are numerous cases of institutions conducting reviews only to affirm their current LMS.

Both surveys offer example questions asking how faculty & students currently use the existing LMS.

Literature Review

Lang, L. & Pirani, J.A. (2016). The 2015 enterprise application market in higher education: Learning management systems. Educause. Retrieved from https://library.educause.edu/~/media/files/library/2016/12/cdsm816d.pdf 

Ryan, T.G., Toye, M., Charron, K., & Park, G. (2012). Learning management system migration: An analysis of stakeholder perspectives. International Review of Research In Open &  Distance Learning, 13(1), 220-237.

University of Buffalo. (2017, February). Trends and the future of learning management systems (LMSs) in higher education. Center for Educational Innovation. Retrieved from https://www.buffalo.edu/content/dam/www/ubcei/reports/CEI%20Report%20-%20Trends%20and%20the%20Future%20of%20Learning%20Management%20Systems%20in%20Higher%20Education.pdf

Change Management & Communication Tips

Organizing Communication

It is important to ensure a plan for addressing communication is in place prior to beginning work on a formal review. Areas to consider in developing a communication plan are:

  • Audiences (administration, faculty, adjuncts, students, staff, decision-makers, governing bodies, etc.)
  • Readiness for change
  • Timeline(s) for project
  • Modes of communication and delivery channels

Additionally, it is important to consider the evidence that will be needed to justify the time and expense of a review. Here are a few questions you may want to consider in this preparation phase:

  • Is there a governing body that needs to approve the formal evaluation/review? 
    • If one is in place, should it be informed prior to a formal evaluation/review?
    • Who has a voting voice in the process of decision-making?
  • Should students and/or staff government be involved in the decision?
    • If so, at what point of the process?
  • Should faculty, students, and staff be queried up front on LMS use/practices (surveys, focus groups, combination, etc.)?
  • Is there a commitment to making an LMS change, or is the evaluation/review to simply investigate available LMS options?

Recommendations:

Above all, communicate the decision and process early and often to faculty, staff, and students. Consider the following:

  • Create a website that will be the central repository for all things LMS review and commit to regularly updating that site
  • If there is a newsletter (centers for teaching and learning, departmental, university), put regular updates in that publication
  • Create graphics and market the review process on digital signage around campus

Any avenue available to be transparent in process and decision-making will pay dividends in the future, no matter what eventual decision is made.

Resources