The evaluate phase of the LMS Review Process is the most time intensive phase as system administrators determine which systems adequately improve learning outcomes, promote instructional innovation, enhance student engagement and offer functionality for different modes of course offering.
Use the tabs below to review evaluation considerations.
It is at this evaluation phase that a User Needs Analysis (different from the initial user survey analysis in the Prepare Phase tab) would be helpful in determining what users need or want from their LMS. Creating a functional requirements list for an LMS user survey will be helpful. Utilize the RFP document from the Initiate tab to assist.
Once an institution has arrived at a set of final LMS vendors to evaluate, consider how each system will reveal differentiation. For some institutions, in addition to running pilots, this will entail developing a comprehensive rubric to aid in a more systematic and detailed analysis of a given system's qualities. Creating a technical rubric will allow LMS support staff, charged with the care and maintenance of the system, a voice in the decision-making process. This type of technical rubric includes a method of scoring the LMS on technical features, functionality, administrative tools, and support criteria. View the resources below to help in this initial user needs analysis process.
Institutions will need to have a sense of current LMS costs before they evaluate LMS alternatives. If the institution currently has a vended LMS, the financial considerations process will be straightforward. If, however, the system is a self-hosted, open source LMS, they will need to determine the amount of staff hours that support the LMS, as well as any infrastructure and support costs. These "indirect" costs are challenging to ascertain, but will make it easier to compare the true costs of changing systems.
Vended LMS systems are often offered on a cloud or SAAS (software as a service) model where they carry the burden of maintaining uptime, patching and update schedules. Further, cloud-based services tend to have robust backup and redundancy contingencies articulated in SLA (service level agreements) in case of system or network failures.
LMS vendors will often provide different support packages at varying price points. Typically, the premium support packages include 24/7 dedicated Help Desk service levels. Vended LMS systems typically price out a subscription to their service per student FTE of the institution. It is not uncommon to see multi-year pricing proposals that heavily discount unit FTE fees in earlier years that then increate dramatically as the contract matures. This approach can be beneficial in that it can offset implementation costs or the costs of having to run two vended systems concurrently for a period of time.
Institutions opting for an open-source/locally hosted option will carry the burden of maintaining servers, performing patching, implementing security measures, etc. These costs both in terms of hardware and staff hours, but also in terms of opportunity costs, must be considered. Cost-related items to consider include:
By this step, typically the earlier vetting process to arrive at systems to evaluate has been achieved either through research, an RFP process, or another method. Institutions should now begin working with the selected vendor(s) to fully evaluate the capacities of their systems. One common way to do this is by running a pilot test of the LMS. Keep in mind that vendors usually charge for these pilots. The terms typically will include the following stipulations:
As you plan your trial period with a pilot group of users, consider the following:
Pilot participant survey examples above offer questions on what users wish the LMS could do and their experience with the system being reviewed.
During the trial agreement with the vendor, there is opportunity to test backend authentication connections. Consider these areas when testing:
As the pilot period concludes, institutions should begin collecting feedback from the pilot participants. Surveys are a great way to do this, and if carefully crafted, can give a sense of how users regard using the systems and how their learning was impacted by the systems (refer back to the User Trials/Pilots tab for links to survey examples). Focus group themes will emerge and the technical rubric will also give insight into what the support burden might be for the evaluated system. Analyzing these three dimensions as to how well they align with institutional goals will, ideally, reveal a clear path forward.
As you begin the decision-making process, consider the following action steps:
Black, E.W., Beck, D., Dawson, K., Jinks, S., & DiPietro, M. (2007). The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments. Techtrends: Linking Research & Practice to Improve Learning, 51(2), 35-39.
Brown, M., Dehoney, J., & Millichap, N. (2015, April 27). The next generation digital learning environment: A report on research. Educause. Retrieved from https://library.educause.edu/resources/2015/4/the-next-generation-digital-learning-environment-a-report-on-research
Brush, T. (2016, April). Relative utility of three models for user evaluations of learning management systems: A higher-ed institution decision context (Doctoral dissertation). Retrieved from https://scholarworks.iu.edu/dspace/handle/2022/20909
Fritz, J. & Whitmer, J. (2017, February 27). Learning analytics research for LMS course design: Two studies. Educause. Retrieved from https://er.educause.edu/articles/2017/2/learning-analytics-research-for-lms-course-design-two-studies
Goodrum, D. (2016, April). Relative utility of three models for user evaluations of learning management systems: A higher-ed institution decision context (Doctoral dissertation). Indiana University, Bloomington, IN. Retrieved from https://scholarworks.iu.edu/dspace/handle/2022/20909
Steel, C. & Levy, M. (2009). Creativity and constraint: Understanding teacher beliefs and the use of LMS technologies. Retrieved from https://www.ascilite.org/conferences/auckland09/procs/steel.pdf
Wright, C., Lopes, V., Montgomerie, T., Reju, S., & Schmoller, S. (2014, April 21). Selecting a learning management system: Advice from an academic perspective. Retrieved from https://er.educause.edu/articles/2014/4/selecting-a-learning-management-system-advice-from-an-academic-perspective
The system decision needs to be communicated institutionally from the top, down. As you determine your communication channels and craft your messages, consider the following tips:
