Overview - Averia Default Web Page - Stephen Burd



Learning Management System Insights from the 2016 Faculty Technology SurveyOverviewUNM conducts annual surveys of students and faculty to gauge technology trends/needs and satisfaction with technology-related services provided by various units at UNM. This document examines the responses to those questions to inform a decision about whether to proceed with an upgrade or replacement of UNM Learn.Survey SpecificsThe survey was conducted online during the period March 11 through April 8. 1828 full-time faculty from main and branch campuses were asked to complete the survey. 387 faculty responded to the survey (21% response rate) with 328 completing all questions asked. The 2016 response rate is typical for annual faculty technology surveys. Faculty email invites included web links specific to each faculty member which will enable survey responses to be cross-referenced to Banner-related data such as faculty rank, academic unit, and hire date.The 2016 faculty survey included 27 questions in the following groups:10 questions - instructional use of technology other than LMSs and related UNM services5 questions - satisfaction with IT services and use of cloud, email, and calendaring services12 questions - experience with LMSs including both UNM Learn and other products/servicesLMS-Related Questions and ResponsesFor purposes of this survey, an LMS was operationally defined as follows:A learning management system (LMS) is a web application that provides an instructor with a way to create and deliver educational content/experiences, support and monitor student participation, post and receive student assignments.LMS-Related QuestionsThe 12 LMS-related questions are summarized here (numbered 16-27 as they were in the survey):I use an LMS for class sizes of _____ students. (0-9, 10-24, 25-49, 50-99, 100-199, 200+)Please indicate your level of agreement with the following statements about the LMS: The LMSis critical to my teachingis very useful as a tool to enhance my teachingis very useful as a tool to enhance student learningcreates efficiencies for my instructionHow many years of experience do you have using LMS technology? (0-1, 2, 3, 4, 5-10, 11+)Do you currently use UNM Learn? (Yes, No)Q19 NO BranchWhat factors influence your decision not to use UNM Learn (Check all that apply)?instructor ease of usestudent ease of useease of maintaining personal web sitestraining availabilitysupport qualityLMS tool usefulnessLMS featuresLMS flexibilityLME reliability/stabilityclass sizelack of 3rd party content integrationprefer a different systemQ19 YES BranchDescribe how you use an LMS to support your instructional activities (Select all that apply).distribute documents and learning materials to studentsdistribute video to studentsweb conferencinggradebookquiz//exam deliveryonline assignment submissiongraded online discussionscommunication between instructor and studentstudent-to-student messaginggroup collaborationlearning analyticsfully online class deliveryhybrid class deliveryflipped classroomPlease indicate your satisfaction with the following aspects of UNM Learn:ease of use in generalease of use from a mobile devicesupports meaningful interaction with studentssystem availabilitysystem response timeinitial use trainingongoing training and professional developmentposting contentmanaging assignmentsmonitoring/management enrollmententering grades and other progress informationreliable assignment receiptintegrating 3rd party contentintegration with other UNM systemsoverall satisfactionDo you have experience with an LMS that you prefer over UNM Learn? (Yes, No)Q23 YES BranchWhat is your preferred LMS? (Angel, Canvas, Desire2Learn/Brightspace, Moodle, WebCT, Faculty/locally-developed, Other)Why is the above your preferred LMS (Check all that apply)?instructor ease of usestudent ease of useusefulness of LMS toolsfeaturesflexibilityreliability/stability3rd-party content integration If my preferred LMS was available for use at UNM, I would invest time in moving my content to that LMS from UNM Learn. (Yes, No)Please provide additional comments comparing your experience with the other LMS to UNM Learn.Survey RespondentsA few observations about respondent characteristics can be gleaned from the survey responses:43% claim 5 or more years of LMS experience (Q18). Thus, it appears that survey respondents are more likely than non-respondents to be users of technology in general and the LMS in particular. If that characterization is true, the good news is that survey respondents are probably well-informed. The bad news is that they probably don’t represent the faculty as a whole.287 respondents (approximately 77%) currently use UNM Learn. This is a fairly large sample from which to gather responses to the detailed UNM Learn questions. However, there’s no way to know how (un)representative of the faculty as a whole are those respondents.53 respondents (approximately 16%) have experience with an LMS that they prefer to UNM Learn. The sample is large enough to gather some useful information about comparisons between UNM Learn and other LMSs. But it isn’t large enough to have high confidence in the accuracy of those comparisons nor whether they would reflect the attitudes of the entire faculty should UNM switch to a different LMS.Faculty Use of UNM LearnQ21 asked UNM Learn users how (which features) they use. The top responses in decreasing order of frequency are:Distributing documents and learning materials (95%)Gradebook (74%)Online assignment/paper submission (63%)Quizzes/exams (52%)Class-related communication between the instructor and students (50%)The least frequent responses were web conferencing (8%) and student-to-student messaging (14%). Use of group collaboration, graded discussions, and analytics were also relatively low (17-22%). Note that Q21 was only asked of faculty members that currently use UNM Learn. Thus, it’s an open question whether past or present users of other LMSs use those LMS in ways similar to the way other faculty use UNM Learn.Responses concerning class delivery modality (fully online, hybrid, and flipped) showed no clear pattern. Fully online and hybrid had a nearly equal number of responses (15-17%). Flipped classroom was about half as many responses (9%) but that isn’t surprising given the relative newness of that class delivery modality. By omission, we might be able to assume that the respondents that didn’t choose any of the other modalities use UNM learn to teach traditional face-to-face classes (up to 59%).Q22 measured faculty satisfaction with UNM Learn feature-by-feature and overall. Overall, approximately 59% of respondents that currently use UNM Learn were satisfied or very satisfied, 21% were neutral, and 19% were dissatisfied or very dissatisfied. 53 respondents answered yes to Q23 (use another LMS they prefer over UNM Learn). If those respondents are assumed to be less-than-satisfied users of UNM Learn then the overall satisfaction for UNM Learn is somewhere between 49% and 59%. Exactly where in this range is the correct value is unclear since there’s no way to know how many respondents that answered yes to Q23 also answered Q22 (i.e., they currently use UNM Learn and currently use or previously used another LMS).Faculty Non-Use of UNM Learn88 respondents answered no to Q19 indicating that they do not currently use UNM Learn. Q20 asked why and provided a set of specific responses and an open-ended response option (choose all that apply). Of the specific responses, the most common in descending order of frequency were:Ease of use for faculty (18.6%)Ease of use for students (10%)Usefulness (lack thereof) of LMS tools (10%)Ease of maintaining personal websites (9%)Availability of training (9%)Flexibility of the system (9%)Open-ended responses underscored the above responses with emphasis on poor ease-of-use, system clunkiness and inflexibility, and the utility of other automated tools such as Dropbox, Google Classroom, and traditional web sites for supporting paring the latter point with the responses to Q21 and assuming that users of both UNM Learn and other LMS use them in similar ways, it’s apparent that many faculty prefer other tools to support the relatively small number of LMS functions that they employ (e.g., document distribution, gradebook, and communication with students). A few of the open-ended responses to this question and to Q27 also noted that utility of using tools such as Drobox that students already know and/or will use after graduation. Ease-of-use also factors into the reasons to use other tools and LMSs as shown by the large number of respondents that point out ease-of-use as either a reason not to use UNM Learn or to be less-than-satisfied with UNM Learn.Faculty Experience with Other LMSs49 respondents described other LMSs with which they have experience. The top responses were:Canvas (16)Other (14)Specific descriptions accompanying the “other” responses were all over the map. Other specific alternatives such as Moodle and Desire2Learn had 5 or fewer responses. The sample sizes for the various UNM learn alternatives are all too small to draw definitive comparisons to Blackboard or among the alternatives.Q25 asked why respondents preferred another LMS to UNM Learn. Similar to responses to earlier questions, ease-of-use (lack thereof) was the most frequently cited reason. Other frequent responses included LMS tool usefulness, flexibility, and reliability/stability.Open-ended responses to Q27 underscored the pre-supplied response alternatives with many anecdotes describing UNM Learn’s poor ease-of-use and other shortcomings and praising the relative strengths of a chosen alternative.Finally Q26 asked if current users of other LMSs would invest the time to transfer content from UNM Learn to their preferred LM. Of 46 respondents, 37 (80%) said yes.Putting It All TogetherThe picture painted by the survey responses is complex. It’s provides neither a strong affirmation of UNM Learn nor a strong mandate for change. As such, the survey can be mined for pieces of information that will support one direction or another. But defending either choice will require further effort to fill in the missing pieces. As such, analysis and discussion of the survey results is only a starting point for deciding whether/how to improve, upgrade, or replace UNM Learn.Some key takeaways from the survey include:UNM Learn is underutilizedEase-of-use, complexity, and training are problem areasUNM Learn Under-UtilizationSome UNM faculty don’t use any LMS. Some use an LMS other than UNM Learn. Some faculty cobble together LMS-like capabilities using tools such as web sites and GoogleDocs. Among faculty that do use UNM Learn, many use only a handful of its functions.Whether underutilization is a “problem” is a matter of perspective – faculty, student, or institutional. Since faculty aren’t mandated to use UNM Learn and since their opinions of its utility vary widely, it’s little surprise that there’s significant underutilization. Faculty teaching online and those teaching large classes have obvious motivations to use the LMS though some of those faculty choose to use tools other than UNM Learn. At the other end of the spectrum, some faculty simply see no need for UNM Learn. For faculty teaching classes such as those with low enrollments, infrequent assignments, and minimal document distribution, relatively small benefits (e.g., 7-24 access to documents) accrue only to the students with the costs of learning to use UNM Learn and maintaining course-related content being borne by faculty and support staff. Given the number of faculty avoiding UNM Learn or complaining about ease-of-use, it’s reasonably clear that a significant number of faculty members don’t value the perceived benefits to themselves or student more highly than the costs that they themselves incur.The faculty survey doesn’t illuminate the student perspective though a separate student survey with LMS-related questions to be completed in May will partly fill that gap. There are some obvious benefits to students from a well-used LMS including accessibility of documents and learning materials, access to grades, online assignment submission and feedback, collaboration with instructors and peers, and more efficient learning. But these benefits are only realized if instructors make good and full use of its features. As the survey results show, many faculty use the LMS as a combination of a static web site, assignment drop box, and Internet-accessible grade book. Features such as calendar integration, online quizzes/exams, modularized and programmed learning, and learning and web site analytics are seldom used. Yet it is these features that LMS-related research points to as having the most significant impact on learning outcomes.Another student perspective is that of tool overload. To the extent that faculty use different LMSs, varying tool sets in place of an LMS, and use UNM learn in widely different ways, students face a relearning process with each course they take. They must know how to use multiple LMSs/tools and must adjust to the myriad ways in which LMS/tool use varies from class to class. Some students negotiate this maze easily but others struggle. It seems obvious that there must be some negative impact on learning outcomes and grades for some students – though there is no research to back up that assumption.LMS underutilization has multiple impacts on UNM. These include:Inability to tightly and uniformly integrate other systems with the LMSLoss of learning outcomes that might or could be achieved through better LMS useComplicating support for faculty and studentsModern organizations are expected to use technology to seamlessly connect various organizational functions and systems. At its best, this integration improves quality and efficiency and reduces costs. But achieving those results requires the right technology and widespread or universal use. Lacking those features, the benefits are poorly achieved. Early alerts through LoboAchieve are a case in point. They have yet to deliver their promised benefits due to underuse and poor integration with UNM Learn. Achieving UNM-wide learning improvement through learning analytics is another case in point. Too few faculty use UNM learn in a standardized enough way to provide sufficient learning data for mining, analysis, and downstream improvement efforts.Ease-Of-Use, Training, and Support – A Gordian Knot?Many current UNM Learn users cite ease-of-use and training and reasons not to use UNM Learn or as barriers to its effective use. At the same time, support organizations report low utilization of available support and training services. This apparent paradox has left both sides unhappy and struggling for a solution. Training and support complexity is a cost borne throughout UNM though it’s primarily concentrated within Extended Learning (EL) and, to a lesser extent, IT Services. EL supports faculty and student users but it struggles to do – chiefly because of high resource requirements, UNM Learn’s complex user interface and features, and the myriad ways in which it’s used from course to course. In many ways, the freedom that faculty have to choose their own LMS/tools and decide exactly how to use it creates a training and support quagmire. Faculty who see significant benefits to LMS use and take advantage of available training and support services are generally satisfied. But many faculty underutilize available services.The issues of ease of use, tool complexity, training, and support are tightly intertwined. Poor ease-of-use and high tool complexity generates greater need for training and support. But it’s unclear whether adequate training and support can ever be a sufficiently strong antidote to what some faculty view as a much-too-complex and clunky LMS. Would upgrading UNM Learn or switching to another LMS solve this problem? That’s a question that they faculty survey wasn’t design to answer directly.Where To From Here?The purported benefits of widespread LMS use by faculty include:Enabling hybrid and online delivery modesSaved time/effort for common instructional tasksImproved instructional outcomesCapturing student performance data to enable downstream capabilities such as alerts, timely intervention, learning analytics, and data-driven improvementThe faculty technology survey provides some evidence that UNM is achieving the first benefit. The results paint a noisy picture with respect to the second and third benefits. Some faculty have achieved those benefits but others haven’t – primarily due to LMS complexity and underutilization. The fourth benefit is unachievable without much higher and more effective LMS utilization. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download