Macquarie University | Tertiary Education
[pic] |Emerging Technologies Grant Scheme
2008 Project Final report form
| |
1. Title of Project
2. Project Leader and contact details
3. Completion date
2009 Adobe Connect Trial – Executive Summary
Report prepared by: Dr Matt Bower
On behalf of: Dr Matt Bower, Dr Meeri Hellstén, Christa Jacenyik-Trawoger
This report summarises the findings derived from the Adobe Connect Trial that occurred during Semester 1 of 2009 at Macquarie University. For the full report please see the Adobe Connect 2009 Trial Evaluation document. This trial was funded by a MQ Emerging Technologies Grant awarded in October of 2008.
The technology
Adobe Connect is a web-conferencing system that provides the following facilities through any Flash enabled web-browser:
• General Presentation Delivery – PowerPoints, general documents
• Webcam – multiple speeds, ability to stream
• VoIP – adjustable broadcast quality to suit connection
• Text Chat – send to all or selected individuals
• Whiteboard – various colors/fonts/transparency levels, drag-and drop, undo, document overlay capabilities
• File Upload/Download – selected from computer or Breeze content repository
• Polling – with presenter access to individual responses
• Attendee List – including status indicator (‘fine’, ‘slower’ etc)
• Web Launcher – launches all users to the same URL
• Notepad – to summarise, provide instructions, etc
• Screen Sharing (Windows only) – desktop or window, with remote control capabilities.
[pic]
Figure 1: Screenshot of web-conferencing system showing pods (clockwise from top-left) ‘Camera and Voice’, ‘Share’, ‘File Share’, ‘Chat’, ‘Note’ and ‘Attendees List’
The meeting ‘host’ (or super-user) can spontaneously adjust the access control of ‘presenters’ and ‘participants’ to each of the tools. Each of these tools (or ‘pods’) can be instantly resized, drag-and-dropped, created or deleted. As well, a room can have several pre-designed layouts, which can be navigated between at the click of a tab. Up to five breakout rooms can be created based on one of the layouts in the meeting, allowing groupwork collaboration on tasks. Finally, all sessions have the capacity to be recorded.
The Teaching and Learning Context
Five academics volunteered to complete the teaching and learning context survey, providing information about a subset of the trial participants. Of the five units, two were from the discipline of education; the remaining three units were from actuarial studies, statistics and linguistics. The number of students expected to participate in each of the units were 8, 8, 20, 21, and 45. Unit convenors used Adobe Connect to provide external students with real time and interactive access to lecturer and student presentations and to discussions; another purpose was to provide recordings for archival access. Except for one unit, which was offered exclusively to off-shore students, all other units offered enrolment to both internal and external students.
A broad range of discipline areas were represented, from statistics, to education, to actuarial studies and linguistics. The structure of the units in which Adobe Connect was being used varied widely, to incidental small group discussion sessions, to optional 2 hour weekly discussion groups to weekly 3 hour classes to six full day classes. Participants opted into the trial because they valued the ability to hold online synchronous meetings with students, the ability to communicate using a variety of modalities (for instance audio, Powerpoint) and the ability to record sessions. Two units planned to use Adobe Connect for assessment purposes. In both instances it was anticipated that students would be required to give presentations using Adobe Connect.
For more detail about the teaching and learning context, see Section 4.1 of the Adobe Connect 2009 Trial Evaluation document.
The Capability of the Technology
A “Software Capability Analysis” survey was distributed to both unit convenors and students who participated in the trial. Unlike the “Teaching and Learning Context” survey, which focused on characteristics of the participants and units, this survey concentrates on the usability and utility of the Adobe Connect system for teaching and learning purposes.
Eight participants completed the survey. The survey instrument did not record whether respondents were staff or students. Again, the low number of responses does not allow for statistically significant conclusions to be determined. Instead, the survey can be regarded as a viewport into participants’ perceptions of learning and teaching with Adobe Connect, from which qualitative insights can be drawn.
For a complete description of Software Capability Analysis results please see Section 4.2 of the Adobe Connect 2009 Trial Evaluation document.
1 Learning and teaching processes
1 Extent to which the technology supported pre-identified teaching and learning processes/activities
Five of eight respondents (57.5%) found Adobe Connect to be “very effective” or “effective” in supporting teaching and learning activities, in contrast to 3 respondents (37.5%), who rated it as “not very effective”.
Adobe Connect supported a range of pedagogical practices that otherwise could not have occurred:
• Remote presentations by lecturers (interactive teacher centred modes)
• Remote presentations by students to peers and lecturer (peer learning activities)
• Class discussions involving both internal and external students (multimodal learing)
• Creation of a distributed yet interactive learning community
• Self-access learning using archived presentations, materials and activities (multimodal, flexible learning).
In terms of pedagogical practices that supported research candidate supervision, Adobe Connect allowed:
• supervisor – candidate collaboration on documents (mentoring and peer learning, co-authoring, work based learning)
• students to interact not only with supervisors but with each other
• better time and task management (by enabling more effective communication)
• supervisors to offer academic and affective support to their candidates remotely.
To not have web-conferencing at the University restricts the remote pedagogies that can be applied, for both classes and higher degree research supervision.
2 Strengths and weaknesses of the system
Some of the reported difficulties can be directly attributed to the use of Adobe Connect, while others are independent of the software. The difficulties experienced and the need for further software training are factors attributable to the software. One lecturer felt that problems with the video function disadvantaged online students because they could not see the lecturer or other students.
One lecturer thought that using Adobe Connect was “better than nothing”, but would have preferred a different system (Live Classroom). Another lecturer stated that Adobe Connect was well suited to the limited purpose it was used for. A software-independent problem related to difficulties in combining face-to-face students with online students in the same classroom. In particular, this lecturer found it difficult to ensure that online students could hear face-to-face students. As indicated earlier, this was ultimately resolved by using a roving microphone.
When asked whether there were any things Adobe Connect could not do, survey respondents described technical problems mostly related either to the audio or video function. Audio problems included feedback and the lecturer’s microphone slipping out of range. With regards to video, one lecturer would like all participating students to be able to see both each other and the lecturer and another reported that they could not attach large video files. One request which is neither related to the audio or video function, was the possibility (or ability?) to support languages other than English. And finally, one lecturer found it difficult to monitor the lecture while at the same time drawing attention to questions posted in the chat pod, which relates to general web-conferencing skills development. The range of issues identified here indicate that web-conferencing should not be viewed as a simple and magical “do-all” fix for remote teaching but rather as a tool that can support remote teaching that requires a degree of user expertise to operate effectively.
Only one respondent thought that the ability to flexibly adjust the interface supported collaboration, and only once users had become familiar with the system. The remaining survey participants said that they either did not use this feature or described problems. The ability to flexibly adjusting the interface to meet the collaborative and cognitive demands of the learning episode was one reason for trialling the Adobe Connect system above other systems. However it appears that novice users generally do not capitalise on this feature and may need training and so support to understand how to leverage this potential.
3 Issues encountered / potential issues arising in other contexts
In terms of capabilities that respondents identified as important when using Adobe Connect, respondents identified both technological and pedagogical skills. Although, in general, no particular technical skills beyond general IT knowledge were necessary to use Adobe Connect, survey respondents said that technical problems required them to develop trouble shooting skills. Pedagogically, one respondent also pointed out that new strategies are required to interpret student progress and to stimulate interaction in the virtual classroom environment. For our teachers to facilitate successful web-conferencing sessions they need to possess both technological and pedagogical capabilities (highlighting the critical role of professional development to be able to use the web-conferencing system effectively).
The majority of teaching tips and advice related either to technical problems on one hand and blending delivery of face-to-face and online students on the other. Technology-related advice mainly addresses managing technological problems, such as having a back-up plan in case problems occur or being flexible with starting and finishing times until all participants are sufficiently familiar with technology. One respondent suggested encouraging the use of reaction buttons (agree, disagree, hand raising, etc.).
With respect to combining face-to-face classroom delivery with online participation, survey respondents suggested the use of a roving microphone to ensure online participants can hear contributions of students attending face-to-face as well as planning activities for online participants while face-to-face participants are engaged in group discussion. Others advised to include structured sections or stated that they themselves needed more experience in aligning face-to-face participation with online participation. One respondent suggest they would use of Skype instead of Adobe Connect, another thought that online participants preferred the flexibility of asynchronous delivery. Combining face-to-face teaching with web-conferencing requires an added level of expertise because to some extent the teacher needs to simultaneously manage, monitor and integrate two environments.
Respondents also identified several important issues in relation to quality assurance and compliance, of which the main concerns were copyright, intellectual property and plagiarism. The uploading of files which are made open to all participants, elicited concerns about the protection of intellectual property and potential breaches of copyright laws. In addition, there was concern about quality assurance because the originality of external students’ work was difficult to track. This is particularly of concern in postgraduate supervision contexts where unpublished ideas and research data are seen as potentially sensitive to unwanted exposure prior to publication. It should be noted that these are of concern in any situation where users have the capacity to upload digital content, and it is not related to the Adobe Connect or web-conferencing technology in particular.
2 Acceptance /usability
Participants were asked to rate their overall satisfaction with Adobe Connect (from “highly satisfactory” to “highly unsatisfactory” and add comments where appropriate. A summary of the Likert scale responses are provided in the figure below.
[pic]
Figure 1 – Participant overall satisfaction with Adobe Connect
Responses indicated that generally people were satisfied with the trial. Overall, Adobe Connect was rated predominantly “satisfactory” in each of the areas of access, ease of use, reliability, speed and connectivity and screen layout. However, with the exception of “screen layout” each feature also received “unsatisfactory” ratings and comments highlight some problems with this software. In particular, comments by some participants called into question the stability of the system. Respondents reported difficulties with the accessing of online meetings or archives and meeting participants dropping out while in sessions. Also, using the video tended to occasionally freeze the view function.
In response to the survey question relating to the advantages of Connect, participants identified several positive features of the system. The fact that Adobe Connect offered a virtual classroom with all elements of a traditional classroom was regarded as positive. However it was noted that this required navigating through pods, which was regarded as a difficulty, particularly when pods had been moved around the screen. Another advantage was in the ability to bring together external and internal students in the same classroom. When doing this with large face-to-face classes it was pointed out that a roving microphone should be used so that on-campus students could be heard by their online peers. Other advantages included features not found in other e-learning platforms, the ability to share desktops to demonstrate software.
In terms of disadvantages, respondents felt that the main problem they experienced with the system was with the audio. “Discussions (audio)” received the highest number of “unsatisfactory” responses (3) and related comments highlight problems with the usability and reliability audio function. In some cases users resorted to chat mode to overcome audio issues. Respondents also expressed problems with accessing archives, using the video function and/or the audio or video not working. In addition, several technical difficulties were reported, such as broken links or presentations only being partly recorded (voice recorded, but not slides). Technical problems were also seen as a reason for making Adobe Connect unreliable for assessment purposes and for causing some anxiety. One respondent suggested that it would be better to share documents in the meeting room, rather than having to download it at either user end. This latter suggestion is actually possible, which indicates that in some circumstances problems can be caused by user capability rather than software functionality.
1 Ease of use
Participants were also asked to identify which features of Adobe Connect they used, to rate their level of satisfaction (“highly satisfactory” to “highly unsatisfactory”) and to provide explanatory comments, where appropriate. . A summary of the Likert scale responses are provided in the figure below.
[pic]
Figure 2 – Participant rating of features of Connect
From the figure it can be seen that the highest rated functions of Adobe Connect were “Screen Sharing”, “Chat”, Whiteboard”, “Note pods” and “Breakout rooms”. Although not used by all trial participants, they uniformly received “highly satisfactory” and “satisfactory” ratings. The remaining functions, namely “Teacher/student presentations”, “File sharing”, Recording for later use” and “Discussions (audio)”, rated predominantly “highly satisfactory” and “satisfactory”. However, each of these functions also received “unsatisfactory” ratings. Thus while there was general approval for the tools, problems were experienced with some of the multimedia elements of the system.
2 Requirements for training and support
On the whole students and staff appeared satisfied with training and support, with staff satisfaction rated higher than student satisfaction. It appears that not all trial participants had attended training sessions provided at the beginning of the trial. At least in one case training was provided by the lecturer rather than by project coordinators. Two other respondents stated that they had not received any training. Similarly, at least one participant was unaware of available documentation, despite several emails notifying participants of the online resources that had been made available. Some participants also noted that no amount of training could overcome some of the technical problems that were being experienced. However, comments also showed an appreciation for the support and coaching received.
The Adobe Connect web-conferencing system requires considerable time-per-user to support (both in terms of providing training and troubleshoot help), and if the system were to be implemented across the university then adequate funds would need to be allocated to providing such support.
Nearly half of all people who requested and were issued with accounts did not end up trying Adobe Connect. The effort required for people to change their modus operandi was for many too great, causing them to revert to familiar (but potentially less effective) behaviours. This indicates that energy is required just to help users get started with such systems if the efficiencies resulting from the use of such systems are to be harvested.
3 Potential to add efficiencies, reduce or increase workload
On the whole respondents indicated that the ongoing workload was “manageable”. Only “preparing materials” received an “unmanageable” rating; one “highly unmanageable” rating was received in the “other” category. Comments indicate that “setting up” and “archiving” were experienced as time-consuming tasks. One respondent reported increased workload caused by the organisation and preparation for teaching in an online setting; however, this was regarded as manageable. The fact that the workload associated with teaching online was generally perceived as manageable is reassuring in terms of encouraging staff to experiment with web-conference based teaching.
3 Hardware / software
Adobe Connect operates on a web-based client-server architecture. Users can access the system as long as they have a web-browser and are connected to the Internet. If users are running an old version of the Adobe Flash Player they may be required to update to the latest version. On first time use users may be prompted to download a plugin that takes approximately 20 seconds to install. This enables some of the more powerful features of the system such as desktop sharing to operate.
The requirements for the Adobe Connect Server machine are reasonably high end, but not excessive (see ). Note that it is also possible to purchase a hosted service which obviates the need to purchase and maintain a server.
Organisational impact
1 Student and staff security, privacy and confidentiality of content and user information
There were no reported issues of student or staff security breaches, other than one instance of the server being hacked when the software was first installed without any antivirus software. Even after the system was reinstalled with anti-virus software he system administrator was still not sure whether the server was actually compromised or whether it had not been configured correctly.
Each user has their own account on the Adobe Connect server and there is a sophisticated permissions system that can be used to allow or prevent access to conference rooms, recordings and resources.
2 Interoperability, integration into MQ IT infrastructure, or other learning technologies including Blackboard
Adobe Connect can authenticate via an LDAP server, allowing it to integrate directly with the existing Macquarie University user authentication system. As it is web-based a hyperlink to a web-conferencing room can be inserted in any document or web-page (including within Blackboard). There are no limits to the amount of rooms that can be established by a user with permissions, meaning that one unit can potentially have several rooms. Adobe also provides documentation outlining how the system can also be integrated directly with Blackboard for single sign-on access (See ).
3 Accessibility – general access and also disability
Adobe Connect is able to adjust the amount of information that is streamed to users based on their bandwidth settings (either dial up modem, ADSL/cable, or LAN). However for an optimal experience it is recommended that users have at least ADSL/cable access to the Internet.
The system allows contribution via text-chat, audio, webcam, whiteboard, screen-sharing, PowerPoint presentation, notepods, voting tools and file uploads. To this extent there are a range of modalities through which users can communicate. If a user has difficulty with listening or speaking then they have visual and textual modalities through which communication can be mediated. Similarly for students with poor literacy a non-textual modality such as audio of visual representation may be used. Because of the multi-modal nature of web-conferencing systems they generally provide a range of alternatives with which students can participate.
Because the system is available online students can have synchronous access to lessons from their home or workplace. The fact that the system allows recording of all sessions means that users with accessibility issues can access the instruction at a later time and process it at a speed that suits them. Thus, through the range of modalities that the system offers, the ability to access rooms and resources from off-campus, and the ability to record all interactions that occur in the environment, Adobe Connect offers greater accessibility than students currently experience in face-to-face lectures.
For a full Section 508 accessibility report for Adobe Connect please see:
4 IP and copyright
Students and academics may choose to upload and broadcast copyright resources in their web-conferencing sessions. As such they need to be aware of copyright legislation and how to abide by it, just as if they were operating in a face-to-face environment. Users need to be particularly mindful of this in light of the recordings of the sessions that they may chose to make available.
Academics concerned about Intellectual Property issues should be careful before they distribute resources via the system, in the same way that they would take care when making resources available via Blackboard. The fact that the information is in digital form makes it particularly simple to replicate and disseminate.
5 Information Management
Adobe Connect comes complete with a resource management system, allowing users to upload, organise and share resources. Each user has an administration area for them to create and manage their meeting rooms. Meeting hosts can also access, edit and organise recordings of sessions.
Within a room the resources that have been previously shared in a room are available from a drop-down menu (for easy retrieval). All text-chat areas, whiteboards and notepods can be cleared at the meeting hosts’ discretion. All transactions and contributions during the meeting can be retrieved from the recording.
6 Use for assessment
Adobe Connect can be used to facilitate remote assessment. For instance, students can be asked to deliver a PowerPoint presentation to the rest of the class. The effectiveness of this approach is dependent on the reliability of the system. In the capability analysis survey one academic noted that the fact that an assessment they conducted was online caused extra anxiety because students had to navigate the technology at the same time they focused on delivering the content. Another academic indicated that they did not use the system for assessment because they did not perceive it to be sufficiently reliable.
Sustainability and Cost
Adobe Connect requires ongoing monitoring and maintenance. As well, a considerable degree of expertise is required to ensure that the server is performing optimally. If Adobe Connect is to be adopted as a part of Macquarie University’s general teaching and learning infrastructure or even as a limited service for some staff and Departments then adequate funding is required to either ensure that a technical officer is allocated appropriate time and training to manage the server or external management services are purchased.
Gavin Perry from Webqem (Australia’s leading distributor of Adobe Connect) provided the following indicative pricing information on the 18th of September:
Connect Pro Server Application:
Approximately A$8,500 Acrobat Connect Pro software licenses lets your Institution host and manage any combination of Acrobat Connect Pro applications from within your company's firewall. This is required for the Acrobat Connect Pro Meeting server to run.
Acrobat Connect Pro Meeting:
Approximately A$171 per Named Organiser license, $309 per Concurrent User, A$21 per Seminar Room seat
Named Organiser: Each individual Named Organizer has the ability to host a meeting with up to 100 total attendees. Named Organizers may create an unlimited number of meeting rooms, however, the Named Organizer can only use one of their rooms at any one time and rooms cannot be accessed unless the Named Organizer is present. Named Organizers must be individuals, not groups or generic logins, and a Named Organizer license cannot be shared between more than one individual.
Concurrent User: A concurrent Connect Pro Meeting user has the ability to attend or host a meeting on the Connect Pro.
Seminar Room: The room size is set based on number of seminar room seats purchased per room, only one meeting can take place in a seminar room at any given time, and a designated Seminar Room Host must be present in the room for the room to be accessed. Minimum purchase quantity of 200 concurrent seats per seminar room. Maximum of 1,500.
All prices above are for a licensed version of Connect Pro. For external hosting Macquarie University would still purchase the software outright, but instead of our IT department hosting and managing the servers, a company like Webqem would. Webqem suggests the hardware including installation would cost approximately $13,000 per server. Only one server would be required assuming the university does not exceed 350 concurrent users with the Connect pro software. If we wanted to separate the database and load balance, then three servers would be required. Webqem’s Bandwidth A standard package that most of their customers are on is A$900 per month which provides 5GB inbound, 40GB outbound per month.
Connect Pro site-wide:
A site-wide quote is difficult to provide without more details of Macquarie’s requirements however, based on an FTE of 2,000 we would be looking at somewhere in the region of A$280,000 plus the 20% M&S. This at this stage is regardless of whether you go named or concurrent.
Maintenance & Support:
Maintenance & Support is calculated at 20% of the total purchase price. Maintenance and Support is mandatory in the 1st year only however, maintaining your Maintenance and Support future proofs your investment and ensures you receive all minor and major updates including service packs, and 1st line technical support directly with Webqem.
The above prices are indicative, in Australian Dollars, exclude GST, are subject to changes in the US dollar exchange rate. Webqem is happy to provide a formal quote once Macquarie provides a clear definition of business requirements.
1 Ability for the technology to grow to meet potential future requirements?
Adobe Connect can service thousands of users simultaneously, however the infrastructure and license needs to be upgraded to support requirements. For this trial we used a 50 concurrent user license which was sufficient to meet the needs of the university during the pilot program. Not everyone in the university (staff or students) is likely to want to use Adobe Connect as soon as it is offered. However the system is scalable up to full campus deployment if required.
2 Likely costs for maintenance, upgrades, training, support
This is highly dependent on the number of users and will most likely change over time as demand for the system grows.
Administering this trial took an inordinate amount of time in terms of maintenance and ongoing support. Any campus-wide implementation of the system should factor in the employment of ongoing staff member to manage the system, training and support. Depending on the level of demand for the system, this could range from 0.2 to 1.0 equivalent full time load.
3 Costs that will or are likely to be imposed on students
The system would impose little or no cost on students other than the bandwidth and computing resources that would be required to access the system. Students who did not have earphones or a microphone for their computer would need to purchase these if they wished to contribute via audio.
4 Feasibility for Faculties, the University and students (incl. costs, upgrades, time resources etc.)
Running this system is highly feasible, and contingent issues have been outlined above. Many other universities run some form of web-conferencing software for staff and students. Faculties and students could run off a centralised Macquarie University server configuration as easily as they can access the Internet.
Overall assessment of the technology
1 Potential for supporting other learning and teaching processes / contexts
Web-conferencing software such as Adobe Connect is the best technological solution available for supporting synchronous online teaching and learning. However it is also possible to run the software in other teaching and learning contexts, for instance:
• Within computer lab tutorial classes to allow multiuser text-chat contributions
• Within computer lab tutorial classes to enable groups of students to draw diagrams using the whiteboard-tool
• Within a lecture to record the presentation and provide a text-chat feed.
The system could also be used for non-academic purposes to provide remote technical support, facilitate collaboration between general staff, and so on.
2 Potential for use across other disciplines, faculties or across the whole university
This trial was run across disciplines and Faculties. There is nothing to prohibit its use across the entire range of subject offerings at Macquarie.
3 Similar or alternative products on the market
Similar or alternative products include:
• Elluminate Live ()
• Wimba Classroom ()
• Dim Dim ().
Elluminate Live and Wimba Classroom are the main competitors to Adobe Connect in the virtual classroom web-conferencing space. Recently open source (less full featured but nevertheless reasonable) alternatives have been entering the market, such as Dim Dim.
4 Assessment of suitability for further investigation, resting or escalation of use with your Faculty /Department or across MQ
A number of Adobe Connect licences have been purchased and trialled across the University. The potential positive impact of this software upon learning and teaching at Macquarie warrants escalation of the system. If the reliability of the system can be established then Adobe Connect provides a vehicle to broaden our customer base, transform our capacity to teach live online classes, and improve the accessibility of our courses. Reliability may either be established by subcontracting external parties to install and maintain our server or opting for a hosted contract with financially bound quality of service criteria.
[pic]
Emerging Technologies Grant Scheme
Adobe Connect Trial Evaluation
Prepared by:
Dr Matt Bower
Dr Meeri Hellstén
Christa Jacenyik-Trawoger
31st September 2009
Introduction
This report outlines the context, method, results and conclusions from the Adobe Connect Trial that occurred during Semester 1 of 2009 at Macquarie University. This trial was funded by a MQ Emerging Technologies Grant awarded in October of 2008.
Background
Based on the perceived need for a web-conferencing system at Macquarie University and the particular value of having such a system to facilitate collaboration between postgraduate research students and their supervisors, the project leaders (Dr Matt Bower and Dr Meeri Hellstén) applied for and received Emerging Technology Grant Funding to trial the Adobe Connect web-conferencing system at the university.
Although Adobe Connect had been successfully trialled previously in the Division of Information and Communication Sciences Graduate Diploma of Information Technology, the outcomes indicated a need for investigation and evaluation of the system in other contexts before the system could reliably be integrated into the Macquarie’s central services or offered to a large sub-section of the university. As well, research conducted by Dr Hellstén into the provision of resources for international students indicated student demand for greater infrastructure (including IT) to facilitate contact with the host institutions.
The trial also provided a valuable point of comparison to the other web-conferencing system being evaluated at Macquarie University (Horizon Wimba). The Adobe Connect system included features that Horizon Wimba did not, including:
• the capacity to redesign the interface based on the communicative and learning requirements of the task
• the ability for each teacher to manage as many classrooms as they require using a stand-alone interface
• the capacity to show and easily manage PowerPoint presentations that include all animations
As well, the latest version of Adobe Connect had additional features that were not present in the 2005-2007 Division of Information and Communication Sciences trial, including:
• increased security
• the capacity to instantly create break-out rooms
• a specialised virtual classroom tool.
The above points provided grounds for trialling this web-conferencing system on campus.
A final part of the rationale for implementing this trial was that once it is setup, access could be provided to members of the Teaching and Learning team and volunteer academics from across the university for trials in other contexts. This furthered the value of this initiative and provided more data with which to validate the efficacy of the technology.
Methodology
Participants were invited to be involved in the trial based on an advertising brochure sent to academics within the Faculty of Human Sciences. As well, lecturers who made inquiries to the Learning and Teaching Centre about the use of web-conferencing in their units for Semester 1 of 2009 were referred on to the project leaders. These requests were honoured by the project leaders in order to provide a broader sample of data with which to evaluate the system.
Training took the form of a one-hour workshop offered in December (approximately 20 participants) and repeated in January (for approximately 15 new users). As well, users were referred to online training and support resources provided at the website that was setup for the project: . In the end 49 people applied for accounts to the server and of these there were 26 who actually used the server for collaboration purposes. Note that this does not include users who did not have an account but were granted guest access to the system by a registered user at the time of using a web-conferencing room.
Initially two feedback surveys were envisaged, Teaching and Learning Context (supervisors / lecturers before start of trial) and Software Capability Analysis (supervisors/lecturers and candidates / students); both surveys have been adapted from the Emerging Technology Evaluation Survey developed by Maree Gosper and Andrew Burrell. However the low number of responses coupled with Adobe Connect system data indicated that many who had expressed interest in the system did not actually use it, which resulted in a non-user survey being implemented to shed further light on why the system was not used.
1 Project Implementation
A brief summary of project tasks and their timings is as follows:
• December 08 Purchase of hardware and setup of the server, testing of the system by key stakeholders, advertising the trial and obtaining expressions of interest for participation, holding a training workshop (approximately 20 participants), establishing a project wiki.
• January-February 08 Interacting with interested participants and promoting awareness of the trial, responding to user questions, providing a further training session (approximately 15 participants), solicit initial feedback, ongoing maintenance of the server, administering the Teaching and Learning Context survey.
• March-June 09 Implementation of the trial throughout Semester 1 involving use by academics and students, ongoing observation and collation of data by the project leaders and research assistant, ongoing maintenance of the system, provision of technical support to users, processing the ethics application for the study, administering the Software Capability Analysis survey.
• July-August 09 Finalise analysis, reporting and dissemination of results.
2 Data Sources used for evaluation
Evaluation of the system was based on four data sources:
1. The “Teaching and Learning Context” online survey
2. The “Software Capability Analysis” online survey
3. The “Adobe Connect Non-user survey”
4. Anecdotal observations by the project leaders derived from using the system, managing the system, and providing support to participants.
In many cases raw data from the surveys has been included in the Results section below. The intention of providing this primary evidence is to enable more reliable inferences to be drawn. The Interpretation of Results section that follows summarises the main thrust of the results and posits corresponding implications. The Summative Comments section provides final evaluative remarks, and following this the future direction of web-conferencing is outlined.
Note that the research conducted in this project was endorsed by the Macquarie University Human Ethics Committee (Ref. HE27MAR2009-R06354).
Results
1 Teaching and Learning Context Survey
The “Teaching and Learning Context” survey was intended for unit convenors only. The survey was designed to elicit information about student and program characteristics of participating units as well as their intended aims and outcomes. Five supervisors / lecturers participated in the survey. Although unit codes and contact details for unit convenors were provided they will not be disclosed in this report for privacy reasons. Due to the low number of responses, survey findings are not statistically meaningful. However, they provide insight into the use of Adobe Connect.
The questions and a summary of the responses are provided below.
Question 1: Which unit are you teaching? (unit code or unit name)
Of the five units, two were from the discipline of education; the remaining three units were from actuarial studies, statistics and linguistics.
Question 2: Unit coordinator
This has been withheld to preserve the anonymity of the respondents.
Question 3: Other staff working on the trial
No respondents indicated that other staff were involved in the trial.
Question 4: Expected number of students in the unit
The number of students expected to participate in each of the units were 8, 8, 20, 21, and 45. This indicates that in this trial, Adobe Connect was predominantly used as an online classroom tool, rather than as a tool for research supervision. This could affect the evaluation of the Adobe Connect system, particularly in terms of evaluating it for the purpose of research supervision.
The fact that the system was used as an online classroom tool is also supported by information regarding the purpose of using Adobe Connect (see Question 9 below). Unit convenors used Adobe Connect to provide external students with real time and interactive access to lecturer and student presentations and to discussions; another purpose was to provide recordings for archival access.
Question 5: Enrolment mode
Except for one unit, which was offered exclusively to off-shore students, all other units offered enrolment to both internal and external students.
Question 6: Students' expected level of technical capability – e.g. with internet
The larger than expected number of students in each unit made it more difficult for convenors to predict their students’ level of technical skills and experience with online learning systems. Unit convenors reported an expected average (3 responses) to high (2 responses) level of technical skills.
Question 7: Students' expected previous experiences with Online Learning Systems, e.g. WebCT
Similar to Question 5, students’ previous experience with online learning systems was rated medium to high (“some experience” - 3 responses, “experienced” – 2 responses).
Question 8: Which of the following best describes the delivery mode in your unit?
The delivery mode was described as “fully online” by two unit convenors and as “blended” by three. This corresponds with data from Question 5 regarding enrolment mode.
Question 9: Provide an overview of the unit which describes the context in which Adobe Connect is to be trialled, e.g. the goals of the unit, its structure (lectures, tutorials, independent modules of work, on-campus sessions, etc.), learning activities and assessment tasks and other characteristics that need to be considered.
Descriptions of the goals included:
To provide students with a foundation knowledge of research skills … to equip them with skills to enable the conduct of an independent research project
Masters level introduction to modern [concepts] using a variety of software tools
Exploring the integration of ICT into courses and learning materials
Teach students … material relevant to their … exam
The Unit is constructed on the basis of providing an introductory breadth of understanding of … education with capacity for in-depth explorations of areas of individual interest and need.
This information again revealed that Adobe Connect was used for online classroom tuition, rather than post-graduate supervision.
The structure of the units in which Adobe Connect was being used varied widely, as indicated by responses:
modules of work that are worked to individually, but with some seminar components to solidify understanding
One 3-hr class per week for 13 weeks
online resources, discussion forum and f2f seminar (available to both on campus and off-campus thru Adobe Connect
6 one-day teaching sessions of 6 hours duration
Seven discussion groups are built on leaning community principles. Adobe Connect was offered as a voluntary trial to introduce each area
Assessment strategies included assignments, projects, portfolio entries and exams.
Question 10: How do you propose to use Adobe Connect? What do you hope to achieve?
Responses included:
We use this as a surrogate classroom to hold seminars and discussions between students and the instructor.
The course is 80% finished. I would have liked to use AC for group conferences and presentations, but I've used it more for teacher presentations and recordings.
Enabling student participation in real time seminar. Recording for archive access. Students present using Adobe Connect
I propose to use adobe connect to enable distance students to view the presentation and listen to the dialogue that goes with it and to allow them to participate interactively
It was offered weekly on a voluntary basis as a value-added component. PPoint presentations that were used and summaries of all discussions that resulted were shared with all participants in the unit.
Question 11: Will Adobe Connect be used for assessment purposes?
Two of the five respondents indicated they would use Adobe Connect for assessment purposes; the remaining three units did not involve Adobe Connect in the assessment process.
Question 12: If Adobe Connect is used for assessment purposes, please provide details.
The two units that used Adobe Connect for Assessment purposes required students to give presentations using Adobe Connect.
2 Software Capability Analysis Survey
The “Software Capability Analysis” survey was open to both unit convenors and students who participated in the trial. Unlike the “Teaching and Learning Context” survey, which focused on characteristics of the participants and units, this survey concentrates on the usability and utility of the Adobe Connect system for teaching and learning purposes.
Eight participants completed the survey. The survey instrument did not record whether respondents were staff or students. Again, the low number of responses does not allow for statistically significant conclusions to be determined. Instead, the survey can be regarded as a viewport into participants’ perceptions of learning and teaching with Adobe Connect, from which qualitative insights can be drawn.
The questions and a summary of the responses are provided below.
Question 1: Rate your overall satisfaction with Adobe Connect – “highly satisfactory” to “highly unsatisfactory”. Please add comments, where appropriate.
A summary of the Likert scale responses are provided in the graph below.
[pic]
Comments included:
I have encountered some difficulties with accessing the program - and some students with limited access to the Internet due to institution requirements have also had issues with accessing both the meetings (live) and the archives of these meetings. The screen layout is good, although somewhat cluttered at times - although you can change these preferences, it would be good if there was a more simple layout to choose. There were some issues with the video streaming so I didn't use this. It didn't work well when all students were connected and interfered with the system causing it to crash.
we had a lot of trouble getting the voice connection established. the other thing about access is that it does not have an online component that allow people to see who comes online and then meet up - it takes phonecalls or emails or other means to decide on meetings which does not encourage use
AC was quite 'flaky': it happened several times (when trying to use it in conference mode) that individuals 'dropped out' (couldn't hear, couldn't see or couldn't make themselves heard). It got to the stage where it was a bit embarrassing to try to use it in conference mode.
There were several issues during semester where the link was broken or students ‘intermittent' access difficulties. It is much more complicated than other virtual classrooms I have used.
Much difficulty getting on line and requests for help has not been successful
It was a trial and error approach. Oral presentations worked well with the PPoint display. There were considerable problems with the oral interactivity among participants, probably due to the variability in microphones being used.
Question 2: Which features of Adobe Connect did you use – rate your level of satisfaction (“highly satisfactory” to “highly unsatisfactory”) and provide explanatory comments, where appropriate.
[pic]
Comments included:
There was an issue with recording for later use where some students weren't able to access (although not sure if this is a problem with their end or not).
Video use with all students was not feasible, and caused the system to crash, ending the session. We needed to continue the discussion using only audio.
Document has to be downloaded on either ends and then one user has to open it on his/her desktop to work on it. would make sense to be able to open it within the room!
The best aspect was using AC as a recording for later use (I had one student attending 'live', so it felt like giving a mini lecture). That worked well almost all of the time, and was rated well by my students, though there were cases where the sound and/or visuals (eg powerpoint slides) didn't appear.
Sometimes the voice was recorded but the slides were not - we never knew why
Not used much
You should have a not applicable response category for these questions…we did not have student presentations and did not use the whiteboard
As a pilot, only those functions indicated were used.
Question 3: What were the main advantages of using Adobe Connect for teaching and learning?
Comments included:
The power of having a real virtual classroom that could take all the elements of a traditional classroom (white board, note-taking facilities, showing presentations and other items of multimedia) and make these accessible for students who are studying by distance.
The best use was as a recording sharing my desktop, particularly for demonstrating software applications. Most of my students weren't available at the class time, so this seemed to suit them well.
blended environment - a class with on-campus and off-campus students could collaborate and all students could present. Archives of the seminars are also valuable although time-consuming to set up.
being able to broadcast the lecture / presentation to remote students around the globe and to allow them to view / listen and participate in the class
Student reaction was appreciative with a consistent participation among those who are able to log on at the required time. It provided a quick and easy way for students to get into some of the background and significant areas for discussion in a topic area.
Interactive and relatively user friendly. Have features other e-learning platforms (e.g. Blackboard) don't provide.
Question 4: What difficulties or challenges did you or your students face? How did you resolve these?
Comments included:
Having to navigate through the different areas (pods) and when given presenter access to show their own points or show video the temptation of moving things around in the classroom...a simple message to them that they shouldn't touch the pods when they were presenting.
It was too nitty-gritty and did not actually get used but once or twice for what it was expected to be used - working on a document together with a supervisor
When we tried actual conferences, re-scheduled to evenings, the problems of flakiness (described above) got in the way.
The system did not seem robust - sometimes the link was broken and sometimes only the voice recorded and not the slides
initially we got the lecturer to repeat questions asked by the live audience before answering them so the online audience could hear both the questions and the answers and not just the answers. Later we switched to using a roving microphone so both live audience and presenter could be heard by the online audience
arranging audio for remote students to ask questions was difficult (too many of them) so we got them to use the text chat window instead"
Audio discussions were problematic (presenter input was fine), so we reverted to chat
Question 5: If used for assessment, please note any issues or difficulties you encountered. How did you resolve these?
Comments included:
I would have liked to use it for individual presentations of assignments and the final project, but the flakiness prevented this and means that I haven't scheduled its use for the final project presentation.
The presentations are assessed, so tech issues added extra anxiety for students. We planned a backup in case.
Question 6: Overall, was Adobe Connect effective in supporting the teaching and learning activities you identified?
A summary of responses is provided in the table below.
[pic]
Comments included:
I think that [the issue with the video] needs to be resolved before it could be considered very effective. I think this is what is missing from distance teaching (the ability to 'read' others who are in the seminars with you and being able to see the teacher).
not straightforward enough - does not allow to concentrate on work in hand - instead must be IT guru
It was better then nothing but worse than Live Classroom
I think it would be good to receive more guidance and practice to use this facility
We used it for a situation where we had a live audience and an online audience
It is a challenge to focus attention on both audiences - and to make sure the 2 audiences can both participate in the class and be aware of each other's dialogue
It served the limited purpose that I used it for well.
Question 7: Were there things Adobe Connect could not do?
Comments included:
Video for everyone
Attach a large video file
Quality of the audio is something that is sometimes a problem - we get audio feedback
Sometimes the lecturer's microphone slips out of position and the audience can't hear what's being said
Need to have someone present to monitor the broadcast and draw attention to questions being posted on the text chat window
Support foreign languages?
Question 8: Do you have any teaching tips and / or advice you would like to pass on?
Comments included:
You need to be flexible enough to overcome issues with the media when presenting (just like being prepared for a conference and assuming that all forms of the technology might fall down), so not to be too tied to presentations of powerpoint or other multimedia.
I think that it is important to ensure students use and get used to the reaction buttons (agree, disagree, hand raising etc) and that these should be spelled out at the beginning of each session (at least until everyone gets the hang of it) so that any questions from the instructor can get a reaction from students even if they don't want to interject with questions.
Flexibility in terms of start and end times in the beginning as students and instructors get used to the rhythm of the virtual classroom, which is very different to a conventional classroom or even blackboard modules where discussion is usually asynchronous.
Use skype instead
Plan your lesson to cater for:
1.the need for students to speak into the mic etc and
2. include some structured sections and
3. what onliners will do during the small group discussions
We got the roving microphone to work ok and this was helpful for the online audience to hear the live audience's questions and commentary and dialogue with the lecturer
Getting the same to happen for the live audience in respect of the online audience is something we still have to address
Students choose online learning for its any time, any place access. They don't like to be tied in general to regular sessions each week given workplace and family commitments. I liked the way we used Adobe Connect as a value-added and optional component of the unit.
Question 9: To what extent did the ability to flexibly adjust the interface (tools, layout) affect your ability to collaborate?
Comments included:
I found it to be very flexible and once everyone in the room had caught up with the system it made collaborating much easier
It was not flexible where I wanted it to be - working on word document
Sometimes the students got lost when they made adjustments and didn't know how to get back (eg lost pods)
N/A
There was not much collaboration as we mainly used it as a way to broadcast our lecture
We are novice users of the software
We have a mixed class so we're still learning how to make it work better and more interactive for both the live and the online audience
Wasn't used much so wasn't an issue.
Question 10: Please list any particular teaching, learning or technical skills required to use Adobe Connect.
Comments included:
You need to be able to adapt and read signs of student interaction through the sometimes minimal signs you receive from them. You need to ensure that your teaching is student-centred and that you use activities that allow them to interact with both yourself as the instructor and with other students. These are not particular to adobe connect, but are more generally applicable to any virtual classroom.
Brains - to figure out how to make Adobe do things that it was meant to be doing but did not
Nothing that couldn't be resolved by having the system working when I needed it.
Patience
The use of speakers to aid in clear voice transmission
What to do when the server crashes?
None. If I can use it anyone ca.n
General IT knowledge. For 'computer idiots', sometimes it takes time to familiar with the functions of Adobe Connect.
Question 11:Were there any issues that arose in relation to quality assurance and compliance with regulatory frameworks? Comment as necessary.
Items that people selected are indicated in the table below.
[pic]
Comments included:
seems like files up loaded here are freely available to all registered, so intellectual property is not protected
not sure whether particular materials can be put on web without violating the copyright law
like other distance learning mode, it's difficult to track originality of students' work.
Question 12: Rate your level of satisfaction with the training and support provided - “highly satisfactory” to “highly unsatisfactory”
Responses to Likert scale items are summarised in the graph below.
[pic]
Comments included:
I have always felt very able to contact any of the coordinators to sort out issues relating to the program
It was not about training or support - Adobe just does not do documents the way it would make sense to me
The initial training and documentation was useful, though didn't prepare me for the ongoing problems. The coordinator (MB) was very willing to help, but sometimes the issues defeated us
Matt was fabulous, but I provided most support for students
I did not receive any training - we had to learn “on the job” but we hired someone with expertise to help us
There was some excellent 1:1 coaching. Other than that, there was no training or documentation provided.
Question 13: After the initial set-up period, how would you rate the ongoing workload implications associated with the normal teaching and learning process – “highly manageable” to “highly unmanageable”
[pic]
Comments included:
Organising and preparing materials is really related to how you can contact students and find a suitable time for everyone to be online and creating materials that are interesting to all levels and stimulate conversation and interaction. Unlike traditional teaching settings, there are obviously higher workload implications but these can also be manageable.
The setting up and archiving were most time-consuming, but during lessons was fine
Question 14: Were there financial costs associated with using Adobe Connect for you and / or students? Please identify these and comment on whether you consider them sustainable.
All but one respondent indicated that there were no costs or did not respond to this question. The person who did respond to this section posted the following comment:
Yes - costs of hiring someone to teach us how to use the system and assist with running the first few teaching sessions.
Question 15: What were the risks associated with using Adobe Connect and how did you manage these risks?
Four respondents indicated that in their opinion there were no identifiable risks. The other respondents suggested the following risks:
The main risk was the class failing completely as students were unable to stay connected. The only way to manage this was to use AC in a different way.
Dependence on archives (seminars are optional). I put up the slides as a backup
Student dissatisfaction, server crash, audio feedback and other audio reception problems for students
Ensuring equity of access among all students; that is, between those who chose to particiapte in AC sessions and those who did not. Overcome by providing PPoints used and chat notes to all students. PPoints were focused on the relevant chapters of the core text prescribed.
Question 16: Would you use Adobe Connect again for similar purposes?
Seven respondents indicated “yes” while one respondent indicated “no”. Comments included:
probably not because it does not allow me to work on my documents with ease
If Live Classroom wasn't available
with more support
I am thinking of using it to expand our teaching of distance students in a range of other units.
Question 17: Are there more effective and efficient ways of undertaking the activities, etc. and achieving the same outcome?
Comments included:
No. Adobe connect (like other virtual classroom programs) provides an abitlity to actually interact with the students (particularly useful in doctoral programs) and guide students without being restricted to only typed discussions. It allows a level of interaction and learning that is most effective compared to other forms of discussion.
Yes
No
Live Classroom and Elluminate
Yes, more support and more effective responses from teh support network
Probably - using similar web conferencing software
Not really
Some students prefer face-to-face learning and teaching activities to e-learning. There are some learning outcomes can't be achieved in the e-environment.
Question 18: What other uses can you see for Adobe Connect in all areas of your academic work (teaching, research, administration, community outreach)?
Comments included:
For teaching, I feel that this could provide a greater outreach for students who are part of oncampus units (as a way of discussing issues related with other students with an instructor as mediator/moderator)
In administration, as a distance member of staff, I think that this could provide a great vehicle for such members of staff to be involved in departmental activities without needing to be in the office.
In terms of community outreach, obviously teaching programs that are offered using adobe connect can be offered to people who would otherwise be unable to attend classes on campus and provides interaction for them where other distance courses can put some people off because of the lack of external organisation (usually the main negative point is that students in distance education need to manage their own time effectively - with adobe this could be organised by the instrcutor via regular connect sessions).
Meetings - we ran our iLecture project across 4 unis and they system worked well. Because it is a net meetings software it works well in this context (not dealing with a range of stduents)
International coordination and research
It has great potential for use in conducting meetings and for distance learning and distance collaboration with colleagues
More opportunities will become apparent as familiarity with the program grows
Conference and workshop presentations.
Question 19: Would you recommend Adobe Connect to your colleagues and what conditions would you put on its use?
Comments included:
Unconditionally. It is a great tool and one that will be the future of distance education.
It depends on what they need to accomplish - if these things are within the reach of Adobe then why not
Only if it were more reliable
With warnings
Yes, must be registered and code and password protected
Yes
Yes. Consider its use as a replacement for “voluntary” face-to-face week-end sessions from time to time that a number of students can't access in the PG area because of distance.
Yes. Adequate training and support on using it.
Question 20: If this type of software was not available to you and your colleagues in the future, what do you see as the possible implications?
Comments included:
I think that this would significantly impede the ability of Macquarie University to provide distance students with the type of interaction that they both require and expect. This is particularly true of postgraduate students, many of whom are searching for a type of academic mentorship that can only be offered through virtual classroom situations (either one on one of more of a seminar style).
Better software from other brands
Eventually students will come to expect it - ours have had several semesters using something similar so the learning environment is reduced without it. They acknowledge the learning community is one of the most valuable assets in the post-grad unit and appreciate the opp to work with a diverse group.
None as there are many other means of communication like this
It would inhibit our ability to provide distance education services
It would reduce the delivery options that are available
Switching back to 'traditional' mode of learning and teaching.
Question 21: Are there other issue or comments you would like to make about Adobe Connect?
Three comments were contributed:
Well done Matt and Christa for trying it
it's probably more difficult to use it in a big class than in a small one, particularly for interactive activities
It was a useful trial with some good outcomes. Thanks for the opportunity to try it.
Question 22: What were the main pedagogies that Adobe Connect supported? How was the software used?
Comments included:
Adobe connect supports all types of pedagogies, from teacher fronted right through to student fronted. Because of the ability to allow students to present, it can provide them with the opportunity to present in front of their peers and instructors. I think that the software is incredibly adaptable.
PhD student/supervisor working on documents
The best type of pedagogy would be group interaction in individual presentations - this didn't succeed. The teacher presentation and recording for later use was useful.
Student presentations, teacher presentations, class discussions (large mic on table with 10 on campus and 8 off campus
Development of an interactive learning communities centred on topics of interest.
Enhance self-access learning skills; flexibility in learning activities; minimize administration work.
Provide post-class learning materials/activities.
Question 23: How has the use of Adobe Connect affected the quality of your student-supervisor relationship? (availability, engagement, efficacy, planning of tasks, etc.)
Comments included:
I think that in terms of supervision, it has allowed students a way to interact not only with me but with each other - particularly as we used this in a doctoral seminar forum.
We discussed the assessments, which was available on archive
Students indicated how much they valued the support.
Positive: better management of time and task. Negative: less 'human' and make the relationship rather distant.
3 Adobe Connect Non-user Survey
Low response rates to the “Teaching and Learning Context” and the “Software Capability Analysis” surveys coupled with evidence from system archives that almost half of the people who had applied for an Adobe Connect account had not used it, led to the development of the “Adobe Connect non-user survey”. The purpose of this survey was to identify reasons for not using Adobe Connect during the trial. Thirteen participants completed this survey.
Question 1: Are you a supervisor or a lecturer
Ten people identified themselves as supervisors, three as students.
Question 2: I did not use Adobe Connect this term because:
the technical requirements exceeded my computers capacity
I now attend supervision meetings/lectures face-to-face
I planned to use it in semester 2 only
I did not find the time to familiarise myself with the software
the training provided did not prepare me well enough
I need more support
I found it too difficult to use
I found software that better suited my needs
other
Five supervisors (50%) indicated that they did not find the time to familiarise themselves with Adobe Connect, the remaining five supervisors (50%) chose “other”. Comments provided reveal reasons for non-use in more detail. Three respondents did not have any need for a web-conferencing system in the semester Adobe Connect was trialled either because they did not run an online unit during this time or because they planned to use it in the coming semester. One explained that they were not teaching staff; it is not clear whether this respondent plans to use Adobe Connect in the future and for what purpose.
Two respondents described that they did not find the time to use Adobe Connect and instead reverted to familiar technology, such as Skype or the telephone. One respondent explained that they attempted to use Adobe Connect but each time encountered technical problems, such as missing audio or video or slow connections.
Reasons given by students for not using Adobe Connect indicated that they attended supervision meetings face-to-face. In one case, the supervisor was not even aware of the software.
Question 3: Are you planning to use Adobe Connect in semester 2?
Four survey respondents (30.8%) indicated that they wanted to use Adobe Connect in the coming semester. Eight (61.5%) stated that they were not interested in future use.
Question 4: Please share your reasons with us:
I no longer have the need for online communication tools
I prefer other media (e.g. telephone, e-mails, Skype, etc.)
I found software that better suits my needs
Other (please specify)
The predominant reasons given for this was a preference for other media, such as telephone, e-mails and Skype (4 respondents) or for software that better suited their needs (2 respondents). One survey respondent explained that although interested they were “... somewhat daunted by the information sheets. It seemed too much effort, and too time consuming.”
4 Observations drawn from managing and supporting the trial
Technical Administration of the Adobe Connect Meeting platform
A great deal of time was spent on administration of the Adobe Connect. As indicated by user feedback, when the Adobe Connect Meeting platform is working well it is a pleasure to use, however the system experienced several technical problems throughout the trial.
In terms of hardware, it was important to acquire a server that could meet the high end system requirements of Adobe Connect, especially because of the potentially large amounts of multimedia information that the server needs to receive, record and distribute. The standard installation program was reasonably straight forward to run, however some problems were experienced in terms of configuring the custom.ini file which stores all of the local settings for the server. These problems may have been caused because we were running a trial license, and were eventually resolved through several communications with Adobe Connect Customer Support in the US/India.
The Adobe Connect service crashed several times throughout the trial. Most times the service crashed while the system was not being used and was able to be restored in time for classes to run. However on three occasions the server was not able to be restored in time (for instance, for weekend classes), resulting in considerable frustration for staff and students.
The main reasons that the Adobe Connect service crashed appeared to be a) problems with restarts after automatic updates by Windows Server 2003 and b) problems caused by conflicts with antivirus software, and c) undiagnosable causes. In most cases the service was restored by rebooting the server and restarting the Adobe Connect services. However in one case a rebuild of the entire server was required.
Antivirus software caused ongoing problems. The University antivirus software (Sofos) initially caused server error messages on the log files which after a few weeks would cause the server to crash. The E-Trust antivirus program was then trialled and after some time spent configuring the software the server ran without failure for six weeks. However the University required that the antivirus software be switched back to Sofos, and even though Sofos staff wrote scripts to ensure that the software was compatible with Adobe Connect, users experienced problems with recordings after the Sofos program was reinstalled (presentation slides and audio in some instances not being recorded). Even though it is possible to simply attribute these recording problems on the antivirus software, it should be noted that similar problems were experienced with the Adobe Connect server being run in the Division of Information and Communication Sciences in 2005-2006, as well as the Webqem[1] server in June of 2008.
The system provides an easy to use interface for administering users and rooms. Users can be imported on mass, and individual users that have been granted appropriate permissions can create as many rooms as they like. Recordings can be edited to remove unwanted sections, and be either made available online or as a downloadable Flashvideo file. However the system does not provide an effective interface for exporting data – there is no capacity to export user data or migrate information from one server to another.
User support for the Adobe Connect Meeting platform
The Adobe Connect software allows users to perform operations that other systems do not, including dragging, dropping and resizing tools within the web conferencing environment on the fly. This allows users to tailor the interface to meet the collaborative and cognitive demands of the learning episode. As well, the system enables users to import Powerpoint presentations (animations preserved) import and markup video, create breakout rooms for student collaboration and manage access to tools using a sophisticated but flexible range permissioning system.
However, participants in this trial did not in general use the more advanced features of Adobe Connect. This appeared to be based on their instructional needs and their lack of familiarity with how the system operated. Through experience in using the system and also by supporting participants to use Adobe Connect, developing the skills and understanding (both technological and pedagogical) to use the software effectively is a medium- to long-term undertaking that requires a concerted effort.
As previously mentioned, introductory training sessions were offered to all staff and students in the trial and a Macquarie Adobe Connect User Resources website was established to provide these participants with a resource centre to provide training and troubleshooting support. However when it came to overcoming technical issues it often fell upon one of the Project Leaders to attempt to resolve the issue. This often required a great deal of time scouring discussion boards and support sites on the Internet to find solutions. When serious issues were still unresolved (such as problems with malfunctioning recordings) the International Adobe Connect support services were solicited (which proved difficult to engage and often bared no results).
Several users experienced difficulties with using audio (volume levels, audio not being broadcast or not being received). It took a degree of experience with the system in order to be able to troubleshoot audio problems. Although audio troubleshooting advice was placed on the support website, a great deal of one-on-one time was spent supporting users to overcome audio problems. This may not be related to the Adobe Connect system per se but rather the use of web-conferencing generally.
Note that other universities have found Adobe Connect to be less than 100% reliable (for Instance, see the Penn State University Adobe Connect Community site available at ).
Interpretation of results
Results from the Teaching and Learning Context Survey indicated that respondents were using the web-conferencing system more for teaching online classes than facilitating collaboration between Higher Degree Research students and their supervisors. A range of disciplines (education, actuarial studies, statistics and linguistics) and class sizes (from 8 to 45) and student enrolment modes (internal, external and off shore) were involved in the trial. In some cases the Adobe Connect system was used for scheduled classes while in other cases it was for optional interaction between teachers and students. It was used to deliver presentations, facilitate discussion and enable collaboration. Thus participants applied Adobe Connect for a wide variety of purposes and in diverse ways, providing a suitably broad range of test cases for trialling the system.
1 User Satisfaction
The Software Capability Analysis Survey indicated that generally people were satisfied with the trial. Overall, Adobe Connect was rated predominantly “satisfactory” in each of the areas of access, ease of use, reliability, speed and connectivity and screen layout. However, with the exception of “screen layout” each feature also received “unsatisfactory” ratings and comments highlight some problems with this software. In particular, the stability of the system was questioned. Respondents reported difficulties with the accessing of online meetings or archives and meeting participants dropping out while in sessions. Also, using the video tended to occasionally freeze the view function.
The highest rated functions of Adobe Connect were “Screen Sharing”, “Chat”, Whiteboard”, “Note pods” and “Breakout rooms”. Although not used by all trial participants, they uniformly received “highly satisfactory” and “satisfactory” ratings. The remaining functions, namely “Teacher/student presentations”, “File sharing”, Recording for later use” and “Discussions (audio)”, rated predominantly “highly satisfactory” and “satisfactory”. However, each of these functions also received “unsatisfactory” ratings. Thus while there was general approval for the tools, problems were experienced with some of the multimedia elements of the system.
Participants indicated that there were several advantages of using the Adobe Connect web-conferencing system. The fact that Adobe Connect offered a virtual classroom with all elements of a traditional classroom was regarded as positive. However it was noted that this required navigating through pods, which was regarded as a difficulty, particularly when pods had been moved around the screen. Another advantage was in the ability to bring together external and internal students in the same classroom. When doing this with large face-to-face classes it was pointed out that a roving microphone should be used so that on-campus students could be heard by their online peers. Other advantages included features not found in other e-learning platforms, the ability to share desktops to demonstrate software.
Respondents felt that the main problem they experienced with the system was with the audio. “Discussions (audio)” received the highest number of “unsatisfactory” responses (3) and related comments highlight problems with the usability and reliability audio function. In some cases users resorted to chat mode to overcome audio issues. Respondents also expressed problems with accessing archives, using the video function and/or the audio or video not working. In addition, several technical difficulties were reported, such as broken links or presentations only being partly recorded (voice recorded, but not slides). Technical problems were also seen as a reason for making Adobe Connect unreliable for assessment purposes and for causing some anxiety.
One respondent suggested that it would be better to share documents in the meeting room, rather than having to download it at either user end. This latter suggestion is actually possible, which indicates that in some circumstances problems can be caused by user capability rather than software functionality.
2 Effectiveness in meeting needs
Five of eight respondents (57.5%) found Adobe Connect to be “very effective” or “effective” in supporting teaching and learning activities, in contrast to 3 respondents (37.5%), who rated it as “not very effective”.
Some of the reported difficulties can be directly attributed to the use of Adobe Connect, while others are independent of the software. The difficulties experienced and the need for further software training are factors attributable to the software. One lecturer felt that problems with the video function disadvantaged online students because they could not see the lecturer or other students.
A software-independent problem related to difficulties in combining face-to-face students with online students in the same classroom. In particular, this lecturer found it difficult to ensure that online students could hear face-to-face students. As indicated earlier, this was ultimately resolved by using a roving microphone.
One lecturer thought that using Adobe Connect was “better than nothing”, but would have preferred a different system (Live Classroom). Another lecturer stated that Adobe Connect was well suited to the limited purpose it was used for.
When asked whether there were any things Adobe Connect could not do, survey respondents described technical problems mostly related either to the audio or video function. Audio problems included feedback and the lecturer’s microphone slipping out of range. With regards to video, one lecturer would like all participating students to be able to see both each other and the lecturer and another reported that they could not attach large video files. One request which is neither related to the audio or video function, was the possibility (or ability?) to support languages other than English. And finally, one lecturer found it difficult to monitor the lecture while at the same time drawing attention to questions posted in the chat pod, which relates to general web-conferencing skills development. The range of issues identified here indicate that web-conferencing should not be viewed as a simple and magical “do-all” fix for remote teaching but rather as a tool that can support remote teaching that requires a degree of user expertise to operate effectively.
The majority of teaching tips and advice again related either to technical problems on one hand and blending delivery of face-to-face and online students on the other. Technology-related advice mainly addresses managing technological problems, such as having a back-up plan in case problems occur or being flexible with starting and finishing times until all participants are sufficiently familiar with technology. One respondent suggested encouraging the use of reaction buttons (agree, disagree, hand raising, etc.).
With respect to combining face-to-face classroom delivery with online participation, survey respondents suggested the use of a roving microphone to ensure online participants can hear contributions of students attending face-to-face as well as planning activities for online participants while face-to-face participants are engaged in group discussion. Others advised to include structured sections or stated that they themselves needed more experience in aligning face-to-face participation with online participation. One respondent suggest they would use of Skype instead of Adobe Connect, another thought that online participants preferred the flexibility of asynchronous delivery. Combining face-to-face teaching with web-conferencing requires an added level of expertise because to some extent the teacher needs to simultaneously manage, monitor and integrate two environments.
Only one respondent thought that the ability to flexibly adjust the interface supported collaboration, and only once users had become familiar with the system. The remaining survey participants said that they either did not use this feature or described problems. The ability to flexibly adjusting the interface to meet the collaborative and cognitive demands of the learning episode was one reason for trialling the Adobe Connect system above other systems. However it appears that novice users generally do not capitalise on this feature and may need training and so support to understand how to leverage this potential.
In terms of capabilities that respondents identified as important when using Adobe Connect, respondents identified both technological and pedagogical skills. Although, in general, no particular technical skills beyond general IT knowledge were necessary to use Adobe Connect, survey respondents said that technical problems required them to develop trouble shooting skills. Pedagogically, one respondent also pointed out that new strategies are required to interpret student progress and to stimulate interaction in the virtual classroom environment. For our teachers to facilitate successful web-conferencing sessions they need to possess both technological and pedagogical capabilities (highlighting the critical role of professional development to be able to use the web-conferencing system effectively).
Respondents identified several important issues in relation to quality assurance and compliance, of which the main concerns were copyright, intellectual property and plagiarism. The uploading of files which are made open to all participants, elicited concerns about the protection of intellectual property and potential breaches of copyright laws. In addition, there was concern about quality assurance because the originality of external students’ work was difficult to track. This is particularly of concern in postgraduate supervision contexts where unpublished ideas and research data are seen as potentially sensitive to unwanted exposure prior to publication. It should be noted that these are of concern in any situation where users have the capacity to upload digital content, and it is not related to the Adobe Connect or web-conferencing technology in particular.
3 Support and training
On the whole students and staff appeared satisfied with training and support, with staff satisfaction rated higher than student satisfaction. It appears that not all trial participants had attended training sessions provided at the beginning of the trial. At least in one case training was provided by the lecturer rather than by project coordinators. Two other respondents stated that they had not received any training. Similarly, at least one participant was unaware of available documentation, despite several emails notifying participants of the online resources that had been made available. Some participants also noted that no amount of training could overcome some of the technical problems that were being experienced. However, comments also showed an appreciation for the support and coaching received.
On the whole respondents indicated that the ongoing workload was “manageable”. Only “preparing materials” received an “unmanageable” rating; one “highly unmanageable” rating was received in the “other” category. Comments indicate that “setting up” and “archiving” were experienced as time-consuming tasks. One respondent reported increased workload caused by the organisation and preparation for teaching in an online setting; however, this was regarded as manageable. The fact that the workload associated with teaching online was generally perceived as manageable is reassuring in terms of encouraging staff to experiment with web-conference based teaching.
The Adobe Connect web-conferencing system requires considerable time-per-user to support (both in terms of providing training and troubleshoot help), and if the system were to be implemented across the university then adequate funds would need to be allocated to providing such support.
Nearly half of all people who requested and were issued with accounts did not end up trying Adobe Connect. The effort required for people to change their modus operandi was for many too great, causing them to revert to familiar (but potentially less effective) behaviours. This indicates that energy is required just to help users get started with such systems if the efficiencies resulting from the use of such systems are to be harvested.
4 Risks
Only one respondent thought that there would be financial costs associated with using Adobe Connect, namely for providing initial training and support for the first few teaching sessions. It is not clear whether this respondent participated in the training provided at the start of the trial period.
Four of the eight respondents did not think that there were any risks associated with using Adobe Connect. Risks that were identified by the other respondents were all technical and included the server crashing, audio not working, and archives failing. Lecturer and student dissatisfaction due to technical problems was also identified as a risk.
Another possible risk was the concern over providing equal access to information for both online students and those participating face-to-face, and in light of varying network connections. This was overcome by providing slides and chat notes to all students and by ensuring that slides focus on the prescribed material in the core text.
Thus, all non-financial risks related to the communication impediment caused by the functionality and reliability of the medium. To this extent evaluating web-conferencing systems is as much about examining what is not communicated as what is communicated.
5 Future use and application
All but one respondent stated that they would use Adobe Connect again. However, comments provided indicate that more support would be welcome and that one user would prefer another application. Three of the eight respondents felt that there were no more efficient ways other than web-conferencing of holding online learning activities. One of them particularly praised the level of interaction and learning afforded by the software. However three other respondents would have preferred the use of other web-conferencing software (such as Live Classroom or Elluminate). Another respondent requested more user support. This indicates that there is definitely a place for a web-conferencing style system at Macquarie, though the particular system that should be used is perhaps debatable.
One respondent raised the pertinent point that some students will because of their dispositions prefer face-to-face teaching to online learning (although the converse may also be true). Web-conferencing may be offered as an alternative mode of engagement rather than a replacement.
Participants felt there were a range of potential applications for Adobe Connect across the university, including:
• allowing off-campus students and staff to participate in on-campus activities, such as classes, departmental meetings
• facilitating collaboration and research across universities and countries
• conducting presentations, conferences and workshops across large distances
• providing outreach services to external students.
One respondent pointed out that further uses would become apparent with increased familiarity.
In terms of recommending Adobe Connect to colleagues, responses ranged from “unconditionally it is a great tool and one that will be the future of distance education” to “only if it were more reliable”. However, five of eight respondents would recommend Adobe Connect to colleagues. The remaining three respondents qualify their responses. One states that it would “depend on what they needed to accomplish”. The other two would “only if it were more reliable” or “with warnings”. Suggested conditions included using Adobe Connect to replace voluntary weekend sessions for external students and adequate support and training.
The majority of respondents (4 out of 7) expressed concern that the withdrawal of Adobe Connect services would reduce delivery options, particularly for external students. Another one thought that post-graduate students in particular would suffer because they would be deprived of a mentoring relationship. Two respondents explained that they would switch to other communications tools or another brand of software. Only one respondent suggested that they would return to “traditional modes of learning and teaching”. In order to move forward it would seem that a web-conferencing system is needed at Macquarie.
6 Pedagogical implications
Adobe Connect supported a range of pedagogical practices that otherwise could not have occurred:
• Remote presentations by lecturers (interactive teacher centred modes)
• Remote presentations by students to peers and lecturer (peer learning activities)
• Class discussions involving both internal and external students (multimodal learing)
• Creation of a distributed yet interactive learning community
• Self-access learning using archived presentations, materials and activities (multimodal, flexible learning).
In terms of pedagogical practices that supported research candidate supervision, Adobe Connect allowed:
• supervisor – candidate collaboration on documents (mentoring and peer learning, co-authoring, work based learning)
• students to interact not only with supervisors but with each other
• better time and task management (by enabling more effective communication)
• supervisors to offer academic and affective support to their candidates remotely.
To not have web-conferencing at the University restricts the remote pedagogies that can be applied, for both classes and higher degree research supervision.
7 Management and Administration
Adobe Connect requires ongoing monitoring and maintenance. As well, a considerable degree of expertise is required to ensure that the server is performing optimally. If Adobe Connect is to be adopted as a part of Macquarie University’s general teaching and learning infrastructure or even as a limited service for some staff and Departments then adequate funding is required to either ensure that a technical officer is allocated appropriate time and training to manage the server or external management services are purchased.
Summative Comments
While it is rewarding to perform an evaluation and derive conclusive results, it is rarely the case, and this trial is no exception. Users were generally satisfied with the Adobe Connect Meeting platform however there were some people who were unsatisfied with the usability and reliability of the system, particularly with relation to the use of audio and the quality of recordings. Participants appreciated the affordances that the Adobe Connect offered (desktop sharing, distributed presentations and discussions, recordings) but the reliability and usability issues dampened their enthusiasm for the system.
Participants indicated that it took time to become comfortable using Adobe Connect, and that user support was required. The workload involved in using web-conferencing was manageable, however the risk of the technology failing was a concern for many. The training was generally appreciated, though labour intensive to deliver. All respondents identified the potential of the software to offer a range of services to students (such as online workshops, international collaboration, remote presentation delivery, outreach to international students), however management and administration of the system is a time consuming and costly undertaking.
Responses from users implied that having some form of web-conferencing system at the university is an imperative. As one respondent put it, not offering web-conferencing services “would significantly impede the ability of Macquarie University to provide distance students with the type of interaction that they both require and expect”. The question then becomes one of whether or not Adobe Connect adequately satisfies the needs of the University, based on the varied user feedback that has been received. This will depend on the discretion of management, based on this evaluation and others that will be or have been conducted (such as that for Elluminate Live).
Looking Forward
In order to continue piloting the use of web-conferencing services across the university an Adobe Connect server is now being operated by the Learning and Teaching Centre, for trial and evaluation purposes. In contrast to the trial license of Adobe Connect which allowed as many account holders as desired but only up to 50 concurrent users, the license being operated in the Learning and Teaching Centre allows five named users to each have up to 99 guests with them at ay one time. That means that although larger classes can be serviced, only five Macquarie University staff members can use the Adobe Connect system with their classes. This may be deemed inadequate depending on the number of people who require the service, and if Adobe Connect is selected as the web-conferencing system of choice for the university, the number of possible account holders on the license will almost certainly need to be increased.
Acknowledgements
Many thanks goes out to all the people who volunteered to participate in this trial. Particular thanks are extended to those who took the time to provide their thoughtful feedback through the online survey forms. As well, we are deeply indebted to our research assistant, Christa Jacenyik-Trawoger, whose diligence and thoroughness kept the project on track in the midst of what seemed to be continually arising obstacles. As well, we are overwhelmingly grateful to our technical assistant, Peter McCarthy, who offered tireless support both as required and with a smile. Finally, sincere thanks to Professor John Hedberg, Dr Maree Gosper and Andrew Burrell who provided valuable governance throughout the duration of the project. Without the help and support of all of these people the project would not have been possible or as successful. It is hoped that outcomes of this project supports Macquarie University’s efforts to provide a high quality information technology infrastructure at the same time as they have informed and supported teaching and learning processes.
-----------------------
[1] Webqem is the leading provider of Adobe Connect in Australia
-----------------------
Web-conferencing - enabling collaboration for Higher Degree students -
Adobe Connect Trial Evaluation
Dr Matt Bower, Dept of Education, matt.bower@mq.edu.au
Dr Meeri Hellstén, Dept of Education, meeri.hellsten@mq.edu.au
Christa Jacenyik-Trawoger, LTC, christa.jacenyik@mq.edu.au
September 2009
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- university city education foundation
- university of education hk
- university of education home
- university degree education reviews
- tertiary education commission
- tertiary education system
- tertiary education pdf
- what is tertiary education definition
- tertiary education data
- tertiary education by country
- tertiary education in ghana
- tertiary education attainment