Abbreviations - Professor Kevin Curran



Using Leap Motion and Gamification to Facilitate and Encourage Rehabilitation in those with Hand InjuriesJamie TaylorB00593273taylor-j10@email.ulster.ac.ukBSc (Hons) Computer ScienceSchool of Computing and Intelligent SystemsUniversity of UlsterSupervisor: Dr. Kevin CurranDecember 2013DeclarationI declare that this is all my own work and does not contain unreferenced material copied from any other source. I have read the University’s policy on plagiarism and understand the definition of plagiarism. If it is shown that material has been plagiarised, or I have otherwise attempted to obtain an unfair advantage for myself or others, I understand that I may face sanctions in accordance with the policies and procedures of the University. A mark of zero may be awarded and the reason for that mark will be recorded on my file.Jamie TaylorAcknowledgementsI would like to extend my sincere thanks and gratitude to Dr Kevin Curran who has been a tremendous help not only throughout this project, but throughout all of my student career for which he has been a part.I would also like to thank James Connolly for his vital contributions to the research and development stages of the project.AbstractInjuries to the hand are more common than those of any other body region and can have considerable financial, time-measured and psychological impact on not only the victim but the community as a whole. Hand rehabilitation aims to return people to their pre-injury roles and occupations and has proved largely successful in doing so with the potential for technology to improve these results further. However, most technology used in hand rehabilitation is based on expensive and non-durable glove-based systems and issues with accuracy are common among those which are not glove-based. We propose an accurate, affordable and portable solution wherein we use the Leap Motion as a tool for hand rehabilitation. User feedback will be given primarily through an animated 3d hand model as the user performs rehabilitative exercises. Exercise results will be recorded for later viewing by patients and clinicians. The system will also include Gamification aspects, techniques which (while proven to increase participation) have seen little to no use in hand-rehabilitation systems.In response to this need for new rehabilitation technologies, we have developed a functional rehabilitation system using the Leap Motion. This system is referred to as the Leap Motion Rehabilitation System or LMRS. The LMRS delivers on the points described above and produces medically relevant data accurate to within 100 milliseconds. The proof-of-concept that is the LMRS coupled with the price and relative accuracy of the Leap Motion in addition to its other unique qualities mean the LMRS represents the beginning of a promising new avenue with regards to the use of technology in rehabilitation.Contents TOC \o "1-3" \h \z \u Abbreviations PAGEREF _Toc386640598 \h 7Table of Figures PAGEREF _Toc386640599 \h 8List of Tables PAGEREF _Toc386640600 \h 9Table of Source Code Listings PAGEREF _Toc386640601 \h 9Table of Graphs PAGEREF _Toc386640602 \h 91: Introduction PAGEREF _Toc386640603 \h 101.1 Existing Approaches PAGEREF _Toc386640604 \h 101.2 Project Aim PAGEREF _Toc386640605 \h 111.3 Chapter Overview PAGEREF _Toc386640606 \h 112: Background and Related Work PAGEREF _Toc386640607 \h 122.1 Hand Injuries: Types, Occurrence and Impact PAGEREF _Toc386640608 \h 122.2 Treatment and Rehabilitation of Hand Injuries PAGEREF _Toc386640609 \h 142.3 The use of Glove-Based Technology in Hand Rehabilitation PAGEREF _Toc386640610 \h 162.4 The use of Non Glove-Based Technology in Hand Rehabilitation PAGEREF _Toc386640611 \h 202.5 The Role and Potential of Gamification PAGEREF _Toc386640612 \h 233: Requirements Analysis and Specification PAGEREF _Toc386640613 \h 263.1 Problem Statement PAGEREF _Toc386640614 \h 263.2 Functional Requirements PAGEREF _Toc386640615 \h 263.3 Non-Functional Requirements PAGEREF _Toc386640616 \h 273.4 Hardware Requirements PAGEREF _Toc386640617 \h 283.5 Software Requirements PAGEREF _Toc386640618 \h 283.6 Development Methodology PAGEREF _Toc386640619 \h 284: Project Planning PAGEREF _Toc386640620 \h 304.1 Milestones and Deliverables PAGEREF _Toc386640621 \h 304.2 Project Plan PAGEREF _Toc386640622 \h 314.3 Time Management PAGEREF _Toc386640623 \h 324.4 Risk Management PAGEREF _Toc386640624 \h 345: Conclusion PAGEREF _Toc386640625 \h 366: Design PAGEREF _Toc386640626 \h 376.1 System Component Overview PAGEREF _Toc386640627 \h 376.2 HCI Rules & Guidelines PAGEREF _Toc386640634 \h 376.3 UI Mock-Ups PAGEREF _Toc386640635 \h 406.4 Diagrams (Use case and Activity Model) PAGEREF _Toc386640636 \h 457: Implementation PAGEREF _Toc386640640 \h 487.1 Technical Overview PAGEREF _Toc386640641 \h 487.2 XML Data Store PAGEREF _Toc386640642 \h 487.3 Patient Side PAGEREF _Toc386640643 \h 497.4 Clinician Side PAGEREF _Toc386640644 \h 658: Evaluation PAGEREF _Toc386640645 \h 738.1 Testing PAGEREF _Toc386640646 \h 738.2 Evaluation against Initial Requirements PAGEREF _Toc386640650 \h 778.3 Future Work and Enhancements PAGEREF _Toc386640653 \h 789: Conclusion PAGEREF _Toc386640654 \h 8010: References PAGEREF _Toc386640655 \h 8111.1: Appendix 1: Clinician User Scenario Test Case PAGEREF _Toc386640656 \h 8511.2: Appendix 2: Patient User Scenario Test Case PAGEREF _Toc386640657 \h 8811.3: Appendix 3: Source Code PAGEREF _Toc386640658 \h 90AbbreviationsVR – Virtual RealityMMORPG – Massively Multiplayer Online Role-Playing GameMOBA – Massively Online Battle ArenaIR – InfraredWBS – Work Breakdown StructureRAG – Red Amber GreenFPS – Frames per SecondIPP – Intel Performance PrimitiveHCI – Human Computer InteractionUI – User InterfaceLMRS – Leap Motion Rehabilitation SystemXML – Extensible Mark-up LanguageWPF – Windows Presentation FoundationTable of Figures TOC \h \z \c "Figure" Figure 1 - Hand Therapy Session PAGEREF _Toc386677867 \h 16Figure 2 - CODA System PAGEREF _Toc386677868 \h 16Figure 3 - 5DT Data Glove Ultra PAGEREF _Toc386677869 \h 17Figure 4 - GloveManager Software PAGEREF _Toc386677870 \h 17Figure 5 - HumanGlove PAGEREF _Toc386677871 \h 18Figure 6 - Graphic Virtual Hand Software PAGEREF _Toc386677872 \h 18Figure 7 - Peregrine Gaming Glove PAGEREF _Toc386677873 \h 19Figure 8 - GloveBox Configuration Software PAGEREF _Toc386677874 \h 19Figure 9 - Rheumatoid Arthritis Hand PAGEREF _Toc386677875 \h 19Figure 10 - Rheumatoid Arthritis Hand PAGEREF _Toc386677876 \h 19Figure 11 - Kinect PAGEREF _Toc386677877 \h 21Figure 12 - Kinect in Hand Rehabilitation Setting PAGEREF _Toc386677878 \h 21Figure 13 - Leap Motion PAGEREF _Toc386677879 \h 22Figure 14 - Leap Motion Schematic View PAGEREF _Toc386677880 \h 22Figure 15 - Leap Motion Internals PAGEREF _Toc386677881 \h 23Figure 16 - Leap Motion Visualizer PAGEREF _Toc386677882 \h 23Figure 17 - Xbox 360 Achievements PAGEREF _Toc386677883 \h 24Figure 18 - Khan Academy Stat Tracking and Achievement System PAGEREF _Toc386677884 \h 24Figure 19 - Gamified Application, Walking through a Forest PAGEREF _Toc386677885 \h 25Figure 20 - CONTRAST Serious Rehabilitation Game PAGEREF _Toc386677886 \h 25Figure 21 - Work Breakdown Structure PAGEREF _Toc386677887 \h 31Figure 22 - Gantt chart PAGEREF _Toc386677888 \h 33Figure 23 - System Architecture PAGEREF _Toc386677889 \h 37Figure 24 - Login Screen PAGEREF _Toc386677890 \h 41Figure 25 - Clinician Main Menu Screenshot PAGEREF _Toc386677891 \h 41Figure 26 - Patient Main Menu Screenshot PAGEREF _Toc386677892 \h 41Figure 27 - Clinician version PAGEREF _Toc386677893 \h 42Figure 28 - Patient Version PAGEREF _Toc386677894 \h 42Figure 29 - Register Patent PAGEREF _Toc386677895 \h 43Figure 30 - Patient Screen PAGEREF _Toc386677896 \h 43Figure 31 - Rehabilitation Exercise Screen PAGEREF _Toc386677897 \h 43Figure 32 - Patient User & Clinician User PAGEREF _Toc386677898 \h 45Figure 33 - Patient User Interactions PAGEREF _Toc386677899 \h 46Figure 34 - Clinician User Interactions PAGEREF _Toc386677900 \h 47Figure 35 - LMRS Login Screen PAGEREF _Toc386677901 \h 49Figure 36 - LMRS Perform Exercises Screen PAGEREF _Toc386677902 \h 51Figure 37 - Metrics used to recognise user initiation of exercises PAGEREF _Toc386677903 \h 54Figure 38 - Vectors used for angle calculation PAGEREF _Toc386677904 \h 57Figure 39 - Demonstration of XNA-powered 3d hand model PAGEREF _Toc386677905 \h 60Figure 40 - LMRS Exercise Results Screen PAGEREF _Toc386677906 \h 60Figure 41 - LMRS Register New Patient Screen PAGEREF _Toc386677907 \h 68Figure 42 - LMRS Remove Patient Screen PAGEREF _Toc386677908 \h 70List of Tables TOC \h \z \c "Table" Table 1 - Non-Operative / Postoperative Hand Therapy Treatments PAGEREF _Toc386312836 \h 15Table 2 - Time Usage during a Typical Day PAGEREF _Toc386312837 \h 34Table 3 - Project Risks and Counter-Measures PAGEREF _Toc386312838 \h 35Table 4 - Exercise Repetition Time Variations PAGEREF _Toc386312839 \h 76Table 5 - LMRS Compared against Functional Requirements PAGEREF _Toc386312840 \h 77Table 6 - LMRS Compared against Non-Functional Requirements PAGEREF _Toc386312841 \h 78Table of Source Code Listings TOC \h \z \c "Listing" Listing 1 - LMRS Data Storage Format PAGEREF _Toc386382600 \h 48Listing 2 - Login Functionality PAGEREF _Toc386382601 \h 50Listing 3 - LMRS Perform Exercises Form Start-up PAGEREF _Toc386382602 \h 52Listing 4 - Leap Listener Get() Addition PAGEREF _Toc386382603 \h 52Listing 5 - LMRS Exercise Calibration PAGEREF _Toc386382604 \h 53Listing 6 - LMRS Exercise Instructions and timer start PAGEREF _Toc386382605 \h 54Listing 7 - LMRS Exercise 1 – Fist Clench core logic PAGEREF _Toc386382606 \h 55Listing 8 - Exercise 2 – Wrist Flexion & Extension Key Metric Conditional PAGEREF _Toc386382607 \h 56Listing 9 - Vector Calculation PAGEREF _Toc386382608 \h 58Listing 10 - Finger Angle Calculation PAGEREF _Toc386382609 \h 59Listing 11 - Pitch & Yaw Calculation PAGEREF _Toc386382610 \h 59Listing 12 - LMRS Toggle Patient Combo-box PAGEREF _Toc386382611 \h 61Listing 13 - Load either Patient or Clinician Main Menu PAGEREF _Toc386382612 \h 62Listing 14 - Load Exercise Combo-box PAGEREF _Toc386382613 \h 62Listing 15 - Exercise Selection Changed PAGEREF _Toc386382614 \h 63Listing 16 - Accept or Reject Call Depending on Tag PAGEREF _Toc386382615 \h 64Listing 17 - Refresh/Reload Chart after Data Change PAGEREF _Toc386382616 \h 64Listing 18 - Chart Type Visibility Toggle PAGEREF _Toc386382617 \h 65Listing 19 - Line Chart XAML PAGEREF _Toc386382618 \h 65Listing 20 - Create and Populate Patient Combo-box PAGEREF _Toc386382619 \h 66Listing 21 - Updating Results Chart on Patient Changed event PAGEREF _Toc386382620 \h 67Listing 22 - LMRS Register Patient PAGEREF _Toc386382621 \h 69Listing 23 - Automatic Username Generation PAGEREF _Toc386382622 \h 69Listing 24 - Finding Patient to Remove PAGEREF _Toc386382623 \h 71Listing 25 - Removing Patient PAGEREF _Toc386382624 \h 72Table of Graphs TOC \h \z \c "Graph" Graph 1 - Exercise 1 - Fist Clench PAGEREF _Toc386382717 \h 74Graph 2 - Exercise 2 - Wrist Flexion & Extension PAGEREF _Toc386382718 \h 75Graph 3 - Exercise 3 - Three Jaw Chuck Pinch PAGEREF _Toc386382719 \h 751: IntroductionThe human hand is one of the most complex creations in existence and the main enabler of our modern lifestyles. Given this intense and extensive use, it should come as little surprise that injuries to the hand are more common than those of any other body region CITATION Mar06 \l 2057 (Trybus, et al., 2006). Injuries such as Repetitive Stress Injuries (RSI’s), lacerations and crushing are just a few common injuries to hand. Such injuries are treated through hand rehabilitation CITATION Deb11 \l 2057 (Amini, 2011). This includes measures such as splinting the hand and prescribing rehabilitation exercises designed to strengthen the muscles in the hand and prevent build-up of scar tissue which would otherwise affect joint movement. Individuals who find themselves afflicted with these kinds of injuries can experience great emotional and psychological since an injury to our hands can threaten our independence and normality in a way few things can. This process is not only time-consuming and costly for the person injured; in the UK, over ?100 million is spent every year treating these kinds of injuries CITATION Jos06 \l 2057 (Dias & Garcia-Elias, 2006). Current rehabilitation is largely analogue, with no technological intervention, primarily due to cost. Data gloves, the most common technological rehabilitation aid, can potentially cost thousands of pounds CITATION Bre10 \l 2057 (O'Donnell, 2010). There is a clear need for something accurate, portable and affordable.1.1 Existing ApproachesAt present, it is common for individuals with hand injuries to undergo rehabilitation using no technical aids. Efforts to improve rehabilitation through the use of technology have led to a number of systems being proposed, these systems are most glove-based, with few alternatives. These glove-based systems are (for the most part) prohibitively expensive CITATION Bre10 \l 2057 (O'Donnell, 2010) and the few alternatives such as Kinect CITATION Aar11 \l 2057 (Bond, 2011) can suffer from portability and accuracy issues. It should also be noted that none of these system take advantage of gamification. Gamification is the use of game-like elements in traditionally non-game like settings and has been proven to increase user enjoyment and participation.1.2 Project AimThis project aims to design and develop a software based system for hand rehabilitation using the Leap Motion, a new, recently released motion-based device which has yet to be investigated as a tool for hand rehabilitation. User feedback comes primarily from an animated 3d hand model which will reflect the users hand movements in real-time. The results from the exercises will then be stored for later viewing by either the patient or a clinician. Furthermore, the project proposes the addition of gamification elements to the proposed system; this is done with the aim of better encouraging patients to adhere to prescribed exercise programs. To achieve this, the project will investigate current techniques and technologies used in the field of hand rehabilitation to better inform the design of the proposed system.1.3 Chapter OverviewThe remainder of this report is structured as follows: chapter two will provide a literature overview, looking at hand injuries, treatments and the role of rehabilitation, current rehabilitation systems using data gloves and other technologies and finally: gamification. Chapter three will then describe the requirements analysis for the proposed system, covering functional and non-functional requirements in addition to essential hardware and software. Chapter four will present project planning efforts, describing project milestones, risk assessment and strategies for optimal time utilisation before concluding with chapter five.2: Background and Related Work2.1 Hand Injuries: Types, Occurrence and ImpactHand injuries are among the most frequent injuries; accounting for between 6.6% and 28.6% of all injuries and 28% of musculoskeletal injuries with the dominant hand being injured in 52.2% of cases. The most frequent place for hand injuries to occur is not the work place as one might suspect. The most frequent place for hand injuries to occur is in fact the home, accounting for 45.3% of all hand injuries, followed by the workplace at 19.7%, with young male manual workers being most at-risk CITATION Mar06 \l 2057 (Trybus, et al., 2006).Common injuries to the hand, treatable with hand rehabilitation include acute issues such as: fractures, lacerations, amputations, burns, surgical repairs of tendons and nerves. This is in addition to more chronic and acquired conditions such as: tendonitis, rheumatoid arthritis, osteoarthritis, RSI’s such as carpal tunnel syndrome CITATION Deb11 \l 2057 (Amini, 2011) and neurological issues (i.e. stroke) CITATION Ame11 \l 2057 (American Society for Surgery of the Hand, 2011). Of all the injuries listed above, fractures of the hand make up for the majority of hand related injuries seen and treated in hand surgery units, with nerve injuries having the most prolonged and profound impact on the patient due to continuing disability CITATION Jos06 \l 2057 (Dias & Garcia-Elias, 2006).In this report, we view the cost and impact of hand injuries as being three-fold, where the impact can be divided into three main categories. These are financial, time and psychological. We look first at the financial impact, progressing to time-related impact and lastly psychological impact.It is estimated that treatment for hand injuries costs the UK approximately ?100 million per year. However, this problem spans much farther than the UK; the US for example, spends approximately $18 billion treating upper extremity disorders and Germany spends approximately €2 billion treating severe trauma with a ratio of 25 patients per 100,000 of the population CITATION Jos06 \l 2057 (Dias & Garcia-Elias, 2006). Looking at RSI as a more specific example, we see that RSI alone is estimated to cost UK employers approximately ?300 million per year CITATION Str08 \l 2057 (Strategy One, 2008). This is again mirrored in other parts of the world, with the US spending approximately $20 billion on RSI compensation each year CITATION Yas97 \l 2057 (Yassi, 1997). Of all the hand injuries described above, amputation is deemed the most expensive, replantation of the hand or some part of the hand can cost up to 1.6 times a patient’s annual salary. Nerve injuries are the second most expensive injury to treat, costing between €51,238 and €31,186 CITATION Hol96 \l 2057 (Holmberg, et al., 1996). Speaking in more general terms, CITATION Mar06 \l 2057 (Trybus, et al., 2006) calculate the mean cost of a hand injury to be $6126.76 or €4507.29. When discussing the financial impact of hand injuries, it is interesting to note the uneven distribution of direct to indirect cost. An example of a direct cost would be that of a surgical procedure whereas examples of indirect cost would include sick leave and outpatient travel. Direct cost was found to make up only 4% of the total expense whereas indirect costs made up the remaining 96% CITATION Mar06 \l 2057 (Trybus, et al., 2006).The impact of hand injuries is not just measured in terms of financial cost to employers through compensation or lost productivity; we can also use time related metrics such as work days lost or treatment duration in days when measuring the impact of injuries to the hand. Reports indicate that hand injuries account for 27% of all work-related injuries requiring more than 1 day of leave CITATION Rai03 \l 2057 (G, 2003). Given that hand injuries are a world-wide concern, it is realistic to suggest that hand injuries can result in millions of work days being lost, as workers are forced to take leave in order to recover from their injuries. RSI for example costs UK employers approximately 3.5 million working days alone, with each affected person taking an average of 13 days off due to their injury CITATION Str08 \l 2057 (Strategy One, 2008). Initial, hospital-based treatment of hand injuries can last anywhere between 1 to 86 days with an average of 9.1 ± 9.3 days. Total treatment duration, time in hospital and aftercare can last between 1 to 420 days with an average of 76.9 ± 67.8 days, meaning hand injuries often take longer to treat than injuries to other regions of the body. It should also be noted that the severity of the injury does, as one would expect, affect the duration of treatment CITATION Mar06 \l 2057 (Trybus, et al., 2006).In addition to the financial and time-related impact observed above, hand injuries can also have a severe psychological impact on those afflicted. It is common for people to view themselves in relation to their occupational role, rank and level of ability CITATION Has02 \l 2057 (Hasselkus, 2002). Injuries that then interfere with one’s occupation or daily routine - such as those involving the hands - can cause severe distress and a strong yearning for a return to normalcy CITATION Has02 \l 2057 (Hasselkus, 2002). Of all the types of hand injuries described here, nerve injuries have the most prolonged and profound psychological impact on the patient, those suffering from a nerve injury in the hand are commonly left with some form of persistent, residual disability that they are forced to contend with for the remainder of their lives. The likelihood of persistent, residual disability after hand injury spans from 1% to 100% with 13.6% of patients being affected on average CITATION Jos06 \l 2057 (Dias & Garcia-Elias, 2006). Psychological issues caused by hand injuries and associated persistent, residual disability includes flashbacks, Post-Traumatic Stress Disorder (PTSD) and concerns with personal appearance CITATION Avi13 \l 2057 (Sousa, et al., 2013).2.2 Treatment and Rehabilitation of Hand InjuriesThis report is focused on the development of a hand rehabilitation system using the Leap Motion; because of this we will be focusing on hand rehabilitation and its use as a treatment for hand injuries to the exclusion of other treatment measures such as surgical procedures.Hand rehabilitation therapy is a form of occupational therapy CITATION Deb11 \l 2057 (Amini, 2011) and is depicted in figure 1. Hand rehabilitation/therapy is focused on “…enabling the client to regain functional use of the traumatized arm and hand … and return to their pre-injury occupations.” CITATION Jan03 \l 2057 (Case-Smith, 2003). The treatment offered by hand therapy can be divided into two main categories; these are preventative, non-operative and post-operative. Using the information presented in CITATION Ame11 \l 2057 (American Society for Surgery of the Hand, 2011), a more complete list of treatment options offered through hand therapy can be compiled and is presented in Table 1. Preventative, Non-operative, ConservativePostoperative RehabilitationManagement of acute or chronic painManagement of open or sutured woundsDesensitization following nerve injury or traumaControl of hypertrophic or hypersensitive scarsSensory re-education after nerve injuryReduction of swellingDesign and implementation of home exercise programs to increase motion, dexterity and/or strengthFabrication of orthoses to protect surgery or increase movementTraining in performance of daily life skills through adapted methods and equipmentInstruction in home exercise programSplint fabrication for prevention or correction of injuryConditioning prior to returning to workTable SEQ Table \* ARABIC 1 - Non-Operative / Postoperative Hand Therapy TreatmentsOf the treatments listed above, it is “design and implementation of home exercise programs…” and “instruction in home exercise programs” that are of particular relevance and interest to this project. CITATION Yaf13 \l 2057 (Lavanon, 2013) Points out that such hand therapy exercises should be “motivating, repetitious, interesting, challenging and graded”, CITATION Deb11 \l 2057 (Amini, 2011) adds that these exercises should incorporate “usual and customary occupation activities…”, this is important, given that the aim of hand therapy as described above is to return patients to their occupational and pre-injury roles. At present, it is common for home exercise programs to be performed without the use of technological aids or systems.Hand therapy offers a high success rate as a treatment for hand injuries. Of those studied and treated in CITATION Jan03 \l 2057 (Case-Smith, 2003), 80% returned to work after an 8 week course of treatment consisting of – on average – 13 hours of treatment. These results are of particular relevance because during this time, the occupational therapist was the patient’s sole provider of rehabilitation services, showing that hand rehabilitation/therapy even in isolation can be greatly successful and beneficial. While hand therapy is already a successful form of treatment for hand injuries CITATION Jan03 \l 2057 (Case-Smith, 2003), there is evidence to suggest that this form of treatment could be improved further through the use of technology CITATION Yaf13 \l 2057 (Lavanon, 2013). Argues “…advanced technology can enrich treatment and help patients…” looking in more detail, we see that technology can be applied to other areas of hand injury treatment beyond rehabilitation. The CODA system seen in figure 2 for example, can be used as a diagnostic motion analysis tool. More relevant to this project however, is the discussion of technology as a rehabilitative tool, in particular, the use of everyday “off the shelf” technology such as the Leap Motion. An example of such a system is described in CITATION Yaf13 \l 2057 (Lavanon, 2013), where a VR system was constructed using the PlayStation EyeToy, a common consumer device. The EyeToy based system was found to be an effective and – more importantly – enjoyable way of exercising, however the system fails to grade exercises. This is something we aim to implement in the proposed system, even enhancing it further through the introduction of gamification elements. 28575069215Figure SEQ Figure \* ARABIC 1 - Hand Therapy SessionFigure SEQ Figure \* ARABIC 2 - CODA System2.3 The use of Glove-Based Technology in Hand RehabilitationGlove-based technology, specifically data-glove technology, is arguably the most common form of technological aid in treatment and managing of hand injuries. Therefore we dedicate a section solely to it. Example applications include motor assessment CITATION Roi12 \l 2057 (Lautman, 2012) and as a tool for rehabilitative exercises CITATION Bre10 \l 2057 (O'Donnell, 2010). This high adoption rate is primarily a result of the richness of the information provided by such systems CITATION Lau08 \l 2057 (Dipietro, et al., 2008). To define a glove-based system, we use the definition provided by CITATION Lau08 \l 2057 (Dipietro, et al., 2008), where a glove-based system is defined as “a system composed of an array of sensors, electronics for data acquisition/processing, power supply and a support for sensors that can be worn on the user’s hand.”. Such gloves are typically made of Lycra onto which sensors are sewn, these sensors then record data of the wearer’s hand movements, joint movement, fingertip positioning and so forth. We now look at a few glove-based systems that show promise in a hand rehabilitation environment.5DT Data Glove UltraThe 5DT Data Glove Ultra, shown in figure 3, developed by Fifth Dimension Technologies CITATION Fif11 \l 2057 (Fifth Dimension Technologies, 2011), is a data glove aimed primarily at Motion Capture and Animation Professionals. The gloves has a total of 14 sensors, uses proprietary optical-fibre flexors and supports 24=16 possible gestures CITATION Lau08 \l 2057 (Dipietro, et al., 2008). The glove communicates with a computer via USB cable or RS 232 serial port through an additional kit (sold separately); another kit is available to allow for wireless operation via Bluetooth (also sold separately), allowing 8 hours of use on a single battery at a range of up to 20 meters. The glove itself has a base unit price of $995 (?608.79); this includes the glove and the ‘GloveManager’ proprietary calibration software, shown in figure 4. The 5DT Data Glove Ultra is available in left and right variants.1333505334033528062865Figure SEQ Figure \* ARABIC 3 - 5DT Data Glove UltraFigure SEQ Figure \* ARABIC 4 - GloveManager SoftwareHuman GloveThe HumanGlove, shown in figure 5, developed by HumanWare CITATION Hum10 \l 2057 (HumanWare, 2010), is a glove-based system developed primarily for use in medicine, rehabilitation, VR and Telerobotics. The glove uses Bluetooth technology by default, emulating an RS 232 port in software and uses a total of 22 hall of effect sensors to measure flexion/extension and abduction/adduction (2 sensors per finger, 2 for the thumb and 2 for the wrist). Like the 5DT described above, the HumanGlove uses proprietary software for calibration, in this case, a package called “Graphical Virtual Hand” shown in figure 6. Pricing information for the HumanGlove is not readily available.266700539751143050165Figure SEQ Figure \* ARABIC 5 - HumanGloveFigure SEQ Figure \* ARABIC 6 - Graphic Virtual Hand SoftwarePeregrine Gaming GloveThe Peregrine Gaming Glove, shown in figure 7, is a glove-based system developed by Peregrine Canada CITATION Per13 \l 2057 (Peregrine, n.d.). The glove is designed for use in games where the number of actions available to the player is vast, games such as MMORPGs or MOBAs. The glove has 18 touchpads, 3 activator pads together with stainless steel conductive traces; allowing support for 30 programmable actions configures using the proprietary GloveBox software shown in figure 8. However, the glove cannot sense flexion/extension or abduction/adduction of the fingers or thumb, instead, the glove detects the thumb as it touches one of the 18 touchpads lining the fingers. The main attraction of the Peregrine Gaming Glove from a hand rehabilitation standpoint is the price; the glove has a unit price of $149.95 Canadian (?84.34) which has allowed students to use the glove in numerous rehabilitation system oriented projects CITATION Bre10 \l 2057 (O'Donnell, 2010), CITATION Roi12 \l 2057 (Lautman, 2012).5619759588510795100330Figure SEQ Figure \* ARABIC 7 - Peregrine Gaming GloveFigure SEQ Figure \* ARABIC 8 - GloveBox Configuration SoftwareWhile glove-based systems offer a wealth of information to developers and researchers, it should be noted that glove-based system suffer from a vast number of flaws. Glove-based systems suffer from robustness and durability issues due to the Lycra fabric, this lack of durability is exacerbated by the price of these systems. Issues of portability when one is tethered to a computer should also be considered (again, the extra cost for wireless options exacerbates this) in addition to the need for constant calibration. The most relevant draw-back of these system from a hand rehabilitation standpoint however, is the simple fact that conditions such as rheumatoid arthritis, shown in figures 9 and 10, can leave a patient unable to even wear the glove.297180101600327660104775Figure SEQ Figure \* ARABIC 9 - Rheumatoid Arthritis HandFigure SEQ Figure \* ARABIC 10 - Rheumatoid Arthritis Hand2.4 The use of Non Glove-Based Technology in Hand RehabilitationThough glove-based systems have proven extremely popular and effective, they do suffer from drawbacks as we have seen. Issues with pricing, durability and simply being unable to wear the glove due to conditions such as rheumatoid arthritis have generated great need, interest and opportunity for non-glove-based systems.OpenCVThe Open Source Computer Vision library or OpenCV is an open source computer vision initiative. Started by Intel in the mid to late 90’s and released to the public in 2000. The project has since been handed over to Willow Garage and Itseez, ensuring a continuing release schedule.OpenCV contains over 500 C/C++ based functions, allowing for a vast array of computer vision based applications, including medical imaging, security and robotics CITATION Bra08 \l 2057 (Bradski & Kaehler, 2008). The library is compatible with a wide range of commercially available camera equipment, the camera uses the position and colour of a pixel to build up a matrix of numbers, this matrix is then passed to the program. OpenCV has been shown to computationally outperform other computer vision libraries such as LTI and VXL CITATION Bra08 \l 2057 (Bradski & Kaehler, 2008). Furthermore, OpenCV can benefit by as much as 20% from IPP, if they are present in the host system. This makes OpenCV a powerful and accessible library for computer vision. Such a resource would potentially be a good supplement for glove-based systems; however, we do not plan to use such a supplement technology in our Leap Motion-based system.KinectThe Kinect, shown in figure 10 is a gesture control device primarily aimed at gaming applications for the Xbox 360 and later Windows based PCs. However, since its initial release, engineers both professional and hobbyist have used it in a wide array of applications ranging from robot guidance CITATION Eva11 \l 2057 (Ackerman, 2011) to hand rehabilitation shown in figure 11 CITATION Aar11 \l 2057 (Bond, 2011). The Kinect is made up of three main sensors, the first of which is an IR depth-finding camera used to read input in the IR spectrum, the second is an IR transmitter and the third is a standard RGB camera. Both the IR depth-finding camera and the RGB camera run at a resolution of 640x480 with a frame-rate of 30 frames per second.The appeal of the Kinect with regards to hand rehabilitation lies in the fact that it is relatively low priced compared to the glove based systems described above, retailing for approximately ?85. This can allow for a high adoption rate among patients, furthermore, the Kinect is not bound by the issue of right VS left handedness, the same unit can be used to train either hand, whereas with glove-based systems, a second glove would have to be ordered. Lastly, the Kinect is much more durable and robust, gloves wear out over time to the point where they must be replaced, which is costly, and a Kinect by comparison may never need to be replaced.While an interesting device, the Kinect does suffer from an array of drawbacks; for example, the device only supports a field of view of 57.8?. However, the main drawback from a hand rehabilitation standpoint is unquestionably its minimum range of 0.6m; there is however third party lenses that try to reduce this with some success CITATION Nyk13 \l 2057 (Pc Mag, n.d.).4762538101733553810Figure SEQ Figure \* ARABIC 11 – KinectFigure SEQ Figure \* ARABIC 12 - Kinect in Hand Rehabilitation SettingLeap MotionThe Leap Motion, shown in figures 12, is a recently released (mass shipping began July 2013) motion-based device for computer interaction developed by Leap Motion Inc. CITATION Lea13 \l 2057 (Leap Motion Inc, 2013) who claim the device offers accuracy to within 0.01mm.The device, as shown in figures13 and 14 is made up of 2 monochromatic IR cameras (the grey dots in figure 13) and 3 infrared LEDs (the red dots in figure 13), giving the device a semi-spherical observational area with a distance of approximately 1 meter. This observational area is smaller than that of the Kinect, which is designed to monitor the entire body; however this allows the Leap Motion to operate at a higher resolution and accuracy where accuracy is defined as “the ability of a 3D sensor to determine a desired position in 3D space” CITATION Fra13 \l 2057 (Weichert, et al., 2013). The IR cameras can run at up to 300 frames per second (as opposed to 30 with the Kinect) while the LEDs generate a 3D pattern of dots made up of IR light CITATION You13 \l 2057 (Anon., 2013). A study on the accuracy of the Leap Motion found that while the claimed 0.01mm accuracy is not achievable, a high precision accuracy of 0.7mm was CITATION Fra13 \l 2057 (Weichert, et al., 2013) achievable. 24892047625Figure SEQ Figure \* ARABIC 13 - Leap MotionFigure SEQ Figure \* ARABIC 14 - Leap Motion Schematic ViewThis makes the Leap far superior to the standard deviation of 1.5cm (15mm) found in the Kinect. How the Leap Motion views the users hands can be seen in figure 15, where the freely bundled “Leap Motion Visualizer” software is demonstrated.In addition to the technical improvements, the Leap Motion enjoys other benefits over previous systems. Firstly, the Leap Motion is more affordable than any other device discussed here - even the Kinect (?85) - the Leap Motion is currently available for ?65. The Leap Motion also benefits from its small size, coming in at 0.5 inches in height, 1.2 inches in width and 3 inches in depth with a weight of only 0.1 pounds CITATION Lea13 \l 2057 (Leap Motion Inc, 2013), making it more portable than any other device discussed here. Another advantage of the Leap Motion (this one it shares with the Kinect) is durability; the Leap Motion is not prone to wear and tear that eventually claims many a glove-based system.247015666755905566675Figure SEQ Figure \* ARABIC 15 - Leap Motion InternalsFigure SEQ Figure \* ARABIC 16 - Leap Motion VisualizerIt is clear that the Leap Motion is more accurate, more affordable and more portable than anything that has come before it. Due to these advantages, we have chosen Leap Motion as the means for delivering the system proposed in this project.2.5 The Role and Potential of GamificationGamification can be defined as “the use of game design elements in non-game contexts” CITATION Seb11 \l 2057 (Deterding, et al., 2011). Gamification is a fast growing initiative, with the aim of increasing motivation and participation among users of non-game applications and is expected to revolutionise all aspects of life in the not too distant future CITATION TCh10 \l 2057 (Chatfield, 2010), CITATION Jes11 \l 2057 (The Pleasure Revolution: Why Games Will Lead the Way, 2011).One of the first examples of gamification been used in a popular commercial product would be the achievement system used in the Microsoft Xbox360 console CITATION Mik11 \l 2057 (Jakobsson, 2011). The achievement system allows users to complete in-game challenges and accumulate “Gamerscore” as shown in figure 16. Due to its success, the system has since been implemented in numerous other platforms including the Sony PlayStation 3 and the popular PC Steam network. An ideal example of game design elements being used in a non-game context however would be the Khan Academy CITATION Kha13 \l 2057 (Khan Academy, 2013). The Khan Academy is a non-profit organisation with the aim of providing “a free world-class education for anyone anywhere”. The site allows users to watch videos on a wide variety of educational topics, complete exercises for which they can build up streaks, earn badges and a Gamerscore-like collection of points in addition to an array of real-time stat tracking tools as seen in figure 17.133350914409715543815Figure SEQ Figure \* ARABIC 17 - Xbox 360 AchievementsFigure SEQ Figure \* ARABIC 18 - Khan Academy Stat Tracking and Achievement SystemMore relevant to this project however, is the use of gamification in a medical and rehabilitation setting. CITATION Kat11 \l 2057 (Gerling & Masuch, 2011) Explore the application of gamification in augmenting the lives of frail elderly people who are no longer able to participate in certain real-life activities due to age (such as a recreational walk through a forest) shown in figure 18. They suggest that if we are able to overcome challenges such as the lack of experience with digital games and systems then elderly users can benefit not only cognitively, physically thanks to increased participation in therapeutic activities, but also socially from the experience, as gamified applications offer the opportunity for friendly competition.Gamification and ‘gamified applications’ such as serious games have been proven to work in medical undertakings such as stroke rehabilitationCITATION JWB09 \l 2057 (Burke, et al., 2009). The authors look at the use of gamified applications in helping those affected by strokes regain control of the affected limbs. Their results show that gamified applications can be used to help solve a common issue experienced by many stroke survivors undergoing therapy. The issue being that the everyday actives assigned to them as part of their rehabilitative therapy are boring and uninteresting. Couple this with the depression that is common among stroke survivors and the result is low user enthusiasm, low participation and poor results in terms of limb functionality regained through therapy. The study proves that gamification can make activities these engaging and stimulating, encouraging user participation and by extension, leading to better results in terms of regained functionality. 135255-2162810Figure SEQ Figure \* ARABIC 19 - Gamified Application, Walking through a ForestFigure SEQ Figure \* ARABIC 20 - CONTRAST Serious Rehabilitation GameThe study that is of most relevance to this project however is that conducted in CITATION Ard \l 2057 (Jacobs, et al., 2013), where the authors investigate the use of gamified applications in arm-hand training for stroke survivors. Here, a proprietary ‘serious game’ (a form of gamified application) named CONTRAST, shown in figure 19 was used wherein the user completes task-oriented exercises involving the manipulation of everyday items. Results of the study show increased user participation and by extension, improved arm-hand functionality. They point out that gamified applications make rehabilitative exercises “meaningful…”However, research would suggest that gamification has yet to be used in a hand-rehabilitation setting despite the fact that both hand rehabilitation and gamification place emphasis on identifying the user/patients personal goals “incorporating usual and customary occupational activities into treatment…” CITATION Deb11 \l 2057 (Amini, 2011), likewise, making the experience relevant to the user is also an essential part of gamification “… it is important to catch the user’s personal goals…” CITATION Gro12 \l 2057 (Groh, 2012).3: Requirements Analysis and Specification3.1 Problem StatementPeople injure their hands in all manner of ways and as a result of which normally have to go through a period of rehabilitation, this rehabilitation period will typically include prescribed exercises aimed at restoring pre-injury functionality. In addition to restoring pre-injury functionality, these exercises are an important part of preventing the build-up of scar-like tissue that would otherwise have a negative effect on the functionality of the patient’s hand. The results of these exercises are typically timed or monitored to identify progress and condition development.This project aims to develop a system to guide the user through rehabilitative exercises whilst tracking their progress. The novelty and contribution however is that we are using the Leap Motion as our measuring device – our technological aid or medium if you will. Though numerous glove-based systems and other solutions such as the Kinect have been experimented with, the Leap Motion has yet to be tested in this setting.Like similar systems which have come before, the system will be able to establish a database connection and store the results of exercises undertaken by the user. These results can later be viewed by the patient or clinician in an easy to interpret graph.Furthermore, we aim to enhance the system by adding gamification elements in an effort to further encourage sustained participation among patients. Gamification has been proven to work in areas one would not normally associate with games; areas such as education and indeed medical systems. However, gamification has yet to be used in the area of hand rehabilitation.3.2 Functional RequirementsIn this report, we view functional requirements as those which directly describe the intended functionality and behaviour of the system – what it should do. The functional requirements identified for this project are:System should allow clinician to create an account for themselves and patients (if the patient has not done so) and allow a patient to create an account for themselves onlySystem should allow for a simple and straightforward login processSystem should allow the user to choose an exercise to performSystem should track the user’s hand movements during the exercise sessionSystem should provide feedback, preferably graphical (such a 3d hand animation) or at the very least text-based feedbackSystem should present user with their results and save them in a database for easy retrieval by the patient or clinicianSystem should allow exercise progress and results to viewed by the patient or clinician in chart formSystem should shutdown gracefully and correctly, confirming user’s wish to close and ensuring all important data is savedSystem should be able to handle unexpected difficulties such as the absence or removal of the Leap Motion or being unable to connect to the data-baseSystem should run on Windows 7 or 8 in accordance with Leap Motion minimum system requirements3.3 Non-Functional RequirementsIn this report, we view non-functional requirements as those which are “not directly concerned with the specific services delivered by the system…” or “may define constraints on the system implementation…” CITATION Ian10 \l 2057 (Sommerville, 2010). The non-functional requirements identified for this project are: System should be robust and durable, the system may potentially see daily use in clinics, not just in a patients home, so robustness and durability are important propertiesSystem should be easy for the user to operate, this is especially in the case of patients whose injuries make interaction with computers difficultSystem should provide guidance on how to perform exercises, this is especially true in the case of home use where a clinician is not there to provide guidance and assistanceSystem should capture accurate readings of the patient’s hand movements, inaccurate readings lead to inaccurate results and ultimately defeat the purpose of such systemsSystem should provide a meaningful use of gamification elements. As described above, gamification is more than simply applying badges and scores to a system, the gamification elements must encourage the patient to pursue end goals which are important to them3.4 Hardware RequirementsThe following hardware is considered essential to the project:At least one PC must be available at any time for development, testing and demonstration purposes. The PC in question must meet the minimum system requirements for the Leap Motion:Windows 7 or 8, Mac OS X 10.6 Snow LeopardAMD Phenom II or Intel Core i32GB RAMUSB 2.0 portLeap Motion Device3.5 Software RequirementsThe following software is considered essential to the project:Windows 7 Operating System (any version)Visual Studio 2010 (any version).NET 4.0XNA Framework 4.0Leap Motion DriversLeap Motion SDK (C# interfaces)3.6 Development MethodologyIt is important in any project that the most suitable development methodology be chosen. Failure to choose correctly can have potentially disastrous results as different methodologies place priority on different aspects of the product and its development process.For this project, we have chosen the Rapid Application Development methodology. This project is largely research-based and dynamic in nature; system requirements and expected functionality may be subject to change as development progresses, going against the traditional Waterfall model, wherein development follows a linear progression from one stage to the next and where concise development strategies and goals are expected for each stage. In addition, this project will include constant user-input from the project supervisor in addition to (hopefully) and others with relevant insight and experience in the problem domain, making methods such as Spiral and Prototyping unsuitable.4: Project Planning4.1 Milestones and DeliverablesIn this report, we define a milestone as a critical point in the projects development and a deliverable as a tangible result of successfully met milestones.The milestones for this project are:Completion of literature reviewCompletion of system specificationCompletion of interim reportCompletion of proto-typeCompletion of application ‘final’ buildCompletion of final reportThe deliverables for this project are:Interim reportFunctioning proto-typeFunctioning ‘final’ buildFinal reportSystem discussion and demonstrationAgain, it is critical that milestones be met and corrective action taken if this proves not to be the case.4.2 Project PlanWe now present a WBS in figure 20 wherein we explicitly list the tasks required to complete the project, breaking those tasks down into smaller sub-tasks where necessary. These tasks are listed in chronological order.Figure SEQ Figure \* ARABIC 21 – Work Breakdown StructureTask 1: Literature ReviewThis task is concerned with the collection and analysis of reference material. Hand injuries, hand rehabilitation, technology used for hand rehabilitation and gamification will be the main areas of focus leading up to the interim report. This will grow to incorporate general software engineering theory when designing and implementing the system.Task 2: Interim ReportThe interim report is concerned with presenting the problem the project aims to solve. Background on hand injuries, hand rehabilitation and current technological efforts will be covered in addition to project planning.Task 3: System Design and SpecificationDecisions related to the technologies used for development will be made here. These include decisions about programming language and environment, as well as possible third party technologies. The system design and layout (class listing, coding standards etc.) will also be decided here. The design and specification will be documented as part of the final report.Task 4: System PrototypeThe first iteration of the system will be implemented, tested and feedback obtained. Testing will be primarily aimed at ensuring the system functions correctly and adheres to the specification. Feedback will be sought from the project supervisor and other relevant individuals if possible.Task 5: System ‘Final’ BuildThe iteration to be used for the final discussion and demonstration will be implemented. Testing will be aimed more at ensuring issues and feedback received on the prototype has been correctly implemented. Feedback will again be sought from the project supervisor and other relevant individuals if possible to ensure that this is so.Task 6: Final ReportThe completion of the final report will be largely focused on discussing whether or not the final system is fit for purpose and whether the project has been successful.4.3 Time ManagementWe now present a Gantt chart in figure 22, detailing the time assigned to each task listed above. The term “float time” refers to a small number of days which are left free to try and absorb sub-tasks that encounter difficulties or run over-time.-1381125231076500 Figure SEQ Figure \* ARABIC 22 - Gantt chart In addition, steps have been taken to measure how time is spent during a typical day. The aim here is to identify ways in which each day can be more effectively utilised. Table 2 shows a typical day, how time is used and how it can be used more effectively going forward.TimeActivityEffective-nessComments8:40 – 9:00Get ready for university80%Essential: Time used efficiently9:00 – 9:15Travel to university100%Time fully optimized9:15 – 11:15Class100%Cannot be skipped11:15 – 2:15Lab session70%Work could be completed faster2:15 – 3:00Relax50%Meeting prep will be done prior.3:00 – 4:00Meet with supervisor80%Use meeting template below.4:00 – 5:30Lunch and MSc course research60%Serviceable/Essential: Could spend a little less time on this5:30 – 8:00Project and non-project coursework80%Other modules cannot be ignored. Time-boxing will ensure optimal time usage here.8:00 – 9:00Dinner100%Essential: Need to eat9:00 – 10:15Project work50%Efficiency lost due to time of day? Move MSc research here instead?Table SEQ Table \* ARABIC 2 - Time Usage during a Typical Day4.4 Risk ManagementFor this project, steps have been taken to identify the main risks; event-driven, evolving in addition to technical and non-technical. In this report we measure risks using the system described in CITATION Joh93 \l 2057 (Turner, 1993) where risks are assessed by multiplying the likelihood of the risk (where 1 = low, 2 = medium and 3 = high) by the consequence of the risk (where 1 = very low, 2 = low, 3 = medium, 4 = high and 5 = very high). We then classify these values using RAG (where 1-5 = green, 6-10 = amber and >10 = red). We present identified risks and counter-measures in table 2. Avoidance refers to reducing the chances of a risks occurrence, whereas contingency refers to accepting that the risk may occur and putting plans in place to deal with the consequences should it occur.DescriptionTypeClassificationRA-GTrigger(s)Response/AlleviationCrashEventTechnical3N/AContingency: keep flash-drive and cloud back-ups.IllnessEventNon-Technical3N/AAvoidance: maintain good personal hygiene. Contingency: Incorporate float-time into schedule.Family issueEventNon-Technical3N/AContingency: Incorporate float-time into schedule.Hardware failureEventTechnical5Recurring system problems.Contingency: use backups and setup backup workstation.Insufficient technical knowledge / Technical implementation difficultiesEvolveTechnical8Recurring difficulties in designing and implementing theory.Avoidance: ensure extensive literature survey and research.Contingency: incorporate float-time into schedule.Missing dead-lines and general falling behindEvolveNon-Technical10Repeatedly failing to meet personal goals.Avoidance: ensure that workload and scope are realistic.Contingency: incorporate float-time into schedule.Table SEQ Table \* ARABIC 3 – Project Risks and Counter-Measures5: ConclusionBy now the report has identified a clear need for an accurate, affordable and portable hand-rehabilitation system. Through the literature review in chapter 2 we have seen that hand injuries are in fact a world-wide issue and can have a potentially devastating impact on individuals as well as those around them. We reviewed current solutions both glove and non-glove-based and have noted that these solutions suffer from issues such as affordability, accuracy and durability.We propose a new system aimed at overcoming the drawbacks described above. By using a new device – the Leap Motion – and the incorporation of gamification, we aim to develop a system that can improve system accuracy and reliability in addition to user participation and enjoyment. The system is intended to feature a real-time, 3D animated model of a human hand; this will provide real-time feedback to the user. In addition, user results will be recorded for later viewing by patients and clinicians; the system will also be one of the first to incorporate aspects of gamification with the aim of improving user enjoyment and by extension, continued user participation. A detailed description of this system has been given in chapter 3, detailing the exact requirements and expected functionality of such a system. This system description is supported by chapter 4, wherein we discuss exactly how we aim to use the resources at our disposal in order to give ourselves the best chance of success.Again, there is a clear need for an accurate, affordable and portable hand-rehabilitation system. The system proposed in this report, through the coupling of new technology – the Leap Motion - and the incorporation of effective yet largely under-utilised theory – gamification -, offers the best opportunity for realising such a system.6: Design6.1 System Component OverviewA visual layout of the system components and how they fit together is now presented; the diagram includes all the major components of the system (the patient/clinician machine, the Leap Motion and the user data). The patient’s machine interacts with the Leap Motion controller and both the patient and the clinician machines interact with the user data (exercise times, results etc…). The most common approach to storing the user data would be in a database of some description. 220980019900902324100230505Clinician Computer6667564135User Data42957759525Leap MotionPatient ComputerFigure SEQ Figure \* ARABIC 23 - System Architecture6.2 HCI Rules & GuidelinesThe system we aim to develop must be suitable for use both by patients and clinicians. In addition to this, both the patient and clinician variants of the system will consist of multiple screens through which the user can navigate and interact with. It is therefore essential that the system use a consistent and easy to comprehend UI scheme throughout. Here we present Shneiderman’s “8 Golden Rules of Interface Design” CITATION Shn05 \l 2057 (Shneiderman B, 2005), these guidelines will be used to inform the systems UI design.Rule 1: Strive for consistency. This rule permeates all aspects of the system. UI layout, colour scheme, format and use of language should be consistent throughout. Exceptions to consistency should be minimized. The system being developed in this project will use the same UI layout, colour-scheme and style of language throughout all its screens. An exception might be calibrating the Leap Motion controller before the user starts a session; in this case, calibration will be done once at the beginning of the session, rather than at the beginning of each individual exercise to minimize these exceptions to consistency.Rule 2: Cater to universalizability. A system should cater to users of diverse backgrounds (experience with the system, technical literacy, age etc…). The system being developed in this project will support this rule though the use of succinct yet informative instructions (good for novice users), tool-tips and the use of short-cuts (good for more experienced users).Rule 3: Offer informative feedback. Any and all user actions (from minor to the more infrequent and major) should be met with a suitable response.The system being developed will provide visual confirmation and feedback when the user interacts with the application. This will range from controls (such as buttons and drop-downs) responding as expected, to the system using dialog screens to confirm the users intent for more major actions (like quitting a session).Rule 4: Design dialog to yield closure. All sequences of actions in a system should have a beginning, middle and an end with informative feedback at the end.The system being developed will provide encouraging and informative at the end of an exercise session, with a screen allowing the user to immediately logout or to return to the main menu (thus providing closure).Rule 5: Prevent errors. The design of a system should make the probability of error miniscule and should an error occur, offer supportive and informative feedback allowing the user to easily correct the error. The system being developed will employ techniques such as greying out non-applicable options (e.g. trying to move to the next exercise before the current exercise is finished), type-checking data entered into text-fields, prompting for confirmation when a user-action may result in a loss of progress or data (e.g. quitting an exercise before completion) and will use a supportive and informative style of language for any and all error messages.Rule 6: Permit easy reversal of actions. As much as possible, actions should be reversible to promote experimentation and familiarisation with the system whilst relieving any anxiety.The system being developed will allow for this by allowing the user to recalibrate and continue a session should they remove their hand from the Leap Motions field of view and by always allowing the user to return to the main menu or the previous screen (if applicable).Rule 7: Support internal locus of control. Experienced users wish to feel like they are controlling and driving their interaction with the system. In addition, surprising or tedious actions build anxiety and dissatisfaction in the user.The system being developed will adhere to this rule by requiring the user to be the initiator, not the responder. Any and all actions must be initiated by the user and can be stopped by the user at any time. The system will never force anything upon the user.Rule 8: Reduce short-term memory load. There is a rule of thumb regarding human information processing in short-term memory: “humans can remember seven plus or minus two chunks of information”. To this end, displays should be kept simple, multiple screens consolidated and sufficient training time allotted for learning any codes, mnemonics or intricacies.The system being developed will adhere to this rule by keeping the number of individual screens to a minimum (a screen with date selection for results giving way to a new screen with a graph could be reduced to one screen for example). In addition, the system will use a consistent layout and theme so as not to surprise or overload the user.6.3 UI Mock-UpsWe now present UI mock-ups for the system, these mock-ups better describe what an average user will see when they interact with the system. Where possible, the UI has been designed to adhere to the rules described above. LoginFigure SEQ Figure \* ARABIC 24 - Login ScreenIn Figure 24 we have the login screen; this screen is re-used for both clinician and patient users. Note the use of calming colours (the blue header) as opposed to colours such as red seen in previous systems (which instil a sense of anger and or panic) along with friendly language (the use of ‘please’ and avoidance of jargon such as ‘credentials’).Main MenuFigure SEQ Figure \* ARABIC 25 - Clinician Main Menu ScreenshotFigure SEQ Figure \* ARABIC 26 - Patient Main Menu ScreenshotFigure 25 and Figure 26 show the main menu for a clinician user; this is where usage paths between clinician and patient users start to diverge. A clinician user has the ability to manage patient accounts (via register and remove) whereas patient users have the option to perform exercises. Both users share the ability to view results and logout.View Exercise Results-4762523495-9525023495Figure SEQ Figure \* ARABIC 27 - Clinician versionFigure SEQ Figure \* ARABIC 28 - Patient Version REF _Ref381625732 \* MERGEFORMAT Figure 27 and REF _Ref381625733 \* MERGEFORMAT Figure 28 demonstrate the view exercise results screen. Here we see with the clinician version in REF _Ref381625732 \* MERGEFORMAT Figure 27 and the patient version in REF _Ref381625733 \* MERGEFORMAT Figure 28. Both versions will allows the user to view results by exercise type as well as being able to specify a timeframe (via start and end dates) for the results. The only variation is the ability of the clinician to view the results of numerous patients, whereas a patient can only view their own results.Manage PatientsFigure SEQ Figure \* ARABIC 29 - Register PatentFigure SEQ Figure \* ARABIC 30 - Patient ScreenFigure 29 shows the register patient screen and Figure 30 shows the remove patient screens. These are exclusive to clinician users. The clinician user must provide a certain amount of information about a new patient before they can be registered. In the event of removing a patient, this is done by searching for the patient’s username.Performing Rehabilitative ExercisesFigure SEQ Figure \* ARABIC 31 - Rehabilitation Exercise ScreenFigure 31 shows the exercise screen which is exclusive to patient users. On the left of the screen, a real-time 3D animated hand model will be used to provide real-time feedback to the user. Information such as exercise instructions, repetition count and time taken will be documented on the left of the screen. Possibilities for continuing to the next exercise include a button (as seen in the image), holding ones hand still or possibly leveraging the Leap Motions built in swipe gesture. The back button will be used to return to the main menu.6.4 Diagrams (Use case and Activity Model)This section outlines various interactions possible between the system and various types of user (both patient and clinician). We first present a use-case diagram which is used to describe the system utilities available to each type of user. We now supplement the use-case diagram above with the following activity models; these are used to describe what a “typical session” with the UI and functionalities described above may look like for a user with access to those system utilities. First, a typical patient user is shown in Figure 32.Figure SEQ Figure \* ARABIC 32 - Patient User & Clinician UserFigure SEQ Figure \* ARABIC 33 - Patient User InteractionsLastly, we present the activity diagram for a typical clinician user:Figure SEQ Figure \* ARABIC 34 - Clinician User Interactions7: Implementation7.1 Technical OverviewThe technologies used in the LMRS are: eXtensibe Mark-up Language (XML), Windows Presentation Foundation (WPF), Leap Motion controller and Microsoft XNA.7.2 XML Data StoreThis initial version of the LMRS uses XML as its storage medium for user data (both credentials and exercise related). XML was chosen due to a combination of factors, these being mainly familiarity and time constraints. The custom XML format used for the LMRS takes on the following form:<?xml version="1.0" encoding="utf-8"?><LMRS_File> <clinicianCollection> <clinician username="c_curranK" password="password0"> <forename>Kevin</forename> <surname>Curran</surname> </clinician> <!-- Additional clinicians here... --> </clinicianCollection> <patientCollection> <patient username="p_taylorJ" password="password1"> <forename>Jamie</forename> <surname>Taylor</surname> <addressLineOne>6 Knock Eden Close</addressLineOne> <addressLineTwo>Town</addressLineTwo> <addressLineThree>Co Antrim</addressLineThree> <postCode>BT53 6UE</postCode> <activity name="Fist_Clench"> <session date="01/01/14"> <rep>1.23</rep> <rep>1.24</rep> <rep>1.55</rep> <rep>1.35</rep> <rep>1.37</rep> </session> <!-- Additional sessions here... --> </activity> <!-- Additional activities here... --> </patient> <!-- Additional patients here... --> </patientCollection></LMRS_File> Listing 1 - LMRS Data Storage FormatWe first have the file element, followed by a collection element (holding either a collection of patient or clinician elements). These patient and clinician elements then hold information such as user name, password, actual name and/or address and in the case of patients: exercise type and repetition times.Meta-data (like session dates and usernames) is stored in attributes, with “actual” data stored as child elements. The “p_” and “c_” sections of a username allow us to quickly identify the user type and immediately jump to the appropriate part of the file with which to work with.7.3 Patient SideWe turn first to the “patient side” of the system; this is the system a typical patient will interact with.We will start with the login screen seen in REF _Ref386139571 \h Figure 35 - LMRS Login Screen. All individual screens in the LMRS are made using WPF and share common traits.Figure 35 - LMRS Login ScreenThe main functionality of this page can be seen in REF _Ref386141450 \h Listing 2. After initial sanity checks (such as ensuring both fields have text), we jump to the appropriate part of the XML data store based on the username prefix mentioned earlier; proceeding to the main menu if the credentials are found to be correct and simply presenting an error and clear the fields if they are not.private void btnLogin_Click(object sender, RoutedEventArgs e){ // Trivial sanity checks ommitted... XDocument lmrsXml = XDocument.Load("LMRS_DataStore.xml"); if (txtUsername.Text.Contains("p_")) { // Find patient user XElement patientCollection = ???? ???????? ????????????lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in ???????? ????????????patientCollection.Elements("patient") ? where (string)el.Attribute("username") == ????????????txtUsername.Text select el; if (patient.Count() == 0) { MessageBox.Show("Please provide a valid username.", "Cannot Login", MessageBoxButton.OK, ???????????? MessageBoxImage.Error); txtUsername.Text = pwbxPassword.Password = ""; } else { XElement userOnRecord = patient.First<XElement>(); if (userOnRecord != null && ???????????? ????????????????pwbxPassword.Password.Equals(???????????? userOnRecord.Attribute("password").Value.ToString())) { UserInfoHelper.user = userOnRecord; UserInfoHelper.foreName = ???? ????????????????????userOnRecord.Element("forename").Value; UserInfoHelper.userName = ????????????????????userOnRecord.Attribute("username").Value; this.NavigationService.Navigate(new ???????????????? ????????????????????LMRS_MainMenu_Patient()); } else { MessageBox.Show("Please provide a valid password.", "Cannot Login", MessageBoxButton.OK, ???????????? ????????????????MessageBoxImage.Error); ???????? pwbxPassword.Password = ""; } } } // btnLogin_Click() Listing 2 – Login FunctionalityThe main menu screen will not be covered as it is a simple collection of calls to the navigation system used to load the next screen depending on the user’s selection of either performing exercises, viewing results or logging out.We will describe the main focus of the patient side of the system and indeed that of the entire LMRS: the perform exercise functionality. This part of the system pulls together WPF, Leap Motion and XNA to deliver the user experience seen in REF _Ref386139716 \h Figure 36.Figure 36 – LMRS Perform Exercises Screen0-190500Upon loading the page, we create a separate thread for the XNA-powered 3d hand model and setup the Leap Motion, giving it its own background worker so it does not block the main thread (which would cause the application to hang).private void Page_Loaded(object sender, RoutedEventArgs e){ // Set XNA hand running IntPtr handle = RenderPanel.Handle; xnaThread = new Thread(new ThreadStart(() => { game = new SkeletalAnimationSample(handle); game.Run(); } )); xnaThread.Start(); // Set Leap Motion running listener = new LeapListener(); controller = new Leap.Controller(); controller.AddListener(listener); leapBackgroundWorker = new BackgroundWorker(); leapBackgroundWorker.WorkerSupportsCancellation = true; leapBackgroundWorker.DoWork += new ????????DoWorkEventHandler(leapBackgroundWorker_DoWork); leapBackgroundWorker.RunWorkerAsync();} Listing 3 – LMRS Perform Exercises Form Start-upThe Leap Listener class in REF _Ref386141622 \h Listing 3 is a class in Leap API used to provide listeners for key system events such as device connected, disconnected and frame to name a few. The implementation used for the LMRS is simply a standard listener with an added get method to help in retrieving the data from the most recent frame (listener can be inherited and extended with custom behaviours for each event).// Additional listener methods omitted...public override void OnFrame(Controller controller){ // Get the most recent frame from the device frame = controller.Frame();}public Leap.Frame LeapFrame{ get { return frame; }} Listing 4 – Leap Listener Get() AdditionThe background workers “DoWork” method is where the core logic of the perform exercise functionality resides. This method contains the code which polls the Leap Motion device, processes this data against the exercise logic and updates the 3D hand model.We start with the calibration phase, this is in fact not to calibrate the Leap Motion device itself, the calibration phase is actually required in order to collect data needed to calculate key metrics which will then be used in the exercise logic.void leapBackgroundWorker_DoWork(object sender, DoWorkEventArgs e){ while (true) { Leap.Frame frameData = ((LeapListener)listener).LeapFrame; if (frameData != null && frameData.Hands.Count > 0) { hand = frameData.Hands[0]; // Calibration #region Clibration if (!calibrated) { if (!calibratePromptGiven) { updateExerciseInstruction("Calibrating,???????????????????? please wait..."); calibratePromptGiven = true; } if (!timer.IsRunning) { timer.Start(); } // Collect hand pitch and vector magnitude readings, // large enough deltas in these parameters are used ??????????????? // to detect user movements and start the timer if (timer.Elapsed.Seconds < 5) { avgMag += (hand.Fingers.Frontmost.TipPosition - ????????????????????????hand.PalmPosition).Magnitude; avgPitch += hand.Direction.Pitch; ++calls; } else { avgMag /= calls; avgPitch /= calls; calibrated = true; timer.Reset(); } } // if (!calibrated) #endregion Listing 5 – LMRS Exercise CalibrationThe calibration process in REF _Ref386143296 \h Listing 5 will be repeated if the user takes their hand away from or goes out of range of the Leap Motion.These key metrics are described below and can also be seen in REF _Ref386139799 \h Figure 37:The magnitude of the vector between the front-most fingertip and the palm. A sizeable change (decrease) indicates that the front-most finger is getting closer to the palm meaning the user has begun to clench their hand into a fist. At this point we start the timer. Exercises 1 and 3 use this logic (?v>n? -> timer start).The average pitch of the hand (rotation about the x-axis). Again, a sizeable delta indicates the user has begun to perform the requested action. Like with exercises 1 and 3 we again use this sizeable change as an indicator to start the timer (? hand pitch >n? -> timer start).Figure 37 – Metrics used to recognise user initiation of exercisesWe now move onto the logic behind the rehabilitation exercises the user is required to perform. We will look at the first exercise where the user is required to go from holding their hand at rest, to forming a clenched fist before finally returning their hand to a resting position, the code for this exercise can be seen in REF _Ref386284111 \h Listing 6.#region exercise1if (currentExercise == 1 && repsPerformed < repsRequired){ if (!exerciseDescGiven) { updateExerciseDescription("Exercise 1 of 3: Fist Clench. " + "\n1) Form a closed fist. \n2) Relax hand. \n\nThe timer will start automatically."); exerciseDescGiven = true; } // Sufficient vector magnitude delta? Start timer if ((hand.Fingers.Frontmost.TipPosition - ???? ????????hand.PalmPosition).Magnitude < 0.9 * avgMag && ????????!timer.IsRunning) { restartTimer(); } Listing 6 – LMRS Exercise Instructions and timer startHere we check the current exercise index and ensure that there are still repetitions left for the user to perform. We then present the description/brief for this exercise if we have not already done so. Lastly, we use the first of the two metrics recorded during calibration to check for a sizable delta in the vector stemming from the palm to the tip of the front-most finger, this means the user has begun to move their hand and so we start the timer.// Exercise part 0 - Form a fist if (hand.Fingers.Count == 5 && !clenchPromptGiven) { // Present time for this rep if (timer.IsRunning) { repTimes[repsPerformed] = timer.ElapsedMilliseconds; updateRepInfo("This rep time: \nRep " + (repsPerformed + 1) + " of 5 : \n" + repTimes[repsPerformed] / 1000.0 + " seconds."); // New best rep time? if (repTimes[repsPerformed] < bestRepTime) { bestRepTime = repTimes[repsPerformed]; updateBestRepInfo("Best rep time: \nRep " + ????????????????????(repsPerformed + 1) + " of 5: \n" + bestRepTime / ?????????????????????1000.0 + "seconds."); } ++repsPerformed; restartTimer(); } // Give prompt if (repsPerformed < repsRequired) { updateExerciseInstruction("Clench your hand ????????????????into a fist"); clenchPromptGiven = true; restPromptGiven = false; } } // Exercise part 1 - Relax hand if (hand.Fingers.Count == 0 && !restPromptGiven) { updateExerciseInstruction("Place your hand at rest"); restPromptGiven = true; clenchPromptGiven = false; }} // if (currentExercise == 1 && repsPerformed < repsRequired)#endregion Listing 7 – LMRS Exercise 1 – Fist Clench core logicIn Listing 7, we first look for 5 fingers; this indicates a hand at rest. When this condition is met, we present the time for the current repetition and store the time if it is a new personal best before prompting the user to continue to the next rep by updating the exercise instruction field. The second condition we check for is a finger count of zero (along with the necessary prompt having been shown/not shown), a finger count of zero indicates a clenched fist. At which point we update the exercise instruction field and toggle the prompt displayed Booleans.The second exercise (Wrist Flexion & Extension) uses similar core logic but uses the second of the two metrics recorded during calibration (hand pitch) to help decide when the timer should be started and instructions. This can be seen in REF _Ref386143885 \h Listing 8.// Lower handif (hand.Direction.Pitch > avgPitch * 2.5 && repsPerformed < ????repsRequired && !lowerPromptGiven){ // ...}// Raise handif (hand.Direction.Pitch < -2.5 * avgPitch && repsPerformed < ????repsRequired && !raisePromptGiven){ // ...} Listing 8 – Exercise 2 – Wrist Flexion & Extension Key Metric ConditionalThe third and final exercise (Three Jaw Chuck Pinch) uses the exact same logic as the first exercise (Fist Clench) and will therefore not be covered. The only notable difference between them is the number of fingers being checked for at each stage of the exercise. Since the third exercise uses the thumb, index finger and middle finger, we check for 3 fingers instead of 5 followed by 1 instead of 0.After the data from the Leap Motion has been collected and used to progress the exercises, the last remaining task for the current loop iteration is to update the 3d hand model. The 3D hand model used in the LMRS is powered by version 3.5 of XNA and uses data from the Leap Motion to animate the bones of the model.Before animating the hand however, we must first calculate two key vectors, the angles between which will then be used to animate the hand model by setting the joints equal to these angles. The first of these two vectors covers the distance between the centre of the palm and the base of the finger in question; the second of the two vectors covers the distance from the base of the finger to the finger tip, these can be seen in REF _Ref386139863 \h Figure 38.Figure 38 – Vectors used for angle calculationHowever, before we can even calculate these two vectors, we must first calculate the base of the finger itself as this data is not readily accessible via the Leap Motion SDK, however, the Leap developers do provide a means to calculate this in the SDK documentation CITATION Lea131 \l 2057 (Leap Motion, 2013). These three vectors (finger base, palm to base and base to tip) along with the resulting angle data are calculated twice. Once for the thumb (using the left-most finger member provided by the Leap Motion SDK) and once for the four remaining fingers (all of which mimic the front-most finger member again provided by the Leap Motion SDK). Due to the Leap Motions lack of skeletal tracking however, it is currently near impossible to reliably identify individual fingers; left-most, right-most and front-most are the only ones reliably accessible through the SDK at present and even then are ambiguous (the left-most finger is the little finger of your left hand and is, at the same time, the thumb of your right hand for example). The code used to calculate these vectors can be seen in REF _Ref386144009 \h Listing 9. // ... } #endregion // Exercise 3/3 // Update hand model updateHandModel(frameData); } } // if (!calibrated) }} // leapBackgroundWorker_DoWork()// Update the hand model based on information from the Leap Motionprivate void updateHandModel(Leap.Frame frame){ // Calculate necessary vectors Leap.Vector fingerBase = -frame.Hands[0].Fingers.Leftmost.Direction * ????????frame.Hands[0].Fingers.Leftmost.Length; fingerBase = fingerBase + ???? ????????frame.Hands[0].Fingers.Leftmost.TipPosition; Leap.Vector palmCenterToBase = fingerBase -????????frame.Hands[0].PalmPosition; Leap.Vector baseToTip = ????????frame.Hands[0].Fingers.Leftmost.TipPosition - fingerBase; Listing 9 – Vector CalculationOnce these vectors have been calculated, they are then used to calculate the angles to which the joints in the 3d hand model will be set. For this, we borrow the following quadratic equation from CITATION Ger05 \l 2057 (Hillerbrand, et al., 2005):qα,β=0.23+1.73d+1.5d2. Originally, this equation was used to define the relationship between the bending angle of the outer-most and middle phalanx (α) and that of the middle and inner phalanx (β), with d denoting the distance between the base joint and fingertip relative to the finger length. For the LMRS, we use this quadratic for the 3d hand model; substituting the values 0.66 and 0.33 for α and β respectively for the index, middle, ring and little finger, for the thumb, we only use the 0.33 value. In addition to this, we multiply by the angle between the finger-base and the palm. This helps the finger bend in a realistic fashion despite only having the angle between the palm and finger base as our only accessible/calculable value. This can be seen in REF _Ref386144177 \h Listing 10 along with some basic clamping to prevent edge case issues from occurring such as fingers bending the wrong way should the Leap Motion lose sight/track of them, this code listing is for the index finger but the same values/angles are applied to the other three (middle, ring and little).// IndexfingerBase = -frame.Hands[0].Fingers.Frontmost.Direction * ????frame.Hands[0].Fingers.Frontmost.Length;fingerBase = fingerBase + ????frame.Hands[0].Fingers.Frontmost.TipPosition; palmCenterToBase = fingerBase - frame.Hands[0].PalmPosition;baseToTip = frame.Hands[0].Fingers.Frontmost.TipPosition - ????fingerBase;// Some basic clampingfloat fingerIndex3 = (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip))) ????> 0 ? 0 : (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip)));????fingerIndex3 = fingerIndex3 <= -1.25f ? -1.25f : fingerIndex3;float fingerIndex2 = (float)(((0.23 + 1.73 * 0.66 + 1.5 * ????(0.66 * 0.66))) * fingerIndex3);float fingerIndex1 = (float)(((0.23 + 1.73 * 0.33 + 1.5 * ????(0.33 * 0.33))) * fingerIndex3); Listing 10 – Finger Angle CalculationWe next calculate a pitch and yaw for the hand model (with pitch describing rotation about the x-axis and yaw describing rotation about the y-axis). To do this we can simply used the normalised direction property of the hand object in the Leap SDK, the only modifications we make are to tone down the yaw as leaving this value unaltered or too high was found to cause difficulties. // Calulate pitch and yawfloat pitch = -frame.Hands[0].Direction.Normalized.Pitch;float yaw = -frame.Hands[0].Direction.Normalized.Yaw; // Tone down yawyaw -= 0.5f;yaw /= 4;// Update handgame.CameraDown(pitch);game.CameraLeft(yaw); Listing 11 – Pitch & Yaw CalculationThe final result is shown in REF _Ref386140100 \h Figure 39.Figure 39 – Demonstration of XNA-powered 3d hand modelWe move now to the final part of the system with which a typical patient user may interact, this being the viewing and graphing of results for that user, this functionality can be seen in REF _Ref386140151 \h Figure 40.Figure 40 – LMRS Exercise Results ScreenThis is the clinician variation of the form being shown, however the main difference between this and the patient variation is the patient combo-box which is not visible for a patient user. We show the fully-featured clinician version to avoid needless repetition. This mention of form variation depending on user type would make the load and unload code a logical place to start; in REF _Ref386145338 \h Listing 12 we see the code responsible for ensuring that the appropriate features are made accessible depending on the user type.public LMRS_ViewExerciseResults(string userType){ InitializeComponent(); // Set the window title WindowTitle = "Leap Motion Rehabilitation System – ????????View Exercise Results"; // Load the XML file and populate the drop-downs lmrsXml = XDocument.Load("LMRS_TestFile.xml"); populatePatientComboBox(); populateExerciseComboBox(); this.userType = userType; if (userType.Equals("clinician")) { lblPatient.Visibility = Visibility.Visible; cmboPatient.Visibility = Visibility.Visible; } else { lblPatient.Visibility = Visibility.Hidden; cmboPatient.Visibility = Visibility.Hidden; }} Listing 12 – LMRS Toggle Patient Combo-boxNote that we are using a custom constructor and decide whether or not to make the patient selection functionality available depending on the user-type argument passed to us. In REF _Ref386145431 \h Listing 13, we see the relevant code from the patient and clinician menus.// LMRS_MainMenu_Patient.xaml.csprivate void btnViewExerciseResults_Click(object sender, ????RoutedEventArgs e){ this.NavigationService.Navigate(new ????????LMRS_ViewExerciseResults("patient"));}// LMRS_MainMenu_Clinician.xaml.csprivate void btnViewExerciseResults_Click(object sender, ????RoutedEventArgs e){ this.NavigationService.Navigate(new ????????LMRS_ViewExerciseResults("clinician"));} Listing 13 – Load either Patient or Clinician Main MenuFrom here on we will assume a typical patient user. Any further code relevant to clinician users will be covered when we look at the clinician side of the system. The first thing to do for a patient user is load the exercise combo box; this can be seen in REF _Ref386145592 \h Listing 14.// Load the exercise combo boxprivate void populateExerciseComboBox(){ // Now load the exercise combo box IEnumerable<XElement> exercises = (from el in ????????selectedPatient.Elements("activity") select el); ????this.exercises.Add(exercises.ElementAt(0).Attribute("name").Value????????.ToString().Replace("_", " ")); ????this.exercises.Add(exercises.ElementAt(1).Attribute("name").Value????????.ToString().Replace("_", " ")); ????this.exercises.Add(exercises.ElementAt(2).Attribute("name").Value????????.ToString().Replace("_", " ")); cmboExercise.ItemsSource = this.exercises; cmboStartDate.ItemsSource = cmboEndDate.ItemsSource = dates; chartDataList = new List<KeyValuePair<string, decimal>>();} Listing 14 – Load Exercise Combo-boxWe take each exercise from the XML data store, substitute the underscores in the XML for plain spaces and add the item to the list. Lastly, we set the combo-box’s data source to this enumerable. Next we look at the code for an exercise changed event, for this we simply fetch the newly selected exercise from the XML data store, load up the associated sessions and repetition times and add the session dates to the date combo-box’s item source. This can be seen.private void cmboExercise_SelectionChanged(object sender, ????SelectionChangedEventArgs e){ // Load available sessions for this exercise, first find the ????// chosen exercise in the file String name = cmboExercise.SelectedItem.ToString().Replace(????????" ", "_"); IEnumerable<XElement> exercise = from el in ????????selectedPatient.Elements("activity")????????where (string)el.Attribute("name") == name select el; chosenExercise = exercise.First<XElement>(); // Now get the sessions and rep times sessions = from el in chosenExercise.Elements("session")??????? select el; repTimes = from el in sessions.Elements("rep") select el; dates.Clear(); int i = 0; foreach (XElement exl in sessions) { dates.Add(exl.Attribute("date").Value); ++i; } // Tag: arbitrary object value that can be used to store custom ????// information about this element. Use it here to avoid ??? ????// triggering unwanted calls to refreshChart cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0; refreshChart();} Listing 15 – Exercise Selection ChangedThe use of the date combo-box’s tag property is to help avoid additional unwanted to calls to refresh chart which were causing crashes during development. The selection changed event for the start and end date combo-box’s is the exact same, we check to see if the call should be accepted or rejected, clamp the end date if necessary and finish with a call to refresh chart.private void cmboStartDate_SelectionChanged(object sender, ????SelectionChangedEventArgs e){ if (((string)cmboStartDate.Tag).Equals("ignoreCall")) { cmboStartDate.Tag = "acceptCall"; return; } // End date cannot be before the start date if (cmboEndDate.SelectedIndex < cmboStartDate.SelectedIndex) { cmboEndDate.SelectedIndex = cmboStartDate.SelectedIndex; } refreshChart();} Listing 16 – Accept or Reject Call Depending on TagThe refresh chart method is responsible for updating the chart when either the exercise or date selection (either start or end date) has been changed. After an exercise change, it is responsible for fetching the session data (between the start and end dates) linked with the newly chosen exercise. After a date change event, it is responsible for loading the newly selected set of session data.// Changing the dates via the combo boxes modifies the dataset used // by the chart.private void refreshChart(){ chartDataList.Clear(); for (int i = 0; i < ((cmboEndDate.SelectedIndex - ????????cmboStartDate.SelectedIndex) + 1) * 5; ++i) { chartDataList.Add(new KeyValuePair<string, ????????????decimal>(sessions.ElementAt(cmboStartDate.SelectedIndex + ????????????(i / 5)).Attribute("date").Value + ": Rep " + ((i % 5) + 1).ToString() + " ", ?????????????Convert.ToDecimal(repTimes.ElementAt<XElement>(i).Value)????????????)); } chrtBarChart.DataContext = chrtLineChart.DataContext = ????????chrtColumnChart.DataContext = null; ????chrtBarChart.DataContext = chrtLineChart.DataContext = ????????chrtColumnChart.DataContext = chartDataList; // Update page title lblHeader.Content = "Results for " + ????????exercises[cmboExercise.SelectedIndex] + " (" + cmboStartDate.SelectedValue + " - " + ????????cmboEndDate.SelectedValue + ")";} Listing 17 – Refresh/Reload Chart after Data ChangeThe three radio buttons on the form simply toggle between three types of graph, these being bar, line and column. The code for each of the radio button’s checked event handlers is the exact same, only toggling the visibility of different graphs.private void rdoBarChart_Checked(object sender, RoutedEventArgs e){ chrtBarChart.Visibility = Visibility.Visible; chrtLineChart.Visibility = Visibility.Collapsed; chrtColumnChart.Visibility = Visibility.Collapsed;} Listing 18 – Chart Type Visibility ToggleTo allow for these different types of graph we use several graph items in the xaml code for the page, the code for each is graph again the same, only the series type and name differ. The line chart in REF _Ref386146922 \h Listing 19 provides an example of this.<!-- Line chart --><chartingToolkit:Chart Name="chrtLineChart" ????Margin="27.151,171.48,28.58,30.009" Grid.Column="1" ????Visibility="Collapsed"> <chartingToolkit:Chart.Axes> <chartingToolkit:LinearAxis Title="Time in Seconds" ????????????Orientation="Y" Interval="0.1"/> <chartingToolkit:CategoryAxis Title="Date and Rep #" ????????? Orientation="X"/> </chartingToolkit:Chart.Axes> <!—- Series type --> <chartingToolkit:LineSeries Title="lineSeries ???? ????????DependentValuePath="Value" IndependentValuePath="Key" ????????ItemsSource="{Binding}"/> <!-- Hide the legend --> <chartingToolkit:Chart.LegendStyle> <Style TargetType="datavis:Legend"> <Setter Property="Width" Value="0" /> </Style> </chartingToolkit:Chart.LegendStyle></chartingToolkit:Chart> Listing 19 – Line Chart XAML7.4 Clinician SideNow we have finished looked at the patient side of the LMRS, we turn to the clinician side of the system.We will start where we previously left off on the patient side: the viewing and graphing of patient results. It was mentioned above that the results page either displays or hides a combo-box of patients depending on the type of user loading the form and some code to that affect was shown. We now look at the code responsible for loading that combo-box.private void populatePatientComboBox(){ // Now load the exercise combo box XElement patientCollection = ????????lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patients = (from el in ????????patientCollection.Elements("patient") select el); int i = 0; int j = 0; foreach (XElement el in patients) { ????????this.patients.Add(patients.ElementAt(i).Element("forename").????????????Value + " " + ????????????patients.ElementAt(i).Element("surname").Value); if (patients.ElementAt(i).Attribute("username").Value == ????????????UserInfoHelper.userName) { j = i; break; } ++i; } cmboPatient.ItemsSource = this.patients; cmboPatient.Tag = "ignoreCall"; cmboPatient.SelectedIndex = 0; selectedPatient = patients.ElementAt(j);} Listing 20 – Create and Populate Patient Combo-boxIn REF _Ref386200939 \h Listing 20 – Create and Populate Patient Combo-box, we add each patient user found in the patient collection element (found in the XML data store). The if statement is used when a patient user is loading the form. The condition stops the loop if the patient currently being added to the combo-box is the same patient user loading the form in which case we stop the loop and return to the constructor where the combo-box (and corresponding label) is then set to hidden, in the case of a clinician user however we continue loading in patients from the XML data store.The event handler for the patient combo-box’s selection changed event simply finds the appropriate patient element and updates the graph.private void cmboPatient_SelectionChanged(object sender, ????SelectionChangedEventArgs e){ if (((string)cmboPatient.Tag).Equals("ignoreCall")) { cmboPatient.Tag = "acceptCall"; return; } // Fetch the newly selected patient XElement patientCollection = ????????lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patients = (from el in ????????patientCollection.Elements("patient") select el); int i = 0; foreach (XElement el in patients) { if(cmboPatient.SelectedItem.ToString().???????????Contains(patients.ElementAt(i).Element("forename").Value) ???????????&& cmboPatient.SelectedItem.ToString().???????????Contains(patients.ElementAt(i).Element("surname").Value)) { selectedPatient = patients.ElementAt(i); break; } ++i; } cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0; cmboExercise.Tag = "ignoreCall"; cmboExercise.SelectedIndex = 0; // Update the graph refreshChart();} Listing 21 – Updating Results Chart on Patient Changed eventAnother unique functionality available only to clinician users is the ability to register new patient users as seen in REF _Ref386140245 \h Figure 41.Figure 41 – LMRS Register New Patient ScreenThe main functionality of this page is held within the register button’s click event seen in REF _Ref386201103 \h Listing 22. Here we ensure that all fields have been completed and that the user does not already exist, if no match is found we create a new user with the credentials being provided, if a match is found, we notify the user and clear the text fields.private void btnRegister_Click(object sender, RoutedEventArgs e){ // Necessary information provided? if (!textFieldsNotComplete()) { MessageBox.Show("Please ensure all text fields have been ????????????completed.", "Missing Information", MessageBoxButton.OK, ????????????MessageBoxImage.Error); } else { // Load the XML file XDocument lmrsXml = XDocument.Load("LMRS_TestFile.xml"); // Does the patient user exist already? String patientUsername = "p_" + txtSurname.Text.ToLower() + ????????????txtForename.Text.ToUpper()[0]; // Get patient collection node XElement patientCollection = ????????????lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in ????????????patientCollection.Elements("patient") where (string)el.Attribute("username") == patientUsername select el; if (patient.Count() != 0) { MessageBox.Show("A patient is already registered with ????????????????those details.", "Patient Already Exists", ????????????????MessageBoxButton.OK, MessageBoxImage.Error); } else { // Add new patient information to record XElement newPatient = new XElement("patient", new XAttribute("username", patientUsername), new XAttribute("password", txtPassword.Text), new XElement("forename", txtForename.Text), new XElement("surname", txtSurname.Text), new XElement("addressLineOne", txtAddress1.Text), new XElement("addressLineTwo", txtAddress2.Text), new XElement("addressLineThree", txtAddress3.Text), new XElement("postCode", txtPostcode.Text), new XElement("Fist_Clench", null), new XElement("Wrist_Flextion_Extension", null), new XElement("Three_Jaw_Chuck_Pinch", null)); // Save and close patientCollection.Add(newPatient); lmrsXml.Save("LMRS_TestFile.xml"); MessageBox.Show("Patient added successfully.", "Patient Added", MessageBoxButton.OK, ????????????????rmation); clearTextFields(); } }} // btnRegister_Click(object sender, RoutedEventArgs e) Listing 22 – LMRS Register PatientThe methods to check for text field completion and to subsequently clear them again are trivial assignment operations and are therefore omitted. Another method worthy of mention is the text changed event handler for the surname field, when this field is modified we automatically generate a username of the format: “p_(surname all lower case)(forename initial in upper case)” i.e. p_smithJ. This is done to ensure consistency across all usernames.private void txtSurname_TextChanged(object sender, ????TextChangedEventArgs e){ if (txtForename.Text != "") { txtUsername.Text = "p_" + txtSurname.Text.ToLower() + ????????txtForename.Text.ToUpper()[0]; }} Listing 23 – Automatic Username GenerationFigure 42 – LMRS Remove Patient Screen097853500The final functionality available only to clinicians (and the last uncovered functionality of the LMRS as a whole) is the ability to remove registered patient users, seen in REF _Ref386140330 \h Figure 42.Like with the register button in the register new patient form, the main functionality for the remove patient form is held in the click event handler for the search button. When this button is pressed we check the XML data store for a user matching the provided username, if one is found we enable the remove button, otherwise we inform the user that no such user exists on record and clear the search field.private void btnSearch_Click(object sender, RoutedEventArgs e){ lmrsXml = XDocument.Load("LMRS_TestFile.xml"); // Find patient user XElement patientCollection = ????????lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in ????????patientCollection.Elements("patient") where (string)el.Attribute("username") == ????????txtUserToRemove.Text select el; if (patient.Count() == 0) { MessageBox.Show("No user found with this username, please ????????????check the username and try again.", "User Not Found", MessageBoxButton.OK, ????????????MessageBoxImage.Error); ??????? txtUserToRemove.Text = ""; } else { userOnRecord = patient.First<XElement>(); txtResults.Text = "User found: \n\n" + ????????????userOnRecord.Element("forename").Value + " " + ????????????userOnRecord.Element("surname").Value + "\n" + userOnRecord.Element("addressLineOne").Value + ????????????"\n" + userOnRecord.Element("addressLineTwo").Value + "\n" + userOnRecord.Element("addressLineThree").Value + ????????????"\n" + userOnRecord.Element("postCode").Value; btnRemove.IsEnabled = true; }} Listing 24 – Finding Patient to RemoveFinally, if a patient user has been found and the clinician user wishes to remove that user we prompt for confirmation, removing the patient user only if the clinician user wishes to proceed.private void btnRemove_Click(object sender, RoutedEventArgs e){ if (MessageBox.Show("Removing this user will permanently delete ????????all related exercise data, proceed?", "Confirm User Removal", MessageBoxButton.YesNo, ????????MessageBoxImage.Question) == MessageBoxResult.Yes) { userOnRecord.RemoveAll(); lmrsXml.Save("LMRS_DataStore.xml"); MessageBox.Show("Patient removed successfully.", "Patient Removed", MessageBoxButton.OK, ????????????rmation); } txtUserToRemove.Text = txtResults.Text = ""; btnRemove.IsEnabled = false;} Listing 25 – Removing Patient8: Evaluation8.1 Testing8.1.1 Unit and Regression TestingThroughout initial development, the most commonly used testing methods were those of unit and regression testing. With unit testing used to test individual functionalities and units of code. Unit testing for the LMRS took the form of compiling and running the system to check that it is first: stable and secondly: that it functions as expected/designed. Regression testing was used in addition to unit testing to confirm correct and expected behaviour after notable changes/refactoring. Regression testing for the LMRS took the form of compiling and re-running the system, with the aim of testing certain functionalities which had seen significant re-working, commonly as a result of the prior mentioned unit testing. This was done before testing the system against more formal, drafted test cases.8.1.2 User Scenario Test CasesThe first formal means of testing the LMRS was through the use of conventional test-cases. These test cases describe the typical usage patterns of both a clinician and patient user. The tests include basic sanity testing (the ability to detect and reject false credentials and other ‘junk data’) in addition to testing the various components of the system (can exercises be performed without issue, does the results graph show correctly etc…). The full test-cases can be found in Appendix 1 and Appendix 2 respectively along with original results and comments.The issues uncovered by these test cases were largely XML related. For example, both crashes on the results screen (for either patient or clinician) were the result of empty XML elements. One due to potentially missing session elements (if a user quits before completing an exercise for example) and the other due to an error in the original remove patient code which would remove all child elements of the patient but not the patient element itself. In addition to this, testing revealed a lack of any suitable prompt for exiting while an exercise session was in progress. A prompt was added in response to this to keep the system in-line with its initial requirements, in this case, the requirement of keeping the user informed at all times.8.1.3 Testing Timing AccuracyOne of – if not the single most – important aspect of the LMRS is that it provides highly accurate timings of exercise repetitions; if the LMRS cannot provide accurate exercise repetition timings then any medical relevance/usefulness of the timings and that of any other LMRS-generated data is dramatically reduced.To measure the accuracy of the LMRS in this respect, the system has been compared against video references; actual video recording of the exercises being performed. The timings recorded by the LMRS are then compared against the timings taken from the video reference (acquired by measuring the time taken in video for a repetition to be performed). Graphs 1, 2 and 3 show the accuracy of the LRMS relative to the video reference for each of the three exercises (Fist Clench, Wrist Flexion & Extension and Three Jaw Chuck Pinch). This data has been collected by performing each exercise three times (three LMRS sessions and three reference videos) and taking the average time for each rep (1, 2, 3 etc…).Graph 1 – Exercise 1 - Fist Clench As we can see in REF _Ref386244565 \h \* MERGEFORMAT Graph 1, deviations in the times recorded by the LMRS compared to those in the reference video are minimal (often around 100 milliseconds). This trend is maintained in exercises 2 and 3, as seen in REF _Ref386244540 \h \* MERGEFORMAT Graph 2 and REF _Ref386244549 \h \* MERGEFORMAT Graph 3.Graph 2 – Exercise 2 – Wrist Flexion & ExtensionGraph 3 - Exercise 3 – Three Jaw Chuck PinchIt is worth noting however that the one consistent area of variation between the LMRS and the reference video is the first repetition of each exercise, this would suggest that adjustments and/or refinements to the values used in the timer related conditions (when to start/restart) may be in order for future iterations.In addition to the repetition timings, the deviation observed for each exercise has also been calculated and are presented in REF _Ref386244658 \h \* MERGEFORMAT Table 4. This is an important – if not critical – metric for the LMRS and any rehabilitation system of this nature and will be a key metric used to judge any future changes or additions made to the LMRS (a more accurate version of the hand model cannot come at the cost of a loss in repetition timing accuracy for example). As with the repetition time data discussed above, this deviation data has been calculated by taking the average deviation for each repetition across the three sessions and then taking the average (the deviation value for rep 1 in exercise 1 for example is the average rep 1 deviation observed for that exercise across the three sessions).Repetition #Exercise 1 – Fist ClenchExercise 2 – Wrist Flexion & ExtensionExercise 3 – Three Jaw Chuck Pinch1-0.109-0.116-0.32220.107-0.0030.1493-0.0310.0350.1244-0.1060.1490.2675-0.1530.1530.076Average-0.05840.04360.0588 Table 4 – Exercise Repetition Time VariationsThe results are encouraging. Despite the Leap Motion being an as yet new and untested device, we see that the average inaccuracy (deviation) in the timings measured by the LMRS is less than 100 milliseconds and rarely is the 100 millisecond barrier broken for any individual repetition.8.2 Evaluation against Initial Requirements8.2.1 Evaluation against Functional RequirementsThe LMRS will now be compared against its initial functional requirements.Req #DescriptionRequirement Met?1System should allow clinician and patients to create accounts for themselvesPartially: Clinicians are responsible for creating patient accounts2System should allow for a simple and straightforward login processMet3System should allow the user to choose an exercise to performNot met: System progresses through exercises in set order4System should track the user’s hand movements during the exercise sessionMet5System should provide feedback, preferably graphical (such a 3d hand animation) or at the very least text-based feedbackMet: LMRS supports real-time feedback through an animated model6System should present user with their results and save them in a database for easy retrieval by the patient or clinicianPartially met: Results can be presented to user but are stored via XML not SQL database7System should shutdown gracefully and correctly, confirming user’s wish to close and ensuring all important data is savedMet8System should be able to handle unexpected difficulties such as the absence or removal of the Leap Motion or being unable to connect to the data-baseMet9System should run on Windows 7 or 8 in accordance with Leap Motion minimum system requirementsPartially Met: Unable to confirm Windows 8 due to dependencies induced by animated 3d model module Table 5 – LMRS Compared against Functional RequirementsOverall, the LMRS meets the majority of the functional requirements set out in the original specification. On the subject of Windows 8 compatibility (requirement 9), this could not be confirmed due to XML dependencies introduced by the 3d animated hand module. If the appropriate XML binaries were present however it stands to reason that the LMRS would be able to run without issue on Windows 8.8.2.2 Evaluation against Non-Functional RequirementsThe LMRS will now be compared against its initial non-functional requirements.Req #DescriptionRequirement Met?1System should be robust and durableMet2System should be easy for the user to operateMet3System should provide guidance on how to perform exercisesMet4System should capture accurate readings of the patient’s hand movementsMet5System should provide a meaningful use of gamification elementsNot Met Table 6 – LMRS Compared against Non-Functional RequirementsUnfortunately, due to time constraints, gamification aspects of the LMRS could not be implemented. However, the research for these elements of the system has largely been done, making gamification an ideal starting point for further enhancements.8.3 Future Work and EnhancementsWhilst developing the LMRS, many a functionality had to be cut and/or changed due to time constraints. Future work or systems in the vein of the LMRS should consider these areas for potential enhancements.The implementation of an SQL database. While the flexibility and portability offered by XML – in addition to time constraints – have resulted in a relatively powerful and easy to work with storage mechanism, actual deployment of the LMRS would require an internet-based storage mechanism.In future, efforts should be made to improve the accuracy of the LMRS. The Leap Motion SDK currently doesn’t offer skeletal tracking which makes any work with the device inherently more difficult than with traditional data gloves. Effort should be made to implement a means by which individual fingers can be reliably identified; this will allow for more accurate readings and the addition of more complex and intricate exercises.Tying into the point made above, unlike traditional data-gloves, the Leap Motion easily allows for two handed exercises and potentially exercises involving items such as cups or other everyday items (through the SDK “point-able” object which allows for tracking of additional items, not just fingers). Due to time constraints, these could not be implemented. However, additional exercise types (two-handed and those with items) represents a good opportunity to build on the strengths of the Leap Motion.9: ConclusionThis project saw us set out with the aim of researching and subsequently developing a rehabilitation system for those with hand injuries and to do so using the Leap Motion as our medium, rather than more traditional technologies like data-gloves. The system was to allow a user to perform rehabilitative exercises while receiving stimulating feedback via a real-time animated model. The system was then required to store this data for later viewing by either the patient or a clinician.Upon completing the system, a more detailed set of functional and non-functional requirements were drafted up to better inform the system design. From these, mock-ups of the UI were created in addition to use-cases describing a typical usage session. Development then began, with C#, WPF, XAML and XNA as the technologies chosen to help with realising the system. The end result is a system that bares striking resemblance to the system described in those initial requirements and depicted in those initial UI mock-ups.After completing development, testing was then carried out to ensure the functional correctness and accuracy of the system. Alongside traditional unit and regression testing, the system has been subject to more formal test-cases and finally to more specialised tests. These more ‘specialised’ tests took the form of comparing the exercise repetition timings as recorded by the system to those observed from a video recording of the same exercise. These tests have proven that any differences between repetition timings as recorded by the system are minimal (rarely above 100 milliseconds and below 100 milliseconds on average) when compared to those observed from a video recording, suggesting the system holds much promise.In closing, the LMRS can be deemed successful. We have been able to craft a functional rehabilitation system using an entirely new medium – the Leap Motion. The price and relative accuracy of this device in addition to its other unique qualities mean the LMRS potentially represents the beginning of a promising new avenue with regards to use of technology in rehabilitation.10: References BIBLIOGRAPHY Ackerman, E., 2011. Top 10 Robotic Kinect Hacks. [Online] Available at: [Accessed 12 December 2013].American Society for Surgery of the Hand, 2011. Hand and Arm Conditions. [Online] Available at: [Accessed 11 November 2013].Amini, D., 2011. The Unique Role of Occupational Therapy in Rehabilitation of the Hand, s.l.: The American Occupational Therapy Association Inc.Anon., 2013. Youtube - Leap Motion Structured Light Pattern. [Online] Available at: [Accessed 15 December 2013].Bond, A., 2011. BSc Thesis: An Automated System for Reading Hand Measurements in Patients with Rheumatoid Arthritis, Londonderry: University of Ulster.Bradski, G. & Kaehler, A., 2008. Learning OpenCV. s.l.:O'Reily.B, S. & C, P., n.d. In: s.l.:s.n.Burke, J. et al., 2009. Optimising Engagement for Stroke Rehabilitation using Serious Games. The Visual Computer, 25(12), pp. 1085-1099.Case-Smith, J., 2003. Outcomes in Hand Rehabilitation Using Occupational Therapy Services. The American Journal of Occupational Therapy, Volume 57, pp. 499-506.Chatfield, T., 2010. Why Gaming Will Dominate the Twnty-First Century. s.l.:Pegasus.Deterding, S., Dixon, D., Khaled, R. & Nacke, L., 2011. From Game Design Elements to Gamefulness: Defining "Gamification". Mindtrek.Dias, J. J. & Garcia-Elias, M., 2006. Hand Injury Costs. INJURY - International Journal of the Care of the Injured, Nov;37(11), pp. 1071-1077.Dipietro, L., Sabatini, A. M. & Dario, P., 2008. A Survey of Glove-Based Systems and their Applications. IEE Transactions of Systems, Man, and Cybernetics, 38(4), pp. 461-482.Fifth Dimension Technologies, 2011. 5DT Data Glove 5 Ultra Product Page. [Online] Available at: [Accessed 6 December 2013].Gerling, K. M. & Masuch, M., 2011. Exploring the Potential of Gamification Among Frail Elderly Persons. Vancouver, s.n.G, R., 2003. Couts des Urgences Mains. Chir Main, Volume 22, pp. 258-63.Groh, F., 2012. Research Trends in Media Informatics: Gamification: State of the Art Definition and Utilization. s.l., s.n., pp. 39-46.Hasselkus, B., 2002. The Meaning of Everday Occupation. NJ: Slack Inc.Hillerbrand, G., Bauer, M., Achatz, K. & Linker, G., 2005. Inverse Kinematic Infrared Optical Finger Tracking. [Online] Available at: [Accessed 22 April 2014].Holmberg, J., Lindgren, B. & Jutemark, R., 1996. Replantation-Revascularization and Primary Amputation in Major Hand Injuries. Resources Spent on Treatment and the Indirect Costs of Sick Leave in Sweden.. Journal of Hand Surgery, Volume 21, pp. 576-80.HumanWare, 2010. Product Page. [Online] Available at: [Accessed 7 December 2013].Jacobs, A. et al., 2013. CONTRAST: Gamficiation of Arm-Hand Training for Stroke Survivors. CHI EA '13 Extended Abstracts on Human Factors in Computing Systems, pp. 415-420.Jakobsson, M., 2011. The Achievement Machine: Understanding Xbox 360 Achievements in Gaming Practices. [Online] Available at: [Accessed 8 December 2013].Khan Academy, 2013. Khan Academy. [Online] Available at: [Accessed 8 December 2013].Lautman, R., 2012. Assessing Hand Movement i Arthritic Patients Using Wearable Glove Technology, Londonderry: s.n.Lavanon, Y., 2013. The Advantages and Disadvantages of using High Technology in Hand Rehabilitation. Journal of Hand Therapy, Volume 26, pp. 179-183.Leap Motion Inc, 2013. Leap Motion Product Page. [Online] Available at: [Accessed 13 December 2013].Leap Motion, 2013. SDK Docuementation - Tracking Hands, Fingers, and Tools. [Online] Available at: [Accessed 22 April 2014].O'Donnell, B., 2010. Hand Rehabilitation Using a Peregrine Gaming Data Glove, Londonderry: University of Ulster.Pc Mag, n.d. Nyko Zoom for Kinect Review. [Online] Available at: [Accessed 13 December 2013].Peregrine, n.d. Peregrine Product Page. [Online] Available at: [Accessed 6 December 2013].Shneiderman B, P. C., 2005. In: Designing the User Interface - Fourth Edition. s.l.:Pearson, pp. 92-93.Sommerville, I., 2010. Software Engineering. 9 ed. s.l.:Addison Wesley.Sousa, A. D. et al., 2013. Psychological Issues In Hand Trauma. ASEAN Journal of Psychiatry, 14(1).Strategy One, 2008. Ergonomics and Repetitive Strain Injury, s.l.: Strategy One for Microsoft.The Pleasure Revolution: Why Games Will Lead the Way. 2011. [Film] Directed by Jesse Schell. s.l.: s.n.Trybus, M., Lorkowski, J., Brongel, L. & Hladki, W., 2006. Causes and Consequences of Hand Injuries. The American Journal of Surgery, July;129(1), pp. 52-57.Turner, J. R., 1993. The Hanbook of Project-Based Management. s.l.:McGraw-Hill Book Co..Weichert, F., Bachmann, D., Rudak, B. & Fisseler, D., 2013. Analysis of the Accuracy and Robustness of the Leap Motion Controller. Sensors, 13(5), pp. 6380-6393.Willow Garage, 2011. Willow Garage Home Page. [Online] Available at: [Accessed 19 December 2013].Yassi, A., 1997. Repetitve Strain Injuries. The Lancet, Volume 349, pp. 943-47.11.1: Appendix 1: Clinician User Scenario Test CaseTest IDDescriptionStepsExpected OutcomeActualOutcomePass/FailTestDateComment(s)0Login with junk credentialsEnter false username and/or passwordSystem should show a failed login promptFailed login prompt shownPASS18/04/141Login with valid credentialsEnter valid username and passwordSystem should login successfullyLogin successfulPASS18/04/142Able to view patient user exercise resultsSelect “View Patient Results” menu option. Toggle between patients, exercises and datesSystem should allow graphs of various types, reflecting changes in patient, exercise and dates Data is shown but crashes if no session data is available/also no guard against duplicate sessionsFAIL18/04/14Added checks for empty session data and attempts to make duplicate sessions under same data now ignored. PASS after retesting3Leaving fields in “Register New Patient” screen emptyLeave any field in the “Register New Patient” screen blank and click “Register”System should show a prompt asking the user to fill in the empty fieldPrompt shownPASS18/04/144Register a duplicate patientEnter the details for an already registered patient user in the “Register New Patient” screen and click “Register” System should show a prompt informing the user that such a patient is already registeredPrompt shownPASS18/04/145Able to register new patient userEnter the details for a new patient user in the “Register New Patient” screen and click “Register”System should show a prompt confirming successful additionPrompt shown, new patient found in XML data storePASS18/04/146Unable to remove non-existing patient userEnter the username of a non-existent patient in the search field of the “Remove Existing Patient” screen and click “Search”System should show a prompt requesting the user to enter a valid patient usernamePrompt shownPASS18/04/147Able to remove existing patient userSelect “Register New Patient”, enter new patient credentials and click “Register”System should show a prompt confirming successful additionPrompt shown, patient removed from XML data storePASS/FAIL18/04/14Empty remaining patient XML element causing results screen crash.8Log outClick “Log Out”System should show a prompt, click OK at which point user should be returned to the login screenPrompt shownPASS18/04/1411.2: Appendix 2: Patient User Scenario Test CaseTest IDDescriptionStepsExpected OutcomeActualOutcomePass/FailTestDateComment(s)0Login with junk credentialsEnter false username and/or passwordSystem should show a failed login promptPrompt shownPASS19/04/141Login with valid credentialsEnter valid username and passwordSystem should login successfullyLogin successfulPASS19/04/142Can perform exercisesFrom the main menu, select the “Perform Exercises” buttonSystem should show “Perform Exercises” screen, exercises should progress logically, with instructions and timing updatedAble to perform exercises, instructions, timing and 3d hand model update and respondPASS19/04/143Progress loss prompt shown when leaving while exercise in progressWhile performing an exercise, click the “Back” buttonSystem should show a prompt, informing of data loss and requiring user confirmation to continueNo prompt shownFAIL19/04/14Prompt added4Can view exercise results from exercise pageWhen exercises have been completed, the “Continue” button should be enabled, click itThe “View Results” screen should be shown. The chart should respond to changes in exercise type, start/end date and chart type.Crash if any exercises have no data recorded against themFAIL19/04/14Added guard against empty sessions5Can view exercise results from main menuFrom the main menu, select the “View Results” optionThe “View Results” screen should be shown. The chart should respond to changes in exercise type, start/end date and chart type.Same as aboveFAIL19/04/146Log outClick “Log Out”System should show a prompt, click OK at which point user should be returned to the login screenPrompt shownPASS19/04/1411.3: Appendix 3: Source CodeLMRS_LeapListener.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using Leap;using System.Diagnostics;// Derives from the default Listener provided by the Leap API,// provides callback implementations for key events.namespace LMRS_Main{ class LeapListener : Listener { private Leap.Frame frame = null; public override void OnConnect(Controller controller) { Console.WriteLine("Leap Motion Connected"); // Register gesutres here... } public override void OnDisconnect(Controller controller) { Console.WriteLine("Leap Motion Disconnected"); } public override void OnInit(Controller controller) { Console.WriteLine("Initilising..."); } public override void OnExit(Controller controller) { Console.WriteLine("Exiting..."); } public override void OnFocusGained(Controller controller) { Console.WriteLine("Gained Focus"); } public override void OnFocusLost(Controller controller) { Console.WriteLine("Lost Focus"); } public override void OnFrame(Controller controller) { // Get the most recent frame from the device frame = controller.Frame(); int STOP; if(frame != null) STOP = 1; } public Leap.Frame LeapFrame { get { return frame; } } }}LMRS_Login.xaml <Page x:Class="LMRS_Main.LMRS_Login" xmlns="" xmlns:x="" xmlns:mc="" xmlns:d="" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366" Title="LMRS_Login" Width="1366"> <Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="52.873*" /> <ColumnDefinition Width="1313.127*" /> </Grid.ColumnDefinitions> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Welcome to the Leap Motion Rehabilitation System" Height="119" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White" Grid.ColumnSpan="2" /> <!-- User credential fields--> <Label FontSize="32" HorizontalAlignment="Left" Margin="357.25,302.948,0,351.534" Name="lblUsername" Width="160" Grid.Column="1">Username:</Label> <TextBox FontSize="32" Height="50" Margin="0,305.806,440.132,351.534" Name="txtUsername" TabIndex="1" Grid.Column="1" HorizontalAlignment="Right" Width="349.981" /> <Label FontSize="32" Height="50" HorizontalAlignment="Left" Margin="371.54,0,0,257.22" Name="lblPasword" VerticalAlignment="Bottom" Width="150" Grid.Column="1">Password:</Label> <PasswordBox FontSize="32" Grid.Column="1" Height="50" Margin="523.014,0,440.132,257.22" Name="pwbxPassword" VerticalAlignment="Bottom" PasswordChar="*"/> <Button Content="Login" Height="46" HorizontalAlignment="Right" Margin="0,661,41,0" Name="btnLogin" VerticalAlignment="Top" Width="150" Click="btnLogin_Click" Grid.Column="1" /> <Button Content="Exit" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnExit_Click" Grid.ColumnSpan="2" /> </Grid></Page>LMRS_Login.xaml.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.Xml.Linq;using System.Data;namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_Login.xaml /// </summary> public partial class LMRS_Login : Page { public LMRS_Login() { InitializeComponent(); // Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - ????????????????Login"; } private void btnLogin_Click(object sender, RoutedEventArgs e) { // Have credentials been provided? if (txtUsername.Text == "" || ???????????? pwbxPassword.Password == "") { MessageBox.Show("Please provide both a username and ????????????????????password.", "Cannot Login", MessageBoxButton.OK, ????????????????????MessageBoxImage.Error); txtUsername.Text = pwbxPassword.Password = ""; } // Is the username valid? else if (!txtUsername.Text.Contains("p_") && ????????????????!txtUsername.Text.Contains("c_")) { MessageBox.Show("Please provide a valid username.", "Cannot Login", MessageBoxButton.OK, ????????????????????MessageBoxImage.Error); txtUsername.Text = pwbxPassword.Password = ""; } else { XDocument lmrsXml = ????????????????????XDocument.Load("LMRS_TestFile.xml"); if (txtUsername.Text.Contains("p_")) { // Find patient user XElement patientCollection = ????????????????????????lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in ????????????????????????patientCollection.Elements("patient") where (string)el.Attribute("username") == txtUsername.Text select el; if (patient.Count() == 0) { MessageBox.Show("Please provide a valid ????????????????????????????username.", "Cannot Login", ????????????????????????????MessageBoxButton.OK, ????????????????????????????MessageBoxImage.Error); txtUsername.Text = ????????????????????????????pwbxPassword.Password = ""; } else { XElement userOnRecord = ????????????????????????????patient.First<XElement>(); if (userOnRecord != null && ????????????????????????????pwbxPassword.Password.Equals(????????????????????????????userOnRecord.Attribute("password").Value.????????????????????????????ToString())) { UserInfoHelper.user = userOnRecord; UserInfoHelper.foreName = ????????????????????????????????userOnRecord.Element("forename").????????????????????????????????Value; UserInfoHelper.userName = ????????????????????????????????userOnRecord.Attribute("username").????????????????????????????????Value; this.NavigationService.Navigate(????????????????????????????????new LMRS_MainMenu_Patient()); } else { MessageBox.Show("Please provide a valid password.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error); pwbxPassword.Password = ""; } } } else { // Find clinician user XElement clinicianCollection = lmrsXml.Root.Element("clinicianCollection"); IEnumerable<XElement> clinician = from el in clinicianCollection.Elements("clinician") where (string)el.Attribute("username") == txtUsername.Text select el; if (clinician.Count() == 0) { MessageBox.Show("Please provide a valid username.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error); txtUsername.Text = pwbxPassword.Password = ""; } else { XElement userOnRecord = clinician.First<XElement>(); if (userOnRecord != null && pwbxPassword.Password.Equals(userOnRecord.Attribute("password").Value.ToString())) { UserInfoHelper.user = userOnRecord; UserInfoHelper.foreName = userOnRecord.Element("forename").Value; UserInfoHelper.userName = userOnRecord.Attribute("username").Value; this.NavigationService.Navigate(new LMRS_MainMenu_Clinician()); } else { MessageBox.Show("Please provide a valid password.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error); pwbxPassword.Password = ""; } } } } } // btnLogin_Click() private void btnExit_Click(object sender, RoutedEventArgs e) { (this.Parent as NavigationWindow).Close(); } }}LMRS_MainMenu_Clinician.xaml<Page x:Class="LMRS_Main.LMRS_MainMenu_Clinician" xmlns="" xmlns:x="" xmlns:mc="" xmlns:d="" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"Title="LMRS_MainMenu_Clinician" > <Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Welcome [CLINICIAN-NAME], Please Select an Option" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/> <Button Content="View Patient Results" Height="85" HorizontalAlignment="Left" Margin="455,181,0,0" Name="btnViewPatientResults" VerticalAlignment="Top" Width="455" Click="btnViewPatientResults_Click"/> <Button Content="Register New Patient" Height="85" HorizontalAlignment="Left" Margin="455,328,0,0" Name="btnRegisterPatient" VerticalAlignment="Top" Width="455" Click="btnRegisterPatient_Click" /> <Button Content="Remove Existing Patient" Height="85" HorizontalAlignment="Left" Margin="455,474,0,0" Name="btnRemovePatient" VerticalAlignment="Top" Width="455" Click="btnRemovePatient_Click"/> <Button Content="Logout" Height="85" HorizontalAlignment="Left" Margin="455,620,0,0" Name="btnLogout" VerticalAlignment="Top" Width="455" Click="btnLogout_Click"/> </Grid></Page>LMRS_MainMenu_Clinician.xaml.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_MainMenu_Clinician.xaml /// </summary> public partial class LMRS_MainMenu_Clinician : Page { public LMRS_MainMenu_Clinician() { InitializeComponent(); // Set the window title and header lblHeader.Content = "Welcome " + UserInfoHelper.foreName + ", Please Select an Option"; this.WindowTitle = "Leap Motion Rehabilitation System - Main Menu"; } private void btnViewPatientResults_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_ViewExerciseResults("clinician")); } private void btnRegisterPatient_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_RegisterNewPatient()); } private void btnRemovePatient_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_RemovePatient()); } private void btnLogout_Click(object sender, RoutedEventArgs e) { if (MessageBox.Show("Are you sure you wish to logout?", "Logout Confirmation", MessageBoxButton.YesNo, MessageBoxImage.Question) == MessageBoxResult.Yes) { this.NavigationService.Navigate(new LMRS_Login()); } } }}LMRS_MainMenuPatient.xaml<Page x:Class="LMRS_Main.LMRS_MainMenu_Patient" xmlns="" xmlns:x="" xmlns:mc="" xmlns:d="" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"Title="LMRS_MainMenu_Patient"> <Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Welcome [PATIENT-NAME], Please Select an Option" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/> <Button Content="Perform Exercises" Height="85" HorizontalAlignment="Left" Margin="455,227,0,0" Name="btnPerformExercises" VerticalAlignment="Top" Width="455" Click="btnPerformExercises_Click"/> <Button Content="View Results" Height="85" HorizontalAlignment="Left" Margin="455,414,0,0" Name="btnResults" VerticalAlignment="Top" Width="455" Click="btnViewExerciseResults_Click" /> <Button Content="Logout" Height="85" HorizontalAlignment="Left" Margin="455,601,0,0" Name="btnLogout" VerticalAlignment="Top" Width="455" Click="btnLogout_Click"/> </Grid></Page>LMRS_MainMenu_Patient.xaml.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_MainMenu_Patient.xaml /// </summary> public partial class LMRS_MainMenu_Patient : Page { public LMRS_MainMenu_Patient() { InitializeComponent(); // Set the window title and header lblHeader.Content = "Welcome " + UserInfoHelper.foreName + ", Please Select an Option"; this.WindowTitle = "Leap Motion Rehabilitation System - Main Menu"; } private void btnPerformExercises_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_PerformExercise()); } private void btnViewExerciseResults_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_ViewExerciseResults("patient")); } private void btnLogout_Click(object sender, RoutedEventArgs e) { if (MessageBox.Show("Are you sure you wish to logout?", "Logout Confirmation", MessageBoxButton.YesNo, MessageBoxImage.Question) == MessageBoxResult.Yes) { this.NavigationService.Navigate(new LMRS_Login()); } } }}LMRS_PerformExercise.xaml<Page x:Class="LMRS_Main.LMRS_PerformExercise" xmlns="" xmlns:x="" xmlns:mc="" xmlns:d="" xmlns:wf="clr-namespace:System.Windows.Forms;assembly=System.Windows.Forms" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"Title="LMRS_PerformExercise" Loaded="Page_Loaded" Unloaded="Page_Unloaded" Width="1366"><Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="124.323*" /> <ColumnDefinition Width="51.444*" /> <ColumnDefinition Width="1190.233*" /> </Grid.ColumnDefinitions> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Perform Exercises" Height="119" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White" Grid.ColumnSpan="3" /> <Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click" Grid.ColumnSpan="3" /> <StackPanel HorizontalAlignment="Left" Margin="32,158.619,0,65.734" Width="644" Grid.ColumnSpan="3"> <WindowsFormsHost Width="640" Height="480" Margin="2"> <wf:Panel x:Name="RenderPanel" BackColor="Black"/> </WindowsFormsHost> </StackPanel> <TextBlock TextAlignment="Center" FontSize="32" Background="Gray" Height="42" Grid.Column="2" Margin="538.159,0,35.725,287.229" Name="txtExerciseInstructions" Text="Exercise Instructions..." VerticalAlignment="Bottom" /> <TextBlock TextAlignment="Center" FontSize="32" Background="LightGray" Margin="538.159,160.619,35.725,350.105" Name="txtExerciseDescription" Text="Exercise Description..." Grid.Column="2" /> <TextBlock TextAlignment="Center" FontSize="32" Foreground="Green" Background="LightGray" Height="164" Margin="538.159,0,356.074,98.601" Name="txtCurrentRepInfo" Text="Current Rep Info..." VerticalAlignment="Bottom" Grid.Column="2" /> <TextBlock TextAlignment="Center" FontSize="32" Background="LightGray" Height="164" HorizontalAlignment="Right" Margin="0,0,35.725,98.595" Name="txtBestRepInfo" Text="Best Rep Info..." VerticalAlignment="Bottom" Width="296" Grid.Column="2" /> <Button Height="46" Margin="0,0,35.725,31" Name="btnContinue" VerticalAlignment="Bottom" Grid.Column="2" HorizontalAlignment="Right" Width="150" IsEnabled="False" Click="btnContinue_Click">Continue</Button> </Grid></Page>LMRS_PerformExercise.xaml.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.Threading;using SkeletalAnimation;using Leap;using ponentModel;using System.Diagnostics;using System.Xml.Linq;namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_PerformExercise.xaml /// </summary> public partial class LMRS_PerformExercise : Page { // For animated hand private SkeletalAnimationSample game; private Thread xnaThread; // For Leap Motion private Thread leapThread; private Leap.Controller controller; private Leap.Listener listener; private BackgroundWorker leapBackgroundWorker; // For exercise progress tracking uint calls = 0; bool calibrated = false; bool calibratePromptGiven = false; int currentExercise = 1; float avgMag = 0.0f; float avgPitch = 0.0f; bool exerciseDescGiven = false; bool clenchPromptGiven = false; bool restPromptGiven = false; bool lowerPromptGiven = false; bool raisePromptGiven = false; uint repsRequired = 5, repsPerformed = 0; long[] repTimes = new long[5]; long bestRepTime = 60000; Stopwatch timer = new Stopwatch(); Hand hand = null; bool exercisesFinished = false; public LMRS_PerformExercise() { InitializeComponent(); // Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - Perform Exercise"; } private void btnBack_Click(object sender, RoutedEventArgs e) { if (!exercisesFinished) { if (MessageBox.Show("Are you sure you wish to quit, progress on your current exercise will be lost?", "Exercises Still in Progress", MessageBoxButton.OKCancel, MessageBoxImage.Question) == MessageBoxResult.OK) { this.NavigationService.Navigate(new LMRS_MainMenu_Patient()); } } else { this.NavigationService.Navigate(new LMRS_MainMenu_Patient()); } } private void btnContinue_Click(object sender, RoutedEventArgs e) { NavigationService.Navigate(new LMRS_ViewExerciseResults("patient")); } private void Page_Loaded(object sender, RoutedEventArgs e) { // Set XNA hand running IntPtr handle = RenderPanel.Handle; xnaThread = new Thread(new ThreadStart(() => { game = new SkeletalAnimationSample(handle); game.Run(); } )); xnaThread.Start(); // Set Leap Motion running listener = new LeapListener(); controller = new Leap.Controller(); controller.AddListener(listener); leapBackgroundWorker = new BackgroundWorker(); leapBackgroundWorker.WorkerSupportsCancellation = true; leapBackgroundWorker.DoWork += new DoWorkEventHandler(leapBackgroundWorker_DoWork); leapBackgroundWorker.RunWorkerAsync(); } private void Page_Unloaded(object sender, RoutedEventArgs e) { // Shutdown hand xnaThread.Abort(); // Shutdown Leap Motion controller.RemoveListener(listener); controller.Dispose(); controller = null; leapBackgroundWorker.CancelAsync(); } void leapBackgroundWorker_DoWork(object sender, DoWorkEventArgs e) { while (true) { Leap.Frame frameData = ((LeapListener)listener).LeapFrame; if (frameData != null && frameData.Hands.Count > 0) { hand = frameData.Hands[0]; // Calibration #region Clibration if (!calibrated) { if (!calibratePromptGiven) { updateExerciseInstruction("Calibrating, please wait..."); calibratePromptGiven = true; } if (!timer.IsRunning) { timer.Start(); } // Collect hand pitch and vector magnitude readings, // large enough deltas in these parameters are used to // detect user movements and start the timer if (timer.Elapsed.Seconds < 5) { avgMag += (hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude; avgPitch += hand.Direction.Pitch; ++calls; } else { avgMag /= calls; avgPitch /= calls; calibrated = true; timer.Reset(); } } #endregion else { // Move onto next exercise/move onto results? if (repsPerformed == repsRequired) { logExerciseResults(); ++currentExercise; repsPerformed = 0; exerciseDescGiven = false; bestRepTime = 60000; if (currentExercise <= 3) { updateExerciseInstruction("Next exercise in three seconds..."); restartTimer(); while (timer.Elapsed.Seconds <= 3) ; timer.Reset(); } else { updateExerciseInstruction("Exercises complete"); game.ResetCameraArcAndRoll(); Dispatcher.Invoke(new Action(() => { btnContinue.IsEnabled = true; })); exercisesFinished = true; } } // Demo exercise 1 - Going from hand at rest to clenched fist #region ex1 if (currentExercise == 1 && repsPerformed < repsRequired) { // Give exercise brief if (!exerciseDescGiven) { updateExerciseDescription("Exercise 1 of 3: Fist Clench. " + "\n1) Form a closed fist. \n2) Relax hand. \n\nThe timer will start automatically"); exerciseDescGiven = true; } // A sufficient delta in the vector between finger-tip and palm signals // the user has begun to moving their hand, start the timer if ((hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude < 0.9 * avgMag && !timer.IsRunning) { restartTimer(); } // Exercise part 0 - Form a fist if (hand.Fingers.Count == 5 && !clenchPromptGiven) { // Present time for this rep if (timer.IsRunning) { repTimes[repsPerformed] = timer.ElapsedMilliseconds; updateRepInfo("This rep time: \nRep " + (repsPerformed + 1) + " of 5 : \n" + repTimes[repsPerformed] / 1000.0 + " seconds."); // New best rep time? if (repTimes[repsPerformed] < bestRepTime) { bestRepTime = repTimes[repsPerformed]; updateBestRepInfo("Best rep time: \nRep " + (repsPerformed + 1) + " of 5: \n" + bestRepTime / 1000.0 + "seconds."); } ++repsPerformed; restartTimer(); } // Go if (repsPerformed < repsRequired) { updateExerciseInstruction("Clench your hand into a fist"); clenchPromptGiven = true; restPromptGiven = false; } } // Exercise part 1 - Relax hand if (hand.Fingers.Count == 0 && !restPromptGiven) { updateExerciseInstruction("Place your hand at rest"); restPromptGiven = true; clenchPromptGiven = false; } // Update xna hand model // updateHandModelExercise1(frameData); } #endregion // Demo exercise 2 - Wrist flexion/extension #region ex2 if (currentExercise == 2 && repsPerformed < repsRequired) { // Start the exercise if (hand.Fingers.Count >= 1) { // Give exercise brief if (!exerciseDescGiven) { updateExerciseDescription("Exercise 2 of 3: Wrist Flexion/Extension. " + "\n1) Tilt your hand upward \n2) Tilt your hand downward \nThe timer will start automatically"); exerciseDescGiven = true; updateExerciseInstruction("Raise your hand from the wrist"); } // Lower hand if (hand.Direction.Pitch > avgPitch * 2.5 && repsPerformed < repsRequired && !lowerPromptGiven) { // Present time for this rep if (timer.IsRunning)// && !firstRep) { repTimes[repsPerformed] = timer.ElapsedMilliseconds; updateRepInfo("Rep " + (repsPerformed + 1) + " complete. \nTime: " + repTimes[repsPerformed] / 1000.0 + " seconds."); // New best rep time? if (repTimes[repsPerformed] < bestRepTime) { bestRepTime = repTimes[repsPerformed]; updateBestRepInfo("Best rep time: \nRep " + (repsPerformed + 1) + " of 5: \n" + bestRepTime / 1000.0 + "seconds."); } ++repsPerformed; } restartTimer(); //firstRep = false; if (!lowerPromptGiven) { updateExerciseInstruction("Lower your hands from the wrist"); } lowerPromptGiven = true; raisePromptGiven = false; } // Raise hand if (hand.Direction.Pitch < -2.5 * avgPitch && repsPerformed < repsRequired && !raisePromptGiven) { if (!raisePromptGiven) { updateExerciseInstruction("Raise your hands from the wrist"); } lowerPromptGiven = false; raisePromptGiven = true; } } // Update xna hand //updateHandModelExercise2(frameData); } #endregion // Demo exercise 3 - Three jaw chuck pinch #region ex3 if (currentExercise == 3 && repsPerformed < repsRequired) { // Give exercise brief if (!exerciseDescGiven) { updateExerciseDescription("Exercise 3 of 3: Three Jaw. " + "\n1) Form a closed fist. \n2) Relax hand. \n\nThe timer will start automatically"); exerciseDescGiven = true; game.ResetCameraArcAndRoll(); } // A sufficient delta in the vector between finger-tip and palm signals // the user has begun to moving their hand, start the timer if ((hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude < 0.9 * avgMag && !timer.IsRunning) { restartTimer(); } // Exercise part 0 - Form a fist if (hand.Fingers.Count == 3 && !clenchPromptGiven) { // Present time for this rep if (timer.IsRunning) { repTimes[repsPerformed] = timer.ElapsedMilliseconds; updateRepInfo("Rep " + (repsPerformed + 1) + " of 5 complete. \nTime: " + repTimes[repsPerformed] / 1000.0 + " seconds."); // New best rep time? if (repTimes[repsPerformed] < bestRepTime) { bestRepTime = repTimes[repsPerformed]; updateBestRepInfo("Best rep time: \nRep " + (repsPerformed + 1) + " of 5: \n" + bestRepTime / 1000.0 + "seconds."); } ++repsPerformed; restartTimer(); } // Go if (repsPerformed < repsRequired) { updateExerciseInstruction("Pinch your fingers and thumb together"); clenchPromptGiven = true; restPromptGiven = false; } } // Exercise part 1 - Relax hand if (hand.Fingers.Count <= 1 && !restPromptGiven) { updateExerciseInstruction("Place your fingers and thumb at rest"); restPromptGiven = true; clenchPromptGiven = false; } } #endregion // Update hand model updateHandModel(frameData); } } // if (!calibrated) else { // ... } } } // leapBackgroundWorker_DoWork() // Update the hand model based on information from the Leap Motion private void updateHandModel(Leap.Frame frame) { // Calculate necessary vectors Leap.Vector fingerBase = -frame.Hands[0].Fingers.Leftmost.Direction * frame.Hands[0].Fingers.Leftmost.Length; fingerBase = fingerBase + frame.Hands[0].Fingers.Leftmost.TipPosition; Leap.Vector palmCenterToBase = fingerBase - frame.Hands[0].PalmPosition; Leap.Vector baseToTip = frame.Hands[0].Fingers.Leftmost.TipPosition - fingerBase; // Calculate angles float thumbIndex2 = -(1.0f - Leap.Vector.ZAxis.AngleTo(palmCenterToBase)); // Some basic clamping thumbIndex2 = thumbIndex2 < -1.75f ? -1.75f : thumbIndex2; // Set the angles for the fingers // Thumb game.Thumb2Radians = thumbIndex2; game.Thumb1Radians = (float)(((0.23 + 1.73 * 0.33 + 1.5 * (0.33 * 0.33))) * thumbIndex2); // Index fingerBase = -frame.Hands[0].Fingers.Frontmost.Direction * frame.Hands[0].Fingers.Frontmost.Length; fingerBase = fingerBase + frame.Hands[0].Fingers.Frontmost.TipPosition; palmCenterToBase = fingerBase - frame.Hands[0].PalmPosition; baseToTip = frame.Hands[0].Fingers.Frontmost.TipPosition - fingerBase; // Some basic clamping float fingerIndex3 = (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip))) > 0 ? 0 : (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip))); fingerIndex3 = fingerIndex3 <= -1.25f ? -1.25f : fingerIndex3; float fingerIndex2 = (float)(((0.23 + 1.73 * 0.66 + 1.5 * (0.66 * 0.66)))) * fingerIndex3; float fingerIndex1 = (float)(((0.23 + 1.73 * 0.33 + 1.5 * (0.33 * 0.33)))) * fingerIndex3; // Calulate pitch and yaw float pitch = -frame.Hands[0].Direction.Normalized.Pitch; float yaw = -frame.Hands[0].Direction.Normalized.Yaw; // Tone down yaw yaw -= 0.5f; yaw /= 4; // Update hand game.CameraDown(pitch); game.CameraLeft(yaw); game.Index3Radians = fingerIndex3; game.Index2Radians = fingerIndex2; game.Index1Radians = fingerIndex1; // Middle game.Middle3Radians = fingerIndex3; game.Middle2Radians = fingerIndex2; game.Middle1Radians = fingerIndex1; // Ring if (currentExercise == 3) { fingerIndex3 = -1.0f; fingerIndex2 = fingerIndex1 = (float)(((0.23 + 1.73 * 0.33 + 1.5 * (0.33 * 0.33))) * fingerIndex3); } game.Ring3Radians = fingerIndex3; game.Ring2Radians = fingerIndex2; game.Ring1Radians = fingerIndex1; // Pinky game.Little3Radians = fingerIndex3; game.Little2Radians = fingerIndex2; game.Little1Radians = fingerIndex1; } private void logExerciseResults() { // Load the XML file XDocument lmrsXml = XDocument.Load("LMRS_TestFile.xml"); // Get patient collection node XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == UserInfoHelper.userName select el; string exerciseName = ""; if (currentExercise == 1) { exerciseName = "Fist_Clench"; } else if (currentExercise == 2) { exerciseName = "Wrist_Flexion_Extension"; } else { exerciseName = "Three_Jaw_Chuck_Pinch"; } IEnumerable<XElement> exercise = from el in patient.Elements("activity") where el.Attribute("name").Value.Equals(exerciseName) select el; XElement e = exercise.First(); // Make sure this isn't a duplicate session (date already on record). IEnumerable<XElement> sessions = from el in e.Elements("session") where el.Attribute("date").Value.Equals(DateTime.Now.Date.ToString("dd/MM/yy")) select el; if (sessions.Count() == 0) { XElement session = new XElement("session", new XAttribute("date", DateTime.Now.Date.ToString("dd/MM/yy")), new XElement("rep", repTimes[0] / 1000.0), new XElement("rep", repTimes[1] / 1000.0), new XElement("rep", repTimes[2] / 1000.0), new XElement("rep", repTimes[3] / 1000.0), new XElement("rep", repTimes[4] / 1000.0)); e.Add(session); lmrsXml.Save("LMRS_TestFile.xml"); } } // Helper methods private void restartTimer() { timer.Reset(); timer.Start(); } private void updateExerciseDescription(string newDescription) { Dispatcher.Invoke(new Action(() => { txtExerciseDescription.Text = newDescription; })); } private void updateExerciseInstruction(string nextInstruction) { Dispatcher.Invoke(new Action(() => { txtExerciseInstructions.Text = nextInstruction; })); } private void updateRepInfo(string repUpdate) { Dispatcher.Invoke(new Action(() => { txtCurrentRepInfo.Text = repUpdate; })); } private void updateBestRepInfo(string newBestRep) { Dispatcher.Invoke(new Action(() => { txtBestRepInfo.Text = newBestRep; })); } }}LMRS_RegisterNewPatient.xaml<Page x:Class="LMRS_Main.LMRS_RegisterNewPatient" xmlns="" xmlns:x="" xmlns:mc="" xmlns:d="" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"Title="LMRS_RegisterNewPatient"><Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Please Enter Patient Details" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/> <Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click"/> <!-- Patient information fields, column 1 --> <Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,160.048,0,0" Name="lblForename" VerticalAlignment="Top" Width="150" FontSize="32">Forename:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.803,160.048,0,0" Name="txtForename" VerticalAlignment="Top" Width="350" FontSize="32" /> <Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,240.072,0,0" Name="lblSurname" VerticalAlignment="Top" Width="150" FontSize="32">Surname:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.784,240.072,0,0" Name="txtSurname" VerticalAlignment="Top" Width="350" FontSize="32" TextChanged="txtSurname_TextChanged"/> <Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,320.096,0,337.244" Name="lblAddress1" Width="155" FontSize="32">Address 1:</Label> <TextBox Height="50" HorizontalAlignment="Left" Margin="295.784,322.096,0,335.244" Name="txtAddress1" Width="350" FontSize="32"/> <Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,0,0,255.791" Name="lblAddress2" VerticalAlignment="Bottom" Width="155" FontSize="32">Address 2:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.803,0,0,255.791" Name="txtAddress2" VerticalAlignment="Bottom" Width="350" FontSize="32" /> <Label Height="50.50" HorizontalAlignment="Left" Margin="107.175,0,0,171.48" Name="lblAddress3" VerticalAlignment="Bottom" Width="155" FontSize="32">Address 3:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.803,0,0,171.48" Name="txtAddress3" VerticalAlignment="Bottom" Width="350" FontSize="32"/> <Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,0,0,90.027" Name="lblPostcode" VerticalAlignment="Bottom" Width="175" FontSize="32">Post Code:</Label> <TextBox Height="50.00" HorizontalAlignment="Right" Margin="0,0,720.216,90.027" Name="txtPostcode" VerticalAlignment="Bottom" Width="350" FontSize="32"/> <!-- Patient information fields, column 2 --> <Label FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,160.048,467.283,0" Name="lblPassword" VerticalAlignment="Top" Width="175">Password:</Label> <TextBox FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,160.048,104.317,0" Name="txtPassword" VerticalAlignment="Top" Width="350" /> <Label FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,240.072,367.283,0" Name="lblConfirmPassword" VerticalAlignment="Top" Width="275">Confirm Password:</Label> <TextBox FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,240.072,104.317,0" Name="txtConfirmPassword" VerticalAlignment="Top" Width="255" /> <Label FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,0,467.283,88.027" Name="lblUsername" VerticalAlignment="Bottom" Width="175">Username:</Label> <TextBox FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,0,104.317,88.027" Name="txtUsername" VerticalAlignment="Bottom" Width="350" IsReadOnly="True"/> <Button Height="46" HorizontalAlignment="Right" Margin="0,0,32,32" Name="btnRegister" VerticalAlignment="Bottom" Width="150" Click="btnRegister_Click">Register</Button> </Grid></Page>LMRS_RegisterNewPatient.xaml.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.Xml.Linq;namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_RegisterNewPatient.xaml /// </summary> public partial class LMRS_RegisterNewPatient : Page { public LMRS_RegisterNewPatient() { InitializeComponent(); // Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - Register New Patient"; } private void btnBack_Click(object sender, RoutedEventArgs e) { LMRS_MainMenu_Clinician mainMenu = new LMRS_MainMenu_Clinician(); this.NavigationService.Navigate(mainMenu); } private void btnRegister_Click(object sender, RoutedEventArgs e) { // Necessary information provided? if (!textFieldsNotComplete()) { MessageBox.Show("Please ensure all text fields have been completed.", "Missing Information", MessageBoxButton.OK, MessageBoxImage.Error); } else { // Load the XML file XDocument lmrsXml = XDocument.Load("LMRS_TestFile.xml"); // Does the patient user exist already? String patientUsername = "p_" + txtSurname.Text.ToLower() + txtForename.Text.ToUpper()[0]; // Get patient collection node XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == patientUsername select el; if (patient.Count() != 0) { MessageBox.Show("A patient is already registered with those details.", "Patient Already Exists", MessageBoxButton.OK, MessageBoxImage.Error); clearTextFields(); } else { // Add new patient information to record XElement newPatient = new XElement("patient", new XAttribute("username", patientUsername), new XAttribute("password", txtPassword.Text), new XElement("forename", txtForename.Text), new XElement("surname", txtSurname.Text), new XElement("addressLineOne", txtAddress1.Text), new XElement("addressLineTwo", txtAddress2.Text), new XElement("addressLineThree", txtAddress3.Text), new XElement("postCode", txtPostcode.Text), new XElement("activity", new XAttribute("name", "Fist_Clench")), new XElement("activity", new XAttribute("name", "Wrist_Flexion_Extension")), new XElement("activity", new XAttribute("name", "Three_Jaw_Chuck_Pinch"))); // Save and close patientCollection.Add(newPatient); lmrsXml.Save("LMRS_TestFile.xml"); MessageBox.Show("Patient added successfully.", "Patient Added", MessageBoxButton.OK, rmation); clearTextFields(); } } } private bool textFieldsNotComplete() { return txtForename.Text != "" && txtSurname.Text != "" && txtAddress1.Text != "" && txtAddress2.Text != "" && txtAddress3.Text != "" && txtPostcode.Text != "" && txtPassword.Text != "" && txtConfirmPassword.Text != ""; } private void clearTextFields() { txtForename.Text = txtSurname.Text = txtAddress1.Text = txtAddress2.Text = txtAddress3.Text = txtPostcode.Text = txtPassword.Text = txtConfirmPassword.Text = txtUsername.Text = ""; } private void txtSurname_TextChanged(object sender, TextChangedEventArgs e) { if (txtForename.Text != "") { txtUsername.Text = "p_" + txtSurname.Text.ToLower() + txtForename.Text.ToUpper()[0]; } } }}LMRS_RemovePatient.xaml<Page x:Class="LMRS_Main.LMRS_RemovePatient" xmlns="" xmlns:x="" xmlns:mc="" xmlns:d="" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"Title="LMRS_RemovePatient"> <Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Select a Patient to Remove" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/> <Label Content="Search for a user i.e. p__smithJ (case sensitive):" FontSize="32" Height="50" Margin="350.105,224.353,315.809,0" Name="lblInstructions" VerticalAlignment="Top" /> <Label Content="User:" FontSize="32" HorizontalAlignment="Left" Margin="455.851,280.084,0,0" Name="label1" Width="80" Height="52.563" VerticalAlignment="Top" /> <TextBox FontSize="32" Margin="541.591,280.084,474.428,0" Name="txtUserToRemove" Height="52.563" VerticalAlignment="Top" TextChanged="txtUserToRemove_TextChanged" /> <Button IsEnabled="False" Content="Search" Height="46" Margin="0,0,32,32" Name="btnSearch" VerticalAlignment="Bottom" HorizontalAlignment="Right" Width="149.921" Click="btnSearch_Click"/> <TextBlock TextAlignment="Center" FontSize="16" Height="128" Margin="325.812,0,290.087,135.755" Name="txtResults" VerticalAlignment="Bottom" /> <Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click"/> <Button Content="Remove" Height="46" HorizontalAlignment="Right" IsEnabled="False" Margin="0,0,32,110.033" Name="btnRemove" VerticalAlignment="Bottom" Width="149.921" Click="btnRemove_Click"/> </Grid></Page>LMRS_RemovePatient.xaml.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.Xml.Linq;namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_RemovePatient.xaml /// </summary> public partial class LMRS_RemovePatient : Page { // The xml file XDocument lmrsXml; // User to remove (if found on file) XElement userOnRecord; public LMRS_RemovePatient() { InitializeComponent(); // Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - Remove Existing Patient"; } private void btnBack_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_MainMenu_Clinician()); } private void txtUserToRemove_TextChanged(object sender, TextChangedEventArgs e) { if (txtUserToRemove.Text.Equals("")) { btnSearch.IsEnabled = false; } else { btnSearch.IsEnabled = true; } } private void btnSearch_Click(object sender, RoutedEventArgs e) { lmrsXml = XDocument.Load("LMRS_TestFile.xml"); // Find patient user XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == txtUserToRemove.Text select el; if (patient.Count() == 0) { MessageBox.Show("No user found with this username, please check the username and try again.", "User Not Found", MessageBoxButton.OK, MessageBoxImage.Error); txtUserToRemove.Text = ""; } else { userOnRecord = patient.First<XElement>(); txtResults.Text = "User found: \n\n" + userOnRecord.Element("forename").Value + " " + userOnRecord.Element("surname").Value + "\n" + userOnRecord.Element("addressLineOne").Value + "\n" + userOnRecord.Element("addressLineTwo").Value + "\n" + userOnRecord.Element("addressLineThree").Value + userOnRecord.Element("postCode").Value; btnRemove.IsEnabled = true; } } private void btnRemove_Click(object sender, RoutedEventArgs e) { if (MessageBox.Show("Removing this user will permanently delete all related exercise data, proceed?", "Confirm User Removal", MessageBoxButton.YesNo, MessageBoxImage.Question) == MessageBoxResult.Yes) { userOnRecord.RemoveAll(); userOnRecord.Remove(); lmrsXml.Save("LMRS_TestFile.xml"); MessageBox.Show("Patient removed successfully.", "Patient Removed", MessageBoxButton.OK, rmation); } txtUserToRemove.Text = txtResults.Text = ""; btnRemove.IsEnabled = false; } }}LMRS_ViewExerciseResults.xaml<Page x:Class="LMRS_Main.LMRS_ViewExerciseResults" xmlns="" xmlns:x="" xmlns:mc="" xmlns:d="" xmlns:chartingprimitives="clr-namespace:System.Windows.Controls.DataVisualization.Charting.Primitives;assembly=System.Windows.Controls.DataVisualization.Toolkit" xmlns:datavis="clr-namespace:System.Windows.Controls.DataVisualization;assembly=System.Windows.Controls.DataVisualization.Toolkit" xmlns:charting="clr-namespace:System.Windows.Controls.DataVisualization.Charting;assembly=System.Windows.Controls.DataVisualization.Toolkit" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"Title="LMRS_ViewExerciseResults" xmlns:chartingToolkit="clr-namespace:System.Windows.Controls.DataVisualization.Charting;assembly=System.Windows.Controls.DataVisualization.Toolkit" Width="1366"><Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="528.73*" /> <ColumnDefinition Width="837.27*" /> </Grid.ColumnDefinitions> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="View Exercise Results" Height="119" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White" Grid.ColumnSpan="2" /> <Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click"/> <!-- Combo boxes for exercise, start and end date selection--> <Label Content="Exercise:" Height="28" HorizontalAlignment="Left" Margin="32,185,0,0" Name="lblExercise" VerticalAlignment="Top" Width="150" /> <ComboBox Height="23" HorizontalAlignment="Left" Margin="30.009,218.637,0,0" Name="cmboExercise" VerticalAlignment="Top" Width="150" SelectionChanged="cmboExercise_SelectionChanged" /> <Label Content="StartDate:" Height="28" Margin="202.918,185,175.812,0" Name="lblStartDate" VerticalAlignment="Top" /> <ComboBox Height="23" Margin="202.918,218.637,175.812,0" Name="cmboStartDate" VerticalAlignment="Top" SelectionChanged="cmboStartDate_SelectionChanged"/> <Label Content="EndDate:" Height="28" HorizontalAlignment="Right" Margin="0,185,1.474,0" Name="lblEndDate" VerticalAlignment="Top" Width="150"/> <ComboBox Height="23" HorizontalAlignment="Right" Margin="0,218.637,1.474,0" Name="cmboEndDate" VerticalAlignment="Top" Width="150" SelectionChanged="cmboEndDate_SelectionChanged"/> <!-- Bar chart --> <chartingToolkit:Chart Name="chrtBarChart" Margin="27.151,171.48,28.58,30.009" Grid.Column="1"> <chartingToolkit:Chart.Axes> <chartingToolkit:CategoryAxis Title="Date and Rep #" Orientation="Y" /> <chartingToolkit:LinearAxis Title="Time in Seconds" Orientation="X" Interval="0.1"/> </chartingToolkit:Chart.Axes> <!-- Chart --> <chartingToolkit:BarSeries Title="barSeries" DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding}" /> <!-- Hide the legend --> <chartingToolkit:Chart.LegendStyle> <Style TargetType="datavis:Legend"> <Setter Property="Width" Value="0" /> </Style> </chartingToolkit:Chart.LegendStyle> </chartingToolkit:Chart> <!-- Line chart --> <chartingToolkit:Chart Name="chrtLineChart" Margin="27.151,171.48,28.58,30.009" Grid.Column="1" Visibility="Collapsed"> <chartingToolkit:Chart.Axes> <chartingToolkit:LinearAxis Title="Time in Seconds" Orientation="Y" Interval="0.1"/> <chartingToolkit:CategoryAxis Title="Date and Rep #" Orientation="X"/> </chartingToolkit:Chart.Axes> <!-- Chart --> <chartingToolkit:LineSeries Title="lineSeries" DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding}"/> <!-- Hide the legend --> <chartingToolkit:Chart.LegendStyle> <Style TargetType="datavis:Legend"> <Setter Property="Width" Value="0" /> </Style> </chartingToolkit:Chart.LegendStyle> </chartingToolkit:Chart> <!-- Column chart --> <chartingToolkit:Chart Name="chrtColumnChart" Margin="27.151,171.48,28.58,30.009" Grid.Column="1" Visibility="Collapsed"> <chartingToolkit:Chart.Axes> <chartingToolkit:LinearAxis Title="Time in Seconds" Orientation="Y" Interval="0.1"/> <chartingToolkit:CategoryAxis Title="Date and Rep #" Orientation="X"/> </chartingToolkit:Chart.Axes> <!-- Chart --> <chartingToolkit:ColumnSeries Title="columnSeries" DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding}"/> <!-- Hide the legend --> <chartingToolkit:Chart.LegendStyle> <Style TargetType="datavis:Legend"> <Setter Property="Width" Value="0" /> </Style> </chartingToolkit:Chart.LegendStyle> </chartingToolkit:Chart> <!-- Toggle between chart types --> <GroupBox Header="Chart Type" Margin="32,0,1.474,223.845" Name="grpChartTypes" Height="99.675" VerticalAlignment="Bottom"></GroupBox> <Grid Height="76.715" Margin="32,0,13.474,223.845" VerticalAlignment="Bottom"> <RadioButton HorizontalAlignment="Left" Margin="44.299,32.867,0,27.151" Name="rdoBarChart" Width="120" IsChecked="True" Checked="rdoBarChart_Checked">Bar Chart</RadioButton> <RadioButton Margin="148.616,33.564,214.35,27.151" Name="rdoLineChart" IsChecked="False" Checked="rdoLineChart_Checked">Line Chart</RadioButton> <RadioButton HorizontalAlignment="Right" Margin="0,33.564,100.03,27.151" Name="rdoColumnChart" IsChecked="False" Checked="rdoColumnChart_Checked" Width="120">Column Chart</RadioButton> </Grid> <!-- Accessible to clinician users only --> <Label Visibility="Hidden" Height="28" HorizontalAlignment="Left" Margin="30.009,274,0,0" Name="lblPatient" VerticalAlignment="Top" Width="150">Patient:</Label> <ComboBox Visibility="Hidden" Height="23" HorizontalAlignment="Left" Margin="28.018,308,0,0" Name="cmboPatient" VerticalAlignment="Top" Width="150" SelectionChanged="cmboPatient_SelectionChanged"/> </Grid></Page>LMRS_ViewExerciseResults.xaml.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.IO;using System.Xml.Linq;namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_ViewExerciseResults.xaml /// </summary> public partial class LMRS_ViewExerciseResults : Page { // The xml file storing user/session data XDocument lmrsXml; // Store information about specific exercise XElement chosenExercise; IEnumerable<XElement> sessions; IEnumerable<XElement> repTimes; // Used to populate combo boxes List<String> exercises = new List<String>(); List<String> dates = new List<String>(); // Used to populate the conditional patient combo box List<String> patients = new List<String>(); // Currently selected patient, used to filter date data XElement selectedPatient; // Used to populate the chart List<KeyValuePair<string, decimal>> chartDataList; // Used to toggle patient controls and ensure proper navigation when back is cliced string userType; public LMRS_ViewExerciseResults(string userType) { InitializeComponent(); // Set the window title WindowTitle = "Leap Motion Rehabilitation System - View Exercise Results"; // Load the XML file and populate the drop-downs lmrsXml = XDocument.Load("LMRS_TestFile.xml"); populatePatientComboBox(); populateExerciseComboBox(); this.userType = userType; if (userType.Equals("clinician")) { lblPatient.Visibility = Visibility.Visible; cmboPatient.Visibility = Visibility.Visible; } else { lblPatient.Visibility = Visibility.Hidden; cmboPatient.Visibility = Visibility.Hidden; } } private void btnBack_Click(object sender, RoutedEventArgs e) { if (this.userType.Equals("clinician")) { this.NavigationService.Navigate(new LMRS_MainMenu_Clinician()); } else { this.NavigationService.Navigate(new LMRS_MainMenu_Patient()); } } // Start things off with the XML parsing by loading the patient combo box private void populatePatientComboBox() { // Now load the exercise combo box XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patients = (from el in patientCollection.Elements("patient") select el); int i = 0; int j = 0; foreach (XElement el in patients) { this.patients.Add(patients.ElementAt(i).Element("forename").Value + " " + patients.ElementAt(i).Element("surname").Value); if (patients.ElementAt(i).Attribute("username").Value == UserInfoHelper.userName) { j = i; break; } ++i; } cmboPatient.ItemsSource = this.patients; cmboPatient.Tag = "ignoreCall"; cmboPatient.SelectedIndex = 0; selectedPatient = patients.ElementAt(j); } private void cmboPatient_SelectionChanged(object sender, SelectionChangedEventArgs e) { if (((string)cmboPatient.Tag).Equals("ignoreCall")) { cmboPatient.Tag = "acceptCall"; return; } // Now load the exercise combo box XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patients = (from el in patientCollection.Elements("patient") select el); int i = 0; foreach (XElement el in patients) { if (cmboPatient.SelectedItem.ToString().Contains(patients.ElementAt(i).Element("forename").Value) && cmboPatient.SelectedItem.ToString().Contains(patients.ElementAt(i).Element("surname").Value)) { selectedPatient = patients.ElementAt(i); break; } ++i; } cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0; cmboExercise.Tag = "ignoreCall"; cmboExercise.SelectedIndex = 0; refreshChart(); } // Load the exercise combo box private void populateExerciseComboBox() { // Now load the exercise combo box IEnumerable<XElement> exercises = (from el in selectedPatient.Elements("activity") select el); this.exercises.Add(exercises.ElementAt(0).Attribute("name").Value.ToString().Replace("_", " ")); this.exercises.Add(exercises.ElementAt(1).Attribute("name").Value.ToString().Replace("_", " ")); this.exercises.Add(exercises.ElementAt(2).Attribute("name").Value.ToString().Replace("_", " ")); cmboExercise.ItemsSource = this.exercises; cmboStartDate.ItemsSource = cmboEndDate.ItemsSource = dates; chartDataList = new List<KeyValuePair<string, decimal>>(); } // Retrieves all sessions for the selected exercise private void cmboExercise_SelectionChanged(object sender, SelectionChangedEventArgs e) { // Load available sessions for this exercise, first find the chosen exercise in the file String name = cmboExercise.SelectedItem.ToString().Replace(" ", "_"); IEnumerable<XElement> exercise = from el in selectedPatient.Elements("activity") where (string)el.Attribute("name") == name select el; chosenExercise = exercise.First<XElement>(); // Now get the sessions and rep times sessions = from el in chosenExercise.Elements("session") select el; repTimes = from el in sessions.Elements("rep") select el; dates.Clear(); int i = 0; foreach (XElement exl in sessions) { dates.Add(exl.Attribute("date").Value); ++i; } // Tag: arbitrary object value that can be used to store custom information about this element // Use it here to avoid triggering unwanted calls to refreshChart cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0; refreshChart(); } private void cmboStartDate_SelectionChanged(object sender, SelectionChangedEventArgs e) { if (((string)cmboStartDate.Tag).Equals("ignoreCall")) { cmboStartDate.Tag = "acceptCall"; return; } // End date cannot be before the start date if (cmboEndDate.SelectedIndex < cmboStartDate.SelectedIndex) { cmboEndDate.SelectedIndex = cmboStartDate.SelectedIndex; } refreshChart(); } private void cmboEndDate_SelectionChanged(object sender, SelectionChangedEventArgs e) { if (((string)cmboEndDate.Tag).Equals("ignoreCall")) { cmboEndDate.Tag = "acceptCall"; return; } // End date cannot be before the start date if (cmboEndDate.SelectedIndex < cmboStartDate.SelectedIndex) { cmboEndDate.SelectedIndex = cmboStartDate.SelectedIndex; } refreshChart(); } // Changing the dates via the combo boxes modifies the dataset used by the chart. private void refreshChart() { if (sessions.Count() != 0) { chartDataList.Clear(); int repOffset = (cmboStartDate.SelectedIndex * 5) < 0 ? 0 : (cmboStartDate.SelectedIndex * 5); for (int i = 0; i < ((cmboEndDate.SelectedIndex - cmboStartDate.SelectedIndex) + 1) * 5; ++i) { chartDataList.Add(new KeyValuePair<string, decimal>(sessions.ElementAt(cmboStartDate.SelectedIndex + (i / 5)).Attribute("date").Value + ": Rep " + ((i % 5) + 1).ToString() + " ", Convert.ToDecimal(repTimes.ElementAt<XElement>(repOffset + i).Value))); } chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = null; chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = chartDataList; // Update page title lblHeader.Content = "Results for " + exercises[cmboExercise.SelectedIndex] + " (" + cmboStartDate.SelectedValue + " - " + cmboEndDate.SelectedValue + ")"; } else { chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = null; } } private void rdoBarChart_Checked(object sender, RoutedEventArgs e) { chrtBarChart.Visibility = Visibility.Visible; chrtLineChart.Visibility = chrtColumnChart.Visibility = Visibility.Collapsed; } private void rdoLineChart_Checked(object sender, RoutedEventArgs e) { chrtLineChart.Visibility = Visibility.Visible; chrtBarChart.Visibility = chrtColumnChart.Visibility = Visibility.Collapsed; } private void rdoColumnChart_Checked(object sender, RoutedEventArgs e) { chrtColumnChart.Visibility = Visibility.Visible; chrtBarChart.Visibility = chrtLineChart.Visibility = Visibility.Collapsed; } }}UserInfoHelper.csusing System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Xml.Linq;using System.Threading;namespace LMRS_Main{ static class UserInfoHelper { static public XElement user; static public string foreName; static public string userName; }} ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download