Final Report.docx - Lehigh CSE



Volleyball Playcharting Pete Biencourt and Nicole WeiglTable of Contents: Executive Summary Introduction and MotivationThe ProblemImportance Proposed SolutionMerits of Solution Project PlanningOrganizationTechnical RequirementsProject Management Software DesignComponentsDependenciesIntercommunicationDesign Techniques Used Implementation Results Project SchedulesOverall ResultsDesignProblemsAlgorithms Verification and ValidationVerification of Correct SoftwareB.Simulations/Testing MethodsC.Lessons Learned from Verification and ValidationDocumentation Future WorkA.Unfinished WorkB.Moving ForwardsDiscussionThings we Learned Executive SummaryThe business of volleyball analytics has yet to reach the magnitude as other sports industries such as baseball and basketball. Although statistical methods do exist for analyzing volleyball, they have not yet been incorporated into standard technologies. The goal of our project was to help bridge the gap left wide open by the current practices of volleyball stat taking. Shot charting remains the standard method for which to collect and analyze the different passes on the court. While shot charting examines the collection of attack trajectories in search of a pattern, it does not look at a play as a single entity. Therefore we sought to create an easy to use mobile application that not only allows data to be stored via tablet input but gives the user a set of calculated patterns based off of an entire play instead of a single attack. We were successful in meeting our initial goals and by the end of the semester have produced an Android mobile application, which is compatible with all tablets. Our application gives the user three main tools to help diagnose patterns for a given team. The first is the ability to enter new data through the user interface. The second is the provided Quick Analysis, which will run quickly enough to be used during time outs or breaks between games to gain valuable knowledge about the trajectory of plays so far. Lastly the user can run a Seasonal Analysis on data collected over a long period of time, from one game to an entire season. Resulting patterns can be shown for a variety of situations including all six possible rotations and a view pertaining to the overall common trajectories when looking at all rotations together. While our final implementation meets the requirements we set for ourselves at the beginning of the semester, it is in no way complete. There are numerous extended capabilities we did not have the time to implement that we feel could prove extremely valuable to coaches or other users of our application. Aside from our extended features, there are a variety of different ways to expand upon our solution to this problem. Many of these ideas were considered in our planning process but we maintained focus on providing users with a clean and simple user interface that provided easily interpreted results. Introduction & Motivation The Problem:Traditional volleyball statistics track players in isolation, but tells us very little about combinations of players, or actions on the court. Traditional statistics simply lack the ability to capture information about how a player performs in a certain situation.Importance:There is a method of stat collection called “shot charting” which involves drawing lines to depict the trajectory of an attack, and then looking for patterns in the collected lines. However, generally shot charting is very limited in scope because the more pieces of a play that get collected, the more difficult it is for a person to identify the patterns. The analysis of this data is interesting because with enough data, it could start to give meaningful recommendations about where a team’s situational strengths and weaknesses lie, which is very advantageous information to have.Proposed Solution:We built an analysis engine that identifies patterns in the flow of a volleyball team’s games, and then correlates that data to a number of kills achieved. By doing this, we are able to find the strongest and weakest scenarios for a team, which allows teams to train under very specific conditions to address these things. It could also be used to analyze an opposing team. With good scouting before a big match, this kind of analysis could be used to discover the opposing team’s biggest weakness, and use it to your advantage. Merits of Solution:Our solution allows us to find patterns in entire plays, which quickly becomes too difficult for humans given the complexity of the plays in even one rotation. This means that given enough data, we can make recommendations to coaches about where their team’s weaknesses and strengths lie. We can also run our analysis based only on the current match, which can help give a team good data about how where the opposing team’s weaknesses and strengths lie.Project PlanningOrganizationClient:For our project our clients consist of Professor Davison, and volleyball coaches/statisticians who want an idea of how their team functions as a unit in certain types of situations. Communication and Meetings with Client:Since each of us worked on a different part of our project, we agreed that we should both maintain communication and attend weekly meetings with our client. We arranged to meet with Professor Davison every Monday at 10:10 AM. Our progress reports were turned in on a weekly basis by Sunday evening. Technical Requirements Development Platform:We’ll be using Android for our UI, Solaris for our backend, and Oracle for our database. Development Environment/Software/Languages:We used Eclipse as our IDE, and Java for our software development. We also used Oracle SQL for our database interactions. Performance Requirements:Most of our application is not performance critical, the piece that is mildly performance critical is the timeout analysis. This is because a typical timeout in volleyball only lasts about a minute, so we would like to have the analysis done in less than 10 seconds. Resources Required:We will need an Android tablet (we will be using a Nexus 7 for this project), access to an Oracle database, and the ability to run a service out of the Sunlab or a similar facility. We are acquiring the tablet privately, and we are working with various faculty members to get access to the others. Testing Methods:We will consider our program correct, when it produces what we deem to be reasonable recommendations based on the data that we give it. Unfortunately, we don’t think it will be possible to track this data with unit testing because of the possibility of a range of variance in the outputs. We will consider the User Experience correct when the GUI performs the actions the user requests and relays any appropriate information about them back to the user. Finally, we will consider the GUI (and the architecture correct) when it meets all of the user specifications. Project Management Brief Description of Project Execution: Our list of tasks is roughly as follows:Our developing environment and repository will be set upWe will acquire a tablet and access to an Oracle databaseWe will set up our databaseWe will mockup GUIThe data layer will be implementedThe engine will be implementedThe UI will be implementedTesting and tweaking will begin Requirements:Programmer-defined zone divisionsNative Android appDatabase schema that supports the following:PlayerZoneSubzoneCoordinates (Destination, Origin)Action IDs (previous, self, next)Play ID (previous, self, next)Half Play ID (previous, self, next)Match IDResult of PlayWhether or not the action ended the playRotationCalculation/recommendation modes Timeout Analysis: must be very fast, and based on match data (maybe match data for the opposing team, but season data for the user's team)Season Analysis: speed is not an issue, all data should be thoroughly examinedThe program should point out patterns (and make recommendations based on them) that it recognizes such as:Defensive WeaknessesDefensive StrengthsOffensive StrengthsServe-Receive Patterns All of the attributes mentioned above based on each rotationClean, Intuitive GUIPrecise drawing capturing for play charting The architecture of the program should be extremely modular to support future extensions.Specifically, it should separate the data entry functionality and the analysisHaving such an interface defined will allow for the analysis engine to be used under many UIs and methods of gathering the necessary data.Risk Areas:Our biggest risk area will be getting the GUI to communicate with the service. To minimize this risk, we will be keeping our I/O interface as simple as possible and using reliable access methods such as SSH to transfer our data. Interfaces:We will need an interface between the database and the engine, and between the engine and the GUI, and obviously the GUI. We will not really be integrating our work with an existing codebase. Self-Evaluation of Progress:We will consider our progress satisfactory if we meet our timeline. However, if something unexpected crops up, we will have to revise our timeline, and we will consider a reasonable amount of progress in resolving the unforeseen issue(s) as satisfactory.Software DesignComponentsDatabase: This is where we keep all of our information about matches and plays. It contains every action and its result and maps each action to a play, and each play to a match.Algorithms: These are the lifeblood of the program. They allow us to analyze a collection of plays and discover meaningful patterns about what typically happens in them. We use these algorithms to discover the patterns that we use to identify the weakest and strongest plays for a team, as well as a team’s typical response to a given player’s serve.Seasonal Analysis:This runs our algorithms for every play stored in the database. This allows us to mine the data for our team over the course of an entire season to discover the strongest and weakest patterns for the team in each rotation individually, and also for all of the rotations together.Quick Analysis:This analysis method runs our algorithms for only the data from the match that is currently being entered. This can be useful in discovering how to defend against/take advantage of the opposing team.User Interface:The UI is responsible for receiving user input and displaying the results produced by our algorithms. When starting the application the user is presented with three main choices, to view the instructions page, add a new match, or run the seasonal analysis. The Instructions page gives a general outline and explanation for the format of results returned along with how to enter new data. The New Match button allows the user to enter new plays, run a quick analysis on those plays, and display the results of the analysis to the screen. The Season Analysis button takes the user to a screen displaying the recommendations for the entire season’s matches. The user has options to view the results for all rotations or for each individual rotation (1-6). DependenciesThe system dependencies are as follows: The GUI depends on the ClientThe Client depends on the ServerThe Server depends on the Engine and DataIODataIO depends on the Database IntercommunicationCommunication between different components is as follows: The server communicates with the engine, and with DataIO.DataIO is the only class allowed to directly communicate with the database.The client communicates with the server.The GUI communicates with the client (they are kind of intertwined).The server acts as the gatekeeper to the backend. No outside classes are allowed to directly communicate with the engine or DataIO. Similarly, DataIO is the gatekeeper for the database.Design Techniques UsedA few of these features were mentioned in our requirements document: "Design for Extensibility"The architecture of the program should be extremely modular to support future extensions.Specifically, it should separate the data entry functionality and the analysisHaving such an interface defined will allow for the analysis engine to be used under many UIs and methods of gathering the necessary data."Make sure the problem is well defined"This technique is clearly defined by the detail provided in our requirements document. The document does a meticulous job of mapping out the problem and detailing our proposed solutionImplementation ResultsInitial and Final Project Schedules (Gantt Chart)Below is our final version of our project schedule. Unfortunately we lost access to our original Gantt chart from the beginning of the semester. However we can detail how our progress matches up to the schedule shown below and our original expectations. After the last week in October we fell a week behind on almost every component of our schedule due to hurricane Sandy. This meant that implementations on the server and for the UI were delayed a week before implementation, causing a delay in the start of Client-Server Communication implementation. UI Delays: Respond correctly to user input: Originally we had planned on giving the user to draw plays by dragging his/her finger across the screen. After this functionality was mostly implemented, we made the decision to change the input method from a dragging motion to points of contact with the ball. Due to the redesign of the input method, the implementation of this feature took longer than expected. Season Analysis Display: This feature was mentioned in our initial requirements but was left out of our Gantt chart planning. Once the Quick Analysis recommended plays could be accurately displayed the seasonal analysis layout was implemented. This took approximately two days to implement once the technology on the back end worked for a regular match analysis. Backend Delays:Client-Server communication: Between the unforeseen issues with Android and the mishap with returning nulls in finally blocks, we lost several days attempting to debug the client-server connection.Engine: We initially missed a couple of edge cases in the engine that caused it to throw some ArrayIndexOutOfBounds exceptions. These exceptions magically broke the debuggers on the computers that they occurred on, which also slowed us down.Overall ResultsOur overall results were very good. The program can successfully collect and interpret user input, and perform pattern analysis on the data it’s given. Additionally, our app can display the results to the user in a way that they can visually interpret. We also think that we succeeded in creating an attractive, and user-friendly GUI.DesignWe had originally planned to have the entire application run natively on Android, but we ultimately decided to run the backend on a remote server for a couple of reasons. The first was that neither of us knew how to work with Android’s flavor of SQL, and the second that we thought that we might not have enough processing power to crunch our data in a timely fashion. The decision to run the backend remotely meant a couple of things for the architecture. The most impactful, was that it required us to use a client-server model to communicate between the frontend and backend. This communication was tricky to get right, and caused us some major problems.ProblemsThroughout the course of the project, we encountered many problems that could not have been resolved by testing. These problems ranged from debugger errors to code inconsistencies. They simply cropped up and substantially slowed our progress. One of these problems was that when testing our Server code, if an ArrayOutOfBoundsException occurred, the debugger became permanently broken on that computer, saying that SignatureParser.class could not be found. We found no way to fix this problem, and when trying to debug the interactions between the GUI and the backend, this was significantly damaging to our development cadence. Another problem that we encountered was with SVN. It would seemingly randomly corrupt our repositories. Until we started checking in small changes very frequently, this continued to hamper our progress. In addition to problems with our SVN and debugger, we encountered a few problems when integrating the android portion of our code with the work done on the back end. Since all the code for the UI was created with Android functionality in mind, many Android-specific tools and methods were utilized. When we began adding components to the Android code to connect with the back end, some of the features built into both sides of the system did not properly match up. After spending time breaking down the problem we eventually added additional classes and methods supported by Android to enable the user interface to accurately interact with the server. AlgorithmsOur program consists of two important algorithms, one to find patterns in plays based on their outcome, and one to find patterns in plays based on the serve. They both leverage many of the same ideas to complete their analysis. Here is a step-by-step breakdown of how each processes the plays that it receives:Patterns From Result Algorithm:Look at the result of every play receivedRemove the plays that do not have the desired resultMark the final destination of each of these actions, to track the frequency of certain areas of the courtPerform a 2D weighted average of these frequencies to find the hottest regionsSearch around the hottest area for actions that ended thereMark origin of each of these actions on a frequency mapPerform a 2D weighted average of these frequenciesGo back to 5 until 3 actions have been foundStitch these actions into a play and return the discovered patternThe first algorithm works backwards, starting at the kill, and search backwards for the actions that facilitated. The second algorithm, however, works by starting with the first action and attempting to find a pattern in the following actions.Patterns From Serving Algorithm:Start from the first action of each given playMark the destination of each action, to track the frequency of certain areas of the courtPerform a 2D weighted average of these frequencies to find the hottest regionsSearch around the hottest area for actions that ended thereIf there are existing actions within the search radius add the discovered point to our pattern and continue, otherwise breakGo back to 2This algorithm attempts to build a pattern from the beginning. This algorithm could potentially return an entire response (pass, set, hit), or it can return only a piece of that, as a given serve may not result in similar sets or hits.Verification & ValidationVerification of Correct SoftwareWe extensively tested the components of the back end by feeding them data that we knew the results for, and checking to make sure that the results were correct (a sort of manual unit testing). We didn’t use unit tests though, because there were major pieces of the engine that did not necessarily have one correct answer. Therefore we were required to use our judgment as to whether or not the results returned were logical and reasonable. This testing mostly occurred while we were in the process of debugging. Specifically, parts were tested as follows:Engine:Using a test main, we passed contrived data to various methodsWe traced the data through the method, verifying that nothing unexpected happened to itWe examined the end result to make sure that a human looking at the same data would come up with a similar patternDataIO:Using a test main, we passed data to the database through our DataIO classWe then used SQLDeveloper to examine the date that had been added to the database, to confirm that no information was being lost or misplacedWe also had the test main retrieve data from the database, and reassemble it into our Java objects and compare the original object with its database companionServer:As we developed the server, we also developed several test clientsWe used these clients to check that the server was communicating the way we expected (as per our server spec)Unfortunately, because of the differences between pure Java and Android, these clients gave us a false sense of security and confidence in our ability to communicateMainActivity: Draw New Plays and Quick Analysis:New PlaysThe main functionality for our application was the ability to enter new data through user input. We were able to test this functionality by testing the bounds of the input.We put constraints on the UI to ensure that plays could not be entered out of bounds, which ensured that no invalid data would be passed to the back end of our system. The final portion of the new play functionality was making sure plays were correctly packaged for the client to later send to the database. Quick Analysis The testing for this feature was split between the features, which relied on the UI, and those dependent upon the back end. The testing performed on the engine covered the necessary features for the quick analysis.From the UI perspective, MainActivity had to receive the plays being returned from the database and correctly draw them on the screen.This feature was tested by double-checking that the point coordinates returned from the server were the same ones drawn on the results screen. One challenge we had to overcome to display results correctly was to translate the coordinates on the UI grid from the different orientation recognized by the analysis algorithm. After adding additional converter classes the points were returned correctly and accurate results were displayed. AnalysisActivity: Seasonal Analysis:The entire basis for this activity in app is to display the seasonal recommendations to the user in seven different views. This was tested by entering data into the backend and ensuring that the different results were drawn on the screen depending on the rotation number selected by the user. We determined this to be successful if the results changed between rotations and indicated a differentiating pattern for the different types returned results (strongest play, weakest, ect). Client:We tested the client against the live version of our serverWe did this by having it send contrived test data to the server and checking to make sure that it got back the response we expectedThis was difficult because before we could test it, we had to trust the other components to function correctlySimulations/Testing MethodsWe used both component level testing and system level testing. System level testing was the most revealing part of our testing because it exposed flaws that only became obvious when we tried to integrate the GUI and back end. Though all of our components worked well in isolation, as we mentioned, the integration was very difficult. This was for a variety of reasons, including miscommunication, difficulties with Android, and networking issues. The system level testing really helped us get a grasp on where the joints between components were malfunctioning, which often caused us to rethink how we were doing something. When these problems arose, we were no longer working in isolation, so we were able to collaborate and come up with much better solutions.Lessons Learned from Verification and ValidationDuring the V&V phase we learned that we should have implemented and tested the integration pieces of the project earlier than we did. In fact, we had identified this as a risk area, and made the mistake of building other functionality first. If we were to do it again, we would probably start by creating our server and Android client, and then building the other functionality as they evolved. We also discovered bugs that would have been easily caught by unit tests that went undiscovered until we started trying to draw plays. For example, in C# there is a convention to put a return statement after a try-catch-finally block, which is enforced by the compiler. Normally you return null or some other error here. C# does not allow return statements in finally. Having remembered the extra return, but forgetting it went outside of the block, a return null was put into several finally blocks. In Java, this overrides any other value you try to return. While this looks natural, a unit test would have immediately picked up on the problem. Unfortunately these things were added after the initial functionality was verified, and were left untested because they seemed so trivial. Again, a unit-testing suite (and the requisite knowledge of JUnit) would have been nice to have in this type of situation. And although it’s more difficult, there were certain parts of this project that would have benefitted from using Test Driven Development, so that we would have been aware of edge cases before coding methods.DocumentationMost of our documentation is the code itself, and the comments that are sprinkled throughout it. However there were a couple of additional documents that we created (not including those required for the class). The first is a short set of instructions for a user to get off the ground with the app. This is native to the UI, and can be seen on the instructions screen. We also created a schema diagram for our database. This was especially helpful to Pete as he developed the DataIO class, as it can be difficult to write complicated SQL joins while trying to keep the organization of the database in memory. To help Nicole visualize the layout of the application we developed many drawings and sketches that helped serve as an outline for the flow of the application. These sketches helped to determine the number of buttons and layers would be needed in order for the UI to provide us with the desired functionality. Arguably the most important non-code document that we created during this project was the server specification. It was absolutely critical to have this document to fall back on when building the server and client to make sure we were following the correct procedures. Especially since the integration was challenging, being able to rule out simple procedural errors quickly was very helpful. Future WorkUnfinished WorkWe completed all of our basic requirements, though we did not have time to finish our extensions. The one extension we did complete was the ability to keep track of the score throughout a game. We did build our app with our proposed extensions in mind, so implementing the remaining ones shouldn’t require a major overhaul of the database, or our general procedures.Moving ForwardThough we will not be working on the project in any official capacity next semester, we may continue to tinker with it. It would definitely be cool to get it into the hands of a coach (potentially the coaching staff at Liberty High School), see what they think, and then use that feedback to drive future development. It would also be interesting to have someone with some experience in sports statistical theory to help us refine our existing algorithms and develop new ones.If we were to pass this project off to someone else, there are a variety of ways they could precede. When we were establishing our original requirements, we talked a lot about using video capture to track players and the ball as they move on the court. A future developer could swap the GUI for a module that gathers information using video capture as an interesting extension.Alternatively, a future developer could continue to work on the project in its current form. There are several quality of life features that could be added with little difficulty (such as match review/playback and qualitative match info such as dates and location). There are also some more challenging problems that would be addressed. For instance, we had to switch from drawing plays to placing endpoints because of the difficulties involved in accurately normalizing human input into lines and vertices (we considered using the ShortStraw algorithm). DiscussionWe believe that our biggest takeaway from this project was that we should have built the most fragile part first. In this case, it was the client-server connection. We agree that we should have built this connection first and then built the functionality on both sides afterwards. This would have effectively made us iterate around one central component as opposed iterating around distinct components then facing the challenge of merging them into one unit. At the beginning of this project, we were quite ambitious about what we thought we could get done. However, as the semester wore on, it became clear that between new technologies, natural disasters, and the sheer scope of our project, we had potentially bitten off more than we could chew. Luckily, we were able to get our base requirements done, but we were not able to implement any of our extension items with the exception of score tracking capabilities. Essentially, the scope of our project placed us in a situation where we were forced to compress the testing of our final product into a short amount of time. Although we were able to do so effectively the time constraint hindered our ability to approach the situation in the most efficient way possible. Things We LearnedWe were lucky when developing this project that each of our skill sets covered major components of the project. When these components overlapped we were fortunate enough to have each other’s experiences to work through problems that cropped up. In the process of fixing such problems each of us was able to learn about new technologies such as server connectivity and Android functionality. We had the opportunity to learn about networking in Android, and its particular flavor of background workers. Though AsyncTasks (Android background workers) are somewhat unintuitive to work with at first, they are pretty powerful, and useful.We also learned the importance of agreeing upon certain conventions before developing. For instance, we neglected to establish a convention for how the frontend and backend handled the coordinates of a court. That was almost cataclysmic for us, but luckily a simple converter class solved the problem. We were also fortunate in that we discovered the problem very rapidly, and did not spend a significant amount of time searching for the root cause. On a very basic level, we learned about the complexities behind sports analysis systems. While initially it is hard to believe similar programs haven’t been created, once we began to lay out all the different components that must be organized the sheer difficulty of Volleyball analytics became clear. While we learned a lot throughout the creation of this project we also recognize the large portion of the problem left unexamined due to time constraints. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download