Table of Figures .edu



Deep Learning CourseFinal ReportInstructor: Dr. Edward A. FoxCS 4624: Multimedia, Hypertext, and Information AccessVirginia Tech, Blacksburg VA 2406105/11/2020Client: Amirsina TorfiTeam: Brendan Bolon | Samuel Burton | Harrison Ellis | Matthew Robinson | Andrew WhelanTable of Contents TOC \o "1-3" \h \z \u Table of Figures PAGEREF _Toc40105559 \h 3Table of Tables PAGEREF _Toc40105560 \h 4Executive Summary PAGEREF _Toc40105561 \h 5Introduction PAGEREF _Toc40105562 \h 5Objective PAGEREF _Toc40105563 \h 5Deliverables PAGEREF _Toc40105564 \h 6Client PAGEREF _Toc40105565 \h 6Team PAGEREF _Toc40105566 \h 6Requirements PAGEREF _Toc40105567 \h 8Open Source Deep Learning Course PAGEREF _Toc40105568 \h 8PyTorch PAGEREF _Toc40105569 \h 8ReStructuredText Documentation PAGEREF _Toc40105570 \h 8Design PAGEREF _Toc40105571 \h 9Implementation PAGEREF _Toc40105572 \h 10Evaluation PAGEREF _Toc40105573 \h 10User Manual PAGEREF _Toc40105574 \h 11Homepage PAGEREF _Toc40105575 \h 11Course Introduction PAGEREF _Toc40105576 \h 12Introduction to Deep Learning Module PAGEREF _Toc40105577 \h 12What is Deep Learning? PAGEREF _Toc40105578 \h 13Why Deep Learning? PAGEREF _Toc40105579 \h 14Applications of Deep Learning PAGEREF _Toc40105580 \h 14Basics Module PAGEREF _Toc40105581 \h 15Logistic Regression PAGEREF _Toc40105582 \h 16Derivatives and The Chain Rule PAGEREF _Toc40105583 \h 17Gradient Descent PAGEREF _Toc40105584 \h 18Simple Neural Networks PAGEREF _Toc40105585 \h 19Simple Logistic Regression PAGEREF _Toc40105586 \h 21Neural Network Module PAGEREF _Toc40105587 \h 22Overview PAGEREF _Toc40105588 \h 23Computation PAGEREF _Toc40105589 \h 24Activation Functions PAGEREF _Toc40105590 \h 25Backpropagation PAGEREF _Toc40105591 \h 26Simple Neural Network Classifier PAGEREF _Toc40105592 \h 27More on Deep Neural Networks Module PAGEREF _Toc40105593 \h 29Regularization PAGEREF _Toc40105594 \h 30Dropout PAGEREF _Toc40105595 \h 31Gradient Setbacks PAGEREF _Toc40105596 \h 32Batch Normalization PAGEREF _Toc40105597 \h 33Final Notes and Citations PAGEREF _Toc40105598 \h 33Developer Manual PAGEREF _Toc40105599 \h 34Course Layout PAGEREF _Toc40105600 \h 34Contribution PAGEREF _Toc40105601 \h 37Methodology PAGEREF _Toc40105602 \h 38Lessons Learned PAGEREF _Toc40105603 \h 46Timeline/Schedule PAGEREF _Toc40105604 \h 46Problems PAGEREF _Toc40105605 \h 47Solutions PAGEREF _Toc40105606 \h 48Solutions PAGEREF _Toc40105607 \h 49Future Work PAGEREF _Toc40105608 \h 49Acknowledgements PAGEREF _Toc40105609 \h 50References PAGEREF _Toc40105610 \h 51Table of Figures TOC \h \z \c "Figure" Figure 1: Class Organization and Structure PAGEREF _Toc40105611 \h 9Figure 2: Homepage View PAGEREF _Toc40105612 \h 11Figure 3: Course Introduction View PAGEREF _Toc40105613 \h 12Figure 4: Deep Learning Module PAGEREF _Toc40105614 \h 12Figure 5: What is Deep Learning Page PAGEREF _Toc40105615 \h 13Figure 6: Why Deep Learning Page PAGEREF _Toc40105616 \h 14Figure 7: Applications of Deep Learning Page PAGEREF _Toc40105617 \h 14Figure 8: Basics Module PAGEREF _Toc40105618 \h 15Figure 9: Logistic Regression Page PAGEREF _Toc40105619 \h 16Figure 10: Derivations and Chain Rule Page PAGEREF _Toc40105620 \h 17Figure 11: Gradient Descent Page PAGEREF _Toc40105621 \h 18Figure 12: Neural Network Architecture Page PAGEREF _Toc40105622 \h 19Figure 13: Neural Network Documentation PAGEREF _Toc40105623 \h 20Figure 14: Logistic Regression Documentation Page PAGEREF _Toc40105624 \h 21Figure 15: Neural Network Module PAGEREF _Toc40105625 \h 22Figure 16: Neural Network Overview Page PAGEREF _Toc40105626 \h 23Figure 17: Neural Network Computation Page PAGEREF _Toc40105627 \h 24Figure 18: Activation Functions Page PAGEREF _Toc40105628 \h 25Figure 19: Backpropagation Page PAGEREF _Toc40105629 \h 26Figure 20: Neural Network Classifier Page PAGEREF _Toc40105630 \h 27Figure 21: Neural Network Classifier Code Documentation PAGEREF _Toc40105631 \h 28Figure 22: Deep Neural Networks Module PAGEREF _Toc40105632 \h 29Figure 23: Regularization Page PAGEREF _Toc40105633 \h 30Figure 24: Dropout Page PAGEREF _Toc40105634 \h 31Figure 25: Gradient Setbacks Page PAGEREF _Toc40105635 \h 32Figure 26: Batch Normalization Page PAGEREF _Toc40105636 \h 33Figure 27: rST Paragraph PAGEREF _Toc40105637 \h 34Figure 28: rST Code Block PAGEREF _Toc40105638 \h 34Figure 29: rST Code Block on Page Example PAGEREF _Toc40105639 \h 35Figure 30: rST Embedded Links PAGEREF _Toc40105640 \h 35Figure 31: rST Embedded Block on Page Example PAGEREF _Toc40105641 \h 35Figure 32: rST Table PAGEREF _Toc40105642 \h 36Figure 33: rST Table on Page Example PAGEREF _Toc40105643 \h 36Figure 34: Pull Request Process PAGEREF _Toc40105644 \h 37Figure 35: Workflow Diagram PAGEREF _Toc40105645 \h 46Table of Tables TOC \h \z \c "Table" Table 1: Example Persona and Goals PAGEREF _Toc40105646 \h 38Table 2: Tasks and Subtasks of Goals for Personas PAGEREF _Toc40105647 \h 41Table 3: Schedule PAGEREF _Toc40105648 \h 47Executive SummaryOur project produced an open source course for deep learning that is hosted on GitHub. Our course is broken up into modules, each with sub-sections that are designed to give users a better understanding of how deep learning works and how to write code for deep learning. The sub-sections for each module have documentation describing the concepts in-depth and are accompanied by PyTorch code to show how the concepts work in practice. Under instruction from our client, Amirsina Torfi, we built this course following the format of courses he has designed before. Users are expected to move sequentially through the modules, as the concepts build on previous concepts.The deep learning course assumes that users have experience with writing code and with computer science concepts. The course also contains a large amount of linear algebra concepts and expects the user to have a background in that field. Users will still have much to gain without this background, but the course was designed with some previous knowledge in mind. Two aspects of our project are the design of the website using reStructuredText and our GitHub repository of PyTorch code. Our documentation was done using reStructuredText. The GitHub repository contains all of the source code for each module, written in PyTorch. Making the course open source and free was done to make it accessible to everyone, as there are not many free open source courses for deep learning. In order to test our course, we gave our fellow classmates as well as a few fellow CS majors access to our course. After taking the course, we were provided constructive feedback from our peers, and we also received feedback via a Canvas quiz that our classmates were able to take as a knowledge test. IntroductionDeep Learning is a subfield of Machine Learning that is focused around creating a multitude of neural networks that are able to learn from data and predict outcomes. Deep Learning is a field that has, in recent years, become increasingly applicable in the workforce due to its many useful applications. Though the demand for deep learning experience has been increasing, there is a lack of open source courses online that individuals can use to learn the basics. So, we have created an open source course, “A Deep Learning Course with PyTorch”, for individuals everywhere to learn the basics of deep learning. ObjectiveOur team has the objective of creating an open source course for deep learning using the PyTorch framework. We will educate our users about what deep learning is, why they should be interested in it, and what possible real-world applications it can have. We will provide resources to teach the fundamentals of linear algebra that are needed in order to implement deep neural networks effectively. We will also educate our users on what neural networks are, how they work, and how to create them using PyTorch. We will ensure that our users have the resources needed so that they can understand all of the basic aspects of what goes into creating a fully functioning deep learning algorithm. A key aspect of this course is that it is open source. Thus, anyone will have the chance to use the provided source code to its full potential, and others can add to its curriculum. PyTorch was chosen as the framework to use because it is generally considered easier to use than the other popular deep learning framework, TensorFlow. We believe it is best to teach our users the basics of deep learning using a framework that is easier to use and understand than others, since this is supposed to be an introductory course.DeliverablesThe deliverables for this capstone project are:An open-source repository of topic documentation hosted on GitHub code developed using Python and PyTorchComprehensive documentation on the following modules:IntroductionBasicsNeural NetworksDeep Neural NetworksClientThe client for our project is Amirsina Torfi, a Ph.D. student at Virginia Tech. He is the founder of Machine Learning Mindset [17], a non-profit organization that strives to enhance how AI is leveraged in today’s world. He is a developer and researcher who has dedicated the bulk of his professional life to Artificial Intelligence and its corresponding subfields such as machine learning and deep learning. His public GitHub repositories currently receive over ten thousand monthly readers, as well as having over five thousand developers from around the world currently benefiting from them [17]. Amirsina has been personally ranked as one of the top 20 GitHub Python developers in the United States. His dedication to this field is a large part of what has inspired our team to complete this open-source project. TeamOur team consists of the following students: Brendan Bolon, Harrison Ellis, Sam Burton, Matt Robinson, and Andrew Whelan. We are all majoring in Computer Science and have plans to graduate from Virginia Tech in the Spring of 2020. We took on the deep learning course because we wanted to learn about how deep learning differs from machine learning, and also explore how deep learning fundamentals could help us in our post-graduation careers. Below there is some background information regarding each member of our team and what role they served during the completion of this project.Sam Burton is interested in cloud architecture and cloud development, specifically on the AWS platform. He will begin his professional career working for AWS as a Solutions Architect, helping companies of all sizes to solve their cloud-related problems. He hopes to one day become an expert in the architecture of cloud computing so that he can apply his problem solving skills to this new, advanced technology that is rapidly changing the world. For the deep learning course, Sam is in charge of creating the skeleton of the GitHub repository. He has created the structure and layout of the course. He has also helped in the research and development of many individual modules. Matt Robinson is most interested in big data analytics. After graduation, he will begin his professional career as a data scientist for IBM, where he will be using cutting edge data mining technologies to identify new patterns and make new discoveries. For the deep learning course, Matt is the team leader, which means he is in charge of submitting assignments as well as in charge of communication with the client. Matt is also responsible for research and PyTorch assistance within each of the modules.Andrew Whelan is a student with a focus on data structures, security, and systems engineering. He will begin his professional career as a Systems Engineer at Northrop Grumman Mission Systems working with data models, research, and maintenance in the defense sector on various projects. With this experience in the defense sector, he hopes to be involved in development security operations, where cyber security is integrated in the development of new technologies. For this deep learning course, Andrew works to create PyTorch source code with Sam, assists in documentation of various concepts, and oversees the basics module in the course. Harrison Ellis is a Computer Science student focused on Software Engineering. Upon graduating, he will start his career at Ally Financial as a Software Engineer on a Digital Channels Technology team. Previous experience applying engineering practices to the development of web-based applications will provide him with a seamless transition to this role. Along with developing client-facing software, he will also gain knowledge of automating deployment processes. Harrison’s primary responsibilities pertaining to this deep learning course include documenting concepts and creating detailed tutorials for the covered topics. Brendan Bolon is studying Computer Science and is most interested in Software Engineering and applications development. Upon graduation Brendan will start his professional at Capital One as a Software Engineer as part of their Technology Development Program. He plans to use his background in full stack application development to help apply and develop the most cutting-edge technologies in the banking industry. Brendan’s primary role in development of the deep learning course has been creating documentation and outlining tutorials for the various topics.RequirementsWorking with our client, we came up with a set of requirements that he wanted us to meet during the creation of the deep learning course. Our biggest requirement is to develop a functional and well documented open-source course that can help users learn about deep learning concepts and some of the algorithms behind those concepts. We chose at the start of the project to write all of our source code using PyTorch, which is an open source Python framework that specializes in machine learning and deep learning applications. Finally, we chose to write all of our documentation in reStructuredText (rST) [6] to stay consistent with our client’s past work on GitHub [17].Open Source Deep Learning CourseOne of the main reasons our client wanted this course to be open source is because of the lack of material on the Internet that teaches deep learning basics. Coming into this project, our team was experienced in Python but had no knowledge about how deep learning algorithms worked. Our plan is to compile all the information we can from every trusted reference that we can find into one GitHub course that anyone can access. By providing source code and documentation that helped our team understand deep learning, we hope to provide users a way to learn basic deep learning concepts in one repository.PyTorchOur client gave us the choice to use PyTorch or TensorFlow as our deep learning framework; we ultimately decided on PyTorch. The main reason we chose PyTorch over TensorFlow was because our client is more familiar with PyTorch and said that it would be easier for him to assist us while writing source code. He also mentioned that the learning curve for TensorFlow is much steeper and that learning it in a semester would be a lot more challenging compared to PyTorch. This was not an easy decision for us because TensorFlow is predominantly used in the workforce, but because we are new to both frameworks and only have a semester to develop code, we chose the more intuitive framework.ReStructuredText DocumentationWe decided to use reStructuredText to document and display our course materials. We chose this framework over markdown because it was consistent with the past GitHub projects that our client has completed and also because of its close association with Python documentation.Design152405269230Figure SEQ Figure \* ARABIC 1: Class Organization and StructureFigure SEQ Figure \* ARABIC 1: Class Organization and Structure1524046482000This deep learning course is organized in the structure described in Figure 1. The overarching course is divided into four modules which are further divided into three to five subtopics. Each subtopic may contain well-commented Python scripts using PyTorch and various linear algebra packages for users to read and experiment with on their own. These modules are designed to be linear for users to follow and to help them focus on material that they require. Some subtopics may include fully written programs or snippets of code to give an idea of certain functions used.ImplementationThis course is built entirely inside of a GitHub repository called deep-learning-course and is a part of the Machine Learning Mindset GitHub repository [17]. We have chosen to have this learning course be on GitHub so that the course itself and all files associated with it are highly available and accessible to those who are interested in diving deep into the source files. The structure of the course is made using ReStructuredText, a type of markdown language that allows for more technical implementations.The course begins with a table of contents as well as a link to a book written by our client, “Deep Learning RoadMap - A Resource Guide”, in the case that any individual would be interested in further reading about deep learning. We then provide a brief introduction to the field of deep learning, and our motivations for creating this course [17]. As you can see in Figure 1, the course is then broken down into four modules, each containing sub-modules that pertain to a specific aspect of deep learning. We wrap up the course with a note and thanks from the creators.EvaluationKeeping this course simple and professional is extremely important. Our code must compile and run correctly, and all information contained within the tutorials must be well formatted and correct so that our course will be credible. For this to happen, we created a system to ensure that every member of the group has conducted the research and taken proper notes when doing so before beginning to write a tutorial or code. Afterwards, our system has each member of the group read over every bit of documentation and code that is created so that no typos are missed, everything is formatted correctly and flows well, and most importantly so that all information contained within our course is correct. We have tested our product by having a small group of our peers, who are interested in deep learning, take our course and evaluate it. The people who participated in the course were able to provide us with feedback in two separate ways. The first way that our peers have provided us with feedback is by taking a pretest before the start of our course and a post-test after the completion of our course. This feedback is important to us because we are able to see if the material we have created is getting absorbed by our peers as we intended. The second way that we have gotten feedback is through an end of survey response where our peers are able to provide us with insight on how to improve the class. From these surveys, we have been able to learn about what we did well and what we have to improve upon to make this the best course possible within our given time frame.Finally, our client will also be reading all of our documentation and code to ensure that it is correct, and representative of how he intends the course to be.User ManualHomepage4089405323840Figure SEQ Figure \* ARABIC 2: Homepage ViewFigure SEQ Figure \* ARABIC 2: Homepage View40894018986500Upon navigating to the course page, a user will be presented with a table of contents and a link to view a book written by our client as shown in Figure 2. Each link in the table of contents will direct the user to the specific position on the page where that module is located, facilitating navigation through the course. Course Introduction2381252390775Figure SEQ Figure \* ARABIC 3: Course Introduction ViewFigure SEQ Figure \* ARABIC 3: Course Introduction View2381250Secondly, users will see an introduction to the course as shown in Figure 3. They will have the opportunity to read the motivation behind why we created this course, and why we think people should be interested. Introduction to Deep Learning Module15049503854450Figure SEQ Figure \* ARABIC 4: Deep Learning ModuleFigure SEQ Figure \* ARABIC 4: Deep Learning Module1504950187325After reading the introduction and motivation, users will see four separate modules like the one above, as can be seen in Figure 4. Each module contains submodules pertaining to a specific area that is needed in order to be able to write a successful deep learning algorithm. Some modules only contain documented tutorials while others contain source code. The Introduction to Deep Learning module begins by containing documentation that teaches users what deep learning is, why it is important, and different applications for it. To see the tutorials, users will simply click on the word, “Tutorial”. Once they have clicked, they will be redirected to a new page such as the one in Figure 5. 04371975Figure SEQ Figure \* ARABIC 5: What is Deep Learning PageFigure SEQ Figure \* ARABIC 5: What is Deep Learning Page045910500What is Deep Learning?The “What is Deep Learning” page gives a brief background on deep learning and goes into supervised learning and the different types of data on which to train deep learning algorithms. This section is divided into 5 subsections, Background, Supervised Learning, Structured Data, Unstructured Data, and Neural Networks as shown in Figure 5.Why Deep Learning?03295650Figure SEQ Figure \* ARABIC 6: Why Deep Learning PageFigure SEQ Figure \* ARABIC 6: Why Deep Learning Page00The “Why Deep Learning page” as shown in Figure 6, explains why deep learning is important and has been gaining popularity in recent years. This page specifically highlights three main reasons: Data, Advancements in Computation, and Innovation in Algorithms.Applications of Deep LearningFigure SEQ Figure \* ARABIC 7: Applications of Deep Learning PageFigure 7 shows the “Applications of Deep Learning” page explains some of the different real-world and current applications of deep learning.Basics Module11548534937760Figure SEQ Figure \* ARABIC 8: Basics ModuleFigure SEQ Figure \* ARABIC 8: Basics ModuleThe basics module as shown in Figure 8 contains content for basics of linear algebra that users will need to know in order to understand how deep learning works. The sub-topics of this module are logistic regression, derivatives and the chain rule, and gradient descent. These tutorials are also accessed by clicking on the blue, “Tutorial” link. Logistic Regression04273550Figure SEQ Figure \* ARABIC 9: Logistic Regression PageFigure SEQ Figure \* ARABIC 9: Logistic Regression PageFigure 9 shows the top of the Logistic Regression Page. The logistic regression documentation page begins with a brief description introductory paragraph and is followed by links to the three separate sections. The notation section explains some commonly used logistic regression notation. The next section explains the sigmoid function and the final section shows how cost functions can be used to assess the sigmoid function and thus train parameters on data.Derivatives and The Chain RuleFigure SEQ Figure \* ARABIC 10: Derivations and Chain Rule PageThe tutorial page shown in Figure 10 discusses some basic calculus and how it is used in deep learning. The section’s primary purpose is to provide some clarifications about information discussed in the gradient descent section.Gradient Descent04857750Figure SEQ Figure \* ARABIC 11: Gradient Descent PageFigure SEQ Figure \* ARABIC 11: Gradient Descent Page00The gradient descent tutorial page (see Figure 11) begins with a visual representation of the concave cost function, J(w, b). The page provides examples of how gradient descent works on high and low levels, and also how it pertains to machine learning.Simple Neural NetworksFigure SEQ Figure \* ARABIC 12: Neural Network Architecture PageThe module on neural networks architecture is the first time in this course that users will be able to attempt writing code based on steps provided in this module. The module begins by giving users basic information about neural networks such as what they are and what their architecture looks like (see Figure 12). The module then provides the steps (see Figure 13) that should be taken by the users in order to write their own simple neural network, with pictures of code provided. The source code is also available under the label “code” for users to look at a completed simple neural network.Figure SEQ Figure \* ARABIC 13: Neural Network DocumentationOnce this module has been completed, users are able to navigate to the next module using a link at the bottom of the page. Simple Logistic RegressionFigure SEQ Figure \* ARABIC 14: Logistic Regression Documentation PageThe simple logistic regression tutorial page (see Figure 14) gives an in-depth breakdown of the logistic model that we trained. This model takes input from an online PyTorch dataset that contains images of handwritten numbers (ranging 0-9) and outputs its prediction of what number is being represented. The example output displayed at the end of the documentation shows the loss statistics as the model gets more accurate after hundreds of iterations.Neural Network ModuleFigure SEQ Figure \* ARABIC 15: Neural Network ModuleFigure 15 shows the outline of the Neural Networks Module. This module explains what neural networks are and how they are used in deep learning. This module starts off with a brief introductory overview of the module. Computation is the next section, followed by the activation function section, which explains what activation functions are and how they are used between layers of the neural network. The last section is backpropagation which explains how neural networks are modified to get more accurate results. All of the tutorials can be accessed by clicking on the tutorial links.OverviewFigure SEQ Figure \* ARABIC 16: Neural Network Overview PageThe overview tutorial (see Figure 16) gives an explanation about what a neural network is and how it fits into deep learning. The tutorial also shows how a neural network is represented graphically and in putationFigure SEQ Figure \* ARABIC 17: Neural Network Computation PageFigure 17 shows the Computation Page. This computation section describes how to compute the final value for a neural network on a specific training item. This computation is also called forward propagation.Activation FunctionsFigure SEQ Figure \* ARABIC 18: Activation Functions PageFigure 18 is the Activation Functions page which outline various activation functions that can be used in the creation of a neural network. This section shows all the different activation functions and explains how activation functions are used. The activation function section also explains why ReLU is the most widely used activation function.BackpropagationFigure SEQ Figure \* ARABIC 19: Backpropagation PageThe backpropagation section (see Figure 19) describes how to calculate the derivatives of the cost function working from right to left across the network.Simple Neural Network ClassifierFigure SEQ Figure \* ARABIC 20: Neural Network Classifier PageFigure 20 shows the Neural Network Classifier Page which outlines a simple neural network classifier code. Users will be able to follow the implementation of the code step by step and learn what each function does on a sample CIFAR10 dataset, a popular dataset that can be imported into Python. This section is inspired by a PyTorch tutorial found online [10] but will go further into detail with every concept used in the tutorial. Figure SEQ Figure \* ARABIC 21: Neural Network Classifier Code DocumentationFigure 21 shows the step by step organization of the Neural Network Classifier page.More on Deep Neural Networks ModuleFigure SEQ Figure \* ARABIC 22: Deep Neural Networks ModuleThe “more on deep neural networks” module (see Figure 22) gives a more in-depth overview of neural networks. The first two sections and last section of the module explain the different strategies to fix the problem of over-complex neural networks. The third section, Gradient Setbacks, discusses problems and limitations associated with gradient descent. The tutorials for each subject can be accessed by clicking on the tutorial link next to the topic.RegularizationFigure SEQ Figure \* ARABIC 23: Regularization PageFigure 23 shows the Regularization page that explains the problem of overfitting and how regularization can be used to fix it. In this section, the main two methods for regularization are explained.DropoutFigure SEQ Figure \* ARABIC 24: Dropout PageFigure 24 shows the Dropout Page. This section explains the dropout method and how it can be used to solve overfitting for overly complex neural networks.Gradient SetbacksFigure SEQ Figure \* ARABIC 25: Gradient Setbacks PageThe gradient setbacks section (see Figure 25) discusses issues and limitations that come with gradient based deep learning. The section also discusses recommended improvements for the setbacks.Batch NormalizationFigure SEQ Figure \* ARABIC 26: Batch Normalization PageThe tutorial on batch normalization (see Figure 26) discusses how to improve performance and results on networks with many hidden layers. The standardization methods are described as well as the overall effects of using batch normalization. Final Notes and CitationsThe page ends with directions on how to submit a pull request for the GitHub repository, a note from the creators, and citations for the course.Developer ManualThe developer manual gives aspiring developers insight on how we created our Deep Learning Course repository and ways that they can continue the work we have completed.Course LayoutAll of our content is developed in Python and reStructuredText. We chose Python as our programming language because it is very easy to learn and has extensive deep learning libraries, such as PyTorch. We decided to use reStructuredText as our markup language because it offers a wide array of markup features for Python development and is consistent with our client’s past projects. Below we have listed some examples of reStructuredText documentation that we used within our repository:Figure SEQ Figure \* ARABIC 27: rST ParagraphFigure 27 shows a reStructuredText format of paragraphs. Paragraphs written in reStructuredText require no specific markup to differentiate them from other elements in a document. They can be separated by headings to make them easier to read, but to add onto a section of the repository, a developer can simply keep writing directly onto the end of a given section.Figure SEQ Figure \* ARABIC 28: rST Code BlockCode blocks differ slightly from paragraph blocks in reStructuredText (see Figure 28) but are still fairly easy to grasp. Instead of using images to represent code within our documentation, we used a built-in feature of reStructuredText to display the code. This is done by indenting the code segment that you would like to display beneath the text line “.. code:: python”. After doing this, you have a nicely highlighted code block that fits seamlessly into the documentation, as seen in Figure 29.Figure SEQ Figure \* ARABIC 29: rST Code Block on Page ExampleFigure SEQ Figure \* ARABIC 30: rST Embedded LinksWe use embedded links all throughout our repository for a number of reasons. In the section above, we have links to three references and then a link to the next section of the course titled “Why Deep Learning?” (see Figure 30). These links can also be used in the midst of a paragraph which is a very useful quality of reStructuredText. The code is displayed in Figure 31.Figure SEQ Figure \* ARABIC 31: rST Embedded Block on Page ExampleFigure SEQ Figure \* ARABIC 32: rST TableFigure 32 shows the structure of tables in reStructuredText. Tables are used on the homepage of our course, with one for each module. The primary purpose of these is to keep the sections organized. The tables that can be seen in Figure 33 make it easier for the user to know what a module covers and how to move through it in an orderly fashion. Figure SEQ Figure \* ARABIC 33: rST Table on Page ExampleContributionFigure SEQ Figure \* ARABIC 34: Pull Request ProcessFigure 34 shows the page on the pull request process. This process for contributions to the course are described in the Pull Request Process section of the homepage.MethodologyList of possible users and goals that our system supports PersonaGoalStudents: People pursuing a degree in a computer related field. Often use online tutorials to learn more. Students could also be people in the workforce who have graduated from university but would still like to learn deep learning. 1. Learn the linear algebra needed to write effective deep learning algorithms2. Write a neural network algorithm in PyTorch for deep learning.3. Learn the various applications of Deep LearningTeachers: Individuals who are teaching classes in a computer related field. They will see the importance of keeping up to date with new, groundbreaking technologies. 1. Help students experiment in writing their own deep learning algorithms in PyTorch, and possibly add to the course content2. Give feedback to Course CreatorsTable SEQ Table \* ARABIC 1: Example Persona and GoalsTable 1 contains example personas and their goals to view our system on the user side.Break down of each goal into units of tasks and subtasksStudent:Teacher:Table SEQ Table \* ARABIC 2: Tasks and Subtasks of Goals for PersonasTable 2 shows steps to achieve a goal and its subtasks on the user side. This is outlined in the creator and system side in the following section implementation based service for each attack.Implementation based service for each taskStudents Goal 1: Learn the linear algebra needed to write effective deep learning algorithmsSub-task: Navigate to course pageInputs: Link to course, Web search/GitHub searchOutputs: Deep learning course pageLibraries, Functions, & Environments: Web browser, GitHubSub-task: Visit linear algebra overview documentationInputs: Link provided to linear algebra documentation from course pageOutputs: Detailed documentation of linear algebra conceptsLibraries, Functions, & Environments: GitHubSub-task: Read our breakdown of linear algebra fundamentalsInputs: Documentation of linear algebra conceptsOutputs: Students understanding essential linear algebra conceptsLibraries, Functions, & Environments: GitHub, Coursera Deep learning Course, Practical Linear Algebra for Machine Learning By: Amirsina TorfiSub-task: Make connections to basic deep learning conceptsInputs: Knowledge of basic linear algebra concepts and basic deep learning conceptsOutputs: Students realizing the math behind deep learning fundamentalsLibraries, Functions, & Environments: GitHub, Coursera Deep Learning Course, Practical Linear Algebra for Machine Learning By: Amirsina TorfiStudents Goal 2: Write a neural network algorithm in PyTorch for deep learning.Sub-task: Navigate to course pageInputs: Link to course, Web search/GitHub searchOutputs: Deep learning course pageLibraries, Functions, & Environments: Web browser, GitHubSub-task: Scroll to Table of ContentsInputs: User scrollOutputs: Table of Contents sectionLibraries, Functions, & Environments: Web Browser, GitHubSub-task: Click on Neural NetworkInputs: User clickOutputs: Neural Network section on GitHub pageLibraries, Functions, & Environments: Web Browser, GitHubSub-task: Click on Neural Network Architecture DocumentationInputs: User clickOutputs: Neural Network Architecture Documentation pageLibraries, Functions, & Environments: Web Browser, GitHub, reStructuredText hyperlink/layoutSub-task: Read Course MaterialInputs: Course DocumentationOutputs: Student understanding/referenceLibraries, Functions, & Environments: N/ASub-task: Click on Python CodeInputs: User clickOutputs: Basic Neural Network python code GitHub repositoryLibraries, Functions, & Environments: Web Browser, GitHub, reStructuredText hyperlink/layout, PyTorchStudents Goal 3: Learn various applications of deep learningSub-task: Navigate to open source Deep Learning CourseInputs: Link to course, Web search/GitHub searchOutputs: Deep learning course pageLibraries, Functions, & Environments: Web browser, GitHubSub-task: Surf course and click “Applications” sub-topicInputs: Link to Applications documentationOutputs: Documentation of relevant Deep Learning ApplicationsLibraries, Functions, & Environments: GitHub, Coursera Deep Learning Course, Journal/web referencesSub-task: Read page about applicationsInputs: Documentation of Deep Learning ApplicationsOutputs: Knowledge about different uses of deep learningLibraries, Functions, & Environments: GitHub, Coursera Deep Learning Course, Journal/web referencesSub-task: Click links to references at bottom of page to further explore more deep learning applicationsInputs: List of references used in researchOutputs: More data and applications that can be further studied/analyzedLibraries, Functions, & Environments: GitHub, journal/web referencesTeachers Goal 1: Help students experiment with writing their own deep learning algorithm in PyTorchSub-task: Help students with any questions they have when writing they own deep learning algorithmsInputs: Students questions related to their codeOutputs: Successfully implemented deep learning algorithms written by students using PyTorchLibraries, Functions, & Environments: PyTorchSub-task: Encourage students to use information in modules to write deep learning algorithmsInputs: Users viewing individual module tutorialsOutputs: Students initial attempts at PyTorch deep learning algorithmsLibraries, Functions, & Environments: PyTorchSub-task: Tell students to navigate and learn about each individual module Inputs: Tutorial page with links to each moduleOutputs: Students learn the basics of deep learningLibraries, Functions, & Environments: GitHubSub-task: Provide students with course linkInputs: Students clicking on the link to the courseOutputs: Students have access to the courseLibraries, Functions, & Environments: GitHubTeachers Goal 2: Give feedback to Course CreatorsSub-task: Navigate to course pageInputs: Link to course, Web search/GitHub searchOutputs: Deep learning course pageLibraries, Functions, & Environments: Web browser, GitHubSub-task: Scroll to Table of ContentsInputs: User scrollOutputs: Table of Contents sectionLibraries, Functions, & Environments: Web Browser, GitHubSub-task: Click on Contact UsInputs: User clickOutputs: Contact Us SectionLibraries, Functions, & Environments: Web Browser, GitHub, reStructuredText hyperlink/layoutSub-task: Click on email hyperlink: deeplearningcourse@Inputs: User clickOutputs: User proprietary e-mailing systemLibraries, Functions, & Environments: Web Browser, GitHub, user email server, reStructuredText hyperlinkSub-task: Write email and send to Course CreatorsInputs: User email/messageOutputs: sent email to course creatorsLibraries, Functions, & Environments: Web Browser, GitHub, user email serverSet of workflows02800350Figure SEQ Figure \* ARABIC 35: Workflow DiagramFigure SEQ Figure \* ARABIC 35: Workflow Diagramlefttop00Figure 35 shows the complete workflow diagram for the student and user side along with the system design to supplement how our course assists in achieving user goals.Lessons LearnedTimeline/ScheduleDue to the nature of our project and team, an immense amount of research was and is required. We created our schedule to ensure that we had allocated 2-3 weeks to create each module. We cycled through a 2-3 week process of research, compilation of notes, experimentation with code, implementation/testing of code, and writing documentation. Table 4 shows the “due date” or milestone date in the first column, topics covered in the second column, and action taken on said topics in the last column. Table 3 outlines the cycle of our process of research and implementation.DateTopics CoveredAction Taken02/14/2020Linear Algebra Review/PyTorch Review, Machine Learning Course ReviewResearch, Notes02/21/2020Introductory ModuleBasicsNeural NetworksDeep Neural NetworksPyTorchGitHub Page SetupResearch, Notes02/28/2020Introductory Module: What, Why, ApplicationsDocumentation03/06/2020Basics Module:Logistic RegressionDerivatives + Chain RuleGradient DescentResearch, Code Implementation03/13/2020Basics Module:Logistic RegressionGradient DescentCode Implementation03/20/2020Basics Module:Logistic RegressionDerivatives + Chain RuleGradient DescentDocumentation, Module Organization in GitHub page03/27/2020Neural Networks:OverviewNN ArchitectureRepresentationResearch, Documentation, Code experimentation04/03/2020Neural Networks:ComputationActivation FunctionsBackpropagationResearch, Code Experimentation04/10/2020Neural Networks:OverviewNN ArchitectureRepresentationComputationActivation FunctionsBackpropagationDocumentation, Code Implementation04/17/2020Deep Neural Networks:RegularizationDropoutGradient SetbacksBatch NormalizationResearch, Code Experimentation04/24/2020Deep Neural Networks:RegularizationDropoutGradient SetbacksBatch NormalizationDocumentation, Code Implementation04/26/2020Testing/Peer ReviewsEvaluationTable SEQ Table \* ARABIC 3: ScheduleProblemsThe biggest problem that we have faced so far this semester is the communication difficulties we have faced due to COVID-19. Prior to the shutdown of Virginia Tech, we met weekly with our client and met as a team twice a week on campus to work on the course. After the shutdown, we have done most of our client communication through Zoom and Slack. Zoom serves as a decent alternative to our weekly meetings, but it makes it hard for our client to help us with our coding problems. In terms of our team meetings, we still try and meet up twice a week, but it is definitely more challenging to make progress compared to when we could utilize campus resources.A second problem that we faced was the learning curve we all dealt with when trying to understand deep learning concepts and reStructuredText. Deep learning algorithms are very powerful, but they are not easy to totally grasp, so we had to take a lot of time researching and practicing different techniques. We were also all new to reStructuredText, so it took some time before we could start exporting all of our documentation onto the course webpage.The last major problem that we faced was deciding how we should balance our time between coding and documentation. This course is designed to help teach users deep learning basics. That requires a lot of documentation on how each algorithm is designed to work. With that being said, providing relevant source code is a difficult task that requires a lot of time and critical thinking.SolutionsWe tried to address all of the communication issues that we faced due to COVID-19 by being more active on Slack and by making sure that we communicated over slack about meeting upcoming deadlines. It has taken a lot more planning and out of class conversations to make sure that we are on track, but that is the only way to ensure that we are completing our work in a timely manner. In terms of dealing with the steep learning curve associated with deep learning, we all took the first four weeks of the project auditing a Coursera Deep Learning Class [13]. The videos and readings provided by this class gave us the basic understanding of deep learning that we needed to dive right into creating the course. Specifically, it gave us a lot of information that we could include in our documentation and a good base to start writing some source code. More research was definitely needed outside of this course, but it was very helpful in balancing the learning curve. Learning reStructuredText was new, but learning it was not nearly as difficult. It took some time but once we all had a feel for it, it became second nature.When deciding how to balance our time, we made the decision toward the beginning of the project to spend 60% of our time working on documentation and 40% of our time writing relevant source code. The reason we decided to divide our time up like this is because our client emphasized that our biggest objective is to provide helpful, well written explanations of deep learning concepts. The code portion of our course serves more as an example of how these concepts are used practically.SolutionsWe tried to address all of the communication issues that we faced due to COVID-19 by being more active on Slack and by making sure that we communicated over slack about meeting upcoming deadlines. It has taken a lot more planning and out of class conversations to make sure that we are on track, but that is the only way to ensure that we are completing our work in a timely manner. In terms of dealing with the steep learning curve associated with deep learning, we all took the first four weeks of the project auditing a Coursera Deep Learning Class [13]. The videos and readings provided by this class gave us the basic understanding of deep learning that we needed to dive right into creating the course. Specifically, it gave us a lot of information that we could include in our documentation and a good base to start writing some source code. More research was definitely needed outside of this course, but it was very helpful in balancing the learning curve. Learning reStructuredText was new, but learning it was not nearly as difficult. It took some time but once we all had a feel for it, it became second nature.When deciding how to balance our time, we made the decision toward the beginning of the project to spend 60% of our time working on documentation and 40% of our time writing relevant source code. The reason we decided to divide our time up like this is because our client emphasized that our biggest objective is to provide helpful, well written explanations of deep learning concepts. The code portion of our course serves more as an example of how these concepts are used practically.Future WorkOur team has now completed all four modules and their corresponding subtopics, written source code where it was needed, used reStructuredText to create the format and layout of the course, and completed extensive research pertaining to the topic of deep learning. To continue expanding upon this course following graduation, more coding tutorials will be added to enhance the quantity and quality of content that is included in the course for our users. Since this course is open source, contributions will be taken from any individual who believes they are able to contribute good content. The GitHub course page will continue to be maintained and regularly updated with any information the team feels is necessary to include in the course. AcknowledgementsWe would like to acknowledge our client, Amirsina Torfi:Ph.D. Student at Virginia TechTop 20 GitHub Python developer in the U.S. by GitAwardTop 100 GitHub Python developer in the world by GitAwardFounder of Machine Learning Mindset: Email: amirsina.torfi@GitHub: [1] Anon. 2018. 4.7. Forward Propagation, Backward Propagation, and Computational Graphs Colab Open the notebook in Colab. (2018). Retrieved April 25, 2020 from [2] Anon. 2019. ReLu. (May 2019). Retrieved April 7, 2020 from [3] Anon. 2019. torchvision.datasets. (2019). Retrieved April 25, 2020 from Anon. 2019. [4] Artem Oppermann. 2019. Deep Learning Academy. (October 2019). Retrieved April 25, 2020 from [5] Asad Mahmood. 2019. Logistic Regression on MNIST with PyTorch. (April 2019). Retrieved April 25, 2020 from [6] Eli Bendersky. 2017. reStructuredText vs. Markdown for technical documentation. (February 2017). Retrieved April 7, 2020 from [7] Elvis. 2020. A Simple Neural Network from Scratch with PyTorch and Google Colab. (February 2020). Retrieved April 25, 2020 from [8] F D. 2017. Batch normalization in Neural Networks. (October 2017). Retrieved April 25, 2020 from [9] Jason Brownlee. 2019. 14 Different Types of Learning in Machine Learning. (November 2019). Retrieved April 7, 2020 from [10] Jason Brownlee. 2019. A Gentle Introduction to Batch Normalization for Deep Neural Networks. (December 2019). Retrieved April 25, 2020 from [11] Jason Brownlee. 2019. A Gentle Introduction to Dropout for Regularizing Deep Neural Networks. (August 2019). Retrieved April 25, 2020 from [12] Jürgen Schmidhuber. 2014. Deep learning in neural networks: An overview. (October 2014). Retrieved April 7, 2020 from [13] Ng, Andrew, et al. “Deep Learning by Deeplearning.ai.” Coursera, [14] Originally published by Suryansh S. 2020. Gradient Descent: All You Need to Know. (April 2020). Retrieved April 25, 2020 from [15] Ritchie Ng. 2020. Logistic Regression with PyTorch. (2020). Retrieved April 25, 2020 from [16] Sagar Sharma. 2019. Activation Functions in Neural Networks. (February 2019). Retrieved April 6, 2020 from [17] Torfi, Amirsina, et al. “Machine Learning With Python.” GitHub, Machine Learning Mindset, 1 Dec. 2019, [18] Vikashraj Luhaniwal. 2019. Why Gradient descent isn't enough: A comprehensive introduction to optimization algorithms in... (May 2019). Retrieved April 25, 2020 from [19] “Training a Classifier.”?Training a Classifier - PyTorch Tutorials 1.5.0 Documentation, tutorials/beginner/blitz/cifar10_tutorial.html ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download