Artificial Intelligence and Machine Learning
Artificial Intelligence and Machine Learning
Vijay Gadepally Jeremy Kepner, Lauren Milechin, Siddharth Samsi
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work
supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
? 2020 Massachusetts Institute of Technology. Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS
Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
Slide contributions from: Siddharth Samsi, Albert Reuther, Jeremy Kepner, David Martinez, Lauren Milechin
Outline
? Artificial Intelligence Overview ? Machine Learning Deep Dives
? Supervised Learning ? Unsupervised Learning ? Reinforcement Learning ? Conclusions/Summary
AI and ML - 2 VNG 010720
1
What is Artificial Intelligence?
Narrow AI: The theory and development of computer systems that perform tasks that augment for human intelligence such as perceiving, classifying, learning, abstracting, reasoning, and/or acting
General AI: Full autonomy
AI and ML - 3 VNG 010720
Definition adapted from Oxford dictionary and inputs from Prof. Patrick Winston (MIT)
Big Data
AI. Why Now?
Compute Power
Machine Learning Algorithms
Source: DARPA/ Public domain
Source: DARPA/ Public domain
Convergence of High Performance Computing, Big Data and Algorithms that enable widespread AI development
AI and ML - 4 VNG 010720
2
AI Canonical Architecture
Sensors Sources
? IEEE. Figure 1 in Reuther, A., et al, "Survey and Benchmarking of Machine Learning Accelerators," 2019 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, USA, 2019, pp. 1-9, doi: 10.1109/HPEC.2019.8916327. All rights reserved. This content is excluded from our Creative Commons license. For more information, see
Structured Data
Algorithms
e.g.:
Human- Machine Teaming (CoA)
Users (Missions)
Data Conditioning
Information
Unstructured Data
? Knowledge-Based
? Unsupervised and Supervised Learning
? Transfer learning
? Reinforcement Learning
? etc
Knowledge
Human Human-Machine Complement Machine
Insight
Spectrum
CPUs
GPUs
Modern Computing
TPU
Neuromorphic
Explainable AI
Metrics and Bias Assessment
Robust AI
Verification & Validation
AI and ML - 5 VNG 010720
GPU = Graphics Processing Unit CoA = Courses of Action TPU = Tensor Processing Unit
Custom . . .
Security (e.g., counter AI)
Quantum
Policy, Ethics, Safety and Training
Select History of Artificial Intelligence
1950
1960
1970
1980
1990
AI Winters 1974?1980 and 1987?1993
2000
2010
2017
1950 - Computing
Machinery and Intelligence "Turing
Test" published by MIND vol. LIX
1956 - Dartmouth Summer Research Project on AI
J. McCarthy, M. Minsky, N. Rochester, O. Selfridge, C. Shannon, others
1955 - Western Joint Computer Conference Session on Learning Machines
W.A. Clark
G. Dineen
O. Selfridge
Generalization of Pattern Programming Pattern Recognition
Recognition in a Self-
Pattern
and Modern
Organizing System Recognition
Computers
(MIT LL Staff)
(MIT LL Staff) (MIT LL Staff)
1957 - Frank
Rosenblatt Neural
Networks Perceiving and Recognizing
A u to m atio n
1957 - Memory Test Computer, first computer to simulate the operation of neural networks
1959 - Arthur Samuel "Some studies in machine learning using the Game of checkers" IBM Journal of R&D
1958 - National Physical Laboratory in the UK Symposium on the Mechanization of Thought Processes
1960 - Recognizing handwritten characters, Robert Larson of SRI AI Center
1961 - James Slagle, Solving Freshman Calculus (Minsky Student) MIT
J. Forgie
and J. Allen
1979 - An
Assessment of AI from
a Lincoln Laboratory Perspective
Internal MIT LL
publication
1982 - Expert Systems Pioneer DENDRAL project at Stanford
1984 - Hidden Markov models
1988 - Statistical Machine Translation
1989 -
Convolutional Neural Networks
1986?present
The return of neural networks
1994 - Human-level
spontaneous speech recognition
1997 - IBM Deep Blue
defeats reigning chess champion (Garry Kasparov)
2001?present The availability of very large data sets
2007 - DARPA Grand Challenge ("Urban Challenge")
2012 - Team from U. of Toronto (Geoff Hinton's lab) wins the ImageNet Large Scale Visual Recognition Challenge with deep-learning software
2015 - DeepMind achieves human expert level of play on Atari games (using only raw pixels and scores)
2005 - Google's Arabic and Chinese to English translation
2011 - IBM Watson defeats former Jeopardy! champions (Brad Rutter and Ken Jennings)
2014 - Google's GoogleNet Object classification at near human performance
2016 - DeepMind AlphaGo defeats top human Go player (Lee Sedol)
2016 - DARPA Cyber Grand Challenge
AI and ML - 8 VNG 010720
Adapted from: The Quest for Artificial Intelligence, Nils J. Nilsson, 2010 and MIT Lincoln Laboratory Library and Archives
3
Artificial Intelligence Evolution
* Waves adapted from John Launchbury, Director I2O, DARPA
AI and ML - 9 VNG 010720
Four Waves of AI
REASONING*
Handcrafted Knowledge
LEARNING
Statistical Learning
Perceiving Learning
Abstracting Reasoning
Perceiving Learning
Abstracting Reasoning
CONTEXT
Contextual Adaptation
Perceiving Learning
Abstracting Reasoning
ABSTRACTION
System Evolution
Perceiving Learning
Abstracting Reasoning
Lots of data enabled
non-expert systems
Adding context to AI systems
Ability of system to abstract
Spectrum of Commercial Organizations in the Machine Intelligence Field
AI and ML - 10 VNG 010720
Source: Shivon Zilis, 2016,
? Shivon Zilis and James Cham, designed by Heidi Skinner. All rights reserved. This content is excluded from our Creative Commons license. For more information, see
4
Data is Critical To Breakthroughs in AI
Year Breakthroughs in AI
Datasets (First Available)
Algorithms (First Proposed)
1994 Human-level read-speech recognition
Spoken Wall Street Journal articles and other texts (1991)
Hidden Markov Model (1984)
1997 IBM Deep Blue defeated Garry Kasparov 700,000 Grandmaster chess games, aka Negascout planning algorithm
"The Extended Book" (1991)
(1983)
2005 Google's Arabic- and Chinese-to-English 1.8 trillion tokens from Google Web and
translation
News pages (collected in 2005)
Statistical machine translation algorithm (1988)
2011
IBM Watson became the world Jeopardy! 8.6 million documents from Wikipedia,
champion
Wiktionary, Wikiquote, and Project
Gutenberg (updated in 2010)
Mixture-of-Experts algorithm (1991)
2014 Google's GoogleNet object classification ImageNet corpus of 1.5 million labeled Convolutional neural network
at near-human performance
images and 1,000 object categories (2010) algorithm (1989)
2015 Google's Deepmind achieved human parity in playing 29 Atari games by learning general control from video
Average No. of Years to Breakthrough:
Arcade Learning Environment dataset of Q-learning algorithm (1992) over 50 Atari games (2013)
3 years
18 years
AI and ML - 11 VNG 010720
Source: Train AI 2017,
AI Canonical Architecture
Sensors Sources
Structured Data
Data Conditioning
Unstructured Data
Information
Algorithms e.g.:
? Knowledge-Based ? Unsupervised
and Supervised Learning ? Transfer learning
? Reinforcement Learning
? etc
Knowledge
Human- Machine Teaming (CoA)
Human Human-Machine Complement Machine
Spectrum
Users (Missions) Insight
CPUs
GPUs
Modern Computing
TPU
Neuromorphic
Explainable AI
Metrics and Bias Assessment
Robust AI Verification & Validation
AI and ML - 12 VNG 010720
GPU = Graphics Processing Unit CoA = Courses of Action TPU = Tensor Processing Unit
Custom . . .
Security (e.g., counter AI)
Quantum
Policy, Ethics, Safety and Training
5
Unstructured and Structured Data
DDaattaa CCoonnddiittiioonniinngg
Algorithms
HumanMachine Teaming
Modern Computing
Robust AI
Structured Data Types
Speech
Sensors
Network Logs
Metadata
Unstructured Data Types
Social Media
Human Behavior
Reports
Side Channel
Data Conditioning/Storage Technologies
- Data to Information -
Technologies
Capabilities Provided
Infrastructure/Databases
? Indexing/Organization/Structure ? Domain Specific Languages ? High Performance Data Access ? Declarative Interfaces
Data Curation
? Unsupervised machine learning ? Dimensionality Reduction ? Clustering/Pattern Recognition ? Outlier Detection
Data Labeling
? Initial data exploration ? Highlight missing or incomplete data ? Reorient sensors/recapture data ? Look for errors/biases in collection
AI and ML - 13 VNG 010720
Often takes up 80+% of overall AI/ML development work
Machine Learning Algorithms Taxonomy
DDaattaa CCoonnddiittiioonniinngg
Algorithms
HumanMachine Teaming
Modern Computing
Robust AI
Algorithms*
* "The Five Tribes of Machine Learning", Pedro Domingos
AI and ML - 14 VNG 010720
DNN = Deep Neural Networks SVM = Support Vector Machines Exp. Sys. = Expert Systems
Symbolists (e.g., exp. sys.)
Bayesians (e.g., naive Bayes)
Analogizers (e.g., SVM)
Connectionists (e.g., DNN)
Evolutionaries (e.g., genetic programming)
Artificial Intelligence Machine Learning Neural Nets Deep Neural Nets
Image Adapted From "Deep Learning" by Ian Goodfellow, Yoshua Bengio and Aaron Courville
6
Modern AI Computing Engines
DDaattaa CCoonnddiittiioonniinngg
Algorithms
HumanMachine Teaming
Modern Computing
Robust AI
CPU GPU TPU
What It Provides to AI
? Most popular computing platform
? General purpose compute
? Used by most for training algorithms (good for NN backpropagation)
? Speeds up inference time (domain specific architecture)
Computing Class
Neuromorphic ? Active research area
Custom
? ? ?
Quantum
? Ability to speed up specific computations of interest (e.g. graphs)
? Benefits unproven until now ? Recent results on HHL
(linear system of equations)
AI and ML - 15 VNG 010720
GPU = Graphics Processing Unit TPU = Tensor Processing Unit
HHL = Harrow-Hassidim-Lloyd quantum algorithm
Selected Results
Alexnet comparison: Forward-Backward Pass
SpGEMM Performance using Graph Processor (G102)
1E1+01144
d1E1+01133 n Seco1E1+01122 er esP1E1+01111 g Ed1E1+01100
Traversed
11E0+99 11E0+88
Traversed Edges / Second
ASIC Graph Processor (Projected) FPGA Graph Processor (Measured) Cray XK7 Titan (Measured) Cray XT4 Franklin (Measured)
16k Nodes, 64 Racks
256 Nodes Rack
64 Nodes Chassis
8 Nodes Mini-Chassis
4k Nodes 16 Racks
1024 Nodes 4 Racks
Embedded Applications
Data Center Applications
11E0+77 11E0+11
11E0+22
11E0+33
11E0+44
11E0+55 Watts
1E1+066
11E0+77
11E0+88
11E0+99
Neural Network Processing Performance
? IEEE. Figure 2 in Reuther, A., et al, "Survey and Benchmarking of Machine Learning Accelerators," 2019 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, USA, 2019, pp. 1-9, doi: 10.1109/HPEC.2019.8916327. All rights reserved. This content is excluded from our Creative Commons license. For more information, see
Peak GOps/Second
1T0ruTeeNroarOthps/W 1 TeraOpMso/WvidiusX 100 GigaOpsT/PWUEdge
MIT Eyeriss
WaveSystem DGX-2
TPU1
Arria
DGX-1
WaveDPU Turing
DGX-Station
TPU3 V100
Goya
TPU2
GraphCoreNode
Nervana P100
GraphCoreC2
Xavier
K80
TrueNorthSyPshi7290F
2xSkyLakeSP
Phi7210F
JetsonTX2
JetsonTX1
Legend
Computation Precision
Int8 Int16 Float16 Float16 -> Float32 Float32 Float64
Form Factor
Chip Card System
Computation Type
Inference Training
AI and ML - 16 VNG 010720
Peak Power (W)
Reuther, Albert, et al. "Survey and Benchmarking of Machine Learning Accelerators." arXiv preprint arXiv:1908.11348 (2019).
7
DDaattaa CCoonnddiittiioonniinngg
Algorithms
HumanMachine Teaming
Modern Computing
Robust AI
AI and ML - 17 VNG 010720
Robust AI: Preserving Trust
Confidence Level in the Machine Making
the Decision
Confidence Level vs. Consequence of Actions
High
Best Matched to Machines
Low Low
Machines Augmenting
Humans
Consequence of Actions
Best Matched to
Humans
High
Robust AI Feature Explainable AI
Metrics
Validation & Verification
Security
Policy, Ethics, Safety, and Training
AI and ML - 18 VNG 010720
Importance of Robust AI
Issue
User unfamiliarity or mistrust leads to lack of adoption
Unknown relationship between arbitrary input and machine output
Algorithms need to meet mission specifications
System vulnerable to adversarial action (both cyber and physical)
Unwanted actions when controlling heavy or dangerous machinery
Example
Solutions
Seamless integration, model expansion, transparent uncertainty
Explainability, dimensionality reduction, feature importance inference
Robust training, "portfolio" methods, regularization
Model failure detection, red teaming
Risk sensitivity, robust inference, high decision thresholds
8
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- future of retail operations winning in a digital era
- measuring news sentiment federal reserve bank of san
- the marketpulse corelogic
- big news truvada is going generic on september 30 2020
- vital statistics rapid release
- important changes for 2020
- wmc report gender and race representation on five big
- artificial intelligence and machine learning
- january 2021 calendarlabs
- ocean freight market update dhl
Related searches
- artificial intelligence companies usa
- best artificial intelligence companies
- artificial intelligence companies
- artificial intelligence stocks to buy
- artificial intelligence stocks
- best artificial intelligence investments
- cheap artificial intelligence stocks
- machine learning and artificial intelligence
- artificial intelligence vs deep learning
- artificial intelligence companies in america
- artificial intelligence company stocks
- artificial intelligence investment funds