Efficient and Flexible Implementation of Machine Learning ...

Efficient and Flexible Implementation of Machine Learning for ASR and MT

Albert Zeyer1,2, Nick Rossenbach1,2, Parnia Bahar1,2, Andr? Merboldt1, Ralf Schl?ter1,2

1RWTH Aachen University 2AppTek

Introduction,

Purpose of this Tutorial

? Overview of existing deep-learning frameworks for human language technology (HLT) related tasks ? Discuss core features of higher-level frameworks in context of sequence processing ? Discuss application oriented features in context of machine translation, speech recognition and other tasks ? Compare needs of researchers to requirements for production systems ? Compare our framework RETURNN to other frameworks

What this tutorial is not about:

? Designing full pipelines for HLT tasks ? Explaining how to write complete RETURNN setups

Introduction, RETURNN tutorial | RWTH i6 | 25 Oct page 2 of 8

Introduction,

Target Audience

This tutorial is mostly targeting:

? Active framework and toolkit contributors ? Developers working on HLT related software ? Researchers interested in flexible neural architecture design

Prerequisite knowledge:

? Being familiar with sequence tasks and HLT ? Basic knowledge about deep learning platforms and frameworks (e.g. TensorFlow, PyTorch, MXNet...) ? Experience with designing and training neural networks

Introduction, RETURNN tutorial | RWTH i6 | 25 Oct page 3 of 8

Introduction,

Terminology: Framework

Machine learning/deep learning software can have many different terms:

? TensorFlow:

? "end-to-end open source machine learning platform" (Homepage / README) ? "Open Source Machine Learning Framework" (GitHub description) ? "open source software library for numerical computation using data flow graphs" (README 2015-2018)

? PyTorch:

? "open source machine learning framework" (homepage) ? "open source machine learning library" (Wikipedia)

? Keras:

? "the Python deep learning API"

? OpenSeq2Seq, ESPNet, Fairseq:

? "... toolkit ..."

We will discuss the conceptual and implementation aspects, not specific tools or scripts: In this presentation we will stick to the term "framework"

Introduction, RETURNN tutorial | RWTH i6 | 25 Oct page 4 of 8

Introduction,

Content and Speakers

Part 1: Implementation of Machine Learning for Sequence Processing Frameworks Overview Flexibility vs. Efficiency vs. Simplicity Concepts Recurrency Training Native Operations Conclusion

Part 2: Specific Models & Applications Introduction Machine translation (RNN/Transformer-based encoder-decoder-attention) Speech recognition (Hybrid HMM, Attention, other end-to-end approaches) Language modeling (RNN/LSTM, Transformer) End-to-end speech translation Text-to-speech

Conclusion

Introduction, RETURNN tutorial | RWTH i6 | 25 Oct page 5 of 8

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download