Simplify Data Conversion from Spark to Deep Software ...

Simplify Data Conversion from Spark to Deep Learning

Liang Zhang Software Engineer @ databricks

About Me

Machine Learning Team @ Databricks

Master in Carnegie Mellon University

Liang Zhang in/liangz1/

Agenda

Why should we care about data conversion between spark and deep learning frameworks?

Pain points Overview of the Spark

Dataset Converter Demo Best Practices

Motivation: Data Conversion from Spark to DL

? Images from driving camera: Detect traffic lights

? Large amount of data - TBs

? New images arriving every day

? Data cleaning and labeling

? Train the model with all available data and periodically re-train with new data

? Predict the label of new images

TensorFlow

Spark DataFrame

?

PyTorch

Pain points: Data Conversion from Spark to Deep Learning frameworks

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download