Intro to Spark and Spark SQL
Intro to Spark and Spark SQL
AMP Camp 2014 Michael Armbrust - @michaelarmbrust
What is Apache Spark?
Fast and general cluster computing system, interoperable with Hadoop, included in all major distros
Improves efficiency through:
> In-memory computing primitives > General computation graphs
Improves usability through:
> Rich APIs in Scala, Java, Python > Interactive shell
Up to 100? faster
(2-10? on disk)
2-5? less code
Spark Model
Write programs in terms of transformations on distributed datasets Resilient Distributed Datasets (RDDs)
> Collections of objects that can be stored in memory or disk across a cluster
> Parallel functional transformations (map, filter, ...) > Automatically rebuilt on failure
More than Map & Reduce
map filter groupBy sort union join leftOuterJoin rightOuterJoin
reduce count fold reduceByKey groupByKey cogroup cross zip
sample take first partitionBy mapWith pipe save ...
Example: Log Mining
Load error messages from a log into memory,
then interactively search foBrasveTaRraDrnDsif
oormuesd RpDaD
tternsmessages
val
lines
=
spark.textFile("hdfs://...")
Cache 1
val
errors
=
lines.filter(_
startswith
"ERROR")
results
Worker
val
messages
=
errors.map(_.split("\t")(2))
messages.cache()
tasks
lines
Driver
Block 1
messages.filter(_
contains
"foo").count()
messages.filter(_
contains
"bar").count()
. . .
Result: sfuclal-lteedxttose1arTcBh doaf tWa iknip5e-d7iaseinc# ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- big data analytics with hadoop and spark at osc
- three practical use cases with azure databricks
- running apache spark applications cloudera
- unified data access with spark sql
- spark sql edu
- bootstrapping big data with spark sql and data frames
- spark sql tutorialspoint
- pyspark of warcraft europython
- advanced analytics with sql and mllib
- data import databricks