Communication Patterns - Stanford
Communication Patterns
Reza Zadeh
@Reza_Zadeh |
Outline
Life of a Spark Program
The Patterns
Shuffling
Broadcasting
Other programming languages
Life of a Spark Program
Life of a Spark Program
1)Create some input RDDs from external data or parallelize a collection in your driver program.
2)Lazily transform them to define new RDDs using transformations like filter() or map()
3)Ask Spark to cache() any intermediate RDDs that will need to be reused.
4)Launch actions such as count() and collect() to kick off a parallel computation, which is then optimized and executed by Spark.
Example Transformations
map()
intersection()
flatMap()
filter()
distinct()
groupByKey()
mapPartitions()
reduceByKey()
mapPartitionsWithIndex()
sortByKey()
sample()
join()
union()
cogroup()
cartesion()
pipe()
coalesce()
repartition()
partitionBy()
...
...
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- pyspark sql s q l q u e r i e s intellipaat
- pyspark sql cheat sheet python qubole
- spark walmart data analysis project exercise
- cheat sheet pyspark sql python lei mao s log book
- apache spark computer science ucsb computer science
- communication patterns stanford
- advanced analytics with sql and mllib
- with pandas f m a vectorized m a f operations cheat sheet
- communication patterns stanford university
Related searches
- free watercolor patterns to trace
- seeing patterns in vision
- free watercolor patterns to print
- watercolor painting patterns free printable
- world history patterns of interaction
- gordon s functional health patterns questions
- geometric patterns in vision
- sentence patterns in english grammar
- seeing patterns in eyes
- gordon s functional health patterns pdf
- functional health patterns assessment tool
- free wood craft patterns catalogs