Communication Patterns - Stanford University
[Pages:25]Communication Patterns
Reza Zadeh
@Reza_Zadeh |
Outline
Shipping code to the cluster
Shuffling
Broadcasting
Other programming languages
Outline
Shipping code to the cluster
Life of a Spark Program
1)Create some input RDDs from external data or parallelize a collection in your driver program.
2)Lazily transform them to define new RDDs using transformations like filter() or map()
3)Ask Spark to cache() any intermediate RDDs that will need to be reused.
4)Launch actions such as count() and collect() to kick off a parallel computation, which is then optimized and executed by Spark.
Example Transformations
map() flatMap()
intersection() distinct()
filter()
groupByKey()
mapPartitions()
reduceByKey()
mapPartitionsWithIndex() sortByKey()
sample()
join()
union()
cogroup()
cartesion() pipe() coalesce() repartition() partitionBy() ... ...
Example Actions
reduce() collect() count() first() take() takeSample() saveToCassandra()
takeOrdered() saveAsTextFile() saveAsSequenceFile() saveAsObjectFile() countByKey() foreach() ...
Sending your code to the cluster
RDD ? Stages ? Tasks
RDD Objects
DAG Scheduler Task Scheduler
Worker
Cluster
manager
Threads
DAG
TaskSet
Task
Block
manager
rdd1.join(rdd2) .groupBy(...) .filter(...)
build operator DAG
split graph into stages of tasks
submit each stage as ready
launch tasks via cluster manager
retry failed or straggling tasks
execute tasks
store and serve blocks
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- spark walmart data analysis project exercise
- communication patterns stanford university
- cheat sheet pyspark sql python lei mao s log book
- apache spark computer science ucsb computer science
- communication patterns stanford
- pyspark sql s q l q u e r i e s intellipaat
- pyspark sql cheat sheet python qubole
- 1 introduction to apache spark brigham young university
- advanced analytics with sql and mllib
Related searches
- stanford university philosophy department
- stanford university plato
- stanford university encyclopedia of philosophy
- stanford university philosophy encyclopedia
- stanford university philosophy
- stanford university ein number
- stanford university master computer science
- stanford university graduate programs
- stanford university computer science ms
- stanford university phd programs
- stanford university phd in education
- stanford university online doctoral programs