Elastic Spark Programming Framework (ESPF)
[Pages:51]Elastic Spark Programming Framework (ESPF)
A Dependency-Injection Based Programming Framework for Spark Applications
Bruce Kuo, Software Engineer, APAC Data, email: bruce3557@yahoo-
1
Outline
Motivation & Related Work Prerequisite Programming Framework Integration with Components Conclusion Q&A
2
Motivation & Related Work
3
Native Spark Application
public class GainsChartDataGeneration { public static void main(String[] args) { String sortedPredictionResultTable = args[0]; String gainTable = args[1];
Initialization
SparkConf conf = new SparkConf(); JavaSparkContext sc = new JavaSparkContext(conf); HiveContext sqlContext = new HiveContext(sc.sc());
Main logic
DataFrame dataFrame = sqlContext.table(sortedPredictionResultTable) .select("target", "score");
// Generate schema ... StructType schema = DataTypes.createStructType(newFields);
long totalCount = dataFrame.count();
4
List seqList = new ArrayList();
for (long i = 100; i ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- the delta lake series features the data and ai company
- apache spark github pages
- cheat sheet for pyspark github
- delta lake cheatsheet the data and ai company
- the ai thunderdome with sahara spark and swift using
- spark change schema of dataframe
- apache spark continuous processing in structured streaming and
- pandas cheat sheet python data analysis library
- log analysis example the data and ai company
- objective getting started with dataframes
Related searches
- what is the elastic clause
- elastic clause example in history
- elastic clause used in history
- examples of elastic clause
- what does the elastic clause do
- the elastic clause
- examples of elastic clause use
- elastic clause quizlet
- elastic clause definition for kids
- elastic clause definition quizlet
- elastic clause examples
- elastic clause of constitution