Www.stlogic.co.kr



02177000CDAP 4.3.3 ?????? ????? TOC \h \z \t "?? 1,2,??,1" CDAP ??? ??? : ?? PAGEREF _Toc509568171 \h 1?? - CDAP ?? PAGEREF _Toc509568172 \h 2CDAP ???? PAGEREF _Toc509568173 \h 4??: ?? ???? ?? ?? ??? PAGEREF _Toc509568174 \h 5?? : New York Times XML Feed Data ?? PAGEREF _Toc509568175 \h 14?? : ?? ?? ????? ?? PAGEREF _Toc509568176 \h 26?? : IoT ?? ??? ?? ? ??? PAGEREF _Toc509568177 \h 38?? : ?? ?? ?? ?? ?? ?? PAGEREF _Toc509568178 \h 52CDAP ????? PAGEREF _Toc509568179 \h 64CDAP ??? ?? ?? (?? ? ???) PAGEREF _Toc509568180 \h 66???? PAGEREF _Toc509568181 \h 71CDAP ???? PAGEREF _Toc509568182 \h 87??? ?? ???? PAGEREF _Toc509568183 \h 92??? ?? ???? PAGEREF _Toc509568185 \h 113???? ?? PAGEREF _Toc509568184 \h 113??? ?? ??? PAGEREF _Toc509568186 \h 128DataPrep ?? PAGEREF _Toc509568187 \h 130?? ?? PAGEREF _Toc509568188 \h 133?? (??) ? ?? ??? PAGEREF _Toc509568190 \h 136Directives PAGEREF _Toc509568191 \h 139Catalog Lookup PAGEREF _Toc509568192 \h 139Change Column Case PAGEREF _Toc509568193 \h 141Changing Case PAGEREF _Toc509568194 \h 143Cleanse Column Names PAGEREF _Toc509568195 \h 145Columns Replace PAGEREF _Toc509568196 \h 147Copy PAGEREF _Toc509568197 \h 149Cut Character PAGEREF _Toc509568198 \h 151Decode PAGEREF _Toc509568199 \h 153Diff Date PAGEREF _Toc509568200 \h 155Drop PAGEREF _Toc509568201 \h 157Encode PAGEREF _Toc509568202 \h 159Extract Regex Groups PAGEREF _Toc509568203 \h 161Fail on condition PAGEREF _Toc509568204 \h 163Fill Null or Empty PAGEREF _Toc509568205 \h 165Filter Row If Matched PAGEREF _Toc509568206 \h 167Filter Row If True PAGEREF _Toc509568207 \h 169Filter Rows On PAGEREF _Toc509568208 \h 171Find and Replace PAGEREF _Toc509568209 \h 173Flatten PAGEREF _Toc509568210 \h 175Format Date PAGEREF _Toc509568211 \h 179Format UNIX Timestamp PAGEREF _Toc509568212 \h 182Generate UUID PAGEREF _Toc509568213 \h 184Hash PAGEREF _Toc509568214 \h 186Increment a transient variable PAGEREF _Toc509568215 \h 191Indexsplit Indexsplit PAGEREF _Toc509568216 \h 192Invoke HTTP PAGEREF _Toc509568217 \h 193JSON Path PAGEREF _Toc509568218 \h 196Keep PAGEREF _Toc509568219 \h 201Mask Number PAGEREF _Toc509568220 \h 203Mask Shuffle PAGEREF _Toc509568221 \h 205Merge PAGEREF _Toc509568222 \h 207Parse AVRO File PAGEREF _Toc509568223 \h 210Parse AVRO Encoded Messages PAGEREF _Toc509568224 \h 212Parse as CSV PAGEREF _Toc509568225 \h 213Parse as Date PAGEREF _Toc509568226 \h 216Parse as Excel PAGEREF _Toc509568227 \h 218Parse as Fixed Length PAGEREF _Toc509568228 \h 219Parse as HL7 PAGEREF _Toc509568229 \h 221Parse as JSON PAGEREF _Toc509568230 \h 224Parser as Log PAGEREF _Toc509568231 \h 227Parse as Simple Date PAGEREF _Toc509568232 \h 232Parse as XML PAGEREF _Toc509568233 \h 233Parse XML to JSON PAGEREF _Toc509568234 \h 234Quantize PAGEREF _Toc509568235 \h 235Rename PAGEREF _Toc509568236 \h 238Send to Error PAGEREF _Toc509568237 \h 240Set Charset PAGEREF _Toc509568238 \h 242Set Column PAGEREF _Toc509568239 \h 243Set Columns PAGEREF _Toc509568240 \h 245Set Record Delim PAGEREF _Toc509568241 \h 249Set Type PAGEREF _Toc509568242 \h 250Set a transient variable PAGEREF _Toc509568243 \h 251Split By Separator PAGEREF _Toc509568244 \h 252Split Email PAGEREF _Toc509568245 \h 253Split to Columns PAGEREF _Toc509568246 \h 256Split to Rows PAGEREF _Toc509568247 \h 259Split URL PAGEREF _Toc509568248 \h 262Stemming PAGEREF _Toc509568249 \h 265Swap PAGEREF _Toc509568250 \h 267Table Lookup PAGEREF _Toc509568251 \h 269Text Distance PAGEREF _Toc509568252 \h 271Text Metric PAGEREF _Toc509568253 \h 274Trimming Spaces PAGEREF _Toc509568254 \h 277URL Decode PAGEREF _Toc509568255 \h 280URL Encode PAGEREF _Toc509568256 \h 282Write as CSV PAGEREF _Toc509568257 \h 284Write as JSON Map PAGEREF _Toc509568258 \h 286Write as JSON object PAGEREF _Toc509568259 \h 288XPath Directives PAGEREF _Toc509568260 \h 290Functions PAGEREF _Toc509568261 \h 291JSON Functions PAGEREF _Toc509568262 \h 291Type Functions PAGEREF _Toc509568263 \h 294Geofence Functions PAGEREF _Toc509568264 \h 297Data Quality Functions PAGEREF _Toc509568265 \h 299Date Manipulation Functions PAGEREF _Toc509568266 \h 302Data Definition Manipulation Functions PAGEREF _Toc509568267 \h 305Services PAGEREF _Toc509568268 \h 308Administration and Management Services PAGEREF _Toc509568269 \h 308Connection Properties PAGEREF _Toc509568270 \h 314Connection Service PAGEREF _Toc509568271 \h 315??? ?? PAGEREF _Toc509568272 \h 322 ?? ?? ??(Request Format Specification) PAGEREF _Toc509568273 \h 324Schema Registry PAGEREF _Toc509568274 \h 327Data Prep Service PAGEREF _Toc509568275 \h 329CDAP ??? ??? : ??CDAP? ?? ?? ?????!CDAP? Hadoop ???? ???? ?? ?? ??, Apache 2.0 ????, ?? ?????? ??? ?????. ?? Hadoop ??? ???? ????? ???? ?? ? ????(on-premise)?? ??? ??? ?? ??????? ??, ?? ? ???? ?? ???? ???? ?? API? ??? UI? ?????.? ???? CDAP? UI ??? ???? ??? ???? ??? ??? ?? ???? ???? ?? ???????. ? ????? ?? ????? UI ??? ?? ?? ???? ???? ????.CDAP? ????? CDAP ? ??????? ???? ?? ?? ???(?? ???, ??? ???)? ??????.?? - CDAP ??Cask Data Application Platform (CDAP)? ? ??? ??????? ?? ?? ???(Unified Integration Platform) ???. ?? ??? ??? ???? ??? ? ???? ??? ??? ??? ???? ?? ? ? ?? ???? ???? ?? ???(abstraction)? ???? ??? ?? ??? ???? ???. ??? ??, ?? ? ??? ?????? ?????? ??, ??, ?? ? ????(govenance)? ??? ?? Big Data ??? ?????. CDAP? ???? Big Data? ??? ? ? ??? ??? UI? ?????.CDAP? ??? ??? ?? ? ?? ??? ?????? ?????.Data Preparation ? ??? ??? ? ?? ?? ???(point-and-click transformation directives)? ???? ??(explore), ??(cleanse), ??(prepare) ? ??(transform) ? ? ?? ??? UI???. Data Preparation(??? ??)? ?? ??? ??????.Data Pipelines? ???? ??? ??(data sources), ??(transformations) ? ??(sinks)? ???? ??? ??? ??? ??????? ?? ? ? ????. ?? ???? ??? ???? ??? ??? ???? ??? ??? ???? ? ??? ?? ??? ??? ?? ??, ??? ? ?? ?? ??? ?? ???? ?????. ??? ??? ?? ???? ??? ??????.CDAP? ?? ?? ??Cask Data Application Platform (CDAP)? Apache ? Hadoop??? ??? ??????? ?? ? ???? ?? ?? ??? ?????.CDAP? ??? ????? ??? ??? ??? ?? ??? ? ??? ???? ??? ??? ?????:Data Collection: Data Preparation ? Data Pipelines ? ? ??, ?????? ?? ??? ???? ??? ???? ?????? ??? UI? ?????.Data Exploration: Data Preparation? ?? ???? ???? ?? ???? ?? ?? ? ? ????. ??? ??? ???? ?? ???? ??? ??? ?? ? ?? ????.Data Processing: ???? Data Preparation ?? ? ??? ?? ????? ??? ??? ??? ?? ? ? Spark ? MapReduce ???? ?? ?????. ????? ??? ?? ???? ?? ???? ?? ? ? ????.Data Storage: CDAP? ???? ????? ?? ?? ??? ? RESTful ?????? ???? ?? ??? ??? ?????. ??? ??? ???? ?? ??? ???? ?????? ?? ??? ?? ? ?????? ?? (?? ??, Avro ?? RDBMS)? ???? ?? ?????. ???, ??? ??? ??? ?? ?? ??? ?? ??? ?? ? ??? ????.CDAP? ???? ??? ??? ? ???? ??? ???? ???? ???? ??? ??? ??? ??? ?? ?? ? ? ????. CDAP? ????? CDAP? ???? ?????? ? ????? ?? ????? CDAP? ??????.?? ???CDP? Data Preparation, Data Pipelines ? Metatada?? ???? ?? ??? ??? ?? Java ??????? ???? ???? ?? ? ? ?? ?? ?? ???(Unified Integration Platform) ???. ???? ??? ?? ??????? ???? ?? ?? ??? ???? ??????.CDAP ??????? ?????? CDAP UI? ???? ??? ???? ?? ??? ???? ??? ???? ? ??? ??? ??? ?????.?? ???? ?? ?? ???: ? ??????? CDAP? Data Preparation? Data Pipelines? ???? MySQL ??????? ??? ?? ???? ??(clean), ??(prepare) ? ??(store)?? ??? ?????. CDAP? ??? ??? ???? ??, ?? ??? ???? ?? ? CDAP ??? ??? ?? ??? ?????.?? ??? XML ?? ??? ??: ? ??????? CDAP? Data Preparation? Data Pipelines? ???? New York Times XML ???? ??? ??? ??? ??? ???? ???? ??? ?????.?? ?? ??? ?? ??: ? ??????? CDAP? Data Preparation? Data Pipelines? ???? ?? ???? ???? ???? ?? ??? ???? ?? ?? ??? ??? ???? ??? ?????.IoT ?? ??? ?? ? ???: ? ?????? CDAP? Data Preparation? Data Pipelines? ???? JSON ???? ??? IoT ?? ???? ??, ??, ??? ? ???? ??? ?????.?? ?? ?? ?? ??: ? ?????? CDAP? ???? CDAP? ??? ? ?? ?????? ???? ?? ??? ???? ??? ?? ??(Machine Learning)? ???? ???? ??? ?????.??: ?? ???? ?? ?? ?????? ??????? CDAP? Data Preparation ? Data Pipeline? ???? MySQL ???????? ?? ???? ??(clean), ??(prepare) ? ??(store)?? ??? ?????. CDAP? ??? ??? ???? ??, ?? ??? ???? ?? ? CDAP ??? ??? ?? ??? ?????.?????? ?? ???? ???? ???? ????? ?? ?? ?? ??? ??? ??? ?? ???? ????? ???. ??? ? ???? ? ?? ?? ??? ????.?????, ??? ?? ???? ???? ???? ???.??? ?? ???? ??, ????? ?? ?? ? ? ?? (Roads ?? Courts? ??) Avenues ? ???? ??? ???. ???????? ???? ? ??? ???? ???? .zip ??? ??????? ??? ??????.Zipfile??? ??????? ????? ????????? ?? ???? ???? ???. demo.sql? ?? ???? ?????. shell?? MySQL? ??? (mysql -u root? ???)?? (CREATE DATABASE demo;? ????) demo?? ??????? ????. ?? ?? mysql ? ??????(‘exit;’’).shell?? demo.sql? ??? ????? ???? mysql -u root -p demo < demo.sql? ??????. ?????? demo?? ?? ?? ???? ?? customer ???? ??????.CDAP? ?? ??? ???? Data Preparation? ??????. ?? ??? ? (?? ???? ?? ?? ?? ?? ??? ???? ?? ? ? ??)?? “Add Connection”? ??????. ??? “Database” ??? ??????.MySQL ????? ???? ?? ?? ? ??? ?? ???. ??? ?? ?? ????? ???? ??? ??? “Cask Market” ? ??????. ?? ?? ?? ??? “Drivers”? ??????. MySQL JDBC ????? ??? ?? ???? ???? ?? ????? ??????. “Add Connection” ?????? ??????? ??? ????? (??? ??? ??? ? ?? ???). host? localhost?, port? 3306? ???? username/password? ?????? (?? ??? ???? ??? ?? root ????). “Test Connection”? ???? ??? ????? ????Database?? demo? ??????.??????? ????? ??? ?????? ??? ??????. ?? ??? “Database” ?? ??? ????. ?? ?? customer ???? ??????. ??? ? / ? ???? ??? ?? ???? ? ? ????.???? states.json ??? ???? ???. ?? ?? ??? ?? ???? ?? ?? ???? ???? ???? states.json? ??? ??? ?????. ??? ? ? ???? ??? ? ? ????.?(State) ?? ??????? ??? nagivation ???? “Califonia”? ?? “CA”? ?? ?? ? ?(state) ??? ??? ??? ?????. ??? ?? ????? ?? ? ??? ?????. ?? ??? states.json ???? ?? ? ??? ?? ? ? ??? ?? ? ?? ?? ????. ?? ??? ??? ???? ?? ???? ???? ???? ? ? ????.??(mapping)? ???? states.json ?? ???. Data Prep UI ?? ? ?? ??????. body?? ?? ???(▼)? ???? “Parse” ? “JSON”? ??????. ? ???? ? ? ??????.? ? ???? ??? ?? array???? ??(parsing)? ?? array?? ? JSON ?? ???? ??(parsing)?? ?? ?????. title? ???? text? ??? body_name ??? name??, body_abbreviation? abbreviation?? ??????.?? “Create Pipeline”? ???? “Batch”? ??????. ??, Pipeline UI? ???? ???, “File” ?????? “Wrangler” ?????? ??? ? ? ?? ????. ? “Wrangler”????? Wrangler?? ?? ??? ????(? ? ??? JSON? ?? ??)? ?????.?? ??? ??? “Sink”? ???? “CDAP Table Dataset”? “Avro Time Partitioned Dataset”????? ?? ??????. “Wrangler”????? output? “CDAP Table Dataset”? ??????. “CDAP Table Dataset” ????? ???? “Row Field”? “name”? ??????.Pipeline ??? “StateNamePipeline”?? ??????. ?? ?? “Deploy”? ???? ? ?????? ??????. ??? “Run”? ???? ? ?????? ??????.?? ???? ? ??? ???? ????? ?????? ? ??? ? ?? CDAP Table Dataset? ??????.?? ???? ? ?? ?????? ?? ? ??? ???? ?? ? ????. Data Preparation?? ???? customer ?? ??????.null state ?? ?? ??? ?? ? ? ???? null state ?? ??? ???????. ??? ??? state? ??? ?? ???(▼)? ??????. ?? ??? ?? Filter? ?? ? ??, Remove Rows? ???? If ??? value is empty? ???? ?? ?? ?? ?? ?? ?? ??? ??? ?????.?? : ref :`table-lookup <table-lookup> `???? ???? ?? state ??? ??????.CDAP?? ???? ??? ???? ?? Data Preparation?? ???? ?????. table-lookup ???? CDAP ???? ??? ???? ???? ?? ??? ?? ?? ???? ???? ? ???? ??????. ?? ?? StateNameTable? ???? ??? ? ??? ?????.???? table-lookup <column> <table> ??? ???. ? ?? ?? state?? ???? StateNameTable???. ?? ??? ?? ?? ??? ?? ?????? ?? ??? (table-lookup state StateNameTable)? ?????.? ?? state_abbreviation? ???? ?? ? ? ?? ????.?? ??? ?? ????? ?? ? ??? ? ?? ?? ?? ??? ?? ???? ??? ??? ???? ?????. ??? ?? ?? ???? ??? ???? Data Preparation? ???? ??? ???? ???? ?????. ??? ??? ?? ?? Directives? ???? ?? ? ? ????. ??? ?? ?? “x”? ???? ?? ??? ?????.? ?? ?? ? ??? ???? ???? ? ?? ??? ? ????. state? ??? ?? ???(▼)? ???? Delete Column? ??????. ?? state_abbreviation? ??? ?? ? ????. ? ??? ?????? ???? ?? ? ? ?? ???. ? ??? “State”? ??????.??? ?(States) ???????, ???, ??? ? ??? ??? ????? ???????. ??? state? ?? CA, OR ?? WA ??? ?? ???? ?? ?? ???? ???.??? ??? state ??? ??? ?? ???(▼)?? ??????. ? ???? ???? Filter? ??????. Keep Rows? ???? ?? ?? ??? ???? if ? value matches regex? ??????.CA, OR ?? WA? ???? ???(regex)? ?????. ??? ^ (CA | OR | WA) $? ??? ????.??? ???? ??? ??? ???? state ? ????.??? Street Type????Avenue??? ???? ???? ?? (? ??? ??? ?????) ??? ????? ?????? ???? ??? “Avenue”?? ??? ?? ? ??? ?? ???.? ??? ??? ??? ??? ?? ???? ????. ?(states)? ????, ?? ?? ???? ???? ? ??? ???? ??? ? ? ????. ??? ??? ??? ????.61 Summit Avenue?, ??? ?? “Avenue”? ???? ?? ???? ???? ???? ? ???.? ??? ???? ??, Contains ??? ?????. address ??? ???(▼)? ???? Filter? Keep Value? ??????. ??, If? value contains?? ????, Avenue? ??????. ?? ignore case? ??????. Apply? ???? ??? ?????. ? ? ?? ??? ?? ?? ?? ? ? ?? ????. ???? ??? ? ? ????. ??? Data Preparation?? ??? ??? ?? 100 ? ?? ?????. ?? Data Preparation? ?? ??? ??? ??? ?? ?? ??? ??? ?? ???? ????? ?? ??? ? ? ?? ?? ?????.??? ?? : ??? ??? ??? ??? ????? ???(navigation system)? ??? ? ? ?? ??? ??? ???? ???? ?? ???? ???? ???? ? ?????.????? ????? country ??? ???? ???? country ?? ??? ? ????. country ?? ?? ???(▼)? ???? Delete Column? ??????. USA ?? ??? ?? ????? ???.?? ???? ???? ??? ??? ? ??? ?????.??? ??? ????? ??? ??? ???? ??? ??? ???? ????. Data Preparation ??? ??? ??? ?? ?? ??(100 ? ???) ? ???? ?? Data Pipeline? ???? ? ?????? ??? ??? ??? ????? Spark ?? MapReduce ??? ?????. ?? ?? ??? ?? ???? ?? ??? ??? ?? ???? ?? ? ? ????.Create Pipeline ? ???? Batch Pipeline? ??????. MySQL ??????? ??? ??? ??? ??? ??? Batch? ?????.Data pipeline UI?? Database stage (customer? ???)? Wrangler ????? ???? ?? ?? ? ? ????. Wrangler state? Data Preparation?? ??? ?? ??? ?????.?? ?? “Sink”???? ???? Avro Time Partitioned Dataset sink? ??????. Wrangler ? ??? ? ??? ??????. Avro Time Partitioned Dataset ??? ?????? CampaignSink?? ??? ?????. ????? ?????? ??? CampaignPipeline?? ?????.?? pipeline? ?? ? ? ????. ??? ?? ??? ?? Deploy? ??????. ??? ???? Run? ?????.Pipeline? ???? Avro Time Partitioned Dataset sink? ????????. ?? ???? View Details ?? ??? ?????. ? view? ????? ??? ???? ??????. Excute? ???? ??? ?? ??? SQL ??? ?????. ??? SQL ?? ??? ?????. ? ?? ?? ?? ??? ???? ???? Pipeline ??? ?????.??? ???? ?? CampaignSink ??? ??? ???? RESTful ????? ?? CDAP UI? ?? ?? ??? ? ? ????.?? : New York Times XML Feed Data ????? ??????? CDAP? Data Preparation ? Data Pipelines? ???? New York Times XML ???? ??? ??? ??? ??? ???? ???? ??? ?????.?????? ??? ?? ?????? ??? ?? ??? ??? ??? ??? New York Times? ?? ?? ?? ?? ?????? ??? RSS XML ??? ??? ?????? ? ??? ????. ?? ??? ?? ??? ?? ????? ????? ???.???? ?? ???? ? ?? ?? ?? ????? ????? ???? ??? ??? ?? ?? ????.???? ?? ???? ? ?? ?? ??? ??????? ???? ??? ??? ?? ?? ????.?????? ???? ????? ???? ? ??? ???? ???? .xml ??? ????????.nytimes-world.xml??? ??????? ????? ?????? ?? ??? ???? ??? ???? ????.????? CDAP ?????? Data Preparation ??? ??????. Data Preparation?? ??? ?? ???? ??????. File System?? nytimes-world.xml? ???????.Wrangling the XMLbody ?? XML ???? ?? ?? ?? ??? ???. body ?? ?? ?? ???? Parse> XML to JSON? ??????. Depth? 1? ??? ??????.??? ????? ??? ?? ? ???? ???? ??? ???? ? ?? ?? ????: Columns ? Directives. Columns ???? ???? body_rss_channel_item ?? ????? ????????.??? ?? ?? ??? ?? ? ? ????. ? ?? ?? ?? ?? ???(caret icon)? ???? Keep Selected Columns ? ??????.Keep Selected Columns ? ????? ??? ???? ?? ?? ?? ??? ? ?? ??? ??????.?? ?? ??? ???? ???. ?? body_rss_channel_item? ???? Parse > JSON with Depth 1? ?????.? ???? ?? ? ??????. ? 16 ?? ?? ?????. ? ??? ??? ?? ??? ???? ?? ???? ???? ????.?????, ???? ??? ??? ???? ?? ?? ?? ?????. Columns ??? ??? ?????.body_rss_channel_item_linkbody_rss_channel_item_dc:creatorbody_rss_channel_item_titlebody_rss_channel_item_categorybody_rss_channel_item_pubDate? ? ? ???? ?? ?? ??? ???? Keep Selected Columns ? ?????.?? ?? ??? ??? ??? ??? ?? ????.linkcreatortitlecategorypubDate??? ???? ? ??? ???? ?? ? ? ????.???? ????? ?? ??????? ??? ??? ?? ?????? ??????? ???? ?? ???? ??? ???? ?? ???? ?? ???? ?? ?????? ??????? ????.???? ?? ???? JSON ??? ?? ??? URL? ???? ??? ? ? ????. ??? ???? ?? ??? ???? ??? ? ? ????.?? ???? ??? ?? ? ? ??? ? JSON? ??(parsing)??? ???. ???? ?? ?? ?? ?? ???? Parse> JSON? ??????. ? ???? ???? ?? ? ? ? ??????.?? category_domain? category_content?? ? ?? ? ?? ????.category_domain ?? ?? ???? Filter> Keep Rows> ?? ??? ??? ??????. ?? nyt_geo? ???? ??? ?? ????? ??????. Apply? ??????.??? ??? ????? ??? ??? ????. category_domain ?? ?? ??? ??? ???? ?? ?? ???? ? ??? ???? ? ?? ??? ? ????.???? ??? ???? ???? ???????, ??? ???? ??? ???? ?? ??????? ?? ? ?? ????.URL ?????? ?? ?? ?? ???? ?? ??? ?????. ???? ???? ?? ????. ?? ? ???? ?? ????? ???? ???. ?? ??, ?? URL? ??? ?? ? ????./2016/09/24/world/asia/chinese-medicine-paul-unschuld.htmlExtract Fields ??? ???? ? ??? ?? ? ? ????. Extract Field? URL, ?? ??, SSN ?? ?? ???? ???? ???? ?? ? ? ?? ??? ?? ??? ?????.link ? ?? ???? Extract Fields> Using Patterns ? ??????.??? ???? Extract Fields ???? Start/End pattern ? ??????. ?? ??? ?? ???? ? ??? ?partner=rss&emc=rss ? ??????.Extract? ??????. ?? ?? ?? ???? link_1_1? ??? link? ?????.?? ???String ??? pubDate ?? Date ??(object)? ????? ???. pubDate ??? Parse > Natural Date? ??????. timezone? GMT???.? ???? ???? Date ??? ??? pubDate_1??? ? ?? ?????. pubDate ?? ???? ? ? ?? ??? pubDate? ?? ? ????.??? ?? ??????? ?? ??? ??? KIRK SEMPLE? ?? ??????. Kirk Semple? ?? ?? ???? ???? ??? ???? ?? ????.??? ?? ??????. Format > To TitleCase ? ??????. ?? ?? ?? ??? ??? ???? ????? ? ? ????.????? ????? ???? ?? ???? ??? / ??? ??????? ???? ?? ?????? ?? ? ????.?? Pipleline ??? ??????. Batch Pipeline? ??????.??? ?? ?????? ??? ?????.? ?? Avro Time Partitioned Dataset sink? ???? ???? (?? ????) ? ?? Python ? ?????.???? ?? ??? ???? ??????.??? ? ?? ??? Align ??? ???? ??? ?? ? ? ????.?? Python node configuration ? ???. ?? ??? ??? ?? ????? ??????.def transform(record, emitter, context): if (record['category_content'] == 'Brazil'): emitter.emit(record)? ??? category? 'Brazil'??? ??? ???? ??? ???? ?????.Label? BrazilEvaluator? ???? ?? ?? ??????.?? BrazilEvaluator? ??? ??? ?? configuration ? ???. Label ? Dataset Name? ?? BrazilSink? ??????.???? ???? ????, ?????? ?? ??? ?? ?? ??? ?????.??? ??? Name your pipeline text ? ???? NewYorkTimesPipeline?? ????? ??? ?????.?? ?????? ?? ???? ??? ?? ????.pipeline? ???? ? ?? ??? ??? ?? ? ? ????. ??? ?????? ???? ?? ??? ???? ?? ??? ????? ????? ???.??? ??? Preview? ??????.Run? ???? ???? ?????? ??? ??? ???? ?????. ???? ? ??? ??? ???? ? ? ????. ?? ?? BrazilEvaluator? ???? ???? ???? ???? ?? ? ?? ? ? ????.?? Deploy? ???? ??? ?????? ??? ? ????.?? ????BrazilSink? ???? configuration ? ?? ??? ?? ???? ?? View Details ? ??? ? ????.??? ?? ???? ??? ??? ??? ? ?? ???? ?????. Execute the window ? ??????. ??? ???? ? ???? ???? ??? ?????? ???.?????! New York Times XML ???? ???? ?????? ???? ? ?? ??????? ??????.?? : ?? ?? ????? ????? ??????? CDAP? Data Preparation ? Data Pipelines? ???? ?? ???? ??(ingest)?? ???? ?? ?? ??? ???? ?? ?? ?????? ???? ??? ?????.??????? ?? ???? ??? ?? ?? ???? ?????. ??? ?? ??? ????? ?? ??? ?? ?? ?? ?? ???? ???? ? ???? ??? ?? ??? ??? ???? ???.Greenblatt? Magic Formula? ??? ???? ?? ?? ?????? ????? ???.??? ??? ?? ?? ??? (NYSE)? ???? ????? ?? ??? ??????? ?? ??? ????? ???.????? ???? ? ??? ???? ??? .csv ??? ??????? ??? ??????.stock_data.csv??? ??????? ????? ?????? ?? ??? ???? ??? ????????.????? CDAP ?????? Data Preparation ??? ??????. Data Preparation?? ??? ?? ???? ??????.File System ?? stock_data.csv? ???????.???? ?? ?????? ???? ????. ?????? NYSE? ?? ??? ?????? ? ??? ???? ?? ???? ?? ?? ????.? ??????? Greenblatt? Magic Formula? ??? ???? ??? ???? ????. ??? ?? ??? ?? ??? ????.??? 5 ?? ??? ?? ?? ??(market capitalization)? ??????.????(utility) ? ?? ??(financial stocks)? ?????.??? ?? ???(company's return on capital) = EBIT / (? ?? ??(net fixed assets) + ?? ??(working capital))? ?????.??? ?? ??(market capitalization)? ???? ?? ???? ?? ?? ?? ???? ???? ??? ?????.?? ??? 20 ? ??? ??????.?? ?? ?? (Minimum Market Capitalization) ????, ?? ?? ??(minimum market capitalization)? ????? ???.stock_data.csv ?? ???? ??????. body ?? ?? ?? ?? ??? ???? 'Set First Row as Header' ??? ???? Parse > CSV ? ??????. body?? ?? ?? ???? Delete Column ? ???? body ?? ??????.??? ?? ??? ????? ??(share price)? ?? ???(the number of outstanding shares)? ?? ?? ?????. ??? ??? ?? ??? ? ???:estimated_shares_outstanding * (high + low) / 2estimated_shares_outstanding * (high + low) / 2high? low? ??? ?? ?? ? ?? ????.??? ?? JEXL ???? ???? market_capitalization? ??? ? ????.?? ?? ?? ??? ??? ??? ??? ???? ???? ???. ?? high, low ? estimated_shares_outstanding? ?? ??????.?? ? ??? ?? ???? ??????.??? ??? ??? double? ????. low ? estimated_shares_outstanding? ?? ?? ?????. ? ? ?? ?? double??? ???.?? estimated_shares_outstanding ?? ?? ?? ???? Custom Transform ??? ??????.?? ??? (??? ?? ????? ??? ???? ???) estimated_shares_outstanding ? ???????, ? ??? ? ? ???? ? ???? ???? market_capitalization? ??? ????.????? ?? ??(market capitalization)? $ 50m ?? ??? ?? ??? ?? ?? ????. ???? ???? ???? Data Preparation? ??? ?? ? ?? ??? $ 50m ??? ??? ????. ??? Data Preparation??? ??? ??? 100 ? ? ?? ?????? ???? ?? ??? $ 50m ??? ?? ??? ?? ??? ?? ? ????.??? ?? ??? ?? ??? market_capitalization ?? ???? Filter ??? ??????. ??? ?? Custom condition? “>50000000”?? ??????.?? ? ???? ?? ???? ??? ?? ? ???? ??(financial and utility stocks)? ???? ????. ??? ?? ?? ??(minimum market capitalization) ??? ??? ?????.gics_sector ?? ?? ??? ???? Filter? ??????. ?? “Financials”? ???? ??? “Remove Rows”? ?????? (?? ?? ??). “Utilities”? ?? ? ??? ??????.?? ??? (Return on Capital, ROC) ????? ?? ??? ?? ??? (Return on Capital, ROC) ? ???? ????. ROC? ??? ?? ?????:EBIT / (net fixed assets + working capital)?? ? ??? ?? ???? ??? ????:` earnings_before_interest_and_tax / (fixed_assets + (total_equity - total_liabilities))`? ??? ROC? ????? ?? earnings_before_interest_and_tax, fixed_assets, current_assets ? current_liabilities ?? ??? String?? Double? ???????. ?? set-type ???? ???? ?? ? ? ????. ?? ?? set-type fixed_assets double? ??????? (?? ??? ??? ????? ??).? ?? ?? ? ? total_equity ??? ?? ?? ??? ???? “Custom Transformation”? ??????. earnings_before_interest_and_tax / (fixed_assets + (total_equity - total_liabilities)) ??? ??????.??? total_equity ?? ??????? ? ?? ??? roc? ????.????? ROC? ???? ?? ???? ????? ???.roc ?? ?? ?? ?? ??? ??????. Calculate > Multiply ? ???? 100? ????. ?? roc ?? ?? ???(return on capital)? ???? ?????.?? 20 ? ??? ???? ?? ????? ???Data Preparation? 100 ?? ???? ???? ???? ??? ?? ??? ??? ?? ??? ?? ? ??? ?????. “Create Pipeline”? ???? Batch? ??????.? ????? ?? ?? ???? ???? ?? ???? ????? ROC? ???? ?? 20 ?? ??? ???? ??? ??? ?? ?????? ????.?????? ?? ?? ? ??? ?? ?????.Wrangler (Data Preparation) ??? ??? ?? 20 ?? ??? ??? ??? ???? ??? ?????.Top-N ????? ?? Cask Market?? ?????. CDAP?? ?? ??? ????? ??? ?????. ??? Cask Market ? ? ??? ??????? ?? ?? ? ???? ?? ??? ???? ????.??? ?? ?????? Cask Market? ?????. “Plugins”???? “Top-N”? ??????.Top-N ??????? ?????. “StockPipeline”??? ???? ?????? ???? ???? ?? ????. Top-N ????? ?? ??? ???? ??? ?? Analytics ??? ?????.???? Top-N ??? ???? Avro Time Partitioned Dataset sink? ?????.Avro Time Partitioned Dataset sink? ??? StockSink? ???? “Database Name”? StockSink? ??????.Top-N ?????? ??? “roc”(??? ????? ????)? ???? size? 20?? ?????? (?? 20 ? ??? ????).?? ???? ???? ??? ??????.?? “Deploy”(??? ?? ???? ??)? ??????. ?????? ???? “Run”? ????.StockSink? ???? “View Details”? ??????. ??? ??? ???? ??? ?? 20 ?? ??? ? ? ????. ?? 5 ?? BBBY, BIIB, AME, AMAT ? BMY ???? ?????.NYSE?? ??? ??? ? ?? ????? ?? RESTful ??? ???? ??????? ??? ??? ? ??????.?? : IoT ?? ??? ?? ? ?????? ??????? CDAP? Data Preparation ? Data Pipelines? ???? JSON ???? ??? IoT ?? ???? ??(clean), ??(prepare), ???(mask) ? ??(store)?? ??? ?????.????FitBit ?? ???? JSON ???? ????. ?? ???? ???? ???? ? ??? ??? ?? ?? ?? ?? ??? ???? ?? ???? ??? (data masking)?? ???. FitBit JSON ???? ??(parsing)??, UNIX timestamp? ????, ??? ID? ??? ????, ??? ??(hash)??, CDAP ???? ?????.???(contractor)? ??? ??????? ??? ??? ?????.???????? ???? ? ??? ???? ???? .json ??? ??????? ??? ??????.FitBit_Device.json??? ??????? ????? ?????? ??? ??? ??? ???? ????????. Data Preparation? ?? FitBit_Device.json? “File”? ???????.???? body ?? ????body ?? ?? ?? ???? Parse > JSON (Depth 1)? ?????. ??? ?? array? ? JSON ??? ??? ?? ??????.JSON ???? ?? ????? Parse > JSON ???? ?? ?? ? ? ? ??????.?? body_device_id, body_calories_burnt, body_duration, body_timestamp?? ? ?? ?? ????.?? ?? ??? ???? Delete Column? ???? body_duration ?? ??????. ?? ?? ?? 60??? ? ?? ???? ????.?? ID ???? ????? FitBit ?? ID? ?? ????. ??? ????? ???? ???? ????? ???? ??? ????? ??? ? ? ????. ???? ???? ??? ?? ??? ???? ???? ? ???? ???(mask) ???.??? ??? body_device_id ?? ?? ?? ???? Mask Data ???? ???? ???.??? Show last 4 characters only ? ??????. ????? ??? 4?? ??? ?? ??? ??? ??????. ???(Shuffling)? ?? ??? ??? ? ? ??? ?? ?? / ??? ???? ????.UNIX Timestamp?? ?? ????body_timestamp? 1970 ? 1 ? 1 ? ??? ?? ?? (?)? ???? UNIX timestamp? ??? ?????.???? ??? ???? timestamp? ???(modulo)? ??? ??? 86400?? ???? ???. ??? ??? ??? ???, ?? ??? ??? ?????.body_timestamp % 86400?? body_timestamp ??? String?? ?? ? ????. String?? ?? ??? ?? ? ? ????! ? ??? ????? body_timestamp? float? ?? ??? ??? ???? ???????.??????, ?? ??? ????? ?? ???? ??????.set-type ???? ?? ??? ?? ?? ??? ?????. ??? String? ????? 10 ??? ???? ? ???? Float?? ??????.?? ??? ??? ???? timestamp? ????? ??? ????? ???(modulo)? ??? ? ????. body_timestamp? ?? ?? ??? ???? ??? ??? ?? ? ? ????. Calculate > Modulo ? ??? ?? 86400? ??????.?? ??? ?? ???timestamp? ??? ? ????. ? ?? ?? ?? ??? ??? ?????. ?? ??, ? ?? ??? 3070.0 ?? ???? ????. ?? FitBit ???? ?? ?? 3070.0 ? ?? ???? ?? ?????.??? ??? ??? ?? ?? ?????? ???? ?? ???? ??? ?? ?? ?? ?? ? ????. 13 ?? body_calories_burnt? -7? ?????. ?? ??? ???? (? : ??? ?? ?) ??? ??? ??? ?? ? ???, FitBit?? ?? ?? ? ? ??? ? ???.? ??? ???? ???? ?? body_calories_burnt? ??? ??? String?? Float? ???????. ? ??? set-type body_calories_burnt Float ???? ???? ?? ? ? ????.?? ??? ???? ?? ? ? ????.?? send-to-error ???? ?? ?????. send-to-error? ??????? ?? ? ? ???? ??? ???? ??? ???? ?? ??? ?? ?? ??? ???? ????????.? ???? ???? ??? ?????.? ?? ??? ?? ????Data Preparation??? FitBit.json ??? ?? “record”? ??????. ? ?? ???? ?? ?? ?? ???? ????? ??? ?? ?? ??????. ??? send-to-error? ?? ? ? ?? ???? ??? ?????. ??? ???? ???? ????.send-to-error? ?? ??? ???? ?? ?, ??? ??? ???? ?? ?? ? ?? ?????.? ?? ??? ?? ??? ??? ??? ???? ??? ???? ??? ?? ?? ????.send-to-error ???? ????? ??? ??? ??? send-to-error ?? “x”(??? 12 ?)? ??????.??? ???? ?? ??? ????.???? ??? ??? ?? Filter? ?? ? ? ????. Data Pipelines? ?? ???? ????? ??? ?? ? ? ??? ???? ?????.body_calories_burnt ??? Filter? ??? ?? Custom condition ?? Remove Rows? ??????. ??? 0?? ?? ??????.??? ??? ?? ?? ?????? ? ? ????.????? ??? ??????, ?? ???? Base64? ??? ? ? ?? ??? ?? ????. ??? Base64?? ?? ?? ?????????.?? body_calories_burnt ? body_timestamp? ??? set-type body_calories_burnt String? set-type body_timestamp String? ???? ???? ?????.?? ?? ??? ?? ?? ?? ???? Encode? ??? ?? Base64? ??????.?? ?? ?? ? ??? ??????. ?? ???? Base64 ???? ??????. ?? ?? ??????.???? ?? ?????????, CDAP Table Dataset? ???? ????, ??? ? ???? ??? ????? ?? ? ????.CDAP Table Dataset?? ? ?? ?? ???? ?????. ?? ? ?? ID? ????? ??? ??? 4 ??? Base64 ???? ?? ? ? ????. ?? ??? ?? ??? ??? generate-uuid ???? ?????.UUID? ?? ??????. generate-uuid? ? ?? ?? UUID? ?????. ?? ? ??? ?????? generate-uuid uuid? ???? uuid?? ? ?? ??????.?? ???? CDAP Table Dataset ? ??? ??? ?????.??? ?? ????? Ingest Data ? ??????.Table? ???? ? ?? ??? “FitBitTable”? ??????. ? ?? ????? ?? ?? ???? ? ?? uuid? ???????.Ingest Data ? ??????. ??? ???? Explore Data ? ??????.??? ???? ??? ??????. ?? ??? ???? ???? ?? ? ?? ? ? ????!?? : ?? ?? ?? ?? ?? ????? ??????? CDAP? ???? CDAP? ??? ? ?? ?????(point-and-click interface)? ???? ?? ??? ???? ??? ?? ??(Machine Learning)? ???? ???? ??? ??? ????.??????? ?? ?? ?? ??? ???? ????. ??? ??? ???? ?? ????? ?? ?????.???? ?? ???? ???? ???? ?? ??(Logistic Regression model) ? ?????? ???.????? ??? ???? ??? ???? ??? ???? ?? ?? ????.???????? ???? ? ??? ???? ??? .csv ??? ??????? ?? ??? ??????.fraud_train.csv??? ??????? ????? ?????? ??? ??? ??? ???? ????????. ??? ??? ?? fraud_train.csv? “File”? ???????. ??? ??? ????? ? ??? ??????.??? ?? ? ???? ???? ??(parsing)?????.body ? ?? ?? ???? Parse > CSV? ???? ? ?? ?? ???? ?????.?? ?? body ? ?? ???? Delete Column? ?????. id ?? ?? ???? ???? ???? ??? ?? ????.Amount? ???? ?? ??? ? 8 ?? ?? ??? ????:normalized_total_spent_last_24_hours: ?? 24 ?? ?? ??? ?????.normalized_merchant_fraud_risk : ???? ?? ?? ??.normalized_time_since_last_transaction : ??? ?? ??? ??.normalized_average_transaction : ?? ??(transaction) ??.normalized_time_till_expiration : ??? ?? ? ???? ??.normalized_daily_average_transaction : ?? ?? ?? ????.normalized_change_in_merchant_sales : ?? ??? ?? ??.Amount : ??? ??? ?????.???? ? ?? ?? ?? ??? ??? ???.?? ?? ?? ?????? ?? ??? ???? ??? ??(training) ? ???(testing)? ? ??? ????. ?? ?????, ???? ???? Logistic Regression ????? ??????. ? ????? “fraud_train.csv”??? ??? ? ???.???? ?? ??? ??? ???? ???? ??? ?? ???? ??? ?????.??: ?? ?? ?? ???? ???? ?? ?? ??? ???? ?? ? ??? ???? ??? ????.?? normalized_total_spent_last_24_hours? normalized_average_transaction? ??? 24_hour_to_average_ratio?? ?? ??? ????? ???.?? normalized_total_spent_last_24_hours ??? Double? ???????. ??? String???. ??? ???, ?? ??? ????? ?? ???? ??????.normalized_average_transaction? ?? ? ??? ???? ?? ??? ??? Double????????.?? ?? ?? ??(interaction variable)? ?? ? ????. ? ? ??? ?? ?? ?? ??? ???? Custom Transform? ???? ? ??? ?? ? ? ????. ??? ??? ?? ??? ?? ?? ?? ????. ??? ?? normalized_total_spent_last_24_hours? ???? ??? twentyfour_hour_to_average_ratio? ??? ? ??? ??? ? ?? ???? ???.normalized_total_spent_last_24_hours? ????? ?? ?? ??? Copy Column ? ??????. ? ?? ??? twentyfour_hour_to_average_ratio? ??? Apply? ????.?? ?? ?? ??? ??? ? ????. twentyfour_hour_to_average_ratio? ?? ?? ?? ??? ???? Custom Transform ? ??????. ?? ??? ?? ??? ??????.??? twentyfour_hour_to_average_ratio ?? ???? ??? ? ? ????.????? ?? ?? ??? Double?? ???????. ?? ?? set-type ???? ???? ?? ??????. ?? ??? ???? ?? ?? ?? ????.set-type normalized_merchant_fraud_risk Doubleset-type normalized_time_since_last_transaction Doubleset-type normalized_days_till_expiration Doubleset-type normalized_change_in_merchant_sales Doubleset-type normalized_transaction_time Doubleset-type Amount Doubleset-type Class Double??(Training) : ??????? ?? ????? ???? “Create Pipeline”? ??????. Batch? ??????.?? ??? ??? ?????.???? ?? ??? ???? ???? ?? ???? ?? ????? ???? ????? ???. Cask Market?? ???? ? ????? ?????? ???.??? ?? ??? “Cask Market”? ??????. application ???? “Logistic Regression Analytics”? ?? ????? ??????.?? ?????? “LogisticRegressionTrainer”? ??????.???? ?? ??? ?? ??? “Sink”? ??? “Logistic Regression Trainer”? ????. ? ????? ???? ???? “Wrangler” ????? ????, ?? ??? ?? ??? ? ????.???? ?? ?? ?????? ??? ??? ?? ??? ??? ?????? ?? ? ? ????. ? ?? ?????? ???? ???? ?? ? ?? ??? ??????? ?????.File ????? ?? Label ? reference name? ???? ??? “CreditCardSource”? ????. ????? “Wrangler”? ??? “CreditTransform”??, “LogisticRegressionTrainer”? “FraudTrainerSink”? ??????.Trainer configuration?? Label Field? Class? ??????. ??? ???? ?? ??? ??? ? ?? ?????. ?? FileSet Name? “FraudModel”?? ?????. ? ?? ??? ??? ???? ?? ?????.??? ?????? ?? ??? ?? ??????? ?????.????? ??? ?????? ??? ?? ????? Preview? ??? ?? Run? ??????. ????? ???? LogisticRegressionTrainer? ?? Preview ??? ???? ?? ???? ? ? ????.?? “Deploy”? ???? ?????? ??? ? ????.???(Testing) : Unknown??? ?? “fraud_train.csv”??? ???? ?? Data Preparation?? ?????. “Create Pipeline”? ?? ??????.File ??? ??????. ? ???? “CDAP Stream”??? “Wrangler”????? ??????. Wrangler ?????? ??? ???? ????? ?? ??? ???? ? ?? ??? ??????. “CDAP Stream”? “FraudStream”?? ???? duration? 10m? ??????. “LogisticRegressionClassifier”? ???? ???? “Avro Time Partitioned Dataset”??? ??????. “Wrangler”? “CreditTransform”, “LogisiticRegressionClassifier”? “FraudClassifier”?, “Avro Time Partitioned Dataset”? “FraudSink”? ??????. “FraudClassifier”?????? FileSet Name? “LogisticRegressionModel”?, Prediction Field? “Class”? ?????. “FraudSink”?????? Dataset Name? “FraudSink”? ??????.?? ????? ??? “FraudClassifier”? ???? ?? ? ? ????.???? ??? ?? “View Details”? ??????. ??? ???? ???? ??? ????? ??? ???? ??? ??? ?? ????. ?? ?? ??? ????? ???? ??? ???.9253,-6.18585747766,7.10298492227,-13.0304552639,8.01082339822,-7.88523736132,-3.9745499936,-12.2296077787,44.98175,-0.734303155038,0.435519332206,-0.530865546585,-0.47111960693,0.643214386605,0.713832369658,-1.23457207133,29.956015,-2.29698749922,4.0640433419,-5.95770634517,4.68000806392,-2.0809375996,-1.46327216002,-4.49084686932,104.0434,-1.70678465109,0.291716504769,1.7070939159,0.551977355909,0.234141485112,-0.265204391462,0.0453993133097,38.193174,1.20124498017,-0.599557518154,0.604505150814,-0.807313717744,-0.43403239775,0.823049573638,-0.858524252306,6.41? ????? ?? ??? ?? Send Events to FraudStrem??? ?? ????.?????? ???? ??? “Run”? ??????. ????? ???? “View Details”? ???? ??????? ???? “FraudSink”?? ??? ??? ? ????. Class field? ?? ? ?? ???? ???? “1”? ????, ?? ???? ?????? ?????? ?? ? ? ????.CDAP ?????Data Pipelines? ???? Spark ? MapReduce? ???? ??(transformations) ? ????? ??? ?? ?? ???? ???? CDAP? ??? ???? ???? ?? ? ?? ? ? ???? Data Pipelines? ???? ???? UI? ???? ??? ??? ?? ?? ??? (?? ? ??? ??)? ?? ? ? ??? ?????. ?????? ??(logs) ? ???(metrics)? ?? ??? ? ? ??? ????? ???? ?? ?? ??? ???? ??? ??? ?? ?? ???? ?? ?? ? ? ????.Data Pipelines? Directed Acylic Graph (DAG)? ??? ??? ?????(stages)? ?????. ??? ??? ?????? ?????. ????? ??? (pipeline graph) ?? “??(nodes)”? ????(stage)? ?? 6 ?? ??? ?? ? ? ????:Sources (??)Transforms (??)Analytics (??)Actions (??)Sinks (??)Error Handling (?? ???)Sources(??)? ???? ?? ?? ??????, ?? ?? ??? ??????. ?? ?? ??? UI? ???? ???? ?? ? ? ???? ??? ??? ???? ?? ?? ??? ??? ????.Transforms(??)? ???? ???? ?? ? ?? ???? ?? ? ? ????. ?? ?? ???? ?? ? ? ????. JSON? ??? ? ? ????. Javascript ????? ???? ??? ?? ???? ??? ?? ????.Analytics (??)????? ???? ?? ?? ?? ?? ?? ??(Machine Learning tasks)? ???? ? ?????. ?? ??, ?? ??(batch source)? ???? ??(logistic regression) ?? Naive Bayes? ??? ? ????. CDAP? ??? ?? ??(use cases)? ?? ?? ????? ?????.Action (??)????? ?? ??? ?? ????? ??? ??? ?? ??? ????? ?? ???? ???? ?? ????? ????. ?? ?? Database custom action? ???? ????? ?? ??? ?????? ??? ??? ? ????. ?? HDFS ???? ?? ??? ???? ??(action)? ??? ? ? ????.?? ???, ????? null ?, ?? ?? ?? ?? ?? ??? ???? Error Handler (?? ???)????? ???? ??? ?? ? ? ????. ? ????? ??? ???? ?? ?? ??analytics ????? ??? ?????. ?? ?? ??? ??? ????? ?? ? ??????? ?? ? ? ????.????? ???? Sink(??)? ?????. ??? Avro, Parquet ?? RMBDS? ?? ??? ???? ???? ??? UI? ?????. ??? ???? ??? ???? CDAP UI ?? RESTful API? ???? ?? ? ? ??????.??? ????? ?? ?? ??? ???? ?? ??? ?? ????? ???? ? ? ????.Data Pipelines? ???? ?????? ???? MapReduce ?? Spark? ???? Big Data ??????? ??? ?? ? ? ????.CDAP ??? ?? ??(?? ? ???)CDAP Pipelines? Hadoop?? ??? ??? ??? ??, ??, ??? ? ???? ?? ???, ??? ? ?? ??? ??? ?????. ?? ?? ???? Apache 2.0 ????? ?? ????? ?????.??? ??? CDAP? ????CDAP Studio? ?????. CDAP Studio? ???? ?? ? ??? ?????(click-and-drag)??, ?? ??? ?????? ??????? ??? ??? ??? ?????? ????.CDAP? ???, ?? ? ?? ??? ??? ??? ??? ?? ? ????? ???? ?? ??? ?? ??? ?? ?? ??(operational view)? ?????. ??? ??? CDAP UI, CDAP CLI (Command Line Interface, ?? ? ?????)?? ?? ? ??? ?? ??? ???? CDAP?? ?? ??? ? ????.??? ????? ??? ?? ????(artifacts)?? ??? ??? ??? ??? ?? ?????????.????? “?????? ???”???. ??? ?? ??????? ??? ??????? ???? ?? ??(configuration file)? ?? ??(configuration) ?? ??? ????? ?? CDAP? ?? ??????. ??? ??? ??? ???? ????? CDAP? ?? ?????.????? ??????? ??? ??? ????(stage)? ???? ?? ??? ? ? ????. ? ????? ?? ??? ???? CDAP? ??? ????? ??????.????? ?? ??? ?? ????? ???? ? ?? (????? ??, ???? ??, ???? ??)? ???? ??? ?? ????? ?????.?? ????? ??? ??? ?????? CDAP Studio ? CDAP?? ???? directed acyclic graph (?? DAG)? ?? ?????.??? ??? ???? ??? ??? ????.?? ?? ?? (pre-run operations): ??? ??? ??? ???? ?? ??? ?? ?? (? : ??? ??)??? ?? (data acquisition): ?? ?? ????? ??? ?? ????? ?? (data transformation): ???? ???? ??? ????? ?? (data publishing) : ?? ???? ??? ?? ?? ???? ?? ?? ???? ? ?? (post-run operations): ??? ?? ??? ?????? ????? ??? ???? ?? ?? ?? ?? ??? ??? ?? ??? ?? ??? ???? ??? ?? ??? ?????? ??? ???? ?? ??? ????? ??? ? ????.??? ? ?? ?????? ????? ??? ???? ??(control) ????? ? ?? ???? ?????.??? ???? ??? ??? ? ?????? ?? ?????? ??? ??? ??? ?????. ???? ????? ???? ?? ????? ??? ??? ????? ? ?? (??? ?? ??)? ?? ????? ?????.?? ???? ??? ??? ????? ?? ????? ??? ???? ? ????? ????? ?? ???????. ?? ?? ???? ??? ??? ?? ???? (?? ??? ??? ????? ???? ?) ? ?? ???? (?? ?? ??? ??? ????? ?? ? ?)?? ??? ? ????. ?? ? ????? ? ??? ?? ?? ? ?? ?? ?? ???? ??? ? ????.??? ? ??? ??? ??CDAP?? ??(logical) ??? ??? ???(physical) ??? ??? ??? ????.?? ??? ??? CDAP Studio?? ? ??? ?(view)???. ? ??????? ??? ??? ??? ???? ???? ? ???? ?? ??? ???? ????? ?????.??? ??? ??? ??? ??? ??? CDAP ???????? ??? ???, CDAP? ??? ??? ???? ?? ?? ?? ???? ? ??? ?????.???(planner)? ??? ??? ??? ??? ??? ???? ?????. ???? ??? ??? ??? ?? ???? ?? CDAP ???????? ?????.??? ?? ????? ??? ??? ???? ??(batch) ?? ???(real-time) ? ? ??? ??? ?? ???? (MapReduce ?? Spark)? ??? ? ????.?? ??????? cron ???? ???? ????? ????? ?? ? ? ??? MapReduce ??? ???? ?? ???? ???? ?? ? ????. ?? ??????? ?? ??? ?? ??? ?? ??? ???? ?? ?? ?? ??? ??? ?????.??? ??????? ??? ????? ??(polling)?? ???? ???? ??? ??? ??? ??? ?? ?? ??? ??? ??? ? ? ??? ???????.??? ??? ????, (CDAP? ??? ????) ??? ???? ?? ???? ?? ???? ??? ????? ??????.??? ?? ?????? CDAP ??????? ????? ??? ??? ????? ??? Lifecycle HTTP RESTful API? ???? CDAP PIPELINES UI, CDAP UI, CDAP CLI ? ?? ? ??? ?? CDAP?? ???? ??? ???? ???? ?? ? ? ????.??????? ??, ?? (transformations ?? ?? transforms??? ?) ? ??? ??? ????? ??????? ???. ????? ?? ????? ??? ???? ??? ?????. ??????? CDAP? ??? ?? ?????? ?? ? ????. ?? ???? ??? ?? ????? ???? ??? ??? ?? ? ? ????.? ????? ?? ? ??? ?? ??? ??? ???? ??? ??????.????? ??? ? ????? ?? ????? ??(configuration)? ???? ????? ?? ??? ???????. ??? ????? ????? ???? ??? ??? ???? ???? ?? ????? ?? ?? ??? ?????.? ????? ?? ?? ? ???? ??? ?? ??? ??? ???? ??? ??????.???(Schema)???? ???? ??? ??? ? ???? (????? ?? ?? ?? ? ??? ??? ??? ?? ????)? ?? ????? ??? ???? ???? ?????. ???? ?????? ???? ???? ?? CDAP Application Studio ?? ???? ?? ???? ?? ????? ?? ? ? ????.???? ???? ??? ??? ?? ?????? ???? ??? ??? ?? ? ? ????. ?? ?????? ?? ???? ???? ?? ????? ???? ???? ?? ?? ? ???? ?? ? ? ????.??????? CDAP ??? ??? ??? ??? ? ???? ???. ???? CDAP? ???, ??? ?? ? ??????? ??? CDAP UI? ?? ?? (? : ???? ??? ?? ? ??? ?? ??)? ????? ?????.CDAP ??CDAP ??? ?? ??CDAP Studio ??CDAP ???? ??????? ??? ?? ???CDAP ??CDAP Sandbox? ?? ???? ????, ?????? ??????.CDAP? ???? ? ????? ?? CDAP UI? ??????.CDAP ??? ?? ??CDAP ??? ??? ???? ?? ??? ??? ???? ??? ??? ????? CDAP Pipelines URL? ??????.??? ???? ??? ?? ???? ?????.CDAP Pipelines : ??? ?? ??CDAP Studio ??CDAP Studio? ???? ??? ?? ???? ????? ?? ? ??? ??????.+ ??? ???? ? ??? ??? ??????.?? ?? ??? Studio (Pipelines ??? ??? ??)? ??????. ??,CDAP Studio URL? ??????.Studio? ???? ????? ? ??? ??? ??? ????. ??? ?? Data Pipeline – Batch(batch?? ?????)?? ???? ??? ????? ?????.CDAP Studio : ??? ??? ????? ? ???CDAP ???? ??????? ???? ?? Studio? ??? ?? ??? ?? ?????.CDAP Studio: ?? ??? ???? ???? ???? ?? ??? ??? ???? ?????? ??? ????(application template artifact)? ???? ??? ????. ??? ???? Data Pipeline - Batch? ?????.?? ??? ??? ?? ?????? ????? ???? ?? ?? ?? ??? ????? ???? ???? ????. ???? ?? ????? ????? ? ???? ?? (Source, Transform, Analytics, Sink, Action)? ??? ?? ??? ???? ???? ?????.??? ??? ?? ?? ???? ???? ????. ?? ??? ???? ?? ??? ????? ???? ????? ???? ???? ??? ?? ???? ????? ???? ??? ??? ????.?? ??? ? ??? ??? ????, ? ?? ???? ???? ? ??? ?? ? ?? ? ?? ???? ????.????? ?? ??? ?? ?? (??? ??? ?? ?? ?? ?? ??? ?? ???)? ??? ?? ?? ??? ?? ??? ??? ????:?? : ?? ????? ?? ?? ??? ??? ?? ??? ?? ??? ?????? : ?? ? ??? ??? ?? ??? ?? ??? ??? ? ??? (? : ?? ?? ?? ????)??? : ?? ????? ?? ?? ??? ?? ??? ?? ??? ????? : 8 ?? ??? ????? ?? ? ??? ??? ?? ?? (? : ?? ????)??? ?? ?? ??? ?? ?? ????? ?? ??? ?? ?? ??? ?????.??? ???? ??? ??? ??? ???? ???? ?? ?? ???? ????. ?? ???? ??? ???? ?????. ???? ??? ???? ?? ???? ?????.??, ??? ??? ??? ???!??? ??? ?? ???? ????? ???? ?? ?? ???? ?? ?? ??? ??(parsing)? ? ?? ???? ???? ???? ??? ??? ????.?? ???? Stream ??? ???? ???? Stream ???? ??????.Transform ??? ??? ??? ?? ???? ?? ? ?? LogParser ??? ???? ?? ???? ???? ?????.Sink ??? ??? ??? ?? ???? ???? Sink? ?? ? ?? Table ??? ???? ?? ???? ???? ?????.???? ??? ??? ???:CDAP Studio : ????Stream? ???? ?? ??? ??? ???? LogParser? ?? ??? ??? ??? ??? ?? LogParser? ???? ??????. ?? ???? ??? ?? ????.CDAP Studio : ??????? LogParser? Table? ???? ??? ?????. ?? ???? ??? ?? ???? ? ?????? ?? ? ??? ?? ? ??? ?????.CDAP Studio : ?? ?? (????? ??? ???? ?? ??, ??? ???? ?? ??? ???? ?? ?? ???? ?? ???. ?? ?? ?? ???? ??? ??? ???? ?????.)????? ??? ????? ? ???? ??? ??????. ???? ???? ? ?????? ??? ? ?? ??? ???? ?? ??? ?????. ?? ?? ?? ??? ??? ?? ??? ??? ?????. ? ?? Stream???? ???? Label, Stream Name ? Duration? ?? ???? ?????.CDAP Studio : Stream ???? ?? ??demoStream? ?? ??? ??? 1d? ?? ?? ??? ??????. ?? ??? ???? ???? ?? ?? (??? ?? ???? X), ???? ????? ? ?? ?? ?? ??? ???? ?? ??? ?? ? ????.??? ???? LogParser? ???? ?? ??? ???? ???? CLF? ?? ?? ??? ?????.?? ???? Name? demoTable ? Row Field? ts (timestamp)? ??????.????? ???? ??? ? ???? ???? ?????? ? ? ????. LogParser? ??? ????? ??? ?? ??? ??? ???? ???? ?????.??? ??? ???? ??? ?? ?? ?? ?? ??? ?????? ?? ???? ?????.???? ???? ??? ??? ??? ??????. ? ?? ??? ?????? ??? ?? ?? ??? ?? ??? ???? demoPipeline? ?? ??? ??????. ??? ?? ???? ??? ??? ? ????.?? ??? ?????? ????? ??? ?? ?? ??? ?? validation ??? ??????. ??? ???? ??? ???? ??? ? ????.CDAP Studio : ??? ? ?? ??? ?? ?? ????Validate ??? ???? ??? ??? ?? ???? ?????.Validation success! Pipeline demoPipeline is valid.??? ??? ???? ?? ??? ??????.??? ??? Publish ??? ??????. ??? ?? ??? ?????. ??? ?? CDAP ??????? ???? ??? ? ??? ?? ???? ?????. ??????? ??? ??? ???.??? ?? ?? ???? ?? ?? ???? ??? ??? ? ????. ???? ?? ??? ??? ??????.??? ???? ??? ??????? ?????.CDAP ??? ?? : ?? ??? ??? ??? ?? ??? ?? ? ? ???, ???? ???? ??? ??? ?? ??? ???? ??? ??? ? ???? ?? ?? ?????? ?????.??? ?? ???? ?? ??????? ???? ?????? ???? ????. ???? ? ??? ??? Run (??)? View in CDAP (???)??? :CDAP Pipelines : ?? ??, ??? ?? ????? ??? ???? ?? ??????? ?? ? ? ??? ???? ???? ??? ???. CDAP?? View ??? ??? ?? Datasets ??? ???? demoStream ??? ???? ????? ?????.CDAP demoStream : ??? ??? ?? Actions ??? ?? Status ???Actions ??? ???? Send Events? ??? ?? ?? ???? Upload? ???? ???? ????? ?? ? ????. ?????? CDAP ? ????? ???? CDAP Sandbox? ??? examples / resources / accesslog.txt ??? ????. ? ???? CLF ??? 10,000 ?? ???? ?? ????.??? CDAP? ?? ????? ????? ???? ? ???? 10,000?? ????????.????? ???? ??(back-button)? ???? CDAP ??? ???? ?????.?? ??? ??? ??? ? ????. ?? ??? ?? Run ??? ??????. ??? ?? ??? ???? ??? ???? ?? ???? Start Now? ?? ? ? ????.??? ???? ?? Running? ?? ??? ??? ??? ??? ??? ??? ????.CDAP Pipelines: ???? ?? ??? ????? ??? ?? ??? ??? 0?? 10,000?? ?????. ??? ???? ?? ???? Completed? ?????. Start time ? duration? ?? ??? ????? ???.CDAP Pipelines : demoPipeline ?? ???? demoTable? ??? ?? ??? ??? ? ????. CDAP?? View ??? ???? ?? ? ?? ??? ???? ??? ??? ?? ????. ?? ???? ???? ???? ? ??? ??????. ?? ??? ??? ???? ? ?? ??? ?? ??? ?? ?? Jump??? ????. ? ?? ?? CDAP? ???? ?? ?????. ? ??? CDAP ?? ??? UI? ?? ?????.CDAP Pipelines : ??? ?? ?? ?? ??? Jump ??demoTable ??? ??? ???? ?? ?? ?? ???? ???? ?? Explore ??? ?? ??? ?? ?????:Explore? ?? ? ??Execute SQL ?? :CDAP Pipelines: demoTable ?? ?????? ?? ???? ???? ????? ???? ?? ??(parser)? ??? ?? ??? ???? ???? ?? ? ?? ? ? ????. ? ???? ??? ??? ??, ?? ?? ?? ??? ??? ? ????.CDAP ??? ?? ????? ???????.CDAP ????CDAP? Hadoop ??? ???(data lakes) ? ??? ??? ?? ??? ?? ?? ??? ??? ETL (extract-transform-load)? ??? ?? ??? ??(batch) ? ??? ??? ?? ??? ?? ?? ???(end-users)? ?????. CDAP Studio? CDAP ???? ??? ???? ???? ??? ??? ??? ????? ??? ??? ???? ????? ???? ?? ??? ?????.??, ??, ??, ?? ? ?? ????? ???? ????? ??? ??? ?????:CDAP Studio : ??? ?? ??? ???? ??? ??????? CDAP? ???, ?? ? ?? ??? ??? ???? ? ? ?? ??? ?? ??? ?? ?? ??(operational view)? ?????.CDAP Pipelines : ?? ??? ???? ?? ? ??? ?? ??CDAP ???? ???? ??? ?? ?????? ???? ???? ???? ??? ???? ?? ??? ??? ?? ?????? ???? ??? ? ????. ?? ?????? ???? ???? ??? ??? ??? ? ?? ????? ???? ?????.??? ???? ?? ??? ???? ?? ??? ?????. ?? ??? ???? ????? ?? ?? (??? ??? X), ???? ????? ? ?? ?? ?? ??? ???? ?? ??? ?? ? ????.? ???? ?? ???? ??? ????? ??? ??? ?? ? (?? ?? ??)? ???? ?? ?? ??? ????? ??? ?? ? ? ????.?? ??? ??? ??? ??? ??? ?? ??? ??? ?? ??? ??? ??? ??? ??? ? ????. ??, ??? ??? ?? ???? ??? Demo?? Demo-1? ??????.??? ?? ??CDAP Studio ??? ???? ??(draft)?? ???? ??? ??? ??? ? ????. ??? ?? ??? ???? ??? ??? ?? ??? ? ????.??? ???? ??? ??? ??? ??? ???? Save ??? ??????.CDAP Studio : ??? ? ?? ??? ?? ?????? ???? ??? ?? ??? draft? ?????. ??? ?? ???? ?? ??? ???? CDAP Studio?? ?? ??? ?? ?? ?? ?? ? ? ????.?? ??? ???? ? ??? ? ???? ???? ?? ?? ??? ??? ?????. ??? ??? ???? ???. ??? ??? ??? ??? ??? ???? ???, ??? ??? ??? ??? ??? ??? ?? ? ????.??? (??? ?? ??? ??? ????)??? ????? ????? ??? ???? ?? ??? ???????.???? ???CDAP Studio ??? ??? ???? ?? ??? ??? ?? ? ? ?? ??? ??? ??? ???? ???? ??? ? ????.??? ??? ?? ??? ? ?? ? ? ????.???? ???? ????? ??? ???? ??? ? ????.???? ???? ?? ??? ?? ???? ???? ??? ??? ???? ??? ??? ????. ??? ?? ??? ???? ???? ??? ?? ? ??? ??? ????(artifacts)? ???? ?????.CDAP Studio : Stream ?? ?????? ???? ??? ??????? ??? ??????? ???? ???? ??? ??????.CDAP Studio ??? ??? (? : Stream source plugin)? ??? ???? ?? ???? ?? ????.??? ?? ??? ???? + ??? ??? ?????.???? ??? ??? ????? ??? ??? ? ????. ?? ??? ?? ?? ??? ?? ????? ?? ??? ??? ? ????.????? ???? ???? ???? ??? ?????.???? ??? ? ?? ? ? ??? ??? ?? ??? ?? ? ????.???? ???? ????? ???? ?? ????? ?? ???? ??? ???? “T” ???? ??? ?????.???? ??? ? ??? ?? ??? ???? ??? ???? ????? ??? ? ????.??? ???? ???? ?? ?????? ??? ? ????. ?????? ????? ???? ?? ?? ? ? ??? ??? ????.??? ?? ??????? ??? ????(artifacts)??? ??????. CDAP?? ?? ?? ????? ???? ???? ??? ?? ????? ?? ? ????. ????? ?? ??(configuration file)? ??? ??????? ??? ? ???? ??? ?? ??????.??? ?? ??????? ????? ???? ??? ???? ??????? ??? ? ??? ??(source), ?? (transformations ?? transforms) ? ??(sinks)? ???? ??????.??, ?? ? ??? ?????? ??? CDAP? ???? ?????? ??? ?? ?? ??, ?? ?? ? ?? ?? ? ?? ? ??? ?? ? ? ????. ????? CDAP? ??? ?????? ??? ??? ???? ? ? ????.??? JSON ??? ????? CDAP Studio? ?? ??? ???? ?? ? ? ????.CDAP? ?? ? ?? ???? (??? ?????? ?)? ?????.cdap-data-pipeline (?? ??? ??(batch pipelines) ?)cdap-data-streams (??? ??? ??(real-time pipelines) ?)??? ?? ?? ??? ??? ??? ?? ??????? ??? ? ?????.??? ??? CDAP? ??? ??? ??? (CDAP Studio)? ?????, CDAP CLI ? curl ?? ?? ? ??? ?????, ?? ???? ?? Java ????? ???? ????? ???? ??? ? ????.??? ?? ?? ?????? ??? ??? ?? ? ?? ??? ????.CDAP Studio ???? ? ?? (? : curl, CDAP CLI ?? CDAP UI)CDAP Studio? ???? ?? ??? ??? ????.? ????? ?? ????? ????? ?? ??? ??? ??? ???? ? ??? ??? ????.??? ??(configurations) ? ??(settings)?? ???? CDAP Studio?? ??? ??? ??????.??? ??? ???? ??? ??? ?? ??? ???? ??????.CDAP Studio?? ??? ??? ???? ???? ?? ??? ??? ???? ??????.CDAP Studio?? ??? ??? ??(publish)?? ??? ?? ??? ??? ??? ??? ???? ??? ???? ?????.? ???? ??? ??? ??? ? ????.?? : ?? ?????? ??, CDAP Studio? ?? ??, ??? ? ??? ??? ?????? ?? ??? ??? “??” ?? ??? ??? ?? ? ? ????. ?? ?? ??? ??? ??? ?? ? ??? ???????.?? ? ??? ???? ?? ??? ??? ????.?? ???? ????? ????? ?? ?? ??? ?? ???? CDAP ??? ?? ?? ??(configuration specification)? ?? JSON ???? ?? ??? ???? ? ??? ??? ????.??? ??? ??? ???? ????? JSON ?? ??? ??????.Lifecycle RESTful API ?? CDAP CLI? ???? ?? ??? ?? ??? ??? ??? ??? ?? ??? ???? ???? ??? ??? ??????.?? ? ??? ???? ??? ??? ??? CDAP UI?? ? ? ??? CDAP Studio? ???? ???? ?? ? ? ????.?? ??? ?????? ??? ??(Batch pipelines)? cron ???? ???? ????? ????? ?? ? ? ??? MapReduce ?? Spark ??? ???? ?? ???? ???? ?? ? ????. ?? ?? ?? ??????? ?? ??? ?? ??? ???? ?? ??? (??) ??? ?????.??? ??? ???? ????? ???? ?? ?? ????? ???? ??? ??? ?? ? ? ????. ??? ??? ????? ??? ???? ?? ???? ?? ? ??(post-run actions)? ??? ? ????.???? ??cdap-data-pipeline ?????? ???? ???? ?? ?? ??? ???? ?? ????? ??? ? ????.ActionsBatch Source PluginsTransformation PluginsAnalytics PluginsBatch Sink Plugins??? ???????? ??? ??? (?? ??? ???) “??”??? ??? ?? ?? ?? CDAP ??????? CDAP ???????? “???”??? ??? ??? ??? ?????? ??????.??? ????? ??? ?? (MapReduce ?? Spark)? ??? ??? ???? ? ???? ????? ?? ????. ?? ??? ????? ?? ??? ??? ??? ??? ?? (cdap-data-pipeline ????)?? ???? ????? ?? ?????.??? ?? ????? ??? ??? ???? ?? ? ?? ?? CDAP Studio? ??? ? ????.CDAP Studio? ???? ?? ??? ??? ??????? ??? ?????? ????? ??? ??? ?? (cdap-data-pipeline ????)? ??????.?? ??? ?? ?? ???? ???? ??? ??? ???? ????? ??????. ?? ????, ?? ???? ? ?? ?? ???? ??? ?? ?? ?????? ?? ?? ????? ??? ? ????.????? ??? ??, ??, ??? ??? ?? ???? ?? ???? ??? ?????.?? ???? ?? ?? ?? ?? ?? ? ? ????. ??? (?? ??) ?? ??? ???? ??? ??? ????? ???? ???? ?????.Setting ??? ????Schedule, “Post-run Actions * ? ??? ??? ??? ??? ??? ? ????.?? ?? ?? ??? ???? ?? ??? ??? ?? ??? ?????. ? ??? ?? ?? ??? cron ??????? ?????.?? ? ??(post-run actions)? ??? ? ??? ??? ?? ?????. ??? ??? ?????? ??(completion), ??(success) ?? ??(failure) ? ??? ????? ??? ? ?? ??? ??? ? ????. ?? ? ??? ???? ? ? ??? + ??? ???? ?? ??? ?? ? ? ????.CDAP ??????? ??? ??? ??????. ????? MapReduce? ?????.? ???? ?? ??? ?? ??? ?? ??? ??? ?? ??? ?? ??????.??? ???? ??? ??? ??????.??? ??? ???? ???? ??? ??????.??? ??? ???? ?? ??? ?? ??? ??? CDAP ??????? ?? CDAP ????? ??? ??? ???? ????.??? ??? ??? ???? ???? ??? ??? ???? ??? ?? ???? ???? ?? ??? ??? ? ? ????.? ???? ??? ??? ??? ? ????.?? ? ??? ???? ?? ??? ??? ??? ??? ?? ??? ??? ??? ??? ??? ? ??? ?? ?? : ?? ??? ?? ??? ??? ??????.????CDAP Studio? ???? ??? ???? ?? ??? ??? ?? ??? ??? ? ????. ??? ??? ?? ??? ??? ????? ?? ?? ??? ??? ??? ? ???? ? ??? ??? ??? ??? ????? ??? ? ????.??? ??? ?? ??? ? ?? ?????? ????.?? ??????? ?? ??? (??, ??, ?, ?, ?, ?)? ?? ??? ??? ???? ?????.HourFive-minute increment after the hour, 0 through 55 minutesDayHour (twenty-four hour clock), plus five-minute increment after the hour, 0 through 55 minutesWeekDay of the week, plus hour (twenty-four hour clock), plus five-minute increment after the hour, 0 through 55 minutesMonthDay of the month, plus hour (twenty-four hour clock), plus five-minute increment after the hour, 0 through 55 minutesYearDate, plus hour (twenty-four hour clock), plus five-minute increment after the hour, 0 through 55 minutes??? ??? ?? ?? (? : 6 ?? ???? ?? 31 ?? ??? ??), ?? ?? ????? ???? ?????.? ?? ???? ?? ??? ?? cron ????? ?????.?? cron ?????? ?? ? ?? ??? ?????? ?? ???? ???? ?? ?????. ?? ????? ?? ??? CDAP Master ???? ????? ???? ?? ??? ?? ????.CDAP Studio? ???? ?? ??? ??? ??, ?? ?? ?? ? ? ??? ??? ???? ??? ?? ?? ????. ???? ??? ??? ?? ? ???? ??? ??? ? ??? ??? ??? ?? ? ? ????.HTTP ??? ???? ??? ???? ??? ?? ????. ??? schedule ??? ???? ???? ??? ??? ?? ? ???? ??? ??? ? ??? ??? ??? ? ????. ‘triggering.properties.mapping’??? ?? JSON ??? ??? ??? ?????.{ "arguments": [ { "source": <runtime argument key in the triggering pipeline>, "target": <runtime argument key in the current pipeline> }, ... ], "pluginProperties": [ { "stageName": <stage name in the triggering pipeline>, "source": <plugin property key in the given stage in the triggering pipeline>, "target": <runtime argument key in the current pipeline> }, ... ]}?? ?? ?? ???? ‘nsX’? ??? ?? ‘X’? ?? ?? ???? ‘nsY’? ??? ?? ‘Y’? ??? ?????. ??? ?? ‘X’? ‘output’??? ??? ???? ??? ???? ??? ?? ‘Y’? ‘input’??? ??? ????? ??? ?????. ??? ?? ‘X’? ?? ? ??? ??? ?? ‘Y’? ????? ??? ?? ‘Y’? ??? ?? ‘X’? ?? ????? ??? ??? ???? ????? ???. ?, ??? ?? ‘X’? ‘output’= ‘xyTable’??? ???? ??? ?? ‘Y’? ‘input’= ‘xyTable’??? ????? ????. ?????? ‘triggering.properties.mapping’?? ??? ‘arguments’???? ??????."arguments": [ { "source": "output", "target": "input" }]?? ???, ??? ?? ‘Y’? ‘row’??? ??? ?? ?? ???? ? ? ??? ???? ? ? ? ??? ??? ?? ‘X’? ?? ???? ??? ???. ?? ??, ??? ?? ‘X’? ‘TableSink’???? ???? ?? ‘schema.row.field’? ??? ? ? ??? ‘id’?? ??? ?? ‘Y’? ‘row’= ‘id’ ??? ?? ???? ?? ????. ??? ??? ‘triggering.properties.mapping’?? ??? ‘pluginProperties’???? ??????."pluginProperties": [ { "stageName": "TableSink" "source": "schema.row.field", "target": "row" }]??? ?? HTTP PUT ?? ??? ??? ????.{ "name": "SampleSchedule", "description": "A schedule that launches pipeline Y when pipeline X completes", "namespace": "nsY", "application": "Y", "applicationVersion": "-SNAPSHOT", "program": { "programName": "SmartWorkflow", "programType": "WORKFLOW" }, "properties": { "triggering.properties.mapping": "{\"arguments\":[{\"source\":\"output\",\"target\":\"input\"}],\"pluginProperties\":[{\"stageName\":\"TableSink\",\"source\":\"schema.row.field\",\"target\":\"row\"}]}", ... }, "constraints": [], "trigger": { "programId": { "namespace": "nsX", "application": "X", "version": "-SNAPSHOT", "type": "WORKFLOW", "entity": "PROGRAM", "program": "SmartWorkflow" }, "programStatuses": ["COMPLETED"], "type": "PROGRAM_STATUS" }, "timeoutMillis": 86400000}?? “MapReduce”(mapreduce) ?? “Spark”(spark) ? ?? ??? ??? ???? ??? ??? ? ????.CDAP Studio? ??(Settings) ??? ???? ?? ??? ????? ??? ??? ?? ?? ???? ?? ?? ??? ???? ? ??? ? ? ????."engine": "mapreduce",??? ??? ??? ??? ?? ? ? ??? ?? ??? ?????.????? CDAP Studio? ????? ?? JSON ??? ?? ?? ??? ???? ?? ??? ??? ?? ?? ? ? ????. ?? ??? ??? ??? ??? ??? ? ???? ?????? ?? ???? ??? ? ?? ??? ?? ?????.??? ??, ??? ??? ?? ??? ?? ?? ???? ?????. ??? ?? ??? ?? ??? ?? ?????? ?? ???? ??? ??? ??? ???? ?? ?? ?? ????? ?????.??? ??? ??? ??? ????? ??? ???? ?????. ?? ??? ???? ?????? ??? ??? ?? ? ??? ?? ??????.?? ?? ????? cdap-data-pipeline ?????? ???? ??? ?? ??? ? ????. ?? ??? ?? ????? ???? ??? ????? ??? ??? ?? ??? ??? ? ????.SSH ?? : ?? ????? SSH ??? ???? ?? ????? ??? ?????.?? ? ???? ? ??? CDAP Studio? ????? ?? JSON ??? “postActions”??? ???? ?? ??? ??? ?? ?? ? ? ????. ?? ??? ??? ??? ??? ??? ? ???? ?????? ????? ??? ? ?? ?? ? ????? ?? ?????.??? ??, ??? ??? ?? ??? ??? ?? ???? ?? ??? ?? ??? ? ????. ? ?? ?? ? ??? ???? ???.completion (??? ???? ??? ???)success (????? ??? ???)failure (????? ??? ?? ?)?? ?? ? ????? cdap-data-pipeline ?????? ???? ??? ?? ??? ? ????. ?? ??? ?? ? ????? ???? ?? ??? ????? ??? ?? ??? ?? ??? ?? ? ? ????.sending an emailrunning a database querymaking an HTTP requestReal-time Pipelines??? ?????????? ??? ??? ??? ???? ???? ???? ??? ???? ??? (??) ?? ? ??? ?? ? ?? ?? ??? ??? ? ? ??? ???????. ????? ????? ?? ??? ??? ?? ? ?? ? ??? ?? ???? ??? ?? ???? ????. ??? ??? ??? ?? ??? ????? ???? ??? ???? ?? ????? ?? ???? ?????.???? ??cdap-data-streams ?????? ???? ???? ?? ??? ??? ???? ?? ????? ??? ? ????.Streaming Source PluginsTransformation PluginsAnalytics PluginsBatch Sink Plugins? ???? ???? ?? ?? ????? ?? ??? ???? ???? ????. ??? ??? ?? ????? ?? ??? ?? ? ? ?? ???? ??? ?????.??? ????????? ??? ??? (?? ?? ???? ??) “??”??? ??? ?? ?? ?? CDAP ????? ?? CDAP ???????? “???”??? ??? ??? ??? ?????? ??????.??? ??????? Spark Streaming ?????? ?????.??? ?? ?????? ??? ??? ???? CDAP Studio ?? ?? ? ??? ??? ? ????.CDAP Studio? ???? ??? ??? ??? ????Data Pipeline – Realtime (cdap-data-strams artifact)? ?? ?????? ?? ?????? ????? ???????? ??? ???? ???? ???? ??? ??? ???? ????? ??????. ?? ????? ?? ?? ???? ??? ???? ?? ?????? ?? ?? ????? ??? ? ????.?? ??, ??? ?? ? ?? ?? ??? ??? ?? ?? ??? ?????.?? ??? ???? ??? ??? ?? ??? ??? ? ????. ?? ??? ??? ???? ???? ??? ???? ??? ?????. ??? ? ??? ‘s’, ? ??? ‘m’, ?? ??? ‘h’? ?? ?? ??? ?? ?? ?? ?? ???. ?? ??, ‘10s’? 10 ?? ?????. ??? ??? 10 ??? ???? ???? ??? ???? ?????.? ???? ??? ?? ??? ?? ??? ??? ?? ??? ?? ??????.??? ???? ??? ?? ????? ??? ???? ???? ??? ??????.??? ??? ???? ?? ??? ?? ??? ??? CDAP ??????? CDAP ????? ??? ??? ???? ????.??? ??? ??? ???? ???? ??? ??? ???? ??? ?? ???? ???? ?? ??? ??? ? ? ????.? ???? ??? ??? ??? ? ????.?? ?? ? ?? ? ??? ???? ??? ??? ??? ??? ??? ??? ??? ? ??? ?? ?? : ??? ??? ?? ??? ??? ??????.?? ?? ??? ??? ?? ? ??? ??? ?? ???? ??? ? ????.?? ?? ???? ??? ????? ????? ???? ???. ??? CDAP Studio ?? ???? ??? ??? ???? ??? ?? CDAP Pipeline Plugin Reference ???? ???? ????.?? ??? ??????? ??? ?? ?? ???? ??? ? ???, ? ???? ?? ??? ???? ???? ???? ???? ???? ???? ?? ?????(placeholders)? ?????. CDAP?? ???? ????? ? ?? ??? ????.Property Lookups (????)Macro Functions (??? ??)??? ?? ?? ??? CDAP Studio UI?? ???? CDAP Pipeline Plugin Reference? ???? ????.?? ???? ??(property lookups)? $ {macro-name} ??? ???? ?????. ??? macro-name? ??? ?? ???? ??? ????.$ {user-name} ?? ?? ?? ?? ????? ?? ??? ?????.user-name?? ?? ?? ??? ?? ??, ??? ?? ?? ??. ??,user-name?? ?? ??? ???? ??, ??? ?? ?? ??. ??,user-name?? ?? ?? ?? ????? ??, ??? ?? ?? ??. ??, “macro not found” ??? ??.? ??? ?? ???? ?? ?? (?? ??? ??)? ?? ????? ?????.?? ??, ????? ?? ???? ??? ?? ?? ?? ????. ?? ???? Stream Name ???? ??? ? ????.${stream-name}??? ??? ???? ??? ?? ? - ? ?? ?????.stream-name: myDemoStreamCDAP Pipelines UI?? ??? ?? ??? ???? ???? ??? ??? ?????. ??? ??? ?? ???? CDAP ??? ??? ??? ? ????.??:?? ?? ? ??? ??? ?? ??? CDAP ?? ???? ?? ??? ????. ??? Lifecycle? Preferences HTTP RESTful API? ??? ? ????.CDAP ??? ??? preferences? ?? ??? ?? ?? ????? Preferences HTTP RESTful API? ??????.??? ???? ?? ??? ?? ??? ??? ??(macro functions)? ??? ? ????. ?? ??? ??? ?? ???? ??? ?? ?????.logicalStartTimesecure??? ?? ?? ??logicalStartTime ??? ??? ??? ?? ??? ??? ?? ??? ?????.?? ??? ???? ??? ?? ? ??? ?? ??? ?????. ?? ?? ??? ?? ?????. ? ??? ??? ?? ??, ??? ? ???? ???? ??? ??? ??? ?? ??? ???? ??(substitution)? ?????.${logicalStartTime([timeFormat[,offset [,timezone])}????ParameterDescriptiontimeFormat(Optional)Time format string, in the format of a Java?SimpleDateFormatoffset(Optional)Offset from the before the logical start timetimezone(Optional)Timezone to be used for the logical start time?? ??, ??? ?? ??? ??? ?? ??? 2016-01-01T00 : 00 : 00?? ? ???? ????? ??????.${logicalStartTime(yyyy-MM-dd'T'HH-mm-ss,1d-4h+30m)}??? yyyy-MM-dd’T’HH-mm-ss?? ???? ??? ?? ?? ?? 1d-4h + 30m???. ?? ???? 20.5 ???? ???? ??? ???? 2015-12-31T03 : 30 : 00?? ???? ?????. ?? ???? 2016 ? 1 ? 1 ? ???? 20.5 ???? ?????.?? ???? ??? ??? ?? ?? ??? ???? Secure Store?? ?? ??? ??? ?? ?????. ??(substitution)? ????? ??? ??? ?? ?? ???? ??????. ?? ??? ???? ??? ?? ? ? ?????.?? ??, MySQL ??????? ???? ????? ?? ?? ?? ??? ??? ?? ?? ? ? ????.${secure(password)}??? ???? Secure Store?? ??? ?????.?? ??? (Recursive Macros)???? ?? 10 ???? ?? ? ? ???? (?? ???? ??). ?? ?? ??? ??? ??? ???? ??? ?? ? ??? ??? ?? ? ? ????. ? ? ??? ?? ???? ???? ???? ?????.hostname: my-demo-host.port: 9991server-address: ${hostname}:${port}??? ?? ???? ??? ?? ???? ??? ? ????.server-address: ${server-address}??? ???? ?? ? ????.my-demo-host.:9991??CDAP Studio ??? validation ??? ??? ??? ???? ?? ??? ??? ???? ?????. ?? ? ?? ???? Studio? ?????. ? ??? ???? ??? ??? ??? ??? ?? ? ? ??? ?? ?? ? ? ????.??(Publishing)??? ??? ???? ?? ??? ?? (? : CDAP Application Studio? ?? ?? ?? JSON ??)? ?? ?? ?? ??, ???? ???? ? ?????? ??? ????? ???? ??? ??? ?? (CDAP ??????)? ????.??? CDAP Studio?? ?? Lifecycle RESTful API? ?? curl ???? create app ??? ?? CDAP CLI ??? ?? ?? ? ??? ???? ?? ? ? ????.? ?? ?? ? ??? ???? ??? ??? ??? CDAP UI?? ? ? ??? CDAP Studio? ???? ???? ?? ? ? ????.??? ? ??? ?? ????? ??? ??? ??? ?? ???? ??? ??? ??? ???? ? ??? ? ????.??? ?? ??? ???? ??? ??? ??? ???? ?? ??? ? ???? ???? ?? ????, ??? ? ?? ? ?? ?? ????? ?? ????? ???? ??? ??? ?? ??? CDAP Application Studio ??? ?? ??? ? ??? ?? ???? ?? ??? ? ????. ??? ???? ?? ??? ???? ??? ??? ??? ?? ???? ?? ????? ??? ? ????.CDAP Studio : ?? ??? ???? ????, ??? ?????? ??? (Data Pipeline ?? ETL Real-time)? ??????.?? ?? Template Gallery? ???? ?? ??? ???? ???? ?? ??? ?????. ????? ??? ???? ??? ??? ??? ?? ? ? ????.??? ?? ??? ??????.Data PipelineModel Trainer: Naive Bayes ???(classifier)? ???? ??? ??(Train model)Event Classifier: Naive Bayes ??? ???? ???? ?? ?? ? ???? ??Log Data Aggregator: IP ? HTTP ??? ????? ?? ??? ??(Aggregate)ETL Real-timeKafka to HBase: Kafka?? HBase ???? ????? ??Kafka to Stream: Kafka?? ????? ??? ??Kafka to OLAP Cube: Kafka?? ????? OLAP Cube ??Twitter to HBase: HBase ???? ??? ??? ??? ??Twitter to Stream: ??? ??? ???? ???? ??Amazon SQS to HBase: Amazon Simple Queue Service?? HBase ????? ??? ????ETL Batch (CDAP 3.5.0?? ? ?? ???? ??? ?? Data Pipeline??)Stream to HBase: ????? HBase ???? ????? ??????? ?? ??? ??? ?? ? ? ????. ????? ??? ??? ?? ??? ??? ??? ? ???? ????? CDAP Studio ??? ????.? ???? ??? ??? ??? ??? ???? ?? ?? ???? ????? ? ??? ???? ?? ? ? ????. ?? ??? ??? ?? ??? ? ???? ?? ? ??? ???? ??? ??? ?????. ???? ??? ??? ? ??? ???? Demo-1?? Demo-2? ??? ????? ????.??????? ?? ?? ??? ??? ? ?? ? ?? ??? ????.CDAP Studio ???CDAP pipeline configuration ??? ???CDAP Studio ??? Export ... ??? ???? ??? ?? ?? JSON ??? ??? ? ????.CDAP Studio : ??? ? ?? ?? Button ???, “Export...”??? ???? export ?? ??? ?????.CDAP Studio : ????? ???? Export ?? ???? ? ?? ??? ?? ??? ????. ?? ???? ???? ?? ? ?? ??? ???? ?? ???, JSON ??? ??? ?? ????? ??? ??? ??? ?? CDAP Studio UI ??? ?? ?? ????. ??, “Export”??? ????, ?? ??? ???? ?? ?? ??? ?? ???? ??? ???, ? ???? ?? ??? ??? ??? ?? ?? ??? ???? CDAP Application Studio?? ??? ??? ?? ??? ? ??? ?? ??? ?????. ?? ??? ?? ???? ?????. ??? ???, ? ?? ????? ?????. UI ??? JSON ?? ??? “__ui__”??? ?????.CDAP pipeline configuration ??? ??? Export ??? ????.CDAP Pipelines : ?? ???, ??? ?? ?? ??, ???? Export ?? ??CDAP Studio?? ????? ????? ??? ???? ???? ?? UI ????? ??? ???? ??? ???? ??? ??? ???? JSON? “__ui__”??? ???? ?? ??? ?????.???? ???? ?? ??? ??? ????? ??? ?? ?? ?? ? ??? ??? ?? ? ????.????CDAP Studio ??? ??? ?? ?? JSON ??? ?? ?? Import Pipeline??? ???? ? ??? ??? ?? ? ????.CDAP Studio : ??? ? ?? ??? ?? Button ???, ?? ??? ?? ???? ?????? ???? ??? ???? ?? ????? ?? ? ? ????.CDAP ??? ?? ?? ?? ??? ????? ??? ?? ??? ?????. ?? ??? ??? ?? ??? ???? ?? ? ? ????.??? ?? ??????? ??? ??? ???? ??, ?? ? ?? ? ? ????.CDAP Pipelines UICDAP UICDAP CLILifecycle HTTP RESTful API? ???? ?? ? ??CDAP PIPELINES UI ??? ??? ?? ??CDAP PIPELINES UI ??? ??? ??? ???? ?? ? ? ????.?? ??? ??? ?? ??? ?? ?? ?? ( “?? ??”) ? ? ????. ?? ?? ??? ??? ?????. (??? ????? ??? ???? ? ??? ??? ???? ???.)??? ??? ??? ?? ??? ??? ????? ???? ???.??? ??? ???? ? ???? ?? (??(preferences)? ??? ??(runtime arguments) ??)? ???? ?? ??, ??, ??, ??? ? ??? ?? ??? ??? ?? ??? ? ? ????.??? ????? ??? ?? ? ? ? ??? ? ??? ?? ??? ??? ? ???? ?? ?? ?? ??(configuration settings)? ?? ??? ??? ?? ? ????.?? ?? ?????? (??, source)?? ?? ??? (??, sink)? ?? ??? ??? ??? ? ????. ?????? ??? ??? ??? ??? ??? ???? ?? ? ? ???? ??? ???? ?? ??? ?? ??(input)?? ???? ???.??? ??? ? ?? ??? ???? ??? ??? ?? ? ????. ??? ?? ??? ?? ??? ??? ??????? ??? ??? ? ???? ??? ?? ? ? ????. ??? ??? ?? ????? ? ??? ???? ??? ?? ? ???? ?? ? ? ????.??? ?? ?? CDAP? ???? ???? ?? ? ???? ??? ?????. ???? ?? (??) ??(recursive (nested) expansion)? ???? ??? ??? ?????. ??? ???? ??? ?? ?? ? ??? ???? ?????.??? ??? ?????? ?? ??, ??? ?? ? ?? ??? ???? ??(sourcing)?? ?????. ??(precedence)? ?? ??? ?? ?? ?? ??? ?????.??? ?? ? ?? ??? CDAP UI, CDAP CLI ?? Program Lifecycle? Preferences HTTP RESTful API? ???? ??? ? ????.???? ??? ?? ??? ??? ??? ??? ???? ?? ???? ?????.??? ?? ?????? ??? ?? ???? ??? ??? ??? ??? ???? ??? ?? ?? ???? ???. ??? ??? ?????? ??? ??? ???? ?? ?????.????? ??? ???? ?? ??? ?? ? ??? ???? ??? ?? ? ????. ???? ??? ?? ??(pipeline configuration)?? ?????.??? ?? ? ????? ?? ??, ?????? ?? ? ?? HTTP ?? ??? ?????.????? ??? ???? CDAP ??? ??? ??????.??? ??? ?? ??? ?? CDAP ?????? ? ????? ??? ??? ???? ?? ? ??? ?? ???, ?? ? ???? ? HTTP RESTful API ???? ?????.???? ??(Script transform) ??? ???? ??? ???? ?? ???? ????? ??? ? ? ????.?????? ??? ???? ??? ? ??? ???? ?? ?????.??? ??? ?? ??? ???? ?? CDAP ?????? ? ????? ??? ??? ???? ?? ? ??? ?? ???, ??? ? Metrics HTTP RESTful API? ???? ????.???? ?? ????? ???? ??? ???? ?? ???? ????? ???? ?? ? ????.?? ?? ? ?? (DataGenerator, JavaScript ? Table)? ?? “demoPipeline”??? ??? ??? ??? ?? ?? ??? ?? ?? ?? ? ?? ?? ?(curl) ??? ???? ?? ??? ???? ?? ? ? ????.Linux$ curl -w"\n" -X POST ""["system.app.log.debug","system.app.","system.app.log.warn","system.dataset.store.bytes","system.dataset.store.ops","system.dataset.store.reads","system.dataset.store.writes","system.metrics.emitted.count","user.DataGenerator.records.out","user.JavaScript.record.count","user.JavaScript.records.in","user.JavaScript.records.out","user.Table.records.in","user.Table.records.out","user.metrics.emitted.count"]Windows> curl -X POST ""["system.app.log.debug","system.app.","system.app.log.warn","system.dataset.store.bytes","system.dataset.store.ops","system.dataset.store.reads","system.dataset.store.writes","system.metrics.emitted.count","user.DataGenerator.records.out","user.JavaScript.record.count","user.JavaScript.records.in","user.JavaScript.records.out","user.Table.records.in","user.Table.records.out","user.metrics.emitted.count"]? ?? ??? ??? “user.JavaScript.record.count”? ??? ???? JavaScript ???? ???????.context.getMetrics().count('record.count', 1);??? ?? ??? ???? ?? ? ? ????.Linux$ curl -w"\n" -X POST "localhost:11015/v3/metrics/query?tag=namespace:default&tag=app:etlRealtime6&metric=user.JavaScript.record.count&aggregate=true"{"startTime":0,"endTime":1468884338,"series":[{"metricName":"user.JavaScript.record.count","grouping":{},"data":[{"time":0,"value":170}]}],"resolution":"2147483647s"}Windows> curl -X POST "localhost:11015/v3/metrics/query?tag=namespace:default&tag=app:etlRealtime6&metric=user.JavaScript.record.count&aggregate=true"{"startTime":0,"endTime":1468884338,"series":[{"metricName":"user.JavaScript.record.count","grouping":{},"data":[{"time":0,"value":170}]}],"resolution":"2147483647s"}CDAP CLI? ???? ??? ???? ?? ?? ? ? ????.CDAP CLIcdap > get metric value user.JavaScript.record.count 'app=demoPipeline'Start time: 0End time: 1468884640Series: user.JavaScript.record.count+===================+| timestamp | value |+===================+| 0 | 170 |+===================+?? ??? ?????? ???? ???? ?? ??? ???? ??? ???? ?? ???? ??? ??? ????? ??? ? ?? ?? ?? ??? ??? ? ????. ??? ?? ?? ???? ?? ? ? ??? ??? ???? ?? ??? ?? ??? ??? ?????.??? ?? ????? ?? ??? ??? ?????.JavaScriptPython EvaluatorValidatorXML Parser?? ? ?? ??? ?? ???? ??????.??? ??????? ?? ???? ?? CDAP ?? ??? ???????? ?? ? ? ????.YARN? CDAP ????? ?? ???? ? ?? ?? ??? ?? ?? ??? ??? ??????.??? ?? ?????4.2.x ??? ??? ???????? ?? CDAP? ?? ??? ?? ??? ? ETL ????? ??? ??? ? ????. ? ??? ???? 4.3.3 ??? CDAP Studio?? ??? ?? ?? ??? ????(artifacts)? ??? ??????? ??????? ?? ? ??? ???? ???.? ??? CDAP? ????? ???? 4.2.x ??? ??? ????? ???? ??????? ?? ?? ?? ????? 4.3.3 ??? ????? ??????? ???????.??? ??? ? CDAP? ?? ???? ???.$ java -cp /opt/cdap/master/libexec/cdap-etl-tools-4.3.3.jar co.cask.cdap.etl.tool.UpgradeTool -u ; -e /tmp/failedUpgrades upgrade? ?? ??? CDAP ???? ??? ? ?????. ? ?? ??? ????? ? ? ?? ??? ??? ??? ?? ???????. ??? ???? ??? ????? ? ??? ?? ??? ???? ?? ?? ??? ??? ??????? ?? ? ????. ?? ??, ????? ? ?? ??? ?? ? ?? ? ??? ??? ? ????.?? ?? ???? ?? ETL ?????? ? ????? ? ?? ????.$ java -cp /opt/cdap/master/libexec/cdap-etl-tools-4.3.3.jar co.cask.cdap.etl.tool.UpgradeTool -u ; -n <namespace-id> upgradeETL ??????? ??? ????? ? ?? ????.$ java -cp /opt/cdap/master/libexec/cdap-etl-tools-4.3.3.jar co.cask.cdap.etl.tool.UpgradeTool -u ; -n <namespace-id> -p <app-name> upgrade??? ????? ?? ? ?? ??? ??? ??? ???? ??? ??? ???????.$ java -cp /opt/cdap/master/libexec/cdap-etl-tools-4.3.3.jar co.cask.cdap.etl.tool.UpgradeTool -u ; -a <tokenfile> upgrade?? ?? (?? ??? <testing-security> ? ??? ??? ???) ??? ?? ??? ??? ?? ?? :{"access_token":"AghjZGFwAI7e8p65Uo7OpfG5UrD87psGQE0u0sFDoqxtacdRR5GxEb6bkTypP7mXdqvqqnLmfxOS", "token_type":"Bearer","expires_in":86400}??? ?? ?? (AghjZGFwAI7e8p65Uo7OpfG5UrD87psGQE0u0sFDoqxtacdRR5GxEb6bkTypP7mXdqvqqnLmfxOS)? ??? ???? ?? ??? ?? ???? ?????.???? ??? ????? ????? ???? ??? ???? ??? ?? ?????.???? ?? : ????? ??? ????? ???? ?? : ????? ????? ?????? ?????? JAR ?? : ???????? JDBC ???? ? ?? ????? ??? ??? ????? ?? ?? : ?? ??? ???? ??? ??? ? ????.???? ?? : CDAP?? ?? ? ???? ?????? ????? ???? ???? JAR? ???? ????? CDAP Studio? ?? UI? ????? ???? ? ???? ??? ?? ??? ???? ?? ??? ??????.???? JAR (JDBC ???? ?)? ???? ?? ?????? ??????? ??? ? ? ?? ??? ?? ?? ??? ??????.?? ??? ????CDAP?? ?? ??? ????? ??? ???? ????? ????. ???? ?? ? ????? ??? ???? ?? ??? ??? ???? ??? CDAP Studio? ?? ?? ????? ?? ???? ???? ??? ? ????.???? ??????? ?? ????(artifacts)?? ??? ? ?? ??? (??? ???? ? ???? ?? ? ?? ???????? ?? ??), ????? ?? JAR ??? ???? ?? ???. ?? ?? JAR ??? ??? ???? ?? ??? ????? ???????.??? ????? ?? ?? ?????? ???? ??? ? ????. ??? ????? ??? ?? ????? ???? ??? ? ????. ????? ??? ????? ???? ?? Artifact HTTP RESTful API? ?? ???? ???? ?? ??? ????? ????? CDAP Master? ?? ???? ?? ???? ?????. ??? ??? ??? ? ??? ?????.??? ????? ??????? ??? ????? ????? JAR ??? ???? ?? ??? ?? ??? ????? ???? ???.CDAP Sandbox: $CDAP_INSTALL_DIR/artifactsDistributed CDAP: ???? JAR? ?? ?? ???? ???? ?? cdap-site.xml? app.artifact.dir ??? ???? ??? CDAP? ?? ? ? ????. ?? ????? ?????? ???? ?? ? ? ????. ?? ??? / opt / cdap / master / artifacts???.? ???? JAR?? ????? ??? ? ?? ????? ???? ?? ?? ??? ??????. ?? ??? .jar ??? ?? .json ???? ??? ??? ?? ????? JAR? ??? ???? ???. ?? ?? JAR ??? ??? custom-transforms-1.0.0.jar ? ?? ???? custom-transforms-1.0.0.json ??? ??????. custom-transforms-1.0.0.jar? cdap-data-pipeline ? cdap-data-streams ?????? ??? ??? ??? ???? ??? custom-transforms-1.0.0.json? ?? ??? ?????.{ "parents": [ "cdap-data-pipeline[4.3.3,4.3.3]", "cdap-data-streams[4.3.3,4.3.3]" ]}? ??? custom-transforms-1.0.0.jar? ????? ?? 4.3.3? cdap-data-pipeline ? cdap-data-streams ?????? ??? ? ??? ?????. ??? ??? ???? ??? ([])? ??? ??? ???? ?? ()? ???? ????? ??? ? ?? ? ??? ???? ??? ?? ????. ? :{ "parents": [ "cdap-data-pipeline[3.5.0,4.0.0)", "cdap-data-streams[3.5.0,4.0.0)" ]}? ??? cdap-data-pipeline ? cdap-data-streams ???? ?? 3.5.0 (??)?? 4.0.0 (??)?? ? ????? ??? ? ??? ?????.????? ???? ????? ???? ?? ?? ?? ?? ??? ????? ?? ? ? ????. ?? ??, ???? JAR? ??? JDBC ????? ?? ? ? ????. ??? ?? ????? ??? ?? ???? ??? ???? ??? ?? ? ? ???? ??? ???? ???.{ "parents": [ "cdap-data-pipeline[3.5.0,4.0.0)", "cdap-data-streams[3.5.0,4.0.0)" ], "plugins": [ { "name": "mysql", "type": "jdbc", "className": "com.mysql.jdbc.Driver" } ]}JAR? ???? ?? ??? ???? ??? CDAP CLI ?? (load artifact) ?? HTTP RESTful API ??? ??? ????? ???? ????? ?? ? ? ????. ????? ?? ???? ??? ??, ??? ????? ?? ??? ???? ?? ??? ? ? ????.?? ? ?? ??? ?? Sandbox ???? ????? CDAP Sandbox? ?? ?????? cdap-master ???? Distributed???? ?? ???????.??? ????? ??????? ??? ????? ????? ???? HTTP RESTful API ???? ?? ?? CLI? ??????.HTTP RESTful API? ???? ?? Artifact-Extends ??? ???? ???. ????? ??? ????? JAR ??? ????? ??(manifest file)? ???? ?? ??? Artifact-Version ??? ???? ???.CLI? ??? ? ??? ????? ??? ?? ? ?? ??? ??? ?? ??? ???????.?? ?? RESTful API? ???? custom-transforms-1.0.0.jar? ????? ??? ??????.Linux$ curl -w"\n" -X POST "localhost:11015/v3/namespaces/default/artifacts/custom-transforms" \-H "Artifact-Extends: system:cdap-data-pipeline[4.3.3, 4.3.3]/system:cdap-data-streams[4.3.3, 4.3.3]" \--data-binary @/path/to/custom-transforms-1.0.0.jarWindows> curl -X POST "localhost:11015/v3/namespaces/default/artifacts/custom-transforms" ^-H "Artifact-Extends: system:cdap-data-pipeline[4.3.3, 4.3.3]/system:cdap-data-streams[4.3.3, 4.3.3]" ^--data-binary @/path/to/custom-transforms-1.0.0.jarCLI ?? :CDAP CLIcdap > load artifact /path/to/custom-transforms-1.0.0.jar config-file /path/to/config.json???? ?? ??? config.json? ?????.{ "parents": [ "system:cdap-data-pipeline[4.3.3,4.3.3]", "system:cdap-data-streams[4.3.3,4.3.3]" ]}??? ????? ???? ??? ????? ?? ? ? ?? ???? ?? ?? ‘system:’? ??? ???. ? ?? ??? ????? ??? ?? ??? ????? ????. ??? ????? ???? ?? ???? ???? ????.???? ??? ????? ???? ??? ?? ???? ??? ???? ???? JAR? ?? ? ? ????. ????? ?? ???? ??? ??, ??? ???? ? ?? ??? ???? ?? ??? ? ? ????.RESTful API ?? (???? ??? JAR ????? ??? ??? JAR ??? ?? ???? ?????? ?? ??? ???? ?)Linux$ curl -w"\n" -X POST "localhost:11015/v3/namespaces/default/artifacts/mysql-connector-java" \-H 'Artifact-Plugins: [ { "name": "mysql", "type": "jdbc", "className": "com.mysql.jdbc.Driver" } ]' \-H "Artifact-Version: 5.1.35" \-H "Artifact-Extends: system:cdap-data-pipeline[4.3.3, 4.3.3]/system:cdap-data-streams[4.3.3, 4.3.3]" \--data-binary @mysql-connector-java-5.1.35.jarWindows> curl -X POST "localhost:11015/v3/namespaces/default/artifacts/mysql-connector-java" ^-H "Artifact-Plugins: [ { \"name\": \"mysql\", \"type\": \"jdbc\", \"className\": \"com.mysql.jdbc.Driver\" } ]" ^-H "Artifact-Version: 5.1.35" ^-H "Artifact-Extends: system:cdap-data-pipeline[4.3.3, 4.3.3]/system:cdap-data-streams[4.3.3, 4.3.3]" ^--data-binary @mysql-connector-java-5.1.35.jarCLI ?? (???? ??? ????? ???? ?? ?? JAR ?? ???? ???? ??????.)CDAP CLIcdap > load artifact /path/to/mysql-connector-java-5.1.35.jar config-file /path/to/config.json???? ?? ??? config.json? ?????.{ "parents": [ "system:cdap-data-pipeline[4.3.3,4.3.3]", "system:cdap-data-streams[4.3.3,4.3.3]" ], "plugins": [ { "name": "mysql", "type": "jdbc", "className": "com.mysql.jdbc.Driver" } ]}?? ??Artifact HTTP RESTful API? ???? ???? ?? ??? ???? ???? ????? ????? ?????? ?? ? ? ????. ?? ?? ??? ?? ?? ????? ?? ?? ??? ????? ??? ??????.Linux$ curl -w"\n" -X GET "localhost:11015/v3/namespaces/default/artifacts/custom-transforms/versions/1.0.0?scope=[system | user]Windows> curl -X GET "localhost:11015/v3/namespaces/default/artifacts/custom-transforms/versions/1.0.0?scope=[system | user]CLI ???? :CDAP CLIcdap > describe artifact properties custom-transforms 1.0.0 [system | user]??? ?? ?? ????? ??? ????? ?? ? ?? ???(scope)? ??????. ??? ????? ??? ?? ?? ????? ?? ? ?? ???? ??????.Artifact HTTP RESTful API? ???? ?? ??? ????? ???? ?? ??? ????? ????? ??? ??? ? ??? ??? ? ????. ?? ??, cdap-data-pipeline? ??? ?? ?? ????? ????? ??? ? ? ??? ????? ??? ??????.Linux$ curl -w"\n" -X GET "localhost:11015/v3/namespaces/default/artifacts/cdap-data-pipeline/versions/4.3.3/extensions/transform?scope=system"Windows> curl -X GET "localhost:11015/v3/namespaces/default/artifacts/cdap-data-pipeline/versions/4.3.3/extensions/transform?scope=system"CLI ???? :CDAP CLIcdap > list artifact plugins cdap-data-pipeline 4.3.3 transform system?? ?? ??? ??? ???? ??(transform)? ??? ??? ??? ? ????. ??? ?? ???? ?? ????? ???? ?????. ? ????? cdap-data-pipeline? ??? ?????? ??? ??? ??????. ??(parent)? ??? ??? ?????? ??? ??? ????? ??? ?? ??? ?? ? ???? ???????.???? JAR ??Prebuilt JARs: ?? ??? ???? JAR (? : JDBC ????)? ?????? ????? ?? JAR? ???? JSON ??? ???? ???.JSON ??? ?? ??? ??? ??? ??? ? ???? ???? ? ???? ?? ??? ??????.?? JDBC Driver Plugin ?? :{ "parents": [ "cdap-data-pipeline[4.3.3,4.3.3]" ], "plugins": [ { "name" : "mysql", "type" : "jdbc", "className" : "com.mysql.jdbc.Driver", "description" : "Plugin for MySQL JDBC driver" }, { "name" : "postgresql", "type" : "jdbc", "className" : "org.postgresql.Driver", "description" : "Plugin for PostgreSQL JDBC driver" } ]}?? ?? ????? ???? (?? ????)? ?? ??? ???? ??? ??? ? ????. ??? CDAP Studio(??? ??? ? Pipelines? CDAP Studio??)?? ????(??? ??? ? Developing Pipelines??)? ????? ???? ???? ?? ? ??? ?? ???? ?????. ??????? ???? ? ???? ?? ??? ????? ?? ??? ???? ?? ?? ?? ????? ??? ? ?? ?? ?? ??? ?????.???? ????????? Artifact HTTP RESTful API ?? CDAP CLI? ???? ??? ? ????.CDAP CLI? ?? user ???? ??? ????? CDAP CLI?? ??? ? ????. ??? ??? ???? HTTP RESTful API? ???? system ? user ??? ????? ?? ??? ? ????.?? ??? ????? ??? ?? ?? (JAR ? JSON ??)? ???? ????. ??, CDAP ????? ?? ??? ?????. ??? ??? ?? ? ? ??? ???? ???, system ??? ????? ??, ?? ? CDAP? ???? ? ?????? ?? ?? ? ????. ?? ?????? ??? ???????? ?????? ???? ???? ?????.??? ?? ???Data Preparation? (??? ??? MapReduce ?? Spark ??? ????) ?? ??? ???? ??? ???? ?? ???? ???(100 ? ?)? ?? ???? ?? ? ? ?? ??? ?????????. ?, ??? ???? ???? ?? ??? ??? ??? ??? ??? ? ????.Data Preparation? ?? RDBMS, Kafka, .csv, .json ?? ? ??? ??? ??? ?? ? ? ????. ? ???? ?? ???? ?? ?? ? ? ????.???? ????, Data Preparation? ?? ???? ?? ???? ???? ?? ? ? ????. ?? ??? ???? ??? ??? ?? ? ???? null ? ?? ?? ? ? ????.?? Data Preparation? ?? ???? ?? ? ? ????. Data Preparation? ?? ??? ????? ??? ?? ?? ??? ??? ???? ?? ???????. ?? ????, ?? ????, ???? ?? ???? ?? ??? ?? ? ? ????. ?? ???? ?? ?? ? ? ???? (? : .csv?? .json?? ??).????? ?? ???? ???? ??? ??? ???? Create Pipeline? ???? ??? ??? ?? ? ? ????. ?? ?? ? MapReduce, Spark ?? Spark Streaming ???? ??? ??? ?? ??? ?? ? Pipeline? ?????? (?? ???? ??).DataPrep ??? Data Perp??? Record, Column, Directive, Step ? Pipeline? ??? ?????.??????? ?? ??? ?? ?? ?????.?Column(?)? ???? ?? Java ?? (? ????? ???)? ??? ????.Directive???A Directive is a single data manipulation instruction, specified to either transform, filter, or pivot a single record into zero or more records. A directive can generate one or more steps to be executed by a pipeline.Directive? ??? ???? 0 ? ??? ???? ??, ??? ?? ??(pivot)??? ??? ?? ??? ?? ??????. ???? ??? ??? ?? ??? ?? ??? ??? ?? ? ? ????.??Step(??)? ?? ??? ?? ??? ???? ???? ??? ?? ??? ?????. ??? ??? ???????? 0 ? ??? ???? ?? ? ? ????.??? ??Pipeline(?????)? ???? ?? ? ??? ?????. ???? ??? ???? ??? ??? ?? ??? ?????.??????A directive can be represented in text in this format:???? ?? ???? ???? ??? ? ????:<command> <argument-1> <argument-2> ... <argument-n>???? ???? ???? ??? ?? ???? ???? ?? ???? ??? ?? ? ??? ???? ?? ?? ?? JSON ??? ?????.? :{ "id": 1, "fname": "root", "lname": "joltie", "address": { "housenumber": "678", "street": "Mars Street", "city": "Marcity", "state": "Maregon", "country": "Mari" }, "gender": "M"}?? ?????? ??? ??? ???? :2.9 GHz Intel Core i516 GB 2133 MHz LPDDR3Java 7??? ??? ?? DMD??? ???? ?? ???? ?? ?????.Parsing of CSVDrop columnsSetting defaults on columnChanging caseMasking dataFiltering rows based on an expressionDirectivesparse-as-csv demo , truedrop demodrop demo_12fill-null-or-empty demo_11 N/Auppercase demo_17mask-number demo_18 xxx###drop demo_6drop demo_7fill-null-or-empty demo_5 N/Auppercase demo_3filter-row-if-true demo_9 =~ "CA"mask-number demo_10 xxx##mask-shuffle demo_4??? ? ?? ??? ??????? : ? ??? 1300? ?????, ? ??? 8000? ??????.?? # 1Number of records: 13,499,973Number of bytes: 4,499,534,313 (~ 4GB)Number of columns: 18Performance Numberscount = 13,376,053mean rate = 64998.50 records/second1-minute rate = 64921.29 records/second5-minute rate = 46866.70 records/second15-minute rate = 36149.86 records/second?? # 2Number of records: 80,999,838 (80M)Number of bytes: 26,997,205,878 (~ 26GB)Number of columns: 18Total time: 1294 seconds (21.5 minutes)Performance Numberscount = 80,944,061mean rate = 62465.93 records/second1-minute rate = 62706.39 records/second5-minute rate = 60755.41 records/second15-minute rate = 56673.32 records/second?? (??) ? ?? ???(Excluding (Restricting) and Aliasing Directives)???? ??? ?? ??? ?? ????? ????? ?? “????? ???? ??”??? ?? (??) ? ? ?? ??? ????. ??? ?? ????? ???? ????. ?? ??? ??? “Safe”??? ?? ????? ?? ??? ?? ???? ?? ? ? ????. ??? ?? ?? ? ?? ?? ??? ?? ?? ? ? ????.??? ??? ? ?? ???? ?? ??? ??? ??? ?? ??? ???? ????? ???? ????. ?????? ??, ?? ?????.??? ??? ??? ???? ?? Data Prep?? ??? ???? ???? ???? ???? ??(aliasing) ? ? ?? ??? ???????.FeatureData Prep? ???? ? ?? ??? ????.Exclusion (??, Restriction)AliasingExclusion? ???? ???? ?????? ?? ??? ?? ????? ?? ?? ?? ???? ??? ??? ??? ? ????.Aliasing? ???? ?? ???? ? ??? ?? ? ????.ScopeExclusion? Aliasing ? ? ?? ???? ?????. ?, ??? ??? ?? ???? ???? ?????.????(configuration)? ?? ?? ?? ?? JSON ??? ?????.exclusionsaliases??? ?? ??? JSON ?????.{ "exclusions" : [ "root-directive", ... "root-directive" ], "aliases" : { "alias" : "alias-name", ... "alias" : "alias-name" }}Exclusion????? ????? UDD (User Defined Directives)? ?? ? ? ?? ??? ????? ??? ??? ? ?? ????.Aliases??? ???? ?? ??? ?? ?? ????.?? ????? ???? ?? ??? endpoint? ?????. ??? ????? ?? REST ??? ??????.curl -s -X POST @<path-to-json>/<filename.json> \ ""??? ?? ??? ????:curl -s -X POST --data-binary @/tmp/wrangler-config.json \ \ | python -mjson.tool{ "message": "Successfully updated configuration.", "status": 200}Directives???(directives)? ??? ?? UI?? ???? ???? ? ?????.Catalog LookupCATALOG-LOOKUP ???? ?? ??? (?? ?) ????? ?? ??? ?????. Currently, ? ??? ?? ?? ICD-9 ? ICD-10- (2016,2017) ??? ?? ?? ?????.??catalog-lookup <catalog> <column>??? <catalog> ?? ?? ?? ? ???? ?? ??? ?????. <column> .?? ???? ????? ??? ????.ICD-9ICD-10-2016ICD-10-2017??? ?? ????? ???? ??? ??? ?? : ??? ?? (??)? ?? ? ????,{ "code": "Y36521S"}ICD-10-2016 ????? ?? CATALOG-LOOKUP ??? ?? :catalog-lookup ICD-10-2016 code?? ?? ?? ???? ?????. code_ <catalog> _description (?? ?? ??). ???? ??? ?? ?? ?? null? ?????. ? ??? ?? code_icd_10_2016_description?? ??? ??????.{ "code": "Y36521S", "code_icd_10_2016_description": "War operations involving indirect blast effect of nuclear weapon, civilian, sequela"}??? ? (null)??? ???? ?? ???? ??, ? (NULL) ?? ?? ??? ?????.Change Column CaseCHANGE-COLUMN-CASE ?? ?? ? ??? ??? ?? ???? ?????.??change-column-case lower|upper??? ?? ??? ?????? ?? ? ?? ???? ??????.??? ???? ??? ?? :{ "Id": 1, "Gender": "male", "FNAME": "Root", "lname": "JOLTIE", "Address": "67 MARS AVE, MARSCIty, Marsville, Mars"}? ??? ?? :change-column-case lower? ???? ??? :{ "id": 1, "gender": "MALE", "fname": "Root", "lname": "Joltie", "address": "67 mars ave, marscity, marsville, mars"}? ??? ?? :change-column-case upper? ???? ??? :{ "ID": 1, "GENDER": "MALE", "FNAME": "Root", "LNAME": "Joltie", "ADDRESS": "67 mars ave, marscity, marsville, mars"}Changing CaseUPPERCASE, LOWERCASE ? TITLECASE ???? ???? ? ?? ? / ???? ?????.??lowercase <column>uppercase <column>titlecase <column>? ???? ?? ??? ??? ?????.?? ???? ??? ?? :{ "id": 1, "gender": "male", "fname": "Root", "lname": "JOLTIE", "address": "67 MARS AVE, MARSCIty, Marsville, Mars"}? ??? ??uppercase gendertitlecase fnametitlecase lnamelowercase address? ???? ??? :{ "id": 1, "gender": "MALE", "fname": "Root", "lname": "Joltie", "address": "67 mars ave, marscity, marsville, mars"}Cleanse Column NamesCLEANSE-COLUMN-NAMES ???? ?? ??? ?? ? ??? ?????.?? ? ?? ?? ??? ?? ???[AZ] [az] [0-9] ?? _ ? ??? ?? ??? ?? (_)? ????.??cleanse-column-names?? ???? ??? ?? :{ "COL1": 1, "col:2": 2, "Col3": 3, "COLUMN4": 4, "col!5": 5}? ??? ?? :cleanse-column-names? ???? ??? :{ "col1": 1, "col_2": 2, "col3": 3, "column4": 4, "col_5": 5}Columns ReplaceCOLUMNS-REPLACE ???? ? ??? ???? ?????.??columns-replace <sed-expression>??? <sed-expression> s / data _ // g? ?? sed ??? ??? ?????.??? ??,? ???? ???? ? ?? ?? ? ??? ???? data_? ????.{ "data_name": "root", "data_first_name": "mars", "data_last_name": "joltie", "data_data_id": 1, "data_address": "150 Mars Ave, Mars City, Mars, 8899898", "mars_ssn": "MARS-456282"}? ??? ?? :columns-replace s/^data_//gdata_? ???? ?? ? ??? ? ???? ??? ???? ??? ?????.{ "name": "root", "first_name": "mars", "last_name": "joltie", "data_id": 1, "address": "150 Mars Ave, Mars City, Mars, 8899898", "mars_ssn": "MARS-456282"}?? :? ?????? ?? ?? ???? ????. ??? ? ????? ?????.CopyCOPY ???? ?? ?? ?? ?? ?? ?????.??copy <source> <destination> [<force>]COPY ???? <source> ?? <destination> column.?? <destination> ?? ?? ????. <force> ??? true? ???? ?? ?? ?? ???? ??? ? ????. ????? <force> ??? false? ?????.??? ?? ????COPY ???? <source> ???? <source> ?? ?????. ?? <source> ???? ???? ??? ??? ???? ??? ?????.???? ??? ?? ??? ???? ?????. ?? ?? ?? ??? ??? ?? ??? ?????.?? ???? ??? ?? :{ "id": 1, "timestamp": 1234434343, "measurement": 10.45, "isvalid": true, "message": { "code": 132, "text": "Failure in the temperature sensor" }}?? ??? ?? :copy timestamp datetimecopy message status? ???? ??? :{ "id": 1, "timestamp": 1234434343, "datetime": 1234434343, "measurement": 10.45, "isvalid": true, "message": { "code": 132, "text": "Failure in the temperature sensor" }, "status": { "code": 132, "text": "Failure in the temperature sensor" }}Cut CharacterCUT-CHARACTER ???? ??? ?? ??? ???? ?? ?? ?? ??? ?????.??cut-character <source> <destination> <type> <range|indexes>??? <type><range|indexes> ?? ?? ?????.??? ?? ????CUT-CHARACTER ???? ?? ???? ?? ??? ?? ?? ??? ?????. ? ??? ?? (-b ???, -c ??, -f ??)?? ???? ??? ?? ?????.N? 1???? N ?? ???, ?? ?? ??.N-?N ?? ???, ?? ?? ?????? ? ???N-M? N ???? M’th (??) ???, ?? ?? ????-M ????? M’th (??) ???, ?? ?? ?????? ???? ??? ?? :{ "body": "one two three four five six seven eight"}?? ??? ?? :cut-character body one -c 1-3cut-character body two -c 5-7cut-character body three -c 9-13cut-character body four -c 15-cut-character body five -c 1,2,3cut-character body six -c -3cut-character body seven -c 1,2,3-5? ???? ??? :{ "body": "one two three four five six seven eight", "one": "one", "two": "two", "three": "three", "four": "four five six seven eight", "five": "one", "six": "one", "seven": "one t"}DecodeDECODE ???? RFC-4648? ?? base32, base64 ?? hex ? ??? ? ?? ??????.??decode <base32|base64|hex> <column>??? <column> ???? ???? ?? ?????.??? ?? ???????? ?? ???? ?? ?? ???? ??? ??? US-ASCII ???? ???? ???? ???? ????? ???? ? ?????. ??? ???? ???? ????? ?? ? ? ?? ??? ???(legacy) ??? ?? ? ???????? ?? ???? ??? ? ????.DECODE ???? ?? ??? ??? ?? ? ?? ?????. <column> _decode_ <type> .?? ?? ?? ?? ?? ?? ?????.?? null? ??, ??? ?? null????.??? ?? ???? ??? ???? no-op? ?? ???.? ?? ?? ??? ?? ??? ??? ??? ?????.ENCODE ???? ??????.?? ???? ??? ?? :{ "col1": "IJQXGZJTGIQEK3TDN5SGS3TH", "col2": "VGVzdGluZyBCYXNlIDY0IEVuY29kaW5n", "col3": "48657820456e636f64696e67"}?? ??? ?? :decode base32 col1decode base64 col2decode hex col3? ???? ??? :{ "col1": "IJQXGZJTGIQEK3TDN5SGS3TH", "col2": "VGVzdGluZyBCYXNlIDY0IEVuY29kaW5n", "col3": "48657820456e636f64696e67", "col1_decode_base32": "Base32 Encoding", "col2_decode_base64": "Testing Base 64 Encoding", "col3_decode_hex": "Hex Encoding",}Diff DateDIFF-DATE ???? ? ??? ??? ?????.??diff-date <column1> <column2> <destination>??? ?? ????DIFF-DATE ???? ? Date ?? ?? ??? ????? ( <column1> ???? <column2> )? ??? (?? ? ???) ?? ?? ????.? ???? ?? ???? ?? PARSE-AS-DATE?? PARSE-AS-SIMPLE-DATE???? ???? ??? ? ?? ??? ??? ? ????.? ?? ?? ? ?? ??? ?? ?? ? ? ?? ??? ?? ? ? ????.?? ?? ???? ?? null? ??, ??? ?? null????.? ? ??? ??? now? ??? ?? ?? - ??? ?????. ?? ???? ???? ?? ?? ??? ??? ?? ?????.?? ???? ??? ?? :{ "create_date": "02/12/2017", "update_date": "02/14/2017"}? ??? ?? :diff-date update_date create_date diff_date? ???? ??? :{ "create_date": "02/12/2017", "update_date": "02/14/2017", "diff_date": 17280000}DropDROP ???? ???? ?? ???? ? ?????.??drop <column>[,<column>]*??? <column> ??? ???? ? ?????.??? ?? ????DROP ???? ??? ? ?? ?? ?? ????? ?????. ??? ???? ?? ? ?? ??? ? ????.?? ???? ??? ?? :{ "id": 1, "timestamp": 1234434343, "measurement": 10.45, "isvalid": true}? ??? ?? :drop isvalid,measurement???? ??? ?? ??? ?? ???? ?????.{ "id": 1, "timestamp": 1234434343}EncodeENCODE ???? RFC3248? ?? base32, base64 ?? hex ? ??? ? ?? ??????.(RFC-4648)??encode <base32|base64|hex> <column>??? <column> ???? ???? ?? ?????.??? ?? ???????? ?? ???? ?? ?? ???? ??? ??? US-ASCII ???? ???? ???? ???? ????? ???? ? ?????. ??? ???? ???? ????? ??? ? ?? ??? ??? ??? ?? ? ???????? ?? ???? ??? ? ????.ENCODE ???? ?? ??? ??? ?? ? ?? ?????. <column> _encode_ <type> .?? ?? ?? ?? ?? ?? ?????.?? null? ??, ??? ?? null????.??? ?? ???? ??? ???? no-op? ?? ???.? ?? ?? ??? ?? ??? ??? ??? ?????.DECODE ???? ??????.?? ???? ??? ?? :{ "col1": "Base32 Encoding", "col2": "Testing Base 64 Encoding", "col3": "Hex Encoding"}?? ??? ?? :encode base32 col1encode base64 col2encode hex col3? ???? ??? :{ "col1": "Base32 Encoding", "col2": "Testing Base 64 Encoding", "col3": "Hex Encoding", "col1_encode_base32": "IJQXGZJTGIQEK3TDN5SGS3TH", "col2_encode_base64": "VGVzdGluZyBCYXNlIDY0IEVuY29kaW5n", "col3_encode_hex": "48657820456e636f64696e67"}Extract Regex GroupsEXTRACT-REGEX-GROUPS ???? ??? ??? ???? ?? ?? ?????.??extract-regex-groups <column> <regex-with-groups>? ???? ?? ???? ???? ?? ?? ?????. <regex-with-groups> . ??? $ 0 ??? ??? ?????.??? ?? ?????? ??? ???? ??.? ?? ??? ?????.?? ?? ???? ??? ???? ?? ?? ? ?? ??? ?????. <column> _ <match-count> _ <match-position> .?? ???? ??? ?? :{ "title": "Toy Story (1995)"}? ??? ?? :extract-regex-groups title [^(]+\(([0-9]{4})\).*? ???? ??? :{ "title": "Toy Story (1995)", "title_1_1: "1995"}title_1_1 ??? ?? ??? ????. <column> _ <match-count> _ <match-position> .Fail on conditionFAIL ???? ??? true? ??? ? ??? ?? ??? ?????.??fail <condition>??? <condition> ???? ?? ???? ????? ??? ??? ???? ??? ???? JEXL ??????.??? ?? ????FAIL ???? ?? ???? ??? ??? ?? ? ? ??? ??? ?? ??? ???? ????.??? ? ??? ?? ???? ?????.?? ???? ????? ???? ??? ??? ?????. ??? ?? ? ??? ???? ???? ??? ??? ?? ??? ?????.fail ErrorCount > 1Fill Null or EmptyFILL-NULL-OR-EMPTY ???? null ?? ???? ?? (“”) ? ?? ?? ??? ? ???.??fill-null-or-empty <column> <fixed-value>?? <column> ???? ??? ???? ?????.??? <fixed-value> ?? ??? ? ? ????.??? ?? ????FILL-NULL-OR-EMPTY ???? ? ?? <fixed-value> ?? ?? null ?? ??? ?? (? ???, “”)??? <fixed-value> ?????? ?? ? ??? ?? ? ????.???? ??? JSON ???? null ? ?? ???? ?? ??? ?? ????? ????? ?????.?? ???? ??? ?? :{ "id": 1, "fname": "root", "mname": null, "lname": "joltie", "address": ""}?? ??? ?? :fill-null-or-empty mname NAfill-null-or-empty address No address specified? ???? ??? :{ "id": 1, "fname": "root", "mname": "NA", "lname": "joltie", "address": "No address specified"}Filter Row If MatchedFILTER-ROW-IF-MATCHED ???? ?? ??? ???? ???? ??? ???.? ?? ???? ?? ???? FILTER-ROWS-ON???? ??????.??filter-row-if-matched <column> <regex>??? <regex> ?? ???? ? ?? ?? ???? ??? ?? ??????.??? ?? ????FILTER-ROW-IF-MATCHED ???? ?? ???? ? ?? ?? ???? ?????. ?? ???? ? ?? ???? ???? ????, ??? ??? ?? ???? ???? ?? ??? ?????.???? null?? ?? JSON null ??? ??? ?? null? ?????.??? ???? ??? ?? :{ "id": 1, "name": "Joltie, Root", "emailid": "jolti@", "hrlywage": "12.34", "gender": "Male", "country": "US"}? ??? ?? :filter-row-if-matched country !~ US(??? “??”? ???? ??) ??? ?? ??? ???? ?? ? ? ????.? ??? ?? :filter-row-if-matched (country !~ US && hrlywage > 12)??? ??? (??? “US”? ???? ??) ??? ?? (hrlywage)? 12?? ? ??? ???? ?? ? ? ????.Filter Row If TrueFILTER-ROW-IF-TRUE ???? ??? ???? ???? ??? ???.? ?? ???? ?? ???? FILTER-ROWS-ON ???? ??????.??filter-row-if-true <condition>??? <condition> true ?? false??? ??? ?? ????.??? ?? ????FILTER-ROW-IF-TRUE ???? ? ???? ?? ??? ?????. ?? ??? true?? ???? ?? ???. ??? ??? ??? ?? ???? ???? ?????.??? <condition> ? JEXL ??? ??? ???? ?? ???? ?????? ?? ? ??? ?? ????? ?????.JEXL ???? ???? ??? ?? ??? commons-jexl documentation??? ??????.??? ???? ??? ?? :{ "id": 1, "name": "Joltie, Root", "hrlywage": "12.34", "gender": "Male", "country": "US"}? ??? ?? :filter-row-if-true country !~ US(??? “??”? ???? ??) ??? ?? ??? ???? ?? ? ? ????.? ??? ?? :filter-row-if-true (country !~ US && hrlywage > 12)??? ??? (??? “US”? ???? ??) ??? ?? (hrlywage)? 12?? ? ??? ???? ?? ? ? ????.Filter Rows OnFILTER-ROWS-ON ???? ??? ?? ???? ??????.??filter-rows-on <filter-type> <options>??? <filter-type> ?? ? ?? ??? ?? ??? ?????.???? ?? ?? ? ?? ?? :condition-false <boolean-expression>condition-true <boolean-expression>empty-or-null-columns <column>[,<column>]*regex-match <regular-expression>regex-not-match <regular-expression>??? ?? ????FILTER-ROWS-ON ???? ?? ??? ?? ?? ?? ???? ? ???? ? ?? ?????. ???? ? ?? ?? true ?? true? ???? ???? ?????. ??? ??? ??? ?? ???? ???? ?????.??? ??? ?? ?????. <filter-type> ? ??. ?? ??, condition-true? ???? <boolean-expression> true? ???? ?? ???? ???? ????.??? ???? ??? ?? :{ "id": 1, "name": "Joltie, Root", "emailid": "jolti@", "hrlywage": 12.34, "gender": "Male", "country": "US"}? ??? ?? :filter-rows-on condition-true country !~ 'US'(??? “??”? ???? ??) ??? ?? ??? ???? ?? ? ? ????.? ??? ?? :filter-rows-on condition-true (country !~ 'US' && hrlywage > 12)??? ??? (??? “US”? ???? ??) ??? ?? (hrlywage)? 12?? ? ??? ???? ?? ? ? ????.Find and ReplaceFIND-AND-REPLACE ???? “sed”? ?? ???? ???? ??? ? ?? ???? ???? ?? ????.??find-and-replace <column> <sed-script>??? ?? ????? ???? ??? ?? ?? ??? ???? ? ?? ??? ???????. sed ????? ? ??? ?? ???? ???? ?????.??? ?? ??? ?? ???? ? :find-and-replace <column> s/regex/replacement/g? ???? ?? ???? ???? ? ?? ?? ??? ????.s? “??”? ?????. g? “global”? ????, ?? ??? ???? ?? ??? ???? ?????.???? ?? ???? ? ?? ?? ?? (? ???? ???) ?? ???? ?? ?? ? ?? ?? ?? ??????. ??? (/)? ?? ?? ? “??”??? ???? ???? ?? ?????.?? ?? hello? ?? ??? message ??? world? ???? ??? ??????.find-and-replace message s/hello/world/g??? (? : / usr / local / bin? / common / bin)? ?? ? ?? ??? ????? ? ???? ???? ???? ????? ??????.find-and-replace column s/\/usr\/local\/bin/\/common\/bin//g?? ???? ??? ?? :{ "body": "one two three four five six seven eight"}?? ? ?? ??? ?? :find-and-replace body s/one/ONE/gfind-and-replace body s/two/2/g? ???? ??? :{ "body": "ONE 2 three four five six seven eight",}FlattenFLATTEN ???? ?? ??? ??? ?? ???? ?????.??flatten <column>[,<column>]*??? <column> JSON ?? ? ?? ?????.??? ?? ????FLATTEN ???? ???? ???? ???? ???? ? ?????.? ??? ?? ???? ?? ?? ??? ??? ???? ?? FLATTEN ???? ?? ?? ?? ?? ? ???? ?????.???? 1col2? ??? ????? col3? ?? col2? ? ?? ?? ?????.?? ???[ { "col1": "A" }, { "col1": "B" }, { "col2": [x1, y1], "col3": 10 }, { "col2": [x2, y2], "col3": 11 }]?? ???[ { "col1": "A" }, { "col1": "B" }, { "col2": "x1", "col3": 10 }, { "col2": "y1", "col3": 10 }, { "col2": "x2", "col3": 11 }, { "col2": "y2", "col3": 11 }]?? 2col2 ? col3? ??? ?????.?? ???[ { "col1": "A" }, { "col1": "B" }, { "col2": [ "x1", "y1", "z1" ], "col3": [ "a1", "b1", "c1" ] }, { "col2": [ "x2", "y2" ], "col3": [ "a2", "b2" ] }]?? ???[ { "col1": "A" }, { "col1": "B" }, { "col2": "x1", "col3": "a1" }, { "col2": "y1", "col3": "b1" }, { "col2": "z1", "col3": "c1" }, { "col2": "x2", "col3": "a2" }, { "col2": "y2", "col3": "b2" }]?? 3col2 ? col3? ??? ?????.?? ???[ { "col1": "A" }, { "col1": "B" }, { "col2": [ "x1", "y1", "z1" ], "col3": [ "a1", "b1" ] }, { "col2": [ "x2", "y2" ], "col3": [ "a2", "b2", "c2" ] }]?? ???[ { "col1": "A" }, { "col1": "B" }, { "col2": "x1", "col3": "a1" }, { "col2": "y1", "col3": "b1" }, { "col2": "z1" }, { "col2": "x2", "col3": "a2" }, { "col2": "y2", "col3": "b2" }, { "col3": "c2" }]?? 4? ???? ??? ?? :{ "x": 5, "y": "a string", "z": [ 1,2,3 ]}? ???? ??? ?? ? ?? ?? ? ???? ?????.xyz5“a string”15“a string”25“a string”3???? ?? (? ???? z ?)??? ?? ?? ??? ?????. ? ?? “all”(*) ??? ??? flatten? ?? ??? ???? ?? ???? ??? ??? ?????.Format DateFORMAT-DATE ???? ?? ? ?? ??? ?? ??? ?? ??? ?????.??format-date <column> <format>??? ?? ?????? ? ?? ??? ?? ? ?? ?? ???? ?????. ?? ??? ??? ‘A’?? ‘Z’?? ? ‘a’?? ‘z’? ???? ?? ??? ?? ?? ?? ???? ?? ??? ???? ?? ??? ?????. ??? ??? ?? ?? ??? (‘)? ???? ???? ?? ? ? ????. ? ?? ?? ??? ‘’? ?? ???? ?????. ?? ?? ??? ???? ????. ??? ?? ?? ???? ????? ?? ?? ?? ???? ?????.? ?? ??? ????? ( ‘A’?? ‘Z’??? ?? ?? ??? ‘a’?? ‘z’??? ?? ?).LetterDate or Time ComponentPresentationExmplesGEra designatorTextADyYearYear1996; 96YWeek yearYear2009; 09MMonth in yearMonthJuly; Jul; 07wWeek in yearNumber27WWeek in monthNumber2DDay in yearNumber189dDay in monthNumber10FDay of week in monthNumber2EDay name in weekTextTuesday; TueuDay number of week (1=Monday,…, 7=Sunday)Number1aAM/PM markerTextPMHHour in day (0-23)Number0kHour in day (1-24)Number24KHour in am/pm (0-11)Number0hHour in am/pm (1-12)Number12mMinute in hourNumber30sSecond in minuteNumber55SMillisecondNumber978zTime zoneGeneral Time ZonePascific Standard Time PST; GMT-08:00ZTime zoneRFC 822 Time Zone-0800XTime zoneISO 8601 Time Zone-08; -0800;-08:00??? ??? ???? ?? PARSE-AS-DATE???? ???? ??? ?? - ?? ???? ???? ???? ???.? ??? ?? ????? ?? ? ?? ??? ???? ??? ?????. ??? ??? ??? 2001-07-04 12:08:56?? ?? ??? ?? ???? ?? ?? ? ?? ?? ??? ???? ?? ???? ??? ?????.Date and Time PatternDate Stringyyyy.MM.dd G ‘at’ HH:mm:ss z2001.07.04 AD at 12:08:56 PDTEEE, MMM d, ‘’yyWed, Jul 4, ‘01h:mm a12:08 PMhh ‘o’ ‘clock’ a, zzzz12 o’clock PM, Pacific Daylight TimeK:mm a, z0:08PM, PDTyyyy.MMMM.dd GGG hh:mm aaa2001.July.04 AD 12:08 PMEEE, d MMM yyyy HH:mm:ss ZWed 4 Jul 2001 12:08:56 -0700yyMMddHHmmssZ010704120856-0700yyyy-MM-dd ‘T’ HH:mm:ss.SSSXXX2001-07-04T12:08:56.235-07:00MM/dd/yyyy HH:mm07/04/2001 12:09yyyy.MM.dd2001-07-04Format UNIX TimestampFORMAT-UNIX-TIMESTAMP ???? UNIX timestamp? ??? ?????.??format-unix-timestamp <column> <pattern><column> ??? ?? ? ????.<pattern> ?? ???? ?? ? ? ??? ?? ??????.??? ?? ????FORMAT-UNIX-TIMESTAMP ???? ?? ???? ???? UNIX ?? ??? ?????. ??? <column> ??? ???? ??? ?? ???? ???? ??.????? ?? ???? ?? ?? ? ?? FORMAT-DATE? ??????.Generate UUIDGENERATE-UUID ???? ???? ??? ?? ??? (UUID)? ?????.??generate-uuid <column>??? <column> ???? ?? ?? ? UUID? ?????.??? ?? ????GENERATE-UUID ???? ?? 4? ???? ??? UUID? ?????. UUID? ?????? ?? ?? ?? ?? ????? ???? ?????.?? ???? ??? ???? ???? ???? ??? ??? ???? ????? ?? :{ "x": 1, "y": 2}? ??? ?? :generate-uuid uuid?? ??? ???? ??? (uuid? ?? ?????).{ "x": 1, "y": 2, "uuid": "57126d32-8c91-4c00-9697-8abda450e836"}HashHASH ???? ??? ?????? ?????.??hash <column> <algorithm> [<encode>]??? <column> ??? ??? ?? ?????. <algorithm> ? ????.?? <encode> ? true? ??? ?? ??? ?????? ?? ??? 0? 16??? ??????. ?????, <encode> true? ?????. 16 ?? ???? ???? ???? <encode> ????.??? ?? ????HASH ???? <column> , ?? ??? ??? ??? ????. ? ???? ???? ??? ?? ???? ????.??? ????? HASH ????? ?????.BLAKE2B-160BLAKE2B-256BLAKE2B-384BLAKE2B-512GOST3411GOST3411-2012-256GOST3411-2012-512KECCAK-224??? -224KECCAK-256KECCAK-288KECCAK-384KECCAK-512MD2MD4MD5RIPEMD128RIPEMD160RIPEMD256RIPEMD320SHASHA-1SHA-224SHA-256SHA-384SHA-512SHA-512 / 224SHA-512 / 256SHA3-224SHA3-256SHA3-384SHA3-512Skein-1024-1024Skein-1024-384Skein-1024-512Skein-256-128Skein-256-160Skein-256-224Skein-256-256Skein-512-128Skein-512-160Skein-512-224Skein-512-256Skein-512-384Skein-512-512SM3Tiger???WHIRLPOOL?????? ???? ??? ?? :{ "message": "secret message"}? ??? ?? :hash message SHA3-384??? ?????? ???? ?? ? ?? ????.{ "message": "9cc25835d1ef78b4cd8b36a0c4ad636a6094fbb944b1d880f21c7129a645e819d3be987e8ae2f0f8d6cbebb8452419ef"}? ???? SHA3-384 ????? ???? ??? ?????? ?????. ?? ??? ??????.?? : ????? ???? ?? ????. ?? <encode> ? false? ??? ?? ?? ? ??? ??? ?????.Increment a transient variableINCREMENT-VARIABLE ???? ???? ?? ???? ??(local) ??? ?? ??????.??increment-variable <variable> <value> <expression>??? <variable> ? ?? ?????. <value> ? <expression> ????? ?????.??? ?? ????? ??? ???? ???? ???(scope) ???? ?????. ?? ???? ? ???? ??? ???? ???? ??? ??????.?????? ???? ? ?? ??????.IndexsplitIndexsplitINDEXSPLIT ???? ??? ???? ???? ? ?? ?? ???? ????.Deprecated? ?? ???? ?? ???? SPLIT-TO-COLUMNS?? PARSE-AS-FIXED-LENGTH???? ??????.Invoke HTTPINVOKE-HTTP ???? ??? ??? ??? ???? HTTP POST ??? ????? ???? ??????.??invoke-http <url> <column>[,<column>*] <header>[,<header>*]??? <column> ???? ?? ?? ?????. <url> POST ???? ????.??? ?? ????INVOKE-HTTP ???? ?? REST ???? ???? ???? ??? ???? ? ?????. ? ???? ??? ?? POST ???? JSON ?? ??? ?????. JSON ??? ?? ? ???? ?? ??? ???? ??? ?? ??? ?? ????.???? ?? ?? ???, INVOKE-HTTP ???? ???? JSON ??? ?? ??? ??? ?????. JSON ??? ? ??? ?? ?? ?? ?? ?? ???? ?? ???? ?????. ??? ?? ??? JSON ?????.?? JSON ??? ??? ????? ???. ??? JSON? ?? ???? ????.?? : HTTP ??? ???? ??? ??? ?? ? ???? ???? ??? ???? ???? ?????? ???? ???? ??? ???.HTTP ???? ? ??? ??? ???? ?? ?? ? - ? ??? ??? ? ????. ? :X-Proxy-Server=0.0.0.0,X-Auth-Type=Basic?? : ?? ?? ?? (=)? ???? ???? ?? (,)? ?????.???? ???? ??? ?? :{ "latitude": 24.56, "longitude": -65.23, "IMEI": 212332321313, "location": "Mars City"}??? ??? ??? ?? ??? ?? ???? http : // hostname / v3 / api / geo-find? ?? ???? ??? ? ??? ?????.? ??? ?? :invoke-http latitude,longitude??? POST ??? ?????.POST v3/api/geo-find??? ?? :{ "latitude": 24.56, "longitude": -65.23}???? ?? ?? ??? ??? ? ?? ??? ???? ?????.??? ?? ?? ???? ?? ???? ???? ??? ?? ?? ? ? ????.JSON PathJSON-PATH ???? JSON ???? ??? DSL? ?????.??json-path <source-column> <destination-column> <expression><source-column> “?? ?? ??”?? “$”? ????? ?? ???? ?? ?????.<destination-column> ???? ??? ?? ? ???? ?? ? ?????.<expression> JSON ?? ??????. ??? ??? ??? ??? ?? ??? ??????.??? ?? ???????? ?? XPath ???? XML ??? ?? ???? ?? ?? ???? JSON ???? ?????. “?? ?? ??”? ?? ?? ?? ? ?? ?? $? ?????.Notation??????? “? ???”? ??? ? ????.$.name.first?? “??? ???”:$['name']['first']???OperatorDescription$The root element to query; this starts all path expressions@The current node being processed by a filter predicate*Wildcard; available anywhere a name or numeric are required..Deep scan; available anywhere a name is required.<name>Dot-notated child['<name>' (, '<name>')]Bracket-notated child or children[<number> (, <number>)]Array index or indexes[start:end]Array slice operator[?(<expression>)]Filter expression; must evaluate to a boolean value????? ??? ??? ?? ? ? ????. ??? ?? ??? ?? ???? ?????. ?? ??? ?? ??? ?? ?????.FunctionReturnsOutputMin()The min value of an array of numbersDoublemax()The max value of an array of numbersDoubleavg()The average value of an array of numbersDoublestddev()The standard deviation value of an array of numbersDoublelength()The length of an arrayInteger?? ?????? ??? ??? ?? ? ???? ?? ??????. ???? ??? ??? ????.[?(@.age>18)]??? @? ???? ?? ??? ?????.?? ??? ??? ?? ??? &&? ||? ???? ?? ? ????.??? ???? [? (@. color == ‘blue’)] ?? [? (@. color == “blue”)]? ?? ?? ??? ? ? ???? ??? ???.FliterOperatorDescription==Left is equal in type and value to right (note 1 is not equal to '1')!=Left is not equal to right<Left is less than right<=Left is less than or equal to right>Left is greater than right>=Left is greater than or equal to right=~Left matches regular expression [?(@.name=~/foo.*?/i)]inLeft exists in right [?(@.size in ['S', 'M'])]ninLeft does not exist in rightsizeSize of left (array or string) matches rightemptyLeft (array or string) is empty??? ???? ??? ?? :{ "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99 }, { "category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99 }, { "category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ], "bicycle": { "color": "red", "price": 19.95 } }, "expensive": 10}JSON Path(click link to test)Result$.store.book[*].authorThe authors of all books$..authorAll authors$.store.*All things, both books and bicycles$.store..priceThe price of everything$..book[2]The third book$..book[0,1]The first two books$..book[:2]All books from index 0 (inclusive) until index 2 (exclusive)$..book[1:2]All books from index 1 (inclusive) until index 2 (exclusive)$..book[-2:]Last two books$..book[2:]Book number two from tail$..book[?(@.isbn)]All books with an ISBN number$..book[?(@.isbn)]All books with an ISBN number$.store.book[?(@.price<10)]All books in store cheaper than 10:math:..book[?(@.price<=`['expensive'])] <$..book%5B ?(@.price%3C=$%5B'expensive'%5D)%5D>`__All books in store that are not "expensive"$..book[?(@.author=~/.*REES/i)]All books matching a regex (ignore case)$..*All books$..book.length()The number of booksKeepKEEP ?? ?? ????? ??? ?? ???? ? ?????. ??? DROP ???? ?? ?????.??keep <column>[,<column>]column? ??? ???? ? ?????.??? ?? ????KEEP ???? ??? ??? ???? ??? ?? ???? ?? ?? ?? ????? ?????.?? ???? ??? ?? :{ "id": 1, "timestamp": 1234434343, "measurement": 10.45, "isvalid": true}? ??? ?? :keep id,measurement? ???? ??? :{ "id": 1, "measurement": 10.45}Mask NumberMASK-NUMBER ???? ? ?? ?? ???? ?????.??mask-number <columm> <pattern>??? <column> ??? ? ?? ?? ??? ?????.??? <pattern> ? ?? ????? ? ??? ?? ?????.??? ?? ?????? ???? ????? ?? ?? ?? ?? ?? ?? ???? ?????. MASK-NUMBER? ? ?? ?? ???? ?????. ? ??? ??? ??? ?? ?? ???? ???? ?? ??????.??? ??? ??? ?????.#? ???? ?? ??? ??? ?????.x ?? ?? ??? ???? ?? ??? ??? ??? ????.??? ???? ??? ?? :{ "first": "Root", "last": "Joltie", "ssn": "000-00-0000", "cc": "4929790943424701"}? ??? ?? :mask-number ssn XXX-XX-####? ???? ??? :{ "first": "Root", "last": "Joltie", "ssn": "XXX-XX-0000", "cc": "4929790943424701"}? ??? ?? :mask-number cc XXXXXXXXXXXX####? ???? ??? :{ "first": "Root", "last": "Joltie", "ssn": "000-00-0000", "cc": "XXXXXXXXXXXX4701"}Mask ShuffleMASK-SHUFFLE ???? ? ?? ?? ???? ?????.??mask-shuffle <columm>??? <column> ??? ? ?? ?? ??? ?????.??? ?? ????MASK-SHUFFLE ???? ??? ?? ?? ??? ???? ?? ??? ???? ?? ? ?? ?? ???? ?????. ? ??? ??? ?? ?? ??????. ??? ???? ?? ?? ???? ??? ??? ?????.??, ?? ? ??? ?? ?? ?? ??? ?? ??? ?????.??? ???? ??? ?? :{ "first": "Root", "last": "Joltie", "ssn": "000-00-0000", "cc": "4929790943424701"}?? ??? ?? :mask-shuffle firstmask-shuffle lastmask-shuffle ssnmask-shuffle cc? ???? ??? :{ "first": "Buek", "last": "Bumkyy", "ssn": "089-75-3119", "cc": "0897531194773254"}MergeMERGE ???? ? ?? ?? ???? ???? ? ?? ?????. ? ?? ?? ?? ??? ?? ??? ?? ? ? ?? ?? ? ????.??merge <first> <second> <new column> '<seperator>'??? <first> ? <second> ? ?? ?? ??? ???? ?????. ?? ? ?? ?? ??? ?? ??? ????? ??? ????? ???.??? <new-column> ???? ?? ? ??? ????. ?? ?? ??? ??? ?????.??? <separator> ? ?? ?? ???? ? ??? ?? ?? ??????. ?? ??? ??? ?????. ? : ?? ?? : ‘ ’;.? ???? ?? ?? ???? ????.??? ?? ?????? ? ?? ?? ?? ?????? ???.??? ???? ??? ?? :{ "first": "Root", "last": "Joltie"}?? ??? ?? :merge first last fullname ' 'merge first last fullname '''merge first last fullname '\u000A'merge first last fullname '---'?? ???? ????.?? ??? ?? ?? ?? (‘ ’)???.{ "first": "Root", "last": "Joltie", "fullname": "Root Joltie"}?? ??? ?? ??? ?? (‘’’)???.{ "fname" : "Joltie", "lname" : "Root", "name" : "Joltie'Root"}?? ??? UTF-8 ? ?? ?? ( ‘\ u000A’)???.{ "fname" : "Joltie", "lname" : "Root", "name" : "Joltie\nRoot"}?? ??? ?? ?? ( ‘---’)???.{ "fname" : "Joltie", "lname" : "Root", "name" : "Joltie---Root"}Parse AVRO FilePARSE-AS-AVRO-FILE ???? AVRO ??? ??? ?????. AVRO ??? ??? ??? ??? ??? ?? (???)? ???? ?? ????? ?????? AVRO ??? ??? ??? ???? ???? ????. AVRO ???? JSON ????, ???? ?? ???? ?? ?? ?? ??????.??parse-as-avro-file <column>??? <column> binary octet stream ? AVRO ??? ??? ?? ??? ?????.??? ?? ????? ???? ???? AVRO ??? ??? ???? ?? ??? ???? ??? ?????.INT, FLOAT, DOUBLE, STRING, SHORT?? ??? ??? ???? ? ??? AVRO ???? ? ??????.??? ??? ??? ? ??, ?? ??? ???? ???? ??? ?? ?? ??? ?????.Parse AVRO Encoded MessagesPARSE-AS-AVRO ???? ???? ?? json AVRO ??? ?? ??? ???? ???? ?????. ? ???? Schema Registry? ??? ???? ??? ?????.??parse-as-avro <column> <schema-id> <json|binary> [<version>]??? <column> id? ??? Schema Registry ? ?? ? ???? ???? whoes ?? ??? ? ?? ?????. <schema-id> . ????? ?? <version> ??? ???? ??? ??? ? ????.??? ?? ????PARSE-AS-XML ???? ????? ??? ???? XML ??? ????? ???? ???? ?? ???? ???? ?? ? ? ????.Parse as CSVPARSE-AS-CSV? ?? ???? ??? ??? ??? ???? ?? ??????.??parse-as-csv <column> <delimiter> [<header=true|false>]??? <column> ??? ??? ???? CSV? ????? ?? ???? ?? ?????. <delimiter> . ?? ? ?? ???? ?? ? ???? ???? ?? ?? <header> ???; ????? false? ?????.????? ?? CSV ??? ? ?? ???????. CSV ??? ? ?? ???? ?????.{ "body": "07/29/2013,Consumer Loan,Vehicle Loan,Managing the loan or lease,,,,Wells Fargo & Company,VA,24540,,N/A,Phone,07/30/2013,Closed with explanation,Yes,No,468882"}? ??? ?? :parse-as-csv body ,? ???? ??? :{ "body": "07/29/2013,Consumer Loan,Vehicle Loan,Managing the loan or lease,,,Wells Fargo & Company,VA,24540,,N/A,Phone,07/30/2013,Closed with explanation,Yes,No,468882", "body_1": "07/29/2013", "body_2": "Consumer Loan", "body_3": "Vehicle Loan", "body_4": "Managing the loan or lease", "body_5": null, "body_6": null, "body_7": "Wells Fargo & Company", "body_8": "VA", "body_9": "24540", "body_10": null, "body_11": "N/A", "body_12": "Phone", "body_13": "07/30/2013", "body_14": "Closed with explanation", "body_15": "Yes", "body_16": "No", "body_17": "468882"}? ?? ???? ??? ???? ? ???? ???? ?? ?? ??? ????.[ { "body": "Date,Type,Item,Action,Company" }, { "body": "07/29/2013,Consumer Loan,Vehicle Loan,Managing the loan or lease,Wells Fargo & Company" }]? ??? ?? :parse-as-csv body , true? ???? ??? :{ "body": "07/29/2013,Consumer Loan,Vehicle Loan,Managing the loan or lease,Wells Fargo & Company" "Date": "07/29/2013", "Type": "Consumer Loan", "Item": "Vehicle Loan", "Action": "Managing the loan or lease", "Company": "Wells Fargo & Company"}Parse as DatePARSE-AS-DATE ???? ?? ?? ??? ???? ??? ???? ?? ????.??parse-as-date <column> [<time-zone>]??? ?? ????PARSE-AS-DATE ???? ?? ?? ?? ? ?? ??? ???? ?? ?? ??? ???? ?? ?? ?? ? ?? ??? ?????.?? ?? (02/28/1979), ???? ?? (oct 1st), ?? ?? (?? ??? ??) ? ?? ?? (?? ??? ?? ???) ? ?? ?? ???? ?? ? ??? ?????.? ???? ? ??? ?? ?? ?? ?? ??? ???? ??? ???? ??? ????.??? ???? ? ?? ????. <column> _1? ?? ??? ?????.?? <time-zone> ? ???? ??? UTC? ???? ?????.??? ???? ??? ?? :{ "create_date": "now",}? ??? ?? :parse-as-date create_date US/Eastern? ???? ????? (?? ?? ??? ? ????? ?? ????).{ "create_date": "now", "create_date_1": "Mon May 01 14:13:35 EDT 2017"}Parse as ExcelPARSE-AS-EXCEL? XLS, XLSX ??? ???? ?????.??parse-as-excel <column> <sheet number> | <sheet name>??? <column> Excel? ?? ? ???? ?? ?????. ??? <sheet number> ?? <sheet name> ????? Excel ?? ?? ??? ?????.Parse as Fixed LengthPARSE-AS-FIXED-LENGTH ???? ??? ? ??? ??? ?? ?? ?? ???? ?? ?????.??parse-as-fixed-length <column> <width>[,<width>]* [<padding>]??? ?? ?????? ?? ??? ??? ? ??, ?? ?? ? ?? ?? ??? ??? ??? ??? ??? ??? ??? ?????. ? ??? ?? ??? ?????.?? ?? ??? ??? ? ?? ?? ?? ??? 10 ?? ???? ?? ?? ? ?? ?? ??? 5?? ? ?? ?? ??? 12 ????. ??? ?? ?? ??? ??? ?????.???? ?? ?? <padding> ??? ?? ??? ?????.?? ???? ??? ?? :{ "body": "12 10 ABCXYZ"}? ??? ?? :parse-as-fixed-length body 2,4,5,3? ???? ??? :{ "body": "12 10 ABCXYZ", "body_1": "12", "body_2": " 10", "body_3": " ABC", "body_4": "XYZ}Parse as HL7PARSE-AS-HL7 ???? ?? ?? 7 ?? 2 (HL7 V2) ???? ???? ?? ????. HL7? ????? ???? ???? ?? ?? ???? ???? ??? ?????.??parse-as-hl7 <column>??? <column> HL7 V2 ???, v2.1?? v2.6? ?????.??? ?? ??????? ???? ?? HL7 V2???? ????.MSH|^~\&||.|||199908180016||ADT^A04|ADT.1.1698593|P|2.6PID|1||000395122||LEVERKUHN^ADRIAN^C||19880517180606|M|||6 66TH AVE NE^^WEIMAR^DL^98052||(157)983-3296|||S||12354768|87654321NK1|1|TALLIS^THOMAS^C|GRANDFATHER|12914 SPEM ST^^ALIUM^IN^98052|(157)883-6176NK1|2|WEBERN^ANTON|SON|12 STRASSE MUSIK^^VIENNA^AUS^11212|(123)456-7890IN1|1|PRE2||LIFE PRUDENT BUYER|PO BOX 23523^WELLINGTON^ON^98111|||19601||||||||THOMAS^JAMES^M|F|||||||||||||||||||ZKA535529776HL7 V2 ???? ???? ??? ?? ????? ?????.??? = {????, ????, ... ????}???? = {??, ??, ... ??}?? = {?? ??, ?? ??, ... ?? ??}Component = {?? ?? ??, ?? ?? ??, ... ?? ?? ??}HL7 V2 ???? ? ?? ???????. ????? ??? ??? ?????. ????? ?? ? ??? ???? ??? ?????. ?? ????? ?? ?? ????? ????. MSH (??? ??); PID (?? ??); ? ?? NK1 (??) ????; ? IN1 (??).? ????? ??? ?????. ???? ????? ??? ??? ?? (? : IN1 (??) ????? ?? ?? ??)? ?? ????. ??? ????? (??? ????) |? ?????. ???.??? ?? ??? ?? ? ????. ?? ??? ????? ^ ??? ?????. ?? ??? PID (?? ??) ?????? ? (LEVERKUHN)? ?? ? ???? ??? LEVERKUHN ^ ADRIAN ^ C? ?? ? ?? ?? ??? ????. ?? (ADRIAN); ? ?? ?? ?? (C). ?? ??? ?? ?? ??? ?? ? ????. ?? ?? ??? ????? & ??? ?????.???? ???? ?? ???? ???? HL7 V2 ?? ??? ??? ???? ?? ??? JSON?? ??? ???? ?????. ?? ??? PARSE-AS-JSON???? ???? ??? ? ????.??? ???? ? ???? ?????.parse-as-hl7 body? ??? ???? ???.{ "body": "<original-hl7-message>", "body_hl7_MSH_<field-index>": "<primitive-value>", ... "body_hl7_MSH_<field-index>_<component-index>": "<primitive-value>", ... "body_hl7_PID_<field-index>": "<primitive-value>", ... "body_hl7_PID_<field-index>_<component-index>": "<primitive-value>", ... "body_hl7_NK1_<field-index>>": "<primitive-value>", ... "body_hl7_NK1_<field-index>_<component-index>": "<primitive-value>", ... "body_hl7_IN1_<field-index>>": "<primitive-value>", ... "body_hl7_IN1_<field-index>_<component-index>": "<primitive-value>", ...}? ????? JSON?? ???? ???? PARSE-AS-JSON?? JSON-PATH???? ??? ? ????.Parse as JSONPARSE-AS-JSON ???? JSON ??? ???? ?? ??????. ? ???? String ?? JSONObject ???? ??? ? ????. ???? ???? JSON? ?? ?? ?? ?? ? ??? ???? ? ? ??? ??????.??parse-as-json <column-name> [<depth>]<column-name> JSON ?? ? ???? ? ?????.<depth> JSON ?? ??? ???? ??? ?????.??? ?? ????PARSE-AS-JSON ???? ??? JSON? ? ?? ???? ?? ? ? ?? ???? ?????. JSON ??? ?? ??? ? ?? ??? ?????. ? ?? ? ??? PARSE-AS-JSON ???? ?? ???? ? ?? ??? ? ?? JSON ?? ? ? ????.??? ??? ? ??? JSON ??? ???? ?? ?????. ? ??? ? ???? ?????.???? ???? ? ???? ??? ???? ??? ????.{ "id": 1, "name": { "first": "Root", "last": "Joltie" }, "age": 22, "weigth": 184, "height": 5.8}? ??? ?? :parse-as-json body? ???? ??? :Field NameField ValuesField Typebody{...}Stringbody_id1Integerbody_name{"first": "Root", "last": "Joltie" }JSONObjectbody_age22Integerbody_weight184Integerbody_height5.8Double??? ???? ????? body_name ????? ? ???? ???.Field NameField ValuesField Typebody{...}Stringbody_id1Integerbody_name{"first": "Root", "last": "Joltie" }JSONObjectbody_name_firstRootStringbody_name_lastJoltieStringbody_age22Integerbody_weight184Integerbody_height5.8DoubleParser as LogPARSE-AS-LOG ???? Apache HTTPD ? nginx ??? ?? ??? ?? ??? ?????.??parse-as-log <column> <format>??? ?? ????PARSE-AS-LOG ???? ??? ???? ???? ?? ?? ??? ?????. <format> ?? ?? ?? ??? ??? ???. ?? ?? ?? ??? ???? ??? ???? ??? ??? ?? ?? ???? ???.Apache HTTPD ????? Custom Log Formats? ??? ?? ?? ?? ?? ??? ?????.???????????????? : ?? ? ?? ?? ?? ?? ?? ??? ????? ??? ??? ? ????.parse-as-log body combinedparse-as-log body common???? ?? ??? ??? ?? ?? ??? ?? ??? ??? ? ????. ? ?? format? ?? ?? ??? ???? ?? ??????.nginx? ?? ?? ??? log_format? embedded variables? ?????.???? ??? ??? ??? ????.%h %l %u %t "%r" %>s %b??? ?? ?? ??? ??? ?? ???? ???? ?? ???? ??????.{ "body": "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326"}?? ??? ? ??? ??????.parse-as-log body %h %l %u %t "%r" %>s %bparse-as-log body common? ???? ??? :{ "ip_connection_client_host": "127.0.0.1", "ip_connection_client_host.last": "127.0.0.1" "number_connection_client_logname": null, "number_connection_client_logname.last": null, ... ... "http_path_request_firstline_uri_path": "/apache_pb.gif", "http_ref_request_firstline_uri_ref": null}?? ? ?? ?? ?? :%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\? ?? ?? ?? :127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "" "Mozilla/4.08 [en] (Win98; I ;Nav)"??? ?? ??? ??? ??? ? ????.%t %u [%D %h %{True-Client-IP}i %{UNIQUE_ID}e %r] %{Cookie}i %s \"%{User-Agent}i\" \"%{host}i\" %l %b %{Referer}i??? ??? ?? ? ?? (?? : ? ??? ?? ???) :[03/Dec/2013:10:53:59 +0000] - [32002 10.102.4.254 195.229.241.182 Up24RwpmBAwAAA1LWJsAAAAR GET/content/dam/Central_Library/Street_Shots/Youth/2012/09sep/LFW/Gallery_03/LFW_SS13_SEPT_12_777.jpg.image.W0N539E3452S3991w313.original.jpg HTTP/1.1] __utmc=94539802; dtCookie=EFD9D09B6A2E1789F1329FC1381A356A|_default|1; dtPC=471217988_141#_load_; Carte::KerberosLexicon_getdomain=6701c1320dd96688b2e40b92ce748eee7ae99722; UserData=Username%3ALSHARMA%3AHomepage%3A1%3AReReg%3A0%3ATrialist%3A0%3ALanguage%3Aen%3ACcode%3Aae%3AForceReReg%3A0; UserID=1375493%3A12345%3A1234567890%3A123%3Accode%3Aae; USER_DATA=1375493%3ALSharma%3ALokesh%3ASharma%3Alokesh.sharma%%3A0%3A1%3Aen%3Aae%3A%3Adomain%3A1386060868.51392%3A6701c1320dd96688b2e40b92ce748eee7ae99722; MODE=FONTIS; __utma=94539802.911097326.1339390457.1386060848.1386065609.190; __utmz=94539802.1384758205.177.38.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); __kti=1339390460526,http%3A%2F%2F%2F,;__ktv=28e8-6c4-be3-ce54137d9e48271; WT_FPC=id=2.50.27.157-3067016480.30226245:lv=1386047044279:ss=1386046439530; _opt_vi_3FNG8DZU=42880957-D2F1-4DC5-AF16-FEF88891D24E; __hstc=145721067.750d315a49c64268192826b3911a4e5a.1351772962050.1381151113005.1381297633204.66; hsfirstvisit=http%3A%2F%2F%2F|http%3A%2F%2Fgoogle.co.in%2Furl%3Fsa%3Dt%26rct%3Dj%26q%%26source%3Dweb%26cd%3D1%26ved%3D0CB0QFjAA%26url%3Dhttp%3A%2F%2F%2F%26ei%3DDmuSULW3AcTLhAfJ24CoDA%26usg%3DAFQjCNGvPmmyn8Bk67OUv-HwjVU4Ff3q1w|1351772962000; hubspotutk=750d315a49c64268192826b3911a4e5a; __ptca=145721067.jQ7lN5U3C4eN.1351758562.1381136713.1381283233.66; __ptv_62vY4e=jQ7lN5U3C4eN; __pti_62vY4e=jQ7lN5U3C4eN; __ptcz=145721067.1351758562.1.0.ptmcsr=google|ptmcmd=organic|ptmccn=(organic)|ptmctr=; RM=Lsharma%3Ac163b6097f90d2869e537f95900e1c464daa8fb9; wcid=Up2cRApmBAwAAFOiVhcAAAAH%3Af32e5e5f5b593175bfc71af082ab26e4055efeb6; __utmb=94539802.71.9.1386067462709; edge_auth=ip%3D195.229.241.182~expires%3D1386069280~access%3D%2Fapps%2F%2A%21%2Fbin%2F%2A%21%2Fcontent%2F%2A%21%2Fetc%2F%2A%21%2Fhome%2F%2A%21%2Flibs%2F%2A%21%2Freport%2F%2A%21%2Fsection%2F%2A%21%2Fdomain%2F%2A~md5%3D5b47f34172392487dcd44c1d837e2e54; has_js=1; SECTION=%2Fcontent%2Fsection%2Finspiration-design%2Fstreet-shots.html;JSESSIONID=b9377099-7708-45ae-b6e7-c575ffe82187; WT_FPC=id=2.50.27.157-3067016480.30226245:lv=1386053618209:ss=1386053618209; USER_GROUP=LSharma%3Afalse; NSC_wtfswfs_xfcgbsn40-41=ffffffff096e1a1d45525d5f4f58455e445a4a423660 200 "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)""" - 24516 as Simple DatePARSE-AS-SIMPLE-DATE ???? ?? ???? ?????.??parse-as-simple-date <column> <pattern>??? ?? ????PARSE-AS-SIMPLE-DATE ???? ?? ???? ???? ?? ???? ?????. ?? null??? ?? ??? ??? ?? ? ???? ???? ?? ?? ??? ????. ??? ??? ?? string ????? ???.????? ?? ???? ?? ?? ? ?? FORMAT-DATE? ??????.Parse as XMLPARSE-AS-XML ???? XML ??? ?????. ? ???? XML VTD ??? ???? ?? ??? ?????. XPATH ???? ???? ??? ??? ? ????.??parse-as-xml <column>??? <column> XML. ?? ???? ?? ?????.??? ?? ????PARSE-AS-XML ???? ????? ??? ???? XML ??? ????? ???? ???? ?? ???? ???? ?? ? ? ????.Parse XML to JSONPARSE-XML-TO-JSON ???? XML ??? JSON ??? ?????. ???? string ??? ?? ??? ?????. ? ???? ???? XML? JSON ??? ????? PARSE-AS-JSON???? ???? ??? ???? ? ????.??parse-xml-to-json <column-name> [<depth>]<column-name> ? ????? XML??? ?? ?????.<depth> XML ?? ??? ??? ???? ?? ??? ?????.??? ?? ????PARSE-XML-TO-JSON ???? XML ??? ????? ???? ?? ??? ?? JSON ??? ?????.XML. ??? ??, ?? W ?? ???? ????. ??? ??? ??? JSON ??? ??? PARSE-AS-JSON???? ???? ??? ??? ? ????.???? ?? ??, ????, DTD ? <[[]]> ???? ?????.QuantizeQUANTIZE ???? ??? ??? ???? ?? ??????.??copy <source> <destination> <range_lower1:range_upper1=value>[,<range_lower2:range_upper2=value>*]??? <source> column? ???? ????.??? <destination> column? ??? ??? ???? ????QUANTIZE ???? <source> column.??? ?? ????QUANTIZE ???? <source> ?? <destination> column.??? ??? ?? ??? ???? <value> ? <destination> ?? ?? ?? ? ??? ???? ?? ????.?? ?? ???? ??? ??? ?????.?? ??? ??? ?? ?? ??? ?? ?? ???? ?????.?? <destination> ?? ?? ??? ?? ???? ?? ?? ?? ????? ?????.? ?? <source> ?? ???? ???.??? <value> ??? ??? ?? ?? ???? ? ????.???? ???? ??? ?? :[ { "body": 1 }, { "body": 2 }, { "body": 3 }, { "body": 4 }, { "body": 5 },]? ??? ?? :quantize body body_q 1:2=20,3:4=40,5:10=max?? ???? ????.[ { "body": 1, "body_q": 20 }, { "body": 2, "body_q": 20 }, { "body": 3, "body_q": 40 }, { "body": 4, "body_q": 40 }, { "body": 5, "body_q": "max" },]RenameRENAME ???? ???? ?? ?? ??? ????.??rename <old> <new><old> ??? ?? ?? ?? ?????.<new> ?? ? ?????.??? ?? ????RENAME ???? ??? ? ??? ? ???? ?? ??? ????. ? ?? ?? ???? ??? ??? ?? ? ??? ? ?? ????? ??? ? ????.RENAME ???? ???? ?? ?? ? ????. ???? ? ??? ??? ??? ???? ?????.?? ???? ??? ?? :{ "x": 6.3, "y": 187, "codes": { "a": "code1", "b": 2 }}?? ??? ?? :rename x heightrename y weight? ???? ??? :{ "height": 6.3, "weight": 187, "codes": { "a": "code1", "b": 2 }}Send to ErrorSEND-TO-ERROR ???? ??? ???? ???? ??? ??? ???? ???? ???? ?? ???(error collector)? ????. ?? ???? ??? ??? ?? ??? ???? ?? ??? ??? ? ???? ?????.??send-to-error <condition>??? <condition> ???? ?? ???? ????? ??? ??? ???? ??? ???? JEXL ??????.??? ?? ????SEND-TO-ERROR ???? ?? ???? ??? ?? ???? ??? ?? ???? ????? ????. ? ??? ??? ??? ???? ?? ???? ???? ??? ?? ?????.???? ??? ??? ???? ???? ?? ??? (??? ??)? ?????. ??? false? ???? ???? ???? ?? ?????.??? ? ??? ?? ???? ?????.?????? (DOB)??? ?? ????? ??? ??? ?? ???? ?? ??? ????? ??????.??? ?? ?? ????.??? ?? ?? ?? 1 ?? 130??DOB? ??? ?????.??? ???? ??? ??? ?????. ??? ?? ? ??? ???? ???? ?? ??? ?? ?? ???? ?????.send-to-error Name == nullsend-to-error Age.isEmpty()send-to-error Age < 1 || Age > 130send-to-error !date:isDate(DOB)Set CharsetSET-CHARSET ???? ?? ???? ???? ??? ?? ?? UTF-8 ???? ?????.??set-charset <column> <charset>column? ?? ? ?? ?????.charset? ?? ??? ???? ??? ????.??? ?? ????? ???? ?? ?? ??? charset?? ?????. ?? charset? ???? ?? ????? ?? UTF-8 String?? ?????.Set ColumnSET-COLUMN ???? ? ?? ??? ?? ??? ?????.??set-column <columm> <expression>??? <column> ?? ??? ?????. ?? ?? ??? ?? ??? ???? ??? ?? ?????. ?? ??? ??? ?? ??? ?? ? ?? ??????.??? <expression> ??? Apache Commons JEXL expression???.??? ?? ????SET-COLUMN ???? ? ?? ??? ?? ??? ?????.???? Apache Commons JEXL????? ?????.math : ceil?? string : upperCase? ?? ?? ????? ??? ?? ?? ???? ?? ?? ????? ?? (? : ??? ? ??)? ?? ? ? ????.??? ???? ??? ?? :{ "first": "Root", "last": "Joltie", "age": "32", "hrlywage": "11.79",}?? ??? ?? :set-column name concat(last, ", ", first)set-column is_adult age > 21 ? 'yes' : 'no'set-column salary hrlywage*40*50set column raised_hrlywage var x; x = math:ceil(FLOAT(hrlywage)); x + 1set column salutation string:upperCase(concat(first, ' ', last))? ???? ??? :{ "first": "Root", "last": "Joltie", "age": "32", "hrlywage": "11.79", "name": "Joltie, Root", "is_adult": "yes", "salary": 23580.0, "raised_hrlywage": 13.0, "salutation": "ROOT JOLTIE"}Set ColumnsSET COLUMNS ???? ?? ??? ??? ???? ?????.??set columns <columm>[,<column>*]??? <column> ?? ?? ? ??? ?????.??? ?? ????SET COLUMNS ???? ?? ???? ??? CSV ??? ??? ? ?? ??? ???? ????. ? ??? ? x ?? ?????? ???? ??? ???? ???? ?????.??? ???? ??? ?? :{ "body": "1,2,3,4,5"}PARSE-AS-CSV???? ???? ??? ??? ?? :parse-as-csv body , false?? ???? ??? ????.{ "body": "1,2,3,4,5", "body_1": "1", "body_2": "2", "body_3": "3", "body_4": "4", "body_5": "5"}SET COLUMNS ???? ???? ??? ????.set columns a,b,c,d,e??? ?? ??? ?? ? ??? ?? ???? ?????.{ "a": "1,2,3,4,5", "b": "1", "c": "2", "d": "3", "e": "4", "body_5": "5"}??? ?? (body_5)?? ??? ??? ???? ?????.? ??? ????? ??? ???? ?? ?? ??? ?????.parse-as-csv body , falseset columns body,a,b,c,d,e? ???? ?? :{ "body": "1,2,3,4,5", "a": "1", "b": "2", "c": "3", "d": "4", "e": "5"}?? DROP???? ??????.parse-as-csv body , falsedrop bodyset columns a,b,c,d,e??? ??? ????.{ "a": "1", "b": "2", "c": "3", "d": "4", "e": "5"}?? ??SET COLUMNS ???? ??? ? ???? ?? ?? SET COLUMNS ???? ? ?? ?? ???? ???. ??? ?? ?? ? ???? ????? ??? ??? ??? ?????.?? ??? ??? ?? ? ?? ??? ????. ???? ???? ? ?? ?? ?? ? ?????.? ???? ??? ???? ???? ?? ? “Name of field”??? *? ??? ??, ?? ??? ???? ???? ?? ??? ??? ??? ?????. ?? ??????? ?? ? ??? ?? ???? ?????.Set Record DelimSET-RECORD-DELIM ???? ??? ?? ??? ?????.??set-record-delim <column> <delimiter> [<limit>]??? ?? ????? ???? ??? ?? ?? ( <delimiter> )? ???? ?? ???? ?????. <column> . ????? ?? ( <limit> )? ???? ?? ??? ??? ? ???? ??? ?? ?? ? ? ????.Set Type?? ??? ?? ????set-type <column> <type>??? <column> ???? ?????. <typpe> . ??? ? ?? ??? int, short, long, float, double, string, bytes, boolean???.Set a transient variableSET-VARIABLE ???? ???? ?? ???? ?? ??? ?? ??????.??set-variable <variable> <expression>??? <variable> ~? ??? ?????. <expression>.??? ?? ????? ???? ???? ???? | ? ???? ?????. ?? ???? ? ???? ??? ???? ???? ??? ??????.Split By SeparatorSPLIT-BY-SEPARATOR ???? ?? ??? ???? ?? ?? ? ?? ????.? ?? ???? ?? ???? SPLIT-TO-COLUMNS???? ??????.Split EmailSPLIT-EMAIL ???? ?? ?? ID? ??? ?? ????? ?????.??split-email <column>??? <column> ??? ??? ??? ????.??? ?? ????SPLIT-EMAIL ???? ?? ?? ??? ?? ? ???? ?? ?? ??? ?????.?? ?? ??? ??? ? ???? ? ?? ? ?? ??? ?? ? ??? ?????.column_accountcolumn_domain?? ?? ??? ???? ??? ? ?? ?? ?? ?? ??? ????? ??? ? ?? ??? ?? null? ?????.??? ???? ??? ?? :{ "name": "Root, Joltie", "email_address": "root@",}? ??? ?? :split-email email_address? ???? ??? :{ "name": "Root, Joltie", "email_address": "root@", "email_address_account": "root", "email_address_domain": ""}?? ??? ?? ?? : ???? ?? ?? ?? ??? null ? ?? :{ "email": null}? ??? ??? ????.{ "email": null, "email_account": null, "email_domain": null}??? ???? ??? ???? ??? ??? ID? ?????.[ { "email": "root@" }, { "email": "joltie.xxx@" }, { "email": "joltie_xxx@" }, { "email": "joltie.'@.'root.'@'.@" }, { "email": "Joltie, Root <joltie.root@>" }, { "email": "Joltie,Root<joltie.root@>" }, { "email": "Joltie,Root<joltie.root@" }]???? ???? ?? ???? ?????.[ { "email": "root@", "email_account": "root", "email_domain": "cask.co" }, { "email": "joltie.xxx@", "email_account": "joltie.xxx", "email_domain": "" }, { "email": "joltie_xxx@", "email_account": "joltie_xxx", "email_domain": "" }, { "email": "joltie.'@.'root.'@'.@", "email_account": "joltie.'@.'root.'@'.", "email_domain": "" }, { "email": "Joltie, Root <joltie.root@>", "email_account": "joltie.root", "email_domain": "" }, { "email": "Joltie,Root<joltie.root@>", "email_account": "joltie.root", "email_domain": "" }, { "email": "Joltie,Root<joltie.root@", "email_account": null, "email_domain": null }]Split to ColumnsSPLIT-TO-COLUMNS ???? ?? ???? ???? ?? ?? ?? ?? ????.??split-to-columns <column> <regex>??? <column> ??? ???? ???? ?? ??? ?? ?????. <regex> .??? ?? ????SPLIT-TO-COLUMNS ???? ?? ??? ?? ??? ?? ??? ??? ?? ???? ?? ?? ????. ?? ??? ?? ?????? ????.{ "column": "...", "column_1": "...", "column_2": "...", "column_3": "...", ... "column_n": "..."}???? ???? ?? ???? ??? ? ??? ?? ??? ??? ? ????. ?? Java regular expression? ?????.?? ?? ?? ??? ? ?? ?? ??? ?? ?????. column_1, column_2, through_column_n?? ?? ?? ???? ??? ??? n ??? ???? ? ????.?? :? ???? ??? ??? ?? ???? ??? ? ????.????? <separator> ???? ?? (??) :??? ?? ? ??? 1, ??? ???? 2, ??? ???? 3, ??? 4????? ? ?? ??? ?? ?????.{ "1": "This will be split 1", "2": "This will be split 2", "3": "This will be split 3", "4": "Split 4"}? ???? ??? ?? :{ "id": 1, "codes": "USD|AUD|AMD|XCD"}? ??? ?? :split-to-columns codes \|?? : ??? ?? (|)? ?? ??? ??? ??? ?? ????? ????? ??? ? ???? ?????.??? 4 ?? ?? ???? ? ???? ? ??? ?????.{ "id": 1, "codes": "USD|AUD|AMD|XCD", "codes_1": "USD", "codes_2": "AUD", "codes_3": "AMD", "codes_4": "XCD"}Split to RowsSPLIT-TO-ROWS ???? ?? ??? ???? ?? ?? ?? ???? ???? ???? ?? ? ? ? ?? ?? ??? ??? ????.??split-to-rows <column> <separator>??? <column> ? ???? ????. <separator> ??? ?? ?? ? ? ????.??? ?? ????SPLIT-TO-ROW ???? ?? ??? ?? ??? ?? ??? ??? ?? ???? ?? ?? ????.???? ???? ?? ???? ?? ? ? ??? ?? ??? ??? ? ????. ?? Java ??? ??? ?????.?? <separator> ?? ??? ??? ?? ???? ??? ???? ??? ?? ?? ???? ????.???? ??? ?? ??? ?? ???? n ?? ???? ???? ?? ?? ???? ?? ? n ?? ?? ?????.?? :? ???? ??? ??? ?? ???? ??? ? ????.?????? <separator> ???? ?? \ n (? ??) ?? :??? ????? 1 \ n ????? 2??? ?? ?? ?? ?? ?? ? ?? ?? ???? ?????.[ { "column": "This will be split 1" }, { "column": "This will be split 2" }]? ???? ??? ?? :{ "id": 1, "codes": "USD|AUD|AMD|XCD",}? ??? ?? :split-to-rows codes \|?? : ??? ?? (|)? ?? ??? ??? ??? ?? ????? ????? ??? ? ???? ?????.??? ?? 4 ?? ???? ???? ? ?? ?? ? ??? ?????.[ { "id": 1, "codes": "USD" }, { "id": 1, "codes": "AUD" }, { "id": 1, "codes": "AMD" }, { "id": 1, "codes": "XCD" }]Split URLSPLIT-URL ???? URL? ????, ??, ???, ??, ??, ?? ?? ? ??? ?????.??split-url <column>??? <column> URL? ???? ????.??? ?? ????SPLIT-URL ???? URL? ?? ??? ?????. URL? ?? ? ? ???? ?? ? ??? ???? 7 ?? ? ?? ????.column_protocolcolumn_authoritycolumn_hostcolumn_portcolumn_pathcolumn_filenamecolumn_queryURL? ???? ??? ? ?? ?? ??? throw???. URL ?? ??? null ?? ?? ?? ???? ?????.??? ???? ??? ?? :{ "url": ""}? ??? ?? :split-url url? ???? ??? :{ "url": "", "url_protocol": "http", "url_authority": ":80", "url_host": "", "url_port": 80, "url_path": "/docs/books/tutorial/index.html", "url_filename": "/docs/books/tutorial/index.html?name=networking", "url_query": "name=networking"}???? URL ??? null ? ?? :{ "url": null}???? ??? ?????.{ "url": null, "url_protocol": null, "url_authority": null, "url_host": null, "url_port": null, "url_path": null, "url_filename": null, "url_query": null}StemmingSTEMMING ???? ?? ??? Porter ??? ????? ?????. Porter ???? ??, ??? ? ??? ???? ??? ??? ?????. ??? ?? (??)? ???? ????? ?????. ????? ? 60 ?? ??? 6 ??? ?????.??stemming <column>??? <column> ??? ?? ?? ?? ??? ?? ??? ?? ?? ?????.??? ?? ????STEMMING ???? ??? ? ?? ??? ??? ?????. ? ???? ???? <column> _porter ?. ??? ???? ?? ??? ??? ?? ???? ?????.?? ???? ???? ??? ?? ?? ??? ???? ??? ? ?? ??? ?? ???.{ "word" : { "how are you doing ? do you have apples ?" }}? ??? ?? :stemming word??? ??? ????.{ "word": { "how are you doing ? do you have apples ?" }, "word_porter": { "how", "ar", "you", "do", "do", "you", "have", "appl" }}SwapSWAP ???? ? ?? ? ??? ?? ????.??swap <column1> <colum2>??? ?? ????SWAP ???? ?? ??? <column1> ~? ???? <column2> ? <column2> ~? ???? <column1> . ? ?? ? ??? ???? ???, ???? ??? ?????.?? ???? ??? ?? :{ "a": 1, "b": "sample string"}?? ??? ? ??? ??????.swap a bswap b a? ???? ??? :{ "b": 1, "a": "sample string"}Table LookupTABLE-LOOKUP ???? Table ??? ??? ?? ??? ?????.??table-lookup <column> <table><column> ?? ???? ???? ???? ?? ????.<table> ? ?? ?? ? ?? ?? ??? ??? ?????. <column>??? ?? ????TABLE-LOOKUP ???? ?? ?? ?? ???? ? ???? ?? ??? ??? ??? ??? ?????. ?? ??? string??????. ?? ?? ?? ?? ???? ?? ???? ?? ?? ? ? ??? ??? ???? ?? ? ???? ? ?? ???? ?????.?? ?? ???? customerTable ??? ??? ?? ??? ?????. ??? ???? ?? ?? customerUserId? ?? ?? ?????.? ???? ??? customerTable? ??? ?????.CustomerUserIdCitybobisthemanPalo Alto, CAjoe1984Los Angeles, CArandomUserqwertyNew York City, NY???? ?? ?? ???? ??? ?? ?? :CustomerUserIdProductQuantitybobisthemanApples10joe1984Bicycle1? ??? ?? :table-lookup customerUserId customerTable??? ?? ?? ???? ?????.CustomerUserIdProductQuantityCustomerUserId_CitybobisthemanApples10Palo Alto, CAjoe1984Bicycle1Los Angeles, CAText DistanceTEXT-DISTANCE ???? ??? ??? ??? ???? ??? ??? ???? ? ?? ??? ?? ??? ?????.??text-distance <method> <column-1> <column-2> <destination><method> ??? ??? ??? ???? ? ??? ???? ?????. <column-1> ? <column-2><destination> ?? ??? ?? ? ????. ???? ?? ?? ???. ???? ??? ?????.?? : ? ? ? ?? ?? ? ? ???? ??? ??? ???? ?? ?? ?? ?? ????? ?? ?? ???.??? ?? ??????? ?? ?? ?? ??? ?????.block-distanceblockcosinedamerau-levenshteindiceeuclideangeneralized-jaccardidentityjaccardjarolevenshteinlongest-common-subsequencelongest-common-substringoverlap-cofficientsimon-white?? ???? ??? ?? :{ "tweet1": "CheeseBurgers are God tellin us everything's gonna be cool...", "tweet2": "Forgiveness can open up a whole new perspective on life, Forgiveness is an act of God's grace."}? ??? ?? :text-distance block tweet1 tweet2 distance? ???? ??? :{ "tweet1": "CheeseBurgers are God tellin us everything's gonna be cool...", "tweet2": "Forgiveness can open up a whole new perspective on life, Forgiveness is an act of God's grace." "destination": 26.0}Text MetricTEXT-METRIC ???? ??? ??? ??? ???? ??? ??? ???? ? ?? ??? ?? ??? ???? ??? (0?? 1??)? ?????.??text-metric <method> <column-1> <column-2> <destination><method> ??? ??? ??? ???? ? ??? ???? ?????. <column-1> ? <column-2><destination> ?? ??? ?? ? ????. ???? ?? ?? ???. ???? ??? ?????.?? : ? ? ? ?? ?? ? ? ???? ??? ??? ???? ?? ?? ?? ?? ????? ?? ?? ???.??? ?? ???????? ?? ?? 0? 1 ?????.??? ?? ?? ?? ??? ?????.block-distanceblockcosinedamerau-levenshteindiceeuclideangeneralized-jaccardidentityjaccardjarolevenshteinlongest-common-subsequencelongest-common-substringoverlap-cofficientsimon-white?? ???? ??? ?? :{ "tweet1": "CheeseBurgers are God tellin us everything's gonna be cool...", "tweet2": "Beer is God's gift to humankind."}? ??? ?? :text-metric longest-common-subsequence tweet1 tweet2 distance? ???? ??? :{ "tweet1": "CheeseBurgers are God tellin us everything's gonna be cool...", "tweet2": "Beer is God's gift to humankind." "destination": 0.26229507}Trimming SpacesTRIM, LTRIM ? RTRIM ???? ???? ??? ?? ??, ?? ?? ????? ??? ?????.??trim <column>ltrim <column>rtrim <column>? ???? ??? ??? ?? ??? ?? ???? ?????. <column>???trim ???? UNICODE ?? ??? ?????. ??? TRIM, LTRIM ? RTRIM? ???? ???? ?????.CharacterDescriptiontCharacter tabulationnLine Feed (LF)U000BLine TabulationfForm Feed (FF)rCarriage Return (CR)“”Spaceu0085Next line (NEL)u00A0No Break Spaceu1680OGHAM Space Marku180EMongolian Vowel Separatoru2000EN Quadu2001EM Quadu2002EN Spaceu2003EM Spaceu2004Three Per EM spaceu2005Four Per EM spaceu2006Six Per EM spaceu2007Figure Spaceu2008Puncatuation Spaceu2009Thin Spaceu200AHair Spaceu2028Line Separatoru2029Paragraph Separatoru202FNarrow No-Break Spaceu205FMedium Mathematical Spaceu3000Ideographic Space?? ???? ??? ?? :{ "id": 1, "gender": " male ", "fname": " Root ", "lname": " JOLTIE ", "address": " 67 MARS AVE, MARSCIty, Marsville, Mars"}? ??? ??trim genderltrim fnamertrim lnameltrim address? ???? ??? :{ "id": 1, "gender": "male", "fname": "Root ", "lname": " JOLTIE", "address": "67 MARS AVE, MARSCIty, Marsville, Mars"}URL DecodeURL-DECODE ???? application / x-www-form-urlencoded MIME ??? ???? ???? ??????.??url-decode <column>??? <column> ??? ? ??? ? URL? ?????.??? ?? ?????? ????? URL-ENCODE ????? ???? ?? ?????. ???? ???? ?? ??? a ~ z, A ~ Z, 0 ~ 9 ? -, _,. ? * ? ???? ?????. ?? %? ????? ?? ????? ???? ???? ?????.??? ??? ? ??????.??? ??? ??? ?????.??? a ~ z, A ~ Z ? 0 ~ 9? ???? ????.., -, * ? _ ?? ??? ???? ????.??? ?? (+)? ?? ?? ‘’% xy ??? ???? ???? ???? ??? ?????. ??? xy? 8 ?? ???? ? ?? 16 ?? ?????. ?? ?? ??? ??? ??? ? ?? ??? ????? ???? ?? ?? ???? ?? ?? ???? ???? ??? ?????. ??? ??? ????? ? ???? ??? ??? ?? ? ? ???, ???? ?? ?? ???? ?? ???? ?????.?? : ???? ??? ???? UTF-8? ?????.URL EncodeURL-ENCODE ???? ???? application / x-www-form-urlencoded MIME ???? ??????.Syntax??url-encode <column>??? <column> ??? ? URL? ?????.??? ?? ????? ??? ?? ??? ??? ? ??????.???? ??? ? ? ?? ??? ?????.??? a ~ z, A ~ Z ? 0 ~ 9? ???? ????.., -, * ? _ ?? ??? ???? ????.?? ??`? ??? ?? (+)? ?????.?? ?? ??? ???? ?? ??? ???? ?? ??? ???? ???? ?? ??? ???? ?????. ? ???? 3 ?? ??? % xy? ?????. ??? xy? ???? ? ?? 16 ?? ?????. ??? ??? ???? UTF-8? ????. ??? ???? ?? ???? ???? ??? ???? ?? ???? ?????.?? : ???? ??? ??? UTF-8? ?????.URL-DECODE???? ??????.Write as CSVWRITE-AS-CSV ???? ???? CSV ???? ?????.??write-as-csv <column>??? <column> ???? CSV ??? ?????.??? ?? ????WRITE-AS-CSV ???? ?? ???? CSV? ?????. ?? <column> ?? ???? ?? ???.?? ???? ??? ?? :{ "int": 1, "string": "this, is a string."}? ??? ?? :write-as-csv body? ???? ??? :{ "body": "1,\"this, is a string.\", "int": 1, "string": "this, is a string."}Write as JSON MapWRITE-AS-JSON-MAP ???? ???? JSON ??? ?????.??write-as-json-map <column>??? <column> ???? ?? ??? ?? JSON ?? ?????.??? ?? ????WRITE-AS-JSON-MAP ???? ?? ???? JSON ??? ?????. ?? <column> ?? ???? ?? ???.??? ???? ?? ??? ??? ?? ???? ?????.?? ???? ??? ?? :{ "int": 1, "string": "this, is a string."}? ??? ?? :write-as-json-map body? ???? ??? :{ "body": { "int":1, "string": "this, is a string." }, "int": 1, "string": "this, is a string."}Write as JSON objectWRITE-AS-JSON-OBJECT ???? ??? ??? ???? JSON ??? ???? ?? ?? ???.??write-as-json-object <destination-column> <source-column>[,<source-column>]*??? <destination-column> ? ??? ?? ??? ??? JSON ??? ?????. <source-column> .??? ?? ????WRITE-AS-JSON-OBJECT ???? ??? ????? ??? ?? ?? ?? ???? JSON ??? ?????.??? ???? ?? ??? ??? ?? ??? JSON ???? ?????. NULL? ???? JsonNull? ?????.?? ???? ??? ?? :{ "number": 1, "text": "this, is a string.", "height" : 1.5, "weight" : 1.67, "address" : null}??? ? ???? ?? :write-as-json-object body number,text,height,address? ???? ??? :{ "body": { "number":1, "text": "this, is a string.", "height" : 1.5, "address" : null }, "number": 1, "text": "this, is a string.", "height" : 1.5, "weight" : 1.67, "address" : null}XPath DirectivesXPATH ? XPATH-ARRAY ???? XML ??? XML ??? ??? ?????.??xpath <column> <destination> <xpath>xpath-array <column> <destination> <xpath>??? <column> XPath? ?? ? ?? XML???.??? ?? ?????? XML <column> ??? XPATH ???? ???? ?? ?? PARSE-AS-XML???? ???? ???.PARSE-AS-XML ???? ???? VTDNav ??? ???? XPATH ???? ??? ? ????. VTD Nav? ??? ???? ?? ??? XML ?????.??? <destination> XPath ?? ??? ???? ? ??? ? ??? ?????. XPath? ?? ???? ??? Null ?? ?? ?????.XPath-ARRAY ???? XPath? ?? ?? ?? ? ?? ?? ???? ???. ?? <destination> value? XPath? ?? ?? ?? ??? JSON ?????.Functions??(functions)? ??? ?? UI?? ???? ???? ? ?????.JSON Functions??? ??? ??? ??? ?? ?????. ? ?? ??? PARSE-AS-JSON ???? ?? ?????.?? ?? ??? ?? ??? PARSE-AS-JSON ???? ??? ?? ? ??? ? ????.?? ?????? ?? ?? ??? json ?? ????? ????.?? ?????? ?? ?? ???? ?? ?? ??? ??? ????.{ "name": { "fname": "Joltie", "lname": "Root", "mname": null }, "coordinates": [ 12.56, 45.789 ], "numbers": [ 1, 2.1, 3, null, 4, 5, 6, null ], "moves": [ { "a": 1, "b": "X", "c": 2.8}, { "a": 2, "b": "Y", "c": 232342.8}, { "a": 3, "b": "Z", "c": null}, { "a": 4, "b": "U"} ], "integer": 1, "double": 2.8, "float": 45.6, "aliases": [ "root", "joltie", "bunny", null ]}??? ???? ???? ??? ??? ??? ???? ?? ??? ???? ??????:parse-as-json bodycolumns-replace s/body_//gJSON ?? ??FunctionDescriptionExamplearray_join (aliases, separator)Joins all the elements in an array with a string separator. Returns the array unmodified if an object. Handles nulls.set-column alias _list json:array_join(aliases, ",")array_sum (numbers)Computes sum of the elements. Skips null values. Returns 0 if any elements that are not summable are found.set-column sum j son:array_sum(numbers)array_max (numbers)Finds the maximum of the elements. Skips null values. Returns 0x0.0000000000001P-1022if any issues with an element.set-column max j son:array_max(numb ers)array_min (numbers)Finds the minimum of the elements. Skips null values. Returns 0x1.fffffffffffffP+1023if any issues with an element.set-column min j son:array_min(numbers)Type Functions??? ??? ??? ???? ?????. ? ??? filter-row-if-false, filter-row-if-true, filter-row-on ?? send-to-error ????? ??? ? ????.?? ?? ?????? filter- * ?? send-to-error ?????? ??? ? ????.?? ?????? ?? ?? ??? ?? ???? ?????.?? ?????? ?? ?? ???? ?? ?? ??? ??? ????.{ "name": { "fname": "Joltie", "lname": "Root", "mname": null }, "date": "12/17/2019", "time": "10:45 PM", "boolean": "true", "coordinates": [ 12.56, 45.789 ], "numbers": [ 1, 2.1, 3, null, 4, 5, 6, null ], "moves": [ { "a": 1, "b": "X", "c": 2.8}, { "a": 2, "b": "Y", "c": 232342.8}, { "a": 3, "b": "Z", "c": null}, { "a": 4, "b": "U"} ], "integer": "1", "double": "2.8", "empty": "", "float": 45.6, "aliases": [ "root", "joltie", "bunny", null ]}??? ???? ???? ??? ??? ??? ???? ?? ??? ???? ??????:parse-as-json bodycolumns-replace s/body_//g?? ?? ??? ??? ??? ???? true? ???? ??? ??? false? ?????.FunctionDescriptionExampleisDate(string)Tests if string value is a date fieldfilter-row-if-true type:is Date(date)isTime(string)Tests if string value is a date timefilter-row-if-true type:is Time(time)isBoolean(string)Tests if string value is a boolean fieldsend-to-error !type:isBoolean(boolean)isNumber(string)Tests if string value is a number fieldsend-to-error !type:isNumber(integer)isEmpty(string)Tests if string value is emptysend-to-error !type:isEmpty(empty)isDouble(string)Tests if string value is a double fieldsend-to-error!type:isDouble(double)isInteger(string)Tests if string value is an integer fieldsend-to-error!type:isInteger(integer)Geofence Functions?? ?? ??? ?? ?? ?? ?? ??? ???? ?????. ? ??? filter-row-if-false, filter-row-if-true, filter-row-on, set column ?? send-to-error ????? ??? ? ????.?? ?? ???? ??? geoJson ???? ????? ???. ?? ??? Double ?? ??? ????? ???.?? ?????? ?? ?? ???? ?? ?? ??? ??? ????.{ "type": "FeatureCollection", "features": [ { "type": "Feature", "properties": {}, "geometry": { "type": "Polygon", "coordinates": [ [ [ -122.05870628356934, 37.37943348292772 ], [ -122.05724716186525, 37.374727268782294 ], [ -122.04634666442871, 37.37493189292912 ], [ -122.04608917236328, 37.38175237839049 ], [ -122.05870628356934, 37.37943348292772 ] ] ] } } ]}??? ???? ???? ?? ??? ?? ?? ????? ?? ??? ? ????:set column latitude 37.37220352507352set column longitude -122.04608917236328?? ??(Geofence) ??? ??? ?? ?? ??? ?? ?? ?? ? ??? ??? true? ???? ??? ??? false? ?????.FunctionDescriptionExamplelocationInFence(double, double, String)Tests if point is inside the fencessend-to-error !geo:inFence(latitude,longitude,body)Data Quality Functions??? ??? ??? ???? ?????. ? ??? filter-row-if-false, filter-row-if-true, filter-row-on ?? send-to-error ????? ??? ? ????.?? ?? ?????? filter- * ?? send-to-error ?????? ??? ? ????.?? ?????? ?? ?? ??? ?? ???? ?????.?? ?????? ?? ?? ???? ?? ?? ??? ??? ????.{ "name": { "fname": "Joltie", "lname": "Root", "mname": null }, "date": "12/17/2019", "time": "10:45 PM", "boolean": "true", "coordinates": [ 12.56, 45.789 ], "numbers": [ 1, 2.1, 3, null, 4, 5, 6, null ], "moves": [ { "a": 1, "b": "X", "c": 2.8}, { "a": 2, "b": "Y", "c": 232342.8}, { "a": 3, "b": "Z", "c": null}, { "a": 4, "b": "U"} ], "integer": "1", "double": "2.8", "empty": "", "float": 45.6, "aliases": [ "root", "joltie", "bunny", null ]}??? ???? ???? ??? ??? ??? ???? ?? ??? ???? ??????:parse-as-json bodycolumns-replace s/body_//g?? ?? ??? ??? ??? ???? true? ???? ??? ??? false? ?????.FunctionConditionExampleisDate(string)Tests if string value is a date fieldfilter-row-if-true type:is Date(date)isTime(string)Tests if string value is a date timefilter-row-if-true type:is Time(time)isBoolean(string)Tests if string value is a boolean fieldsend-to-error !type:isBoolean(boolean)isNumber(string)Tests if string value is a number fieldsend-to-error !type:isNumber(integer)isEmpty(string)Tests if string value is emptysend-to-error !type:isEmpty(empty)isDouble(string)Tests if string value is a double fieldsend-to-error!type:isDouble(double)isInteger(string)Tests if string value is an integer fieldsend-to-error!type:isInteger(integer)Date Manipulation Functions??? ??? ??? ???? ?????. ? ??? filter-row-if-false, filter-row-if-true, filter-row-on ?? send-to-error ????? ??? ? ????.?? ?? ?????? filter- * ?? send-to-error ?????? ??? ? ????.?? ?????? ?? ?? ??? ?? ???? ?????.?? ?????? ?? ?? ???? ?? ?? ??? ??? ????.{ "name": { "fname": "Joltie", "lname": "Root", "mname": null }, "date": "12/17/2019", "time": "10:45 PM", "boolean": "true", "coordinates": [ 12.56, 45.789 ], "numbers": [ 1, 2.1, 3, null, 4, 5, 6, null ], "moves": [ { "a": 1, "b": "X", "c": 2.8}, { "a": 2, "b": "Y", "c": 232342.8}, { "a": 3, "b": "Z", "c": null}, { "a": 4, "b": "U"} ], "integer": "1", "double": "2.8", "empty": "", "float": 45.6, "aliases": [ "root", "joltie", "bunny", null ]}??? ???? ???? ??? ??? ??? ???? ?? ??? ???? ??????:parse-as-json bodycolumns-replace s/body_//g?? ?? ??FunctionConditionExampleisDate(string)Tests if string value is a date fieldfilter-row-if-true type:is Date(date)isTime(string)Tests if string value is a date timefilter-row-if-true type:is Time(time)isBoolean(string)Tests if string value is a boolean fieldsend-to-error !type:isBoolean(boolean)isNumber(string)Tests if string value is a number fieldsend-to-error !type:isNumber(integer)isEmpty(string)Tests if string value is emptysend-to-error !type:isEmpty(empty)isDouble(string)Tests if string value is a double fieldsend-to-error!type:isDouble(double)isInteger(string)Tests if string value is an integer fieldsend-to-error!type:isInteger(integer)Data Definition Manipulation Functions??? ??? ??? ???? ?????. ? ??? filter-row-if-false, filter-row-if-true, filter-row-on ?? send-to-error ????? ??? ? ????.?? ?? ?????? filter- * ?? send-to-error ?????? ??? ? ????.?? ?????? ?? ?? ??? ?? ???? ?????.?? ?????? ?? ?? ???? ?? ?? ??? ??? ????.{ "name": { "fname": "Joltie", "lname": "Root", "mname": null }, "date": "12/17/2019", "time": "10:45 PM", "boolean": "true", "coordinates": [ 12.56, 45.789 ], "numbers": [ 1, 2.1, 3, null, 4, 5, 6, null ], "moves": [ { "a": 1, "b": "X", "c": 2.8}, { "a": 2, "b": "Y", "c": 232342.8}, { "a": 3, "b": "Z", "c": null}, { "a": 4, "b": "U"} ], "integer": "1", "double": "2.8", "empty": "", "float": 45.6, "aliases": [ "root", "joltie", "bunny", null ]}??? ???? ???? ??? ??? ??? ???? ?? ??? ???? ??????:parse-as-json bodycolumns-replace s/body_//g?? ?? ??? ??? ??? ???? true? ???? ??? ??? false? ?????.FunctionConditionExampleisDate(string)Tests if string value is a date fieldfilter-row-if-true type:is Date(date)isTime(string)Tests if string value is a date timefilter-row-if-true type:is Time(time)isBoolean(string)Tests if string value is a boolean fieldsend-to-error !type:isBoolean(boolean)isNumber(string)Tests if string value is a number fieldsend-to-error !type:isNumber(integer)isEmpty(string)Tests if string value is emptysend-to-error !type:isEmpty(empty)isDouble(string)Tests if string value is a double fieldsend-to-error!type:isDouble(double)isInteger(string)Tests if string value is an integer fieldsend-to-error!type:isInteger(integer)Services??? ?? ???(services)? RESTful ?????? ???? ???? ???? ??? ?????.Administration and Management ServicesCreating a WorkspaceDeleting a WorkspaceUploading a File to a WorkspaceDownloading a File from a Workspace## Creating a Workspace? RESTful API ??? ?? ??, ? ????? wrangling ? ???? ??? ???? ?? ???? ??? ?????. ?? ??? ? ?? ?? ?? (_) ??? ??? ???? ?????.URLworkspaces/:workspaceidMethodPUTURL ParamsNoneData ParamsNot ApplicableSuccess ResponseCode 200 Content{ 'status': 200, 'message': "Successfully created workspace ':workspaceid'"}Error ResponsesCode 500 Server Error Content{ 'status': 500, 'message': "<appropriate error message>"}??Code 500 Server Error ContentUnable to route to service <url>Sample Call$.ajax({ url: "${base-url}/workspaces/${workspace}", dataType: "json", type : "PUT", success : function(r) { console.log(r); } });????? ???? ???? ??? ??? ?? ??? ???? API ??? ?????.## Deleting a Workspace? RESTful API ??? ?? ??? ?????. ??? ??? ?? ???? ?????.URLworkspaces/:workspaceidMethodDELETEURL ParamsNoneData ParamsNot ApplicableSuccess ResponseCode 200 Content{ 'status': 200, 'message': "Successfully deleted workspace ':workspaceid'"}Error ResponsesCode 500 Server Error Content{ 'status': 500, 'message': "<appropriate error message>"}??Code 500 Server Error ContentUnable to route to service <url>Sample Call$.ajax({ url: "${base-url}/workspaces/${workspace}", dataType: "json", type : "DELETE", success : function(r) { console.log(r); } });????? ???? ???? ??? ??? ?? ??? ???? API ??? ?????.## Uploading a File to a Workspace? RESTful API ??? ??? ?? ??? ??? ???. ??? ? ?? ?? (EOL)? ?? ?? ????.URLworkspaces/:workspaceid/uploadMethodPOSTURL ParamsNoneData ParamsNot ApplicableSuccess ResponseCode 200 Content{ 'status': 200, 'message': "Successfully uploaded data to workspace ':workspaceid' (records 1000)"}Error ResponsesCode 500 Server Error Content{ 'status': 500, 'message': "Body not present, please post the file containing the records to be wrangle."}??Code 500 Server Error ContentUnable to route to service <url>??Code 500 Server Error Content{ 'status': 500, 'message': "<appropriate error message>"}Sample Call$.ajax({ url: "${base-url}/workspaces/${workspace}/upload", type: 'POST', data: data, cache: false, contentType: 'application/octet-stream', processData: false, // Don't process the files contentType: false, success: function(r) { console.log(r); }, error: function(r) { console.log(r); } });## Downloading a File from a Workspace? RESTful API? ?? ??? ???? ???? ??? ???? ???.URLworkspaces/:workspaceid/downloadMethodGETURL ParamsNoneData ParamsNot ApplicableSuccess ResponseCode 200 Content<data stored in workspace>Error ResponsesCode 500 Server Error Content{ 'status': 500, 'message': "No data exists in the workspace. Please upload the data to this workspace."}??Code 500 Server Error ContentUnable to route to service <url>??Code 500 Server Error Content{ 'status': 500, 'message': "<appropriate error message>"}Sample Call$.ajax({ url: "${base-url}/workspaces/${workspace}/download", type: 'GET', success: function(r) { console.log(r); }, error: function(r) { console.log(r); } });Connection PropertiesFollowing are properties for different types of connections.PropertyDescriptionDatabaseusername(String) Username to be used in the connectionpassword(String) Password to be used along with username. Empty should be double-quotedhostname(String) Hostname to connect toport(Integer) Port numberssl(Boolean) true or falseurl(String) JDBC UrlKafkazkqorum (String) Zookeeper quorum string (host1:2181,host2:2181,host3:2181)brokers(String) Kafka BrokerConnection Service?? ???? ?? ??? ???? ???? ?? RESTful API? ?????. ?? ?? ??? ??? ??? ?? ???? ?????.??? ?? ????? ???? ??? ??? ?????.? ?? ??? (POST, $ {base} / connections / create)?? ?? ???? (POST, $ {base} / connections / {id} / update)?? (PUT, $ {base} / connections / {id} / properties? key = <key> &amp; value = <value> )??? ?? ?? ?? ?? (GET, $ {base} / connections / {id} / properties)?? ?? (DELETE, $ {base} / connections / {id})?? ?? (GET, $ {base} / connections / {id} / clone)?? ??? ?? ?? ?? (GET, $ {base} / connections)??? ?? ?? ?? (GET $ {base} / connections / {id})Base??? ???? ?? URL???.: All examples below use a 'default' namespace.?? ? ?? ??????? JSON ?? ????? ??? ??? ? ?? ?????.name (mandatory)description (optional)type (mandatory; one of:)DATABASEKAFKAS3properties (optional)??? ??? ???? ?? JSON ??? ????.{ "name": "MySQL Database", "description": "MySQL Configuration", "type": "DATABASE", "properties": { "hostaname": "localhost", "port": "3306" }}??? ???? ??? ID? ?? ???. ??? ??? ???? ?? ?? ????.{ "status": 200, "message": "Success", "count": 1, "values": [ "mysql_database" ]}?? ??JSON ??cat /Users/nitin/Work/Demo/data/mysql.connection.json{ "name":"MySQL Database", "type":"DATABASE", "description":"MySQL Configuration", "properties" : { "hostaname" : "localhost", "port" : 3306 }}REST API ??? ??curl -s --data "@/tmp/mysql.connection.json" '' | python -mjson.tool{ "count": 1, "message": "Success", "status": 200, "values": [ "mysql_database" ]}Repeat creation will fail?? ?? ??curl -s --data "@/Users/nitin/Work/Demo/data/mysql.connection.json" '' | python -mjson.tool{ "message": "Connection name 'MySQL Database' already exists.", "status": 500}?? ??curl -X DELETE "" | python -mjson.tool{ "status":200, "message":"Success"}?? ??? ????? ????? ?? ?? ? ?? ???? ?????.curl -s -X DELETE "" | python -mjson.tool{ "message": "Success", "status": 200}?? ?? ??curl -s "*" | python -mjson.tool{ "count": 1, "message": "Success", "status": 200, "values": [ { "created": 1494529821, "description": "MySQL Configuration", "id": "mysql_database", "name": "MySQL Database", "type": "DATABASE", "updated": 1494529821 } ]}?????? ??? ??? ??curl -s "" | python -mjson.tool{ "count": 1, "message": "Success", "status": 200, "values": [ { "created": 1494529821, "description": "MySQL Configuration", "id": "mysql_database", "name": "MySQL Database", "type": "DATABASE", "updated": 1494529821 } ]}?? ??curl -s "" | python -mjson.tool{ "count": 1, "message": "Success", "status": 200, "values": [ { "created": 1494527723, "description": "MySQL Configuration", "id": "mysql_database", "name": "MySQL Database", "properties": { "hostaname": "localhost", "port": 3306.0 }, "type": "DATABASE", "updated": 1494527723 } ]}?? ??curl -s "" | python -mjson.tool{ "count": 1, "message": "Success", "status": 200, "values": [ { "created": 1494528015, "description": "MySQL Configuration", "name": "MySQL Database_Clone", "properties": { "hostaname": "localhost", "port": 3306.0 }, "type": "DATABASE", "updated": 1494528015 } ]}??? ?? ??(Fetch)curl -s "" | python -mjson.tool{ "count": 1, "message": "Success", "status": 200, "values": [ { "hostaname": "localhost", "port": 3306.0 } ]}? ?? ?? ?? ?? ?? ????curl -X PUT -s "" | python -mjson.tool{ "count": 1, "message": "Success", "status": 200, "values": [ { "hostaname": "localhost", "port": 3306.0, "ssl": "true" } ]}??? ????? ???? ??? ???? ???? ?????.URLworkspaces/:workspaceid/executeMethodGETURL Params??? ???? ?? ??(query arguments)? ?????. ?? ?? ???? ????? ?? ?? ?? ??? ?????.Requireddirective=[encoded directive]Optionallimit=[numeric]Data ParamsNot ApplicableSuccess ResponseCode: 200 Content:{ 'status': 200, 'message': 'Success', 'items': <count of records>, 'header': [ 'header-1', 'header-2', ..., 'header-n' ], 'value': { { processed record - 1}, { processed record - 2}, . . . { processed record - n} }}Error ResponsesCode: 500 Server Error Content:{ 'status': 500, 'message': "<appropriate error message>"}???? : 500 ?? ?? ?? :Unable to route to service <url>Sample Call$.ajax({ url: "${base-url}/workspaces/${workspace}/execute", data: { 'directive': <directive-1>, 'directive': <directive-2>, ... 'directive': <directive-k> 'limit': <count> } cache: false type : "GET", success : function(r) { console.log(r); } });?? ?? ??(Request Format Specification)Currently, the Data Prep front-end makes a GET call with all the directives in the query arguments. This causes issues on some browsers as there are limits as to how much data can be pushed using this approach. This document describes the format that is used for sending a request to the back-end.?? Data Prep ??? ??(front-end)? ?? ??(query arguments)? ?? ?? ???? ???? GET ??? ????. ? ???? ???? ??? ?? ???? ??(push)? ? ???? ?? ??? ?? ??? ?? ??????? ??? ?????. ? ??? ???(back-end)? ??? ??? ? ???? ??? ?? ?????.?? ??? ??? ?? ?? ?? 1.0? ??? ??? ?? ? ????? ? ?? ??? ???? ??? ??? ?????.JSON ?? ??{ "version": "1.0" "workspace": { "name": string, "results": number }, "recipe": { "directives": [ "string", "string", ... "string" ], "save": boolean, "name": string }, "sampling": { "method": string, "seed": number, "limit": number }}Version? ?? (1.0)? ??? ?????. ??? ?? ?? ?? ??? ?? ?? ? ??? ???????.Workspace? ????? ???? ???? ?? ??(workspace) ? ?? ??? ???? ???? ?? ??? ? (results)? ?? ??? ?????.PropertyMandatoryDescriptionNameYesName of the workspace that Data Prep should operate onresultsYesNumber of records that should be returned in response to execution of the directivesRecipe? ???? ?? ??? ???? ??? ?? ???? ???? ??? ?? ???? ???? ??? ???? ????.FieldMandatoryDescriptiondirectivesYesList of directives to be applied on the datasaveNoIf true, specifies that the directives should be saved. If so, then name should also be specified.nameNoName of the recipe. This option is valid only when save is set to true.Sampling? ????? ?? ???? ????? ??? ?? ??? ?????.FieldMandatoryDescriptionMethodYesType of sampling to be applied while selecting input data. Currently only supports first.seedNoThe random seed to be used when sampling datalimitNoThe number of input records to be read from the source when applying directives???? ? :{ "version": "1.0", "workspace": { "name": "body", "results": 100 }, "recipe": { "directives": [ "parse-as-csv body ,", "drop body", "set-columns a,b,c,d" ], "save": true, "name": "my-recipe" }, "sampling": { "method": "first", "seed": 1, "limit": 1000 }}Schema RegistrySchema Registry provides a serving layer for all types of metadata. It provides a RESTful interface for storing and retrieving schemas (AVRO, Protobuf, etc). It stores a versioned history of all schemas,??? ?????(Schema Registry)? ?? ??? ?? ???? ?? ??? ??? ?????. ??? (AVRO, Protobuf ?) ?? ? ??? ?? RESTful ?????? ?????. ?? ?? ???? ?? ? ??? ?????.??? ??FieldDescriptionIDId of the schema as provided by the user.NameDisplay name for the schema.DescriptionUser facing description about the schemaCreated DateTime in seconds about when the schema was created.Updated DateTime in seconds about when the schema was last updated.VersionTime in seconds about when the schema was last updated.TypeType of the schema, currently supports AVRO and Protobuf-descSpecificationByte array of the specification of schemaRESTful APIsAPIMethodPathResponseDescriptionCreate a schema entryPUT/schemas200 - OK, 500 - Error in backend storeCreates an entry in the schema registry. No schema is registred.Add a schema to schema entryPOST/schemas/{id}200 - OK, 500 - Error adding schema to schema registryAdds a versioned schema to schema registry. POST should use Content-Type:application n/octet-streamDelete all version of schemaDELETE/schemas/{id}200 - OK, 500 - Error deleting schemaDeletes the entire schema entry including all the versions of schema.Delete a sepecific version of schemaDELETE/schemas/{id} /versions/{ve rsion}200 - OK, 500 - Error deleting a version of schemaDeletes a specific version of schema, if schema is not found then a 404 is returned.GET information about a a version of schemaGET/schemas/{id} /versions/{ve rsion}200 - OK, 500 - Backend error, 404 - Schema id not foundInformation about schema version and schema entryGET information about schema entryGET/schemas/{id}200 - OK, 500 - ErrorInformation about schema entryList version for schema availableGET/schemas/{id} /versions200 - OK, 500 - ErrorList the versions of schema.Data Prep ServiceData Prep? CDAP Service? ???? HTTP RESTful ?? ??? ??? ???? ?????. ? ???? ?? ??? ??? ??? ??? ??(parsing)?? ? ??? ???? ????? ???? ?? ?? ? ? ??? ?? ????. ? ???? ??? ??? ??? ???? ?? ?? ????. ?? ?? ??? ??? ?? ????? ???? ???? ? ?????.?? ?? ???? ??? ????.??? ???? ??? ??? :Administration and ManagementDirective ExecutionColumn Type Detection and StatisticsColumn Name Validation?? ?? ?? ??? ??? ???? ??? ? ???? ??? ?????. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download