Hyper-Scale Streaming Data Processing and Storage

[Pages:2]CASE STUDY

Hyper-Scale Streaming Data Processing and Storage

PoC Overview

Impetus conceived and implemented a hyper-scale Stock Ticker application to demonstrate real time processing, storage and visualization of huge volumes of stock-tick data.

Client Overview Challenges

Highlights and Benefits

hyper scale and storage

Our Solution

The solution of the Stock Ticker Proof of Concept can be divided into six segments. Impetus team addressed client requirements in the following ways:

? Data Generator: A multi-threaded process that can generate mocked-up stock tick data from multiple exchanges around the globe, with the flexibility of increasing/decreasing data velocity.

and reports on historical data

Technologies

StreamAnalytix, Oracle NoSQL Database, Kafka, Intellicus, D3.js, Tomcat

? Real-time Data Processing and Storage: A parallel processing engine, which can process high volume of stock tick data and can publish it to the UI in real-time with high data processing rate.

? Data Store: A Data store that stores and makes available huge volume of incoming stock tick data in real-time.

? Reporting on Historical Data: A reporting tool that can generate reports on the historical data. The reports can be viewed on the UI for analysis and enabling business decisions.

? User Interface: A User Interface that can handle huge real-time data and is capable of showing it as graphs in real-time.

Legend System System Function

Ingestion modules capable to streaming generated data

at very high rates (Hyper Ingestion)

Data Generation Ingestion Process Tick Generator

Tick Encoder Tick Publisher

Ingestion Process Tick Generator Tick Encoder Tick Publisher

? Provisioning and Monitoring Tool: A Cluster Management tool.

User Interface Live Views

Historical Views

Streaming to the UI for live and historical data

StreamAnalytix Stream Visualization Support

Data Endpoints Distributed & Parallel Processing

Parallel Storage Calls

Intellicus

Intellicus

Reporting Server connectivity to

Data Cubes

NoSQL for building data cubes for

historical data

Oracle NoSQL Data Store

Parallel calls for high velocity storage and access from NoSQL database

Ankush Cluster Management & Monitoring Ankush for dynamic and flexible provisioning for the entire solution

? 2014 Impetus Technologies, Inc. All rights reserved. Product and company names mentioned herein may be trademarks of their respective companies.

StreamAnalytix is an enterprise grade, visual, big data analytics platform for unified streaming and batch data processing based on best-of-breed open source technologies. It supports the end-to-end functionality of data ingestion, enrichment, machine learning, action triggers, and visualization, StreamAnalytix offers an intuitive drag-and-drop visual interface to build and operationalize big data applications five to ten times faster, across industries, data formats, and use cases.

Visit or write to us at inquiry@

9

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download