Skip to content

The Streamed Data Management Systems

March 22, 2013

 

In it simple definition, data stream is a continuous flow of data produced by a data source. The data sources can be sensors (that monitor real time conditions) or discrete data source that produces high volumes of data at high speed in a continuous way, examples of this later definition are weblogs of a heavy use website or SMS messages flows on heavy loaded SMS server clusters.

Data streams feeds can be stored and manipulated in different ways. In the case of data feed generated by sensors, is common practice to reduce the data stream to discrete values (rows) by dividing the continuous flow of data in (temporary) windows and reducing the data on those windows by identifying trends and/or variations; This discrete values are then persisted in base tables in a database. In the case of feeds of discrete values, like MSM messages, we most decide if we save every occurrence as an unstructured data element (recommended if data mining will be required), or if we reduce blocks of discrete data in similar way as we do with data generated by sensors.

A Streamed Data Management System (SDMS) is responsible for the reception, temporary storage, windowing and reduction of feeds from data streams sources, and to store the reduced data. Given the high speed of the feeds, it is recommended to implement the active part of the SDMS (caption, reduction) in RAM and the pervasive part (base tables containing discrete reduced data) in disk. Given the high volume of data, the passive part (database) tends to grown fast, it is recommended to move the stable data from the base tables and derived tables (if there exists) to the Streamed Data Repository in the Storage Area of the Data Warehouse and to frequently purge the tables in the SDMS.

Advertisements
Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: