AWS Kinesis Data Firehose

Kinesis Data Firehose is an Amazon Web Services data transfer service which can move streaming data to data storage. It will extract data, transform it, and load data in the cloud. When data must be imported from multiple sources that are generating data in realtime and ingested for processing before ending up in a data store, Kinesis Data Firehose acts as the conduit. So as data is generated by user inputs on the web (clickstream data), enterprise applications, smartphones, or other sources KDF can compress and encrypt the data before transferring the data to Amazon cloud storage, such as an Amazon simple storage service (S3) bucket or an outside location.

KDF can transfer the data at several GBs per second, and will backup the data at three different locations in an AWS cloud data region. (Amazon has 6 regions in the United States - Northern Virginia; Ohio; Northern California; Oregon; US-East; and US-West). AWS only charges for the data that is ingested via KDF - there is no minimum fee.

KDF may convert JSON data to other formats in order to save storage space, and it can also convert .csv files or data from structured databases to the JSON format.

So for example financial transaction data can be sent automatically through KDF to a S3 bucket in its raw state, or KDF can transform the data , summing up sales to a particular entity, and then store the data in the bucket.

KDF is designed to ensure that data collected on servers from hundreds of sources does not get lost when the servers go down. It will scale up as more streaming data is generated. Data is actively backed up in the cloud as it is generated.