AWS Kinesis Data Firehose

Kinesis Data Firehose is an Amazon Web Services data transfer service which can move streaming data to data storage. It will extract data, transform it, and load data in the cloud. When data must be imported from multiple sources that are generating data in realtime and ingested for processing before ending up in a data store, Kinesis Data Firehose acts as the conduit. So as data is generated by user inputs on the web (clickstream data), enterprise applications, smartphones, or other sources KDF can compress and encrypt the data before transferring the data to Amazon cloud storage, such as an Amazon simple storage service (S3) bucket or an outside location.

KDF can transfer the data at several GBs per second, and will backup the data at three different locations in an AWS cloud data region. (Amazon has 6 regions in the United States - Northern Virginia; Ohio; Northern California; Oregon; US-East; and US-West). AWS only charges for the data that is ingested via KDF - there is no minimum fee.

KDF may convert JSON data to other formats in order to save storage space, and it can also convert .csv files or data from structured databases to the JSON format.

So for example financial transaction data can be sent automatically through KDF to a S3 bucket in its raw state, or KDF can transform the data , summing up sales to a particular entity, and then store the data in the bucket.

KDF is designed to ensure that data collected on servers from hundreds of sources does not get lost when the servers go down. It will scale up as more streaming data is generated. Data is actively backed up in the cloud as it is generated.

The CLOUD Act

In 2018, Congress passed the Clarifying Lawful Overseas Use of Data Act, the CLOUD Act, amending the Stored Communications Act. The CLOUD Act removed restrictions on U.S. companies’ responding directly to foreign countries investigating serious crimes.

The law allows the federal law enforcement officials to subpoena cloud data stored on servers located in foreign countries. The CLOUD Act authorizes the White House to reach agreements with foreign governments providing for a process to transfer data. State and the AG must confirm that the other country has adequate measures in place to protect the privacy of U.S.citizens’ data.

SSAE-16 and SOC-1 - Assessment of Controls for Financial Reporting

The Tip of the Night for February 26, 2019 discussed the SOC 2 Cloud Security Standard, the cloud security certification offered by AICPA, the American Institute of Certified Public Accountants. When considering whether or not to use a vendor to host data in the cloud, check to see if it has also completed an AICPA SSAE-16 examination performed by one of the Big Four accounting firms. A good vendor will have done both SOC 2 and SSAE-16 examinations. The SOC 1 reports that a SSAE-16 audit issues will review how the data center's controls affect their financial reporting. SSAE-16 stands for Statement on Standards for Attestation Engagements No. 16.

SOC 2 focuses on the security and privacy of data when it's stored and in transit. It checks the security, availability, processing integrity, confidentiality, and privacy of data. SSAE-16, and the SOC 1 Type 1 and Type 2 reports address the sufficiency of internal controls for the purposes of financial reporting.

SSAE-16 is the audit conducted before a SOC 1 report is issued which assesses how comprehensive a data center's controls are. After a first report giving an evaluation of the data center at a given point in time, and second SOC 1 report will be prepared that shows the condition of the data center's control system over time.