Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.

Overview

AWS infrastructure to ship CloudTrail logs from S3 to Kinesis

This repository contains a Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.

Use-cases include shipping CloudTrail logs of an organization trail to a SIEM (such as Splunk) or third-party service provider.

Architecture

In summary:

  • A S3 bucket notification is configured to trigger a SNS event every time AWS delivery a new CloudTrail log file to the S3 bucket.

  • A subscription is set up so that a Lambda function is invoked for every SNS event, i.e. every time AWS delivers a new CloudTrail log file to the S3 bucket.

  • The Lambda function picks up the information that a new file was uploaded, reads this file from S3 and writes it to Kinesis.

Pre-requisites

  • A CloudTrail organization trail must already be configured to ship organization-wide CloudTrail logs to a S3 bucket

  • This S3 bucket ("CloudTrail S3 bucket") must already exist

  • The CloudTrail S3 bucket must be in the same AWS account and region as the one the module is instantiated

Usage

Inputs

Name Description Type Default Required
cloudtrail-bucket-name Name of the S3 bucket in which CloudTrail logs are stored (must exist and properly configured to receive CloudTrail logs prior to calling this module) string n/a yes
cloudtrail-sns-topic-name Name of the SNS topic where information about newly shipped CloudTrail log files are sent string "organization-trail-event-notification-topic" no
cloudwatch-logs-retention-time-days Retention period for the CloudWatch logs of the Lambda function (in days) number 7 no
kinesis-num-shards Number of shards to use in the Kinesis stream number 4 no
kinesis-retention-time-days Retention period of the Kinesis stream (in days) number 7 no
kinesis-stream-kms-key-id ID of the KMS key to use for encrypting the Kinesis stream string "alias/aws/kinesis" no
kinesis-stream-name Name of the Kinesis stream used for aggregation string "cloudtrail-logs-stream" no
lambda-memory Memory to allocate to the Lambda function number 512 no

Outputs

Name Description
kinesis-stream-arn ARN of the newly created Kinesis stream
kinesis-stream-name Name of the newly created Kinesis stream
sns-topic-arn arn of the newly creates SNS topic
sns-topic-name Name of the newly creates SNS topic

Example usage

Sample usage:

provider "aws" {
  region = "eu-west-1"
}

// Ship CloudTrail logs that AWS ships in "my-organization-trail-s3-bucket" into a Kinesis stream "cloudtrail-logs"
module "cloudtrail-to-kinesis" {
  source                      = "github.com/nexthink/terraform-aws-cloudtrail-s3-to-kinesis?ref=v1.0.0"
  cloudtrail-bucket-name      = "my-organization-trail-s3-bucket"
  kinesis-stream-name         = "cloudtrail-logs"
  kinesis-retention-time-days = 7
}

Monitoring

Logs

Logs of the Lambda function are sent to a CloudWatch log group. Invocation rate and success/error rate can be seen directly from the AWS console, under the "Monitoring" tab of the Lamdba function

Observability

The module creates a custom CloudWatch metric NumberOfCloudTrailRecordsShippedToKinesis, allowing to have visibility on the total number of CloudTrail events shipped by the Lambda function. When used conjointly to the default AWS Lambda CloudWatch metrics, it allows to easily build a CloudWatch dashboard to monitor the volume of logs processed by the Lambda function, as well as any potential errors.

CloudWatch dashboard for observability

Unit tests

Run the Lambda function unit tests using:

$ cd lambda
$ make test
AWS_SECRET_ACCESS_KEY= AWS_ACCESS_KEY_ID= python -m unittest discover tests -v
test_extract (test_extract.TestExtract) ... ok
test_entrypoint (test_lambda_entrypoint.TestLambdaEntryPoint) ... 2021-04-09 16:38:33,062 Reading CloudTrail log file s3://my-cloudtrail-bucket/dir/to/cloudtrail.json.gz
2021-04-09 16:38:33,066 Writing CloudTrail log payload to Kinesis (27596 bytes)
ok
test_writes_to_kinesis (test_load.TestLoad) ... 2021-04-09 16:38:33,124 Found credentials in environment variables.
2021-04-09 16:38:33,155 Writing CloudTrail log payload to Kinesis (24 bytes)
ok
test_returns_correct_data_from_s3 (test_transform.TestExtract) ... 2021-04-09 16:38:33,236 Found credentials in environment variables.
2021-04-09 16:38:33,300 Reading CloudTrail log file s3://my-bucket//path/to/cloudtrail-file.json.gz
ok

----------------------------------------------------------------------
Ran 4 tests in 0.532s

OK

End-to-end tests

End-to-end tests use Terratest to spin up real infrastructure, create a S3 bucket, ship a sample CloudTrail log file to it and check that the expected record is produced in Kinesis.

# Pre-requisite: ensure you are authenticated against a test/CI AWS account
# $ aws-vault-exec ci-account

$ cd test/
$ go test

End-to-end tests take around 2-3 minutes to run.

Examples

See the examples/ folder for complete usage examples.

You might also like...
Python binding for Terraform.

Python libterraform Python binding for Terraform. Installation $ pip install libterraform NOTE Please install version 0.3.1 or above, which solves the

The public discord bot, created by: primitt, further developed by: duino-coin team.
The public discord bot, created by: primitt, further developed by: duino-coin team.

Duino Stats Mini A public Duino-Stats Discord bot. Click this link to invite the bot to your server. License Duino Stats Mini distributed under the MI

Aggrokatz is an aggressor plugin extension for Cobalt Strike which enables pypykatz to interface with the beacons remotely and allows it to parse LSASS dump files and registry hive files to extract credentials and other secrets stored without downloading the file and without uploading any suspicious code to the beacon. The Research PACS on AWS solution facilitates researchers' access medical images stored in the clinical PACS in a secure and seamless manner
The Research PACS on AWS solution facilitates researchers' access medical images stored in the clinical PACS in a secure and seamless manner

Research PACS on AWS Challenge to solve Solution presentation Deploy the solution Further reading Releases License Challenge to solve The rise of new

Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automatically.
Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automatically.

DATE_TIME_USERBOT-TeLeTiPs Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automaticall

An script where it logs in your instagram account and follows people and likes their posts

InstaFollower An script where it logs in your instagram account and follows people and likes their posts (uses the tags to fetch people) Requirements:

Simple, yet effective moderator bot for telegram. With reports, logs, profanity filter and more :3

👹 Samurai Telegram Bot Simple, yet effective moderator bot for telegram. With reports, logs, profanity filter and more :3 Description Personal bot, m

Bombcrypto-bot - A python bot that automatically logs in, clicks the new button, and sends heroes to work in the bombcrypto game RichWatch is wrapper around AWS Cloud Watch to display beautiful logs with help of Python library Rich.
RichWatch is wrapper around AWS Cloud Watch to display beautiful logs with help of Python library Rich.

RichWatch is TUI (Textual User Interface) for AWS Cloud Watch. It formats and pretty prints Cloud Watch's logs so they are much more readable. Because

Releases(v1.0.0)
Owner
Nexthink
Nexthink
A VCVideoPlayer Bot for Telegram made with 💞 By @TeamDeeCoDe

VC Video Player How To Host ✨ Heroku Deploy ✨ The easiest way to deploy this Bot is via Heroku. Credit 🔥 |🇮🇳 Louis |🇮🇳 Sammy |🇮🇳 Blaze |🇮🇳 S

TeamDeeCode 6 Feb 28, 2022
Info & tools for reverse engineering the M6 smart fitness band

m6-reveng This repo contains information and tools for reverse engineering the $7 M6 smart fitness band. Hardware The SoC (system-on-a-chip) is a Teli

41 Dec 26, 2022
A telegram mirror bot with an integrated RSS feed reader.

About What is this repo? This is a slightly modified fork which includes some extra features & memes added to my liking. How's this different from the

11 May 15, 2022
Replace sequence_IDs in gff3 based on given genome.fasta

gff-rename Replace the sequence IDs in a gff3 file with a set of provided sequence IDs from a genom.fasta. This is useful when a gff3 file is retrieve

tolkit 1 Nov 12, 2021
Simple web browser to visualize HiC tracks

HiCBrowser : A simple web browser to visualize Hi-C and other genomic tracks Fidel Ramirez, José Villaveces, Vivek Bhardwaj Installation You can insta

The deepTools ecosystem 14 Jun 21, 2022
The implementation of Learning Instance and Task-Aware Dynamic Kernels for Few Shot Learning

INSTA: Learning Instance and Task-Aware Dynamic Kernels for Few Shot Learning This repository provides the implementation and demo of Learning Instanc

11 Jan 02, 2023
Python functions to run WASS stereo wave processing executables, and load and post process WASS output files.

wass-pyfuns Python functions to run the WASS stereo wave processing executables, and load and post process the WASS output files. General WASS (Waves

Mika Malila 3 May 13, 2022
Script Crack Facebook, and Instagram 🚶‍♂

in-mbf Script Crack Facebook, and Instagram 🚶‍♂ Bukti Install Script $ pkg update && pkg upgrade $ pkg install git $ pkg install python2 $ pip2 insta

Yumasaa 5 Dec 27, 2021
Twitter bot to know the number of dislikes of a YouTube video

YT_dislikes is a twitter bot that allows you to know the number of dislikes (and likes) of a YouTube video. Now it is not possible to see the number o

1 Jan 08, 2022
A VCVideoPlayer Bot for Telegram made with 💞 By @ProErrorXD

VC Video Player How To Host ✨ Heroku Deploy ✨ The easiest way to deploy this Bot is via Heroku. Credit 🔥 |🇮🇳 Louis |🇮🇳 Sammy |🇮🇳 Blaze Marsha

丂ムᄊᄊƳ 95 May 17, 2022
universal messaging & notifications api

Pronounced "boat-shahft" What is botschaft? Botschaft is unified messaging & notifications appliance. Want to text yourself when a long-running task c

Tyler M. Kontra 25 Aug 16, 2022
Minimal Python client for the Iris API, built on top of Authlib and httpx.

🕸️ Iris Python Client Minimal Python client for the Iris API, built on top of Authlib and httpx. Installation pip install dioptra-iris-client Usage f

Dioptra 1 Jan 28, 2022
Send GitHub Issues, PRs or Discussions Updates to Wechat

Send GitHub Issues, PRs or Discussions Updates to Wechat

Hollow Man 2 Jul 12, 2022
C Y B Ξ R UserBot is a project that simplifies the use of Telegram.

C Y B Ξ R USΞRBOT 🇦🇿 C Y B Ξ R UserBot is a project that simplifies the use of Telegram. All rights reserved. Automatic Setup Android: open Termux p

FVREED 4 Dec 07, 2022
A simple, fast, and awesome discord nuke bot! The only thing you need to add is your bot token.

SimpleNukeBot A simple, fast, and awesome discord nuke bot! The only thing you need to add is your bot token. Instructions: All you need to do is crea

Bisc 1 Apr 18, 2022
Converts a text file of songs to a playlist on your Spotify account.

Playlist Converter Convert a text file of songs to a playlist on your Spotify account. Create your playlists faster instead of manually searching for

Priya Aggarwal 18 Dec 21, 2022
Generate Heroku-like random names to use in your python applications

HaikunatorPY Generate Heroku-like random names to use in your python applications. Installation pip install haikunator Usage Haikunator is pretty sim

Atrox 116 Nov 15, 2022
The Simple Google Colab Notebook to Download Files from Direct Link to Google Drive with custom name and bulk link support.

Direct Link to Google Drive (Advanced! 🔥 ) The Most Advanced yet Simple Google Colab Notebook to Download Files from Direct Link to Google Drive. 🆕

Dr.Caduceus 14 Jul 26, 2022
Hacktoberfest2021 - Submit Just 4 PRs to earn SWAGS and Tshirts🔥

dont contribute in this repo, contribute only in below mentioned repo Special Note For Everyone ''' always make more then 4 pull request lets you have

Keshav Singh 820 Jan 02, 2023