Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.

Overview

AWS infrastructure to ship CloudTrail logs from S3 to Kinesis

This repository contains a Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.

Use-cases include shipping CloudTrail logs of an organization trail to a SIEM (such as Splunk) or third-party service provider.

Architecture

In summary:

  • A S3 bucket notification is configured to trigger a SNS event every time AWS delivery a new CloudTrail log file to the S3 bucket.

  • A subscription is set up so that a Lambda function is invoked for every SNS event, i.e. every time AWS delivers a new CloudTrail log file to the S3 bucket.

  • The Lambda function picks up the information that a new file was uploaded, reads this file from S3 and writes it to Kinesis.

Pre-requisites

  • A CloudTrail organization trail must already be configured to ship organization-wide CloudTrail logs to a S3 bucket

  • This S3 bucket ("CloudTrail S3 bucket") must already exist

  • The CloudTrail S3 bucket must be in the same AWS account and region as the one the module is instantiated

Usage

Inputs

Name Description Type Default Required
cloudtrail-bucket-name Name of the S3 bucket in which CloudTrail logs are stored (must exist and properly configured to receive CloudTrail logs prior to calling this module) string n/a yes
cloudtrail-sns-topic-name Name of the SNS topic where information about newly shipped CloudTrail log files are sent string "organization-trail-event-notification-topic" no
cloudwatch-logs-retention-time-days Retention period for the CloudWatch logs of the Lambda function (in days) number 7 no
kinesis-num-shards Number of shards to use in the Kinesis stream number 4 no
kinesis-retention-time-days Retention period of the Kinesis stream (in days) number 7 no
kinesis-stream-kms-key-id ID of the KMS key to use for encrypting the Kinesis stream string "alias/aws/kinesis" no
kinesis-stream-name Name of the Kinesis stream used for aggregation string "cloudtrail-logs-stream" no
lambda-memory Memory to allocate to the Lambda function number 512 no

Outputs

Name Description
kinesis-stream-arn ARN of the newly created Kinesis stream
kinesis-stream-name Name of the newly created Kinesis stream
sns-topic-arn arn of the newly creates SNS topic
sns-topic-name Name of the newly creates SNS topic

Example usage

Sample usage:

provider "aws" {
  region = "eu-west-1"
}

// Ship CloudTrail logs that AWS ships in "my-organization-trail-s3-bucket" into a Kinesis stream "cloudtrail-logs"
module "cloudtrail-to-kinesis" {
  source                      = "github.com/nexthink/terraform-aws-cloudtrail-s3-to-kinesis?ref=v1.0.0"
  cloudtrail-bucket-name      = "my-organization-trail-s3-bucket"
  kinesis-stream-name         = "cloudtrail-logs"
  kinesis-retention-time-days = 7
}

Monitoring

Logs

Logs of the Lambda function are sent to a CloudWatch log group. Invocation rate and success/error rate can be seen directly from the AWS console, under the "Monitoring" tab of the Lamdba function

Observability

The module creates a custom CloudWatch metric NumberOfCloudTrailRecordsShippedToKinesis, allowing to have visibility on the total number of CloudTrail events shipped by the Lambda function. When used conjointly to the default AWS Lambda CloudWatch metrics, it allows to easily build a CloudWatch dashboard to monitor the volume of logs processed by the Lambda function, as well as any potential errors.

CloudWatch dashboard for observability

Unit tests

Run the Lambda function unit tests using:

$ cd lambda
$ make test
AWS_SECRET_ACCESS_KEY= AWS_ACCESS_KEY_ID= python -m unittest discover tests -v
test_extract (test_extract.TestExtract) ... ok
test_entrypoint (test_lambda_entrypoint.TestLambdaEntryPoint) ... 2021-04-09 16:38:33,062 Reading CloudTrail log file s3://my-cloudtrail-bucket/dir/to/cloudtrail.json.gz
2021-04-09 16:38:33,066 Writing CloudTrail log payload to Kinesis (27596 bytes)
ok
test_writes_to_kinesis (test_load.TestLoad) ... 2021-04-09 16:38:33,124 Found credentials in environment variables.
2021-04-09 16:38:33,155 Writing CloudTrail log payload to Kinesis (24 bytes)
ok
test_returns_correct_data_from_s3 (test_transform.TestExtract) ... 2021-04-09 16:38:33,236 Found credentials in environment variables.
2021-04-09 16:38:33,300 Reading CloudTrail log file s3://my-bucket//path/to/cloudtrail-file.json.gz
ok

----------------------------------------------------------------------
Ran 4 tests in 0.532s

OK

End-to-end tests

End-to-end tests use Terratest to spin up real infrastructure, create a S3 bucket, ship a sample CloudTrail log file to it and check that the expected record is produced in Kinesis.

# Pre-requisite: ensure you are authenticated against a test/CI AWS account
# $ aws-vault-exec ci-account

$ cd test/
$ go test

End-to-end tests take around 2-3 minutes to run.

Examples

See the examples/ folder for complete usage examples.

You might also like...
Python binding for Terraform.

Python libterraform Python binding for Terraform. Installation $ pip install libterraform NOTE Please install version 0.3.1 or above, which solves the

The public discord bot, created by: primitt, further developed by: duino-coin team.
The public discord bot, created by: primitt, further developed by: duino-coin team.

Duino Stats Mini A public Duino-Stats Discord bot. Click this link to invite the bot to your server. License Duino Stats Mini distributed under the MI

Aggrokatz is an aggressor plugin extension for Cobalt Strike which enables pypykatz to interface with the beacons remotely and allows it to parse LSASS dump files and registry hive files to extract credentials and other secrets stored without downloading the file and without uploading any suspicious code to the beacon. The Research PACS on AWS solution facilitates researchers' access medical images stored in the clinical PACS in a secure and seamless manner
The Research PACS on AWS solution facilitates researchers' access medical images stored in the clinical PACS in a secure and seamless manner

Research PACS on AWS Challenge to solve Solution presentation Deploy the solution Further reading Releases License Challenge to solve The rise of new

Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automatically.
Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automatically.

DATE_TIME_USERBOT-TeLeTiPs Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automaticall

An script where it logs in your instagram account and follows people and likes their posts

InstaFollower An script where it logs in your instagram account and follows people and likes their posts (uses the tags to fetch people) Requirements:

Simple, yet effective moderator bot for telegram. With reports, logs, profanity filter and more :3

👹 Samurai Telegram Bot Simple, yet effective moderator bot for telegram. With reports, logs, profanity filter and more :3 Description Personal bot, m

Bombcrypto-bot - A python bot that automatically logs in, clicks the new button, and sends heroes to work in the bombcrypto game RichWatch is wrapper around AWS Cloud Watch to display beautiful logs with help of Python library Rich.
RichWatch is wrapper around AWS Cloud Watch to display beautiful logs with help of Python library Rich.

RichWatch is TUI (Textual User Interface) for AWS Cloud Watch. It formats and pretty prints Cloud Watch's logs so they are much more readable. Because

Releases(v1.0.0)
Owner
Nexthink
Nexthink
A collection of tools for managing Jira issues for the RHODS project

RHODS-Jira-Tools A collection of tools for managing Jira issues for the RHODS project move_to_qa.py This script handles transitioning a given Jira iss

Alex Corvin 1 Sep 20, 2022
Shuffle and add items from jellyfin to mpd (use in tandem with jellyfin-mopidy and mpd-mopidy). Similar to ncmpcpp's "Add random" feature..

jellyshuf Essentially implements ncmpcpp's add random feature (default hotkey: `) through a script which grabs info from jellyfin api itself. jellyfin

Ethan Djeric 2 Dec 14, 2021
A simple, multipurpose Discord bot.

EpicBot 🏅 A simple, multipurpose Discord bot. • Info EpicBot is a multipurpose Discord bot that was designed to make your Discord life easier and coo

Nirlep_5252_ 130 Dec 29, 2022
A modern, easy to use, feature-rich, and async ready API wrapper improved and revived from original discord.py.

A Python API wrapper that is improved and revived from the original discord.py

Orion 19 Nov 06, 2021
A Discord bot made by QwertyIsCoding

QwertyBot QwertyBot A Discord bot made by QwertyIsCoding Explore the docs » View Demo . Report Bug . Request Feature About The Project This Discord bo

4 Oct 08, 2022
Simple Bot With Python 3.8+ For Converstaion Your Media

Media-Conversation Simple Bot With Python 3.8+ For Converstaion Your Media

Farzin 2 Dec 06, 2021
🚧 finCLI's own News API. No more limited API calls. Unlimited credible and latest information on BTC, Ethereum, Indian and Global Finance.

🚧 finCLI's own News API. No more limited API calls. Unlimited credible and latest information on BTC, Ethereum, Indian and Global Finance.

finCLI 5 Jun 16, 2022
Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.

AWS infrastructure to ship CloudTrail logs from S3 to Kinesis This repository contains a Terraform module to ship CloudTrail logs stored in a S3 bucke

Nexthink 8 Sep 20, 2022
Bot for mirroring one or multiple Twitter accounts in Pleroma/Mastodon.

Stork (pleroma-bot) Mirror one or multiple Twitter accounts in Pleroma/Mastodon. Introduction After using the pretty cool mastodon-bot for a while, I

73 Jan 08, 2023
Asyncevents: a small library to help developers perform asynchronous event handling in Python

asyncevents - Asynchronous event handling for modern Python asyncevents is a small library to help developers perform asynchronous event handling in m

Mattia 5 Aug 07, 2022
Python client for using Prefect Cloud with Saturn Cloud

prefect-saturn prefect-saturn is a Python package that makes it easy to run Prefect Cloud flows on a Dask cluster with Saturn Cloud. For a detailed tu

Saturn Cloud 15 Dec 07, 2022
TON Miner from TON-Pool.com

TON-Pool Miner Miner from TON-Pool.com

21 Nov 18, 2022
A Discord Server Cloner Which Can Clone Any Discord Server In Just Few Minutes

A Discord Server Cloner Which Can Clone Any Discord Server In Just Few Minutes.

samet 4 Jul 23, 2022
Programmeertheorie 2022 - Team Trainspotters - RailNL

Trainspotters Vak: Programmeertheorie 2022 Gekozen case: RailNL Teamnaam: Trainspotters Studenten: Mijntje Meijer, Sam Bijhouwer, Maik Larooij To-do's

Maik Larooij 1 Jan 25, 2022
a Music bot for discord

Bot this is a discord bot made by AnHalfGuy.py#6031(ID: 747864072879603743) and HastagStopAnimalAbuse#5617(ID :349916852308279306) This Bot Is For Mus

A Discord Bot Development 1 Oct 29, 2021
Instagram-follower-bot - An Instagram follower bot written in Python

Instagram Follower Bot An Instagram follower bot written in Python. The bot follows the follower of which account you want. e.g. (You want to follow @

Aytaç Kaşoğlu 1 Dec 31, 2021
42-event-notifier - 42 Event notifier using 42API and Github Actions

42 Event Notifier 42서울 Agenda에 새로운 이벤트가 등록되면 알려드립니다! 현재는 Github Issue로 등록되므로 상단

6 May 16, 2022
domhttpx is a google search engine dorker with HTTP toolkit built with python, can make it easier for you to find many URLs/IPs at once with fast time.

domhttpx is a google search engine dorker with HTTP toolkit built with python, can make it easier for you to find many URLs/IPs at once with fast time

Naufal Ardhani 59 Dec 04, 2022
A simple Telegram bot that converts a phone number to a direct whatsapp chat link

Open in WhatsApp I was using a great app to open a whatsapp chat with a given number directly without saving that number in my contact list, but I fel

Pathfinder 19 Dec 24, 2022
A discord token creator that uses the service capmonster for captcha solving!

Discord Token Creator A discord token creator that uses the service capmonster for captcha solving! Report Bug · Request Feature Features Autojoin dis

dropout 41 Oct 25, 2021