Archive tweets and make them searchable

Overview

Tweeter

Archive and search your tweets and liked tweets using AWS Lambda, DynamoDB and Elasticsearch.

Note: this project is primarily being used a test bed for figuring out best practices with AWS Lambda

Architecture

Requirements

Make sure you have the following installed before you proceed

Setup

Twitter API key setup

Set up a Twitter Developer account. Once you are signed up, create an app.

Add the credentials to SSM Parameter Store:

aws ssm put-parameter --name /tweeter/twitter/consumer_key --value <your consumer key value> --type SecureString --overwrite
aws ssm put-parameter --name /tweeter/twitter/consumer_secret --value <your consumer secret value> --type SecureString --overwrite
aws ssm put-parameter --name /tweeter/twitter/access_token --value <your access token value> --type SecureString --overwrite
aws ssm put-parameter --name /tweeter/twitter/access_token_secret --value <your access token secret value> --type SecureString --overwrite

Elastic Cloud setup

Elastic Cloud is not free

Set up an account with Elastic Cloud. Create a deployment and then an Elastic App Search engine. Retrieve the private key from the Credentials section.

Add the private key to SSM.

aws ssm put-parameter --name /tweeter/es/private_key --value <your password> --type SecureString --overwrite

Also, update the configuration in ui/src/config/engine.json, specifically the values of endpointBase and searchKey.

Deploy the backend

Build and deploy your application for the first time by running the following commands in your shell:

make build
make deploy.guided

The first command will build the source of your application within a Docker container. The second command will package and deploy your application to AWS. Guided deploy means SAM CLI will ask you about the name of your deployment/stack, AWS Region, and whether you want to save your choices, so that you can use make deploy next time.

Deploy the frontend

In the ui directory, deploy the application with AWS Amplify.

amplify publish

Use the SAM CLI to build and test locally

Whenever you change your application code, you'll have to run build command:

make build

The SAM CLI installs dependencies defined in poller/requirements.txt, creates a deployment package, and saves it in the .aws-sam/build folder.

Test a single function by invoking it directly with a test event:

make invoke.poller
make invoke.indexer

Working with localstack

Validate DynamoDB was populated using Localstack:

docker-compose up -d
make invoke.poller
aws --endpoint-url=http://localhost:4566 dynamodb scan --table-name TweetsTable --max-items 10

An event is a JSON document that represents the input that the function receives from the event source. Test events are included in the events folder in this project.

Fetch, tail, and filter Lambda function logs

To simplify troubleshooting, SAM CLI has a command called sam logs. sam logs lets you fetch logs generated by your deployed Lambda function from the command line. In addition to printing the logs on the terminal, this command has several nifty features to help you quickly find the bug.

NOTE: This command works for all AWS Lambda functions; not just the ones you deploy using SAM.

sam logs -n TweeterPoller --stack-name tweeter --tail

You can find more information and examples about filtering Lambda function logs in the SAM CLI Documentation.

CI/CD with GitHub

Backend

Create a user github with the following policy: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-permissions.html.

Create secrets with AWS.

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Frontend

Follow the AWS Amplify console to configure the frontend.

Notes

  1. Set StreamModeEnabled to false in samconfig.toml the first time you run the poller to get all tweets.
  2. Twitter apparently only allows access to a user's most recent 3,240 tweets with this method.
  3. This probably isn't the best use case for Lambda, which has a timeout of 15 minutes. Ideally we should be calling Fargate for long running task. That said, in practice, because of the limitations of Twitter's API, getting all the available tweets is well within that limit.

Appendix: Powertools

Tracing

Tracer utility patches known libraries, and trace the execution of this sample code including the response and exceptions as tracing metadata - You can visualize them in AWS X-Ray.

Logger

Logger utility creates an opinionated application Logger with structured logging as the output, dynamically samples 10% of your logs in DEBUG mode for concurrent invocations, log incoming events as your function is invoked, and injects key information from Lambda context object into your Logger - You can visualize them in Amazon CloudWatch Logs.

Metrics

Metrics utility captures cold start metric of your Lambda invocation, and could add additional metrics to help you understand your application KPIs - You can visualize them in Amazon CloudWatch.

License

MIT

Trellox Tool is written in Python3 and designed to pull and list Trello boards.

TrelloX Trellox Tool is written in Python3 and designed to list and pull Trello boards. It can be used by penetration testers/bug bounty hunters to de

Ali Fathi Ali Sawehli 1 Dec 05, 2021
PyMusic Player is a music player written in python3.

PyMusic Player is a music player written in python3. It harvests r

PythonSerious 2 Jan 30, 2022
A python wrapper for interacting with the LabArchives API.

LabArchives API wrapper for Python A python wrapper for interacting with the LabArchives API. This very simple package makes it easier to make arbitra

Marek Cmero 3 Aug 01, 2022
Role Discord Members (by username) from File

Role Discord Members (by username) from File Bot Setup Navigate to https://discord.com/developers/applications Create a new application Navigate to th

Dylan Orrell 3 Jan 06, 2022
BioThings API framework - Making high-performance API for biological annotation data

BioThings SDK Quick Summary BioThings SDK provides a Python-based toolkit to build high-performance data APIs (or web services) from a single data sou

BioThings 39 Jan 04, 2023
Thread-safe Python RabbitMQ Client & Management library

AMQPStorm Thread-safe Python RabbitMQ Client & Management library. Introduction AMQPStorm is a library designed to be consistent, stable and thread-sa

Erik Olof Gunnar Andersson 167 Nov 20, 2022
Example app to be deployed to AWS as an API Gateway / Lambda Stack

Disclaimer I won't answer issues or emails regarding the project anymore. The project is old and not maintained anymore. I'm not sure if it still work

Ben 123 Jan 01, 2023
A telegram bot writen in python for mirroring files on the internet to our beloved Google Drive

[] Mirror Bot This is a telegram bot writen in python for mirroring files on the internet to our beloved Google Drive. Deploying on Heroku Give Star &

43 Mar 06, 2022
Remedy when Amazon ECR is not running basic scans for container CVEs.

Welcome to your CDK Python project! This is a blank project for Python development with CDK. The cdk.json file tells the CDK Toolkit how to execute yo

4n6ir 4 Nov 05, 2022
Basic Discord python bot

#How to Create a Discord Bot Account In order to work with the Python library and the Discord API, we must first create a Discord Bot account. Here ar

Tustus 1 Oct 13, 2021
Dashboard to monitor the performance of your Binance Futures account

futuresboard A python based scraper and dashboard to monitor the performance of your Binance Futures account. Note: A local sqlite3 database config/fu

86 Dec 29, 2022
Housing Price Prediction Using Machine Learning.

HOUSING PRICE PREDICTION USING MACHINE LEARNING DESCRIPTION Housing Price Prediction Using Machine Learning is to predict the data of housings. Here I

Shreya Shree Padhi 1 Aug 03, 2022
ВКонтакте бот для управления Sugar кошельком

Sugarchain VK ВКонтакте бот для управления Sugar кошельком Установка Установить зависимости можно командой: pip install -r requirements.txt Запуск (из

Vladimir 4 Jun 06, 2021
Hcl.py is an Amino client for Python

Hcl.py Hcl.py Hcl.py is an Amino client for Python. It provides to access aminoapps Web, app and socket servers. Developed BY Kapidev And Upgraded BY

Oustex 3 Dec 02, 2021
An inline real-time media searching robot without any database.

MediaBuddy A Telegram Inline media searching robot without any database. About mediaBuddy is an inline media searching robot. If you have so many movi

Renjith Mangal 28 Oct 21, 2022
A simple tool that allows you to change your default AWS CLI profile.

Select AWS Profile Select AWS Profile (slapr) is a simple tool that lets you select which AWS Profile you want to use and sets it as the default AWS p

Antoni Yanev 2 Nov 09, 2022
List of twitch bots n bigots

This is a collection of bot account names NamelistMASTER contains all the names we reccomend you ban in your channel Sometimes people get on that list

62 Sep 05, 2021
TikTok 4L and 4C checker that doesn't count banned usernames as available

TikTok 4L and 4C checker that doesn't count banned usernames as available. Once a username is available, it will send it to your Discord Webhook.

cliphd 26 May 01, 2022
A Python script that exports users from one Telegram group to another using one or more concurrent user bots.

ExportTelegramUsers A Python script that exports users from one Telegram group to another using one or more concurrent user bots. Make sure to set all

Fasil Minale 17 Jun 26, 2022
Price checker windows application

Price-Checker price checker windows application This application monitors the prices of selected products and displays a notification if the price has

Danila Tsareff 1 Nov 29, 2021