Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]" 

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
Lending-Club-Loans - Using TensorFlow to create an ANN model to predict whether people would charge off or pay back their loans.

Lending Club Loans: Brief Introduction LendingClub is a US peer-to-peer lending company, headquartered in San Francisco, California.[3] It was the fir

Ali Akram 1 Jan 03, 2022
A stack-based systems language that supports structures, functions, expressions, and user-defined operator behaviour

A stack-based systems language that supports structures, functions, expressions, and user-defined operator behaviour. Currently compiles to URCL with plans to add additional formats in the future.

Lucida Dragon 3 Nov 03, 2022
Group Chat Spammer For Discord

Group Chat Spammer For Discord Free and public gc spammer

Dreamy 20 Dec 27, 2022
Telegram bot to stream videos in telegram voicechat for both groups and channels

Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live streams, YouTube videos and telegram media. With record stream support, Schedule streams, and many more

ALBY 9 Feb 20, 2022
Paginator for Dis-Snek Python Discord API wrapper

snek-paginator Paginator for Dis-Snek Python Discord API wrapper Installation: pip install -U snek-paginator Basic Example: from dis_snek.client impo

1 Nov 04, 2021
wrapper for facebook messenger

pyfacebook pyfacebook library for python. Requirements common Help Got a question? File a GitHub issue. Contributing Bug Reports & Feature Requests Pl

Luis Mayta 3 Nov 12, 2021
A simple program to display current playing from Spotify app on your desktop

WallSpot A simple program to display current playing from Spotify app on your desktop How to Use: Linux: Currently Supports GNOME and KDE. If you want

Nannan 4 Feb 19, 2022
Telegram üzerinden paylaşılan kısa linkleri geçmenin daha hızlı bir yolu

Telegram Url skipper Telegramda paylaşılan kısa linkleri geçmenin daha hızlı bir yolu · Hata Raporla · Öneri Yap İçerik Tablosu Kurulum Kullanım Lisan

WarForPeace 6 Oct 07, 2022
asyncio client for Deta Cloud

aiodeta Unofficial client for Deta Clound Install pip install aiodeta Supported functionality Deta Base Deta Drive Decorator for cron tasks Examples i

Andrii Leitsius 19 Feb 14, 2022
WakeNote is a tool that hides notifications from you until you confirm you want to read them, with technology to help prevent the reading of depressing messages first thing in the morning.

By: Seanpm2001, Et; Al. Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afrikaans |

Sean P. Myrick V19.1.7.2 3 Oct 21, 2022
Получение интересной информации о любой пиццерии Додо

dodopizza-abuse Получение инфорации о выбранной пиццерии Додо Установка и запуск на Linux Устанавливаем git и python: apt-get update && apt-get -y ins

Хозя 24 Nov 02, 2022
Source code for Profile REST API

PROJECT PROFILE REST API Creating local development server: We will create a local development server that can run and test our API as we build it. We

1 Mar 29, 2022
An advanced Filter Bot with nearly unlimitted filters!

Unlimited Filter Bot ㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤ An advanced Filter Bot with nearly unlimitted filters! Features Nearly unlimited filters Supports all type of fil

1 Nov 20, 2021
Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automatically.

DATE_TIME_USERBOT-TeLeTiPs Powerful Telegram userbot to turn your PROFILE PICTURE & LAST NAME into a real time clock & to change your BIO automaticall

53 Jan 05, 2023
A Pluggable And Powerful Telegram Manager Bot

FRIDAY-USERBOT 🇮🇳 A Powerful, Smart And Simple Userbot In Pyrogram. Support 🚑 Inspiration & Credits Userge-X Userge Pokurt Pyrogram Code Owners Chs

DevsExpo 371 Mar 26, 2022
Blankly - 🚀 💸 Trade stocks, cryptos, and forex w/ one package. Easily build, backtest, trade, and deploy across exchanges in a few lines of code.

💨 Rapidly build and deploy quantitative models for stocks, crypto, and forex 🚀 View Docs · Our Website · Join Our Newsletter · Getting Started Why B

Blankly Finance 1.4k Jan 03, 2023
Check and write all account info + Check nitro on account

Discord-Token-Checker Check and write all account info + Check nitro on account Also check https://github.com/GuFFy12/Discord-Token-Parser (Parse disc

36 Jan 01, 2023
WhatsApp Status Tracker With Python

Warning!! This Repo is Purly educational purpose Don't use this to stalk on others, which is subjective to crime Pre-Req: Telegram bot of your own wit

Vignesh Karunagaran 10 Dec 09, 2022
Automatic generation of crypto-arts based on image layers

NFT Generator Автоматическая генерация крипто-артов на основе слоев изображения. Установка pip3 install -r requirements.txt rm -rf result/* Как это ра

Zproger 31 Dec 29, 2022