Exercise to teach a newcomer to the CLSP grid to set up their environment and run jobs

Overview

Onboarding Exercise

You will make a list of the most common URLs in all the tweets in /home/aadelucia/files/minerva/raw_tweets_deduplicated/tweets/.

Goals of this exercise

  • Learn how to submit a job to the scheduler
  • Get comfortable with Tweet JSON format
  • Learn strategies for working with large datasets (map-reduce style setup)
  • Refresh on git commands

0. Set up your workspace

You will need to be on the grid for this exercise. You can access the grid with ssh @login.clsp.jhu.edu .

Set up your .bashrc

In order to make the grid really feel like home, we need to personalize it. We can do this by setting environment variables, having specific programs run on login, etc. Different OS have different places to save your preferences. On the grid, your preferences immediately get loaded upon login from ~/.profile. This is dumb because most other automated installed edit your ~/.bashrc to modify environment variables such as your PATH.

We need to edit your ~/.profile to source your .bashrc upon login. It's some weird thing where most programs edit your .bashrc but the system doesn't look at it upon login, so this will fix that. Open your ~/.profile with your favorite editor (I use vim) and add the following

# Code from Steven at CLSP help to read bashrc on login
if [ -n "BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
    . "$HOME/.bashrc"
    fi
fi

Now you can add whatever you want to your ~/.bashrc and it will load on startup. For example, mine saves project directories as environment variables and loads a specific conda environment on login:

export PROJECT=/path/to/project
conda activate fav-env

Set up SSH keys

Okay we will need to set up multiple SSH keys in order to avoid entering our passwords over and over: (1) from your local machine to the grid (2) from the grid to GitHub (3) from your grid home directory to other nodes on the cluster.

  1. On your local machine, check if you have a key already. You can do this with ls ~/.ssh/id_rsa.pub. If that file exists, then you already have a key and skip to step 3.
  2. Generate a new private/public key pair with ssh-keygen -C local machine and accept the defaults. (-C is just a comment, this lets you know what the key is for)
  3. Copy your public key to the grid with ssh-copy-id -i ~/.ssh/id_rsa.pub @login.clsp.jhu.edu
  4. Make sure it works by ssh-ing to the grid. You should not have to type your password anymore. For debugging, contact me.
  5. On the grid, generate a new key for acessing GitHub with ssh-keygen -C CLSP grid and accept the defaults
  6. Copy the contents of ~/.ssh/id_rsa.pub and add it to your GitHub SSH keys (under Settings > SSH and GPG Keys)
  7. Add your Grid SSH key for easy node access with cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

CLSP Grid 101

Think of the grid, also referred to as a cluster or a "supercomputer" on large scales, as a community of well-organized citizens. Each citizen (node) has its role to play in the cluster society. When you log onto the cluster, the login nodes greet you. Their names are login and login2, and are accessed with ssh @login.clsp.jhu.edu . Their job is to welcome you to the grid and so some SCPing for files. That is all, everything else is above their paygrade. Now there are the compute nodes. They are the real heavy lifters, the real workers! They are identified by a letter and a number, such as b01 and c12. To see a full list of the nodes, run qconf -sel. These nodes are where you do all your file editing and debugging. They are accessed by first logging onto the grid and then running ssh like ssh b10. Working only on the compute nodes frees up the login node resources to welcome others to the grid.

It's easy to forget to immediately move to a compute node, so a modified ssh script can be used to access a compute node directly. From your local machine, run

ssh [email protected] -t ssh b10

This accesses the grid through the login node and then immediately ssh's onto a compute node.

Set up Anaconda

The grid has Python 3 but in order to make setting up our own environments easier, we will install the package manager Anaconda. Or, more specifically, Miniconda, which is just a lightweight version of Anaconda (doesn't come with all the packages installed).

Log on to the grid and run the following in your home directory:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod u+x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh

This downloads the Miniconda installation file and begins the installation process. I recommend installing it in your home directory, but you could also put it in the folder you created on a compute node.

After installation is over, run source ~/.bashrc to update your PATH.

To set up an environment for this project, go to the Minerva project root directory and run:

conda env create --file environment.yml
conda activate onboarding

Note: You can add conda activate onboarding to the end of your ~/.bashrc to make this environment load upon login.

Clone this repo (if you haven't already)

You will have to re-clone the repo if you already cloned using HTTPS and would like to use SSH.

  1. Pick a pretty node name that you like (list node names with ls /export/. I'm on /export/fs04/a13) and create a folder with your name in /export/ . The /home directory is small so do not store large datasets or repos there, or it could impact other users.
  2. Clone this repo with git clone [email protected]:AADeLucia/CLSP-grid-onboarding.git

1. Fill in the code

Remember from above, edit on a compute node!

analyze_tweet_urls.py has the starter code. The task is to complete the script so that it does the following:

Loops through the input files, counts the unique URLs, and saves the counts in the output directory in an easily parsible format (one output file per input file)

As I've been messing with tweets I created a small package (littlebird) for the processes I do over and over again (like iterating over GZIP'd Twitter files). It is already installed in the onboarding environment. An example of how to use littlebird is in token_count_features.py. I highly recommend looking at this script for a guide on how to do this exercise and how to use littlebird.

2. Submit the job to run

Okay your script is done, now we want to run it! There are about 86K Twitter files so we need to be smart about this or it will take 5ever. Luckily, we already wrote our script to take in a list of files, so the plan is to use a "job array" to submit multiple small jobs as part of one big job. A starter script that you need to fill in is batch_count_urls.sh.

  1. Edit batch_count_urls.sh to work with your script. I suggest starting with -t 1-2 to make sure your script works.
  2. Submit the job with qsub batch_count_urls.sh
  3. Check the progress of your job by looking at your log file or qstat -u

If your job status is Eqw this means there was an error in the job submission. To read the errors run qstat -j .

Note: when working with a job array, each job gets submitted to run with the most recent version of your code so be careful about editing your code while waiting for it to end.

3. Aggregate the counts

Create a method to aggregate the counts (in analyze_tweet_urls.py) that:

  • Loops over all the files in a given directory
  • Adds up the counts for the unique URLs
  • Saves the output (either a pickle dump or to a tab-separated (TSV) file)
  • Can be called with a command-line flag (e.g. --aggregate)

4. Make a pretty visualization

You have your results, yay! Now I want you to do something pretty with it. Make a word cloud, do a basic plot, whatever you want! This part is mostly to teach you how to use a jupyter notebook on the grid.

  1. Edit the paths in forward_notebook.sh to match your desired directory (home or where your files are). Also, change the PORT variable to a unique number (preferably above 5000).
  2. From the login node (NOT a compute node like b10. To get from a compute node to login run exit) run ~/.forward_notebook.sh to start the Jupyter Notebook. This job will stay running, so you only have to run this script again if something goes wrong.
  3. On your local machine run ssh -L {PORT}:localhost:{PORT} @login.clsp.jhu.edu (where PORT is from the earlier step) and navigate to localhost:{PORT} in your browser. If you are prompted for a token or password, open the notebook_log directory (specified in forward_notebook.sh) and take the notebook URL from the most recent log file.
  4. Make a visualization

5. Bask in the glory

Congratulations, you finished the exercise! Go reward yourself with some ice cream or send me a message so I can compliment you.

Owner
Alexandra
Computer Science graduate student at Johns Hopkins. My work is in natural language processing, data science, and machine learning.
Alexandra
BMI-Calculator: Program to Calculate Body Mass Index (BMI)

The Body Mass Index (BMI) or Quetelet index is a value derived from the mass (weight) and height of an individual, male or female.

PyLaboratory 0 Feb 07, 2022
Better GitHub statistics images for your profile, with stats from private and public repos

Better GitHub statistics images for your profile, with stats from private and public repos

Jacob Strieb 2k Dec 30, 2022
Python decorator for `TODO`s

Python decorator for `TODO`s. Don't let your TODOs rot in your python projects anymore !

Klemen Sever 74 Sep 13, 2022
MeepoBenchmark - This project aims at providing the scripts, logs, and analytic results for Meepo Blockchain

MeepoBenchmark - This project aims at providing the scripts, logs, and analytic results for Meepo Blockchain

Peilin Zheng 3 Aug 16, 2022
Alternative StdLib for Nim for Python targets

Alternative StdLib for Nim for Python targets, hijacks Python StdLib for Nim

Juan Carlos 100 Jan 01, 2023
A companion web application to connect stash to deovr

stash-vr-companion This is a companion web application to connect stash to deovr. Stash is a self hosted web application to manage your porn collectio

19 Sep 29, 2022
banking system with python, beginner friendly, preadvanced level

banking-system-python banking system with python, beginner friendly, preadvanced level Used topics Functions else/if/elif dicts methods parameters hol

Razi Falah 1 Feb 03, 2022
The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss.

The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss. Greater accuracy is achieved thanks to the line-by-line comparison of

197 Nov 14, 2022
Why write code when you can import it directly from GitHub Copilot?

Copilot Importer Why write code when you can import it directly from GitHub Copilot? What is Copilot Importer? The copilot python module will dynamica

Mythic 41 Jan 04, 2023
Usando Multi Player Perceptron e Regressão Logistica para classificação de SPAM

Relatório dos procedimentos executados e resultados obtidos. Objetivos Treinar um modelo para classificação de SPAM usando o dataset train_data. Class

André Mediote 1 Feb 02, 2022
LinuxHelper - A collection of utilities for non-technical Linux users accessible via a GUI

Linux Helper A collection of utilities for non-technical Linux users accessible via a GUI This app is still in very early development, expect bugs and

Seth 7 Oct 03, 2022
Randomly distribute members by groups making sure that every sector is represented

Generate Groups Randomly distribute members by groups making sure that every sector is represented The Scenario Imagine that you have a large group of

Jorge Gomes 1 Oct 22, 2021
Release for Improved Denoising Diffusion Probabilistic Models

improved-diffusion This is the codebase for Improved Denoising Diffusion Probabilistic Models. Usage This section of the README walks through how to t

OpenAI 1.2k Dec 30, 2022
Height 2 LDraw With python

Height2Ldraw About This project aims to be able to make a full lego 3D model using the ldraw file format (.ldr) from a height and color map, currently

1 Dec 22, 2021
Fused multiply-add (with a single rounding) for Python.

pyfma Fused multiply-add for Python. Fused multiply-add computes (x*y) + z with a single rounding. Useful for dot products, matrix multiplications, po

Nico Schlömer 18 Nov 08, 2022
Fetch data from an excel file and create HTML file

excel-to-html Problem Statement! - Fetch data from excel file and create html file Excel.xlsx file contain the information.in multiple rows that is ne

Vivek Kashyap 1 Oct 25, 2021
Ultimate Microsoft Edge Uninstaller!

Ultimate Microsoft Edge Uninstaller

1 Feb 08, 2022
Enjoyable scripting experience with Python

Enjoyable scripting experience with Python

8 Jun 08, 2022
Built with Python programming language and QT library and Guess the number in three easy, medium and hard rolls

password-generator Built with Python programming language and QT library and Guess the number in three easy, medium and hard rolls Password generator

Amir Hussein Sharifnezhad 3 Oct 09, 2021
A minimal configuration for a dockerized kafka project.

Docker Kafka Quickstart A minimal configuration for a dockerized kafka project. Usage: Run this command to build kafka and zookeeper containers, and c

Nouamane Tazi 5 Jan 12, 2022