In GitLab by @szabogtamas on Mar 25, 2020, 19:57

Merges vdj_plot -> master

Added a much faster version of #24
Fixes #24.

Test cases need to be added yet.

I am trying to install Scirpy using anaconda/Jupyter on my desktop window.

**when I try this: conda install -c conda-forge -c bioconda scirpy

I go the error message:**

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

Note: you may need to restart the kernel to use updated packages.


Building graph of deps:   0%|          | 0/2 [00:00<?, ?it/s]
Examining python=3.8:   0%|          | 0/2 [00:00<?, ?it/s]  
Examining scirpy:  50%|#####     | 1/2 [00:00<00:00,  2.94it/s]
Examining scirpy: 100%|##########| 2/2 [00:00<00:00,  5.88it/s]
                                                               

Determining conflicts:   0%|          | 0/2 [00:00<?, ?it/s]
Examining conflict for python scirpy:   0%|          | 0/2 [00:00<?, ?it/s]
                                                                           

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

THEN I tried this: pip install scirpy, I got another error message:

ERROR: Command errored out with exit status 1:
   command: 'C:\Users\tpeng\Anaconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\tpeng\\AppData\\Local\\Temp\\pip-install-jpyjqkeg\\python-levenshtein\\setup.py'"'"'; __file__='"'"'C:\\Users\\tpeng\\AppData\\Local\\Temp\\pip-install-jpyjqkeg\\python-levenshtein\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\tpeng\AppData\Local\Temp\pip-wheel-3n51vnw0'
       cwd: C:\Users\tpeng\AppData\Local\Temp\pip-install-jpyjqkeg\python-levenshtein\
  Complete output (27 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.8
  creating build\lib.win-amd64-3.8\Levenshtein
  copying Levenshtein\StringMatcher.py -> build\lib.win-amd64-3.8\Levenshtein
  copying Levenshtein\__init__.py -> build\lib.win-amd64-3.8\Levenshtein
  running egg_info
  writing python_Levenshtein.egg-info\PKG-INFO
  writing dependency_links to python_Levenshtein.egg-info\dependency_links.txt
  writing entry points to python_Levenshtein.egg-info\entry_points.txt
  writing namespace_packages to python_Levenshtein.egg-info\namespace_packages.txt
  writing requirements to python_Levenshtein.egg-info\requires.txt
  writing top-level names to python_Levenshtein.egg-info\top_level.txt
  reading manifest file 'python_Levenshtein.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '*pyc' found anywhere in distribution
  warning: no previously-included files matching '*so' found anywhere in distribution
  warning: no previously-included files matching '.project' found anywhere in distribution
  warning: no previously-included files matching '.pydevproject' found anywhere in distribution
  writing manifest file 'python_Levenshtein.egg-info\SOURCES.txt'
  copying Levenshtein\_levenshtein.c -> build\lib.win-amd64-3.8\Levenshtein
  copying Levenshtein\_levenshtein.h -> build\lib.win-amd64-3.8\Levenshtein
  running build_ext
  building 'Levenshtein._levenshtein' extension
  error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
  ----------------------------------------
  ERROR: Failed building wheel for python-levenshtein
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\tpeng\Anaconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\tpeng\\AppData\\Local\\Temp\\pip-install-jpyjqkeg\\python-levenshtein\\setup.py'"'"'; __file__='"'"'C:\\Users\\tpeng\\AppData\\Local\\Temp\\pip-install-jpyjqkeg\\python-levenshtein\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\tpeng\AppData\Local\Temp\pip-record-spj3ycwj\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\tpeng\Anaconda3\Include\python-levenshtein'
         cwd: C:\Users\tpeng\AppData\Local\Temp\pip-install-jpyjqkeg\python-levenshtein\
    Complete output (27 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.8
    creating build\lib.win-amd64-3.8\Levenshtein
    copying Levenshtein\StringMatcher.py -> build\lib.win-amd64-3.8\Levenshtein
    copying Levenshtein\__init__.py -> build\lib.win-amd64-3.8\Levenshtein
    running egg_info
    writing python_Levenshtein.egg-info\PKG-INFO
    writing dependency_links to python_Levenshtein.egg-info\dependency_links.txt
    writing entry points to python_Levenshtein.egg-info\entry_points.txt
    writing namespace_packages to python_Levenshtein.egg-info\namespace_packages.txt
    writing requirements to python_Levenshtein.egg-info\requires.txt
    writing top-level names to python_Levenshtein.egg-info\top_level.txt
    reading manifest file 'python_Levenshtein.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*pyc' found anywhere in distribution
    warning: no previously-included files matching '*so' found anywhere in distribution
    warning: no previously-included files matching '.project' found anywhere in distribution
    warning: no previously-included files matching '.pydevproject' found anywhere in distribution
    writing manifest file 'python_Levenshtein.egg-info\SOURCES.txt'
    copying Levenshtein\_levenshtein.c -> build\lib.win-amd64-3.8\Levenshtein
    copying Levenshtein\_levenshtein.h -> build\lib.win-amd64-3.8\Levenshtein
    running build_ext
    building 'Levenshtein._levenshtein' extension
    error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\tpeng\Anaconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\tpeng\\AppData\\Local\\Temp\\pip-install-jpyjqkeg\\python-levenshtein\\setup.py'"'"'; __file__='"'"'C:\\Users\\tpeng\\AppData\\Local\\Temp\\pip-install-jpyjqkeg\\python-levenshtein\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\tpeng\AppData\Local\Temp\pip-record-spj3ycwj\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\tpeng\Anaconda3\Include\python-levenshtein' Check the logs for full command output.

  Stored in directory: c:\users\tpeng\appdata\local\pip\cache\wheels\8c\51\cb\423184c62cc06c302d2f54f9853e5acee6c15a3b04d49a5eb3
  Building wheel for python-levenshtein (setup.py): started
  Building wheel for python-levenshtein (setup.py): finished with status 'error'
  Running setup.py clean for python-levenshtein
  Building wheel for yamlordereddictloader (setup.py): started
  Building wheel for yamlordereddictloader (setup.py): finished with status 'done'
  Created wheel for yamlordereddictloader: filename=yamlordereddictloader-0.4.0-py3-none-any.whl size=4058 sha256=05980b7e37960621917874dd26c59adf1eb9d304ca1f21d7c81d38adc8bc2674
  Stored in directory: c:\users\tpeng\appdata\local\pip\cache\wheels\50\9a\6f\9cb3312fd9cd01ea93c3fdc1dbee95f5fa0133125d4c7cb09a
Successfully built airr yamlordereddictloader
Failed to build python-levenshtein
Installing collected packages: yamlordereddictloader, airr, python-levenshtein, squarify, parasail, pytoml, scirpy
    Running setup.py install for python-levenshtein: started
    Running setup.py install for python-levenshtein: finished with status 'error'

The original issue

Id: 9
Title: List of plots

could not be created. This is a dummy issue, replacing the original one. It contains everything but the original issue description. In case the gitlab repository is still existing, visit the following link to show the original issue:

TODO

Dear authors,

Currently, the ir_neighbors algorithm consumes over 100G of memory of data from 55k cells, causing memory failures on the server. Is there any way to limit memory consumption and is this normal behavior, which parameters can I adapt to control memory usage ?

Kind Regards,

BCR support meta issue.

Initial PR #183 addresses:

• [x] change datastructure

• Instead of TRA_1, TRA_2, TRB_1 and TRB_2 have arm1_primary, arm1_secondary, arm2_primary and arm2_secondary
• have additional 4 columns: arm1_primary_type, ...; These can accept values such as TRA, TRB, TRG, TRD, IGH, IGL (see AIRR locus names)

• [x] adapt chain_categories to identify bona fide vs other pairings (e.g. flag a cell with TRA + IGH).
• [x] clonotype network: separate by receptor_type (per default, no connections between BCR and TCR)
• [x] update glossary and documentation. Make clear that now there are VJ and VDJ chains. (mostly done)
• [x] rename tcr_ to cdr3_ or receptor_ or vdj_ or ir_

To be resolved before next release (v0.5 "adding experimental BCR support")

• [x] #194 vdj_usage is broken with new data structure
• [x] #195 improve BCR-related documentation
• [x] read_tracer() with gamma/delta
• [x] read_bracer()
• [x] #198 Add BCR example dataset

BCR-related issues that can be resolved at a later point

• [ ] #197 function to infer antibody class
• [ ] #196 function to infer somatic hypermutation status
• [ ] #199 BCR-tutorial
• [ ] add support for CDR1 and 2 (#185)

In GitLab by @grst on Mar 20, 2020, 18:16

The current one, sctcrpy is hard to pronounce and remember.

Also it would be nice if the name left the option to expand to BCRs later on.

imm, sc, py, receptor, cr, ... ??

Description of the bug

I try to convert the AnnData after clonal assignment by Scirpy into dandelion format. The result showed that "field productive has invalid bool T + T". I would like to convert it for updating germline sequence of each BCR sequences using dandelion because I did not find this function in Scirpy. However, if you could suggest the other ways. Feel free to let me know.

Minimal reproducible example

import scirpy as ir

ABC_irdata_exclude_orphan_dandelion = ir.io.to_dandelion(ABC_irdata_exclude_orphan)
ABC_irdata_exclude_orphan_dandelion

The error message produced by the code above

~/.conda/envs/dandelion/lib/python3.8/site-packages/airr/schema.py in validate_row(self, row)
    276                 if spec == 'number':  self.to_float(row[f], validate=True)
    277             except ValidationError as e:
--> 278                 raise ValidationError('field %s has %s' %(f, e))
    279 
    280         return True

ValidationError: field productive has invalid bool T + T

Version information

Dear ICBI lab,

I have some questions regarding the Scirpy package, I would really appreciate your help. For my analyses I first merged TCR and transcriptomics data from two different samples (organ 1 and organ 2), subsequently I then merged these two files.

1. a) If using the Scirpy clonotype_network tool the 'sequence' is set to 'nt', are the clusters in the clonotype network (where each node represents a cell) based on identical nucleotide sequences, or is it based on similarity, meaning that one cluster could consists of cells with slightly different nucleotide sequences?

As I understand, each node represents a cell, and edges connect cells belonging to the same clonotype. The function makes visualization of the clonotype-network possible, analogous to the construction of a neighborhood graph from transcriptomics data with the Scanpy package, so that based on the above, it computes a neighborhood graph of CDR3 nucleotide sequences with "scirpy.pp.tcr_neighbors())". However answer to my question I couldn't find, I was hoping you can help me out?

b) Do the lines that connect the cells within a cluster have any meaning?

c) In the plot above, is it correct that the closer the different clusters are together, the more similar their nucleotide sequences are? Meaning that the sequences of the clonotypes consisting of only 2 cells in this case are most different from the clonotype clusters consisting of >5 cells (as they are further apart)?

2. a) The package allows for specifying what organs the clusters consist of:

My question is: how can I select specific clusters in the clonotype network graph (I only would like to include the clonotypes in the network that have identical clonotypes between the two different samples from my data, i.e. organ 1 and organ 2), meaning that in the plot above I would like to filter for the clusters only that display both blue and orange nodes.

b) How can I assign numbers to my clusters based on identical nucleotide sequences shared between two samples?

c) How can I add a legend to my plot, and how can I change the name 'batch' into 'sample'?

3. Using Scirpy, how can one best specify differentially expressed genes (based on the transcriptomics data) between the different clusters based on shared nucleotide sequences between blood and fat, for example cluster 1 vs. rest of the clusters, and cluster 1 and 2 vs. cluster 3 and 4? I have tried implementing Scanpy's tool to rank genes using WIlcoxon, but unfortunately I can't make It work.

sc.tl.rank_genes_groups(adata, 'clonotype', groups=['1','2'], reference=['3','4'], method='wilcoxon') sc.pl.rank_genes_groups(adata, groups=['1', '2'], n_genes=20)

-The key of the observations grouping to be considered would be: "clonotype clusters based on share identical nucleotide sequences between organ 1 and organ 2". -Subset of groups to which comparison wouldl be restricted: "clonotype 1 and clonotype 2" -Comparison: Compare with respect to a specific group -Group identifier with respect to which compare: "clonotype 3 and clonotype 4" “The number of genes that appear in the returned tables”: 100 “Method”: Wilcoxon-Rank-Sum

Thanks in advance,

Josine

I have a question about BD Rhapsody CDR3 VDJ data import. Following scirpy introduction, imported data object is very similar with BD VDJ data.

If you have an interest wiht BD VDJ data, please check the possibility of BD VDJ data import into scirpy. I can share my data to you.

In GitLab by @grst on Mar 2, 2020, 14:21

Merges feat/plot-overhaul -> master

This PR aims at addressing issues in %"plot overhaul".

• [x] Simplify and restructure tools (Don't add to uns when inexpensive, Fixes #25)
• [x] Rudimentary support for figure themes (atm, only a default theme is supported, but can be extended easily, Fixes #18)
• [ ] Simplify and restructure plots

Plot checklist: Make sure every plotting function

• [x] accepts
  • styling kwargs
  • ax object
  • (this is implicitly given by kwargs forwarding. Maybe requires better documentation)
• [x] returns ax object
• [x] has sensible defaults for ax labelling and title

In GitLab by @grst on Jan 30, 2020, 12:59

Tools are functions that work with the data parsed from 10x/tracer and add either

• new columns to obs
• new matrices to obsm (e.g. distance matrices)
• other summary data to uns.

They are usually required as an additional processing step before running certain plotting functions. Here's a list of tools we want to implement.

@szabogtamas, feel free to add to/edit the list.

List of tools

• [x] st.tl.define_clonotypes(adata) assignes clonotypes to cells based on their CDR3 sequences
• [x] st.tl.tcr_dist(adata, chains=["TRA_1, "TRB_1"], combination=np.min) adds TCR dist to obsm (#11)
• [x] st.tl.kidera_dist adds Kidera distances to obsm
• [x] st.tl.chain_convergence(adata, groupby) adds column to obs that contains the number of nucleotide versions for each CDR3 AA sequence
• [x] st.tl.alpha_diversity(adata, groupby, diversityforgroup) Now we were only thinking about calculating diversity of clonotypes in different groups. But the diversity of any group could just as well be calculated.
• [ ] st.tl.sequence_logos(adata, ?forgroup?) Precompute MSAs and sequence logos for plotting with st.pl.sequence_logos.
• [ ] st.tl.dendrogram(adata, groupby) Compute a dendrogram on an arbitrary distance matrix (e.g. from tcr_dist).

Needs discussion

• [ ] st.tl.create_group(group_membership={Group1: ['barcode1', barcode2']} adds a group membership to each cell by adding a column to obsm and the name of the grouping to a list in uns (by default, groups based on samples, V gene usage and even clonotypes could be created at initial run); might call chain_convergence and alpha_diversity functions to calculate these measures right when creating a group

Ideas, might be implemented at later stage

• [ ] Shared Kmers
• [ ] GLIPH
• [ ] Chains recognizing the same eiptopes based on McPAS-TCR
• [ ] epitope reactivity -> query external database
• [ ] tcellmatch (Fischer, Theis et al. )

Description of feature

The first reasonable methods to predict antigen specificity emerge, e.g.

• ERGO-II (https://www.frontiersin.org/articles/10.3389/fimmu.2021.664514/full, https://github.com/IdoSpringer/ERGO-II)

These are conceptually different from querying databases through sequence distance metrics or autoencoders, as they do not simply model the sequence similarity, but explicitly model the specificity.

Would be nice to call them directly from scirpy.

@FFinotello, potentially another good student task.

Description of feature

I have been playing with omniscope's COVID dataset that provides 8M TCR receptors. By doing so, I identified several bottlenecks that make working with >1M cells in scirpy painful or impossible.

This meta issue is to give an overview of the progress improving scirpy's scalability.

graph TB
    subgraph legend
         legend1(could be faster -- minutes)
         OK(OK -- seconds)
         legend2(prohibitively slow -- hours)
         legend3(not profiled yet)
         style legend1 stroke:#ff7f00
         style OK stroke:#4daf4a
         style legend2 stroke:#e41a1c
    end

graph TB
    subgraph preprocessing
      IO --> QC
      QC --> dist_id[ir_dist identity]
      QC --> dist_levenshtein[ir_dist levenshtein]
      QC --> dist_alignment[ir_dist alignment]
      dist_id --> define_clonotypes
      dist_levenshtein --> define_clonotypes
      dist_alignment --> define_clonotypes
      define_clonotypes --> clonotypes
      QC -.-> autoencoder
      autoencoder -.-> clonotypes
      autoencoder -.-> define_clonotypes

      clonotypes[(CLONOTYPES)]
      
      style IO stroke:#ff7f00
      style QC stroke:#4daf4a
      style dist_id stroke:#4daf4a
      style define_clonotypes stroke:#e41a1c
      style dist_levenshtein stroke:#e41a1c
      style dist_alignment stroke:#e41a1c
      style clonotypes stroke:white
   end
   
   subgraph downstream
      clonotypes --> clonotype_network
      clonotypes --> other[other tools]
   end

Action items

1. data structure (#356). The foundation for other changes. Might also speed up saving the anndata object.
2. reading data (#367). User experience can be improved, but not a top priority atm.
3. ir_dist (#304). Needs more scalable methods for computing sequence distances.
4. define_clonotypes (#368). At the very least needs a better parallelization. Maybe there's room for some jax/numba.
5. autoencoder-based embedding (#369). Possible alternative to ir_dist. Maybe it even makes sense to combine ir_dist and define_clonotypes into a single step.

Description of feature

IMO autoencoder-based sequence embedding has a huge potential for finding similar immune receptors, potentially improving both the speed and the accuracy compared to alignment-based metrics. In particular, finding similar sequences is important in two scirpy functions:

• defining clonotypes
• querying immune receptor databases.

For the database query, an online-update algorithm similar to scArches for gene expression would be nice: The autoencoder could be trained on the database (which might have millions of unique receptors) once. A new dataset (which might only have 10k-100k unique receptors), could be projected into the same latent space as the database, significantly improving query time.

An extension to this idea is to embed gene expression and TCR/BCR data into the same latent space.

Existing tools

• Trex by @ncborcherding. Based on keras.
• mvTCR by @b-schubert's lab. Combines receptor/Gex data. Based on pytorch.
• TESSA. Combines receptor/Gex data. Not even sure it's an autoencoder, need yet to check in detail, but it seems to use some clever sequence embeddings.
• There are likely more...

@drEast mentioned he is working on something like that a few months ago. Are you willing to share a few details and if you would be interested in integrating it with scirpy? @adamgayoso, any chance there's AirrVI soon? :stuck_out_tongue_winking_eye:

Description of feature

The define_clonotypes function scales badly. There are two problems with it

• it could be faster (while it relies heavily on numpy, there are parts implemented in Python)
• parallelization doesn't work properly with large data. Due to how multiprocessing is implemented in Python, parallelization involves a lot of copying. If parallelization worked properly, the speed would still be bearable if one throws enough cores at the problem.

Where's the bottleneck of the function?

INPUT:

• 2 distance matrices, one for unique VJ sequences, one for unique VDJ sequences

OUTPUT:

• a clonotype id for each cell

CURRENT IMPLEMENTATION:

1. compute unique receptor configurations (i.e. combining cells with the same sequences into a single entry) (fast)
2. build a lookup table from which the neighbors of each cell can be retrieved (fast enough)
3. loop through all unique receptor configurations and find neighbors (SLOW)
4. build a distance matrix (fast)
5. graph partition using igraph (fast)

ALTERNATIVE IMPLEMENTATIONS I considered but discarded

• reindexing sequence distance matrices such that they match the table of unique receptor configurations
• Then perform matrix operations to combine primary/secondary and TRA/TRB matrices.
• The problem with this approach is that large dense blocks in the sparse matrices can arise if many unique receptors have the same sequence (e.g. same TRA but different TRB).

Possible solutions

• fix parallelization (shared memory)
• reimplement using jax/numba (this may also solve the parallelization and provide GPU support)
• Combine 2-4 into a single step (maybe possible with sequence embedding -- see #369 ). Note that this would be an alternative route and wouldn't replace ir_dist/define_clonotypes completely.
• Special-casing: In the case of omniscope data (which only has TRB chains), the problem simplifies to reindexing a sparse matrix. If using only one pair of sequences per cell, the problem is likely also simpler.

Description of feature

Loading AIRR data with 1.5M rows takes ~10 minutes. This is not too bad, but could be made less annoying

• Make validation optional (I expect the validation of the airr implementation takes a good chunk of that time)
• Parallelization (read different parts of the file in parallel - or first read into pandas, parse several chunks of the dataframe in parallel)
• Progress bar (this at least shows that this will eventually finish)