requirements.txt now includes update to scipy and leidenalg

new tests folder added with 7 unit tests for functions which rely on scipy/leidenalg

cluster function in cluster.py refactored to separate community detection functions for testing

Note: Maybe we should create a "version 1.5.7" branch as mentioned?

Hello, I have installed PhenoGraph 1.5.7 together with the required libraries. PhenoGraph breaks when trying to run the example notebook not "line by line" but in full (have a try with this: PhenoGraph_demo_random.zip). Indeed, PhenoGraph crashes during internal execution of sort_by_size function, starting an infinite loop in which 4 identical plots are created.

When I set n_jobs = 4 or I omit that parameter, the execution crashes during Neighbors computation:

Instead, if I set n_jobs = 1, the execution stops at sort_by_size function:

In both cases the error, related to multiprocessing, is the following:

How could we solve that?

Numpy version is 1.18.5. Running phenograph.cluster(pca_df), where pca_df is either a np.ndarray or pd.DataFrame, gives the following error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-127-8f4599a5b0ca> in <module>
      2 
      3 # Cluster and cluster centrolds
----> 4 communities, graph, Q = phenograph.cluster(pca_df.values)
      5 communities = pd.Series(communities, index=pca_df.index)

~/software/miniconda3/lib/python3.8/site-packages/phenograph/cluster.py in cluster(data, clustering_algo, k, directed, prune, min_cluster_size, jaccard, primary_metric, n_jobs, q_tol, louvain_time_limit, nn_method, partition_type, resolution_parameter, n_iterations, use_weights, seed, **kargs)
    243     else:
    244         del d
--> 245         graph = neighbor_graph(kernel, kernelargs)
    246         print(
    247             "Jaccard graph constructed in {} seconds".format(time.time() - subtic),

~/software/miniconda3/lib/python3.8/site-packages/phenograph/core.py in neighbor_graph(kernel, kernelargs)
     82     :return graph: n-by-n COO sparse matrix
     83     """
---> 84     i, j, s = kernel(**kernelargs)
     85     n, k = kernelargs["idx"].shape
     86     graph = sp.coo_matrix((s, (i, j)), shape=(n, n))

~/software/miniconda3/lib/python3.8/site-packages/phenograph/core.py in parallel_jaccard_kernel(idx)
    152         graph.data[i] = tup[1]
    153 
--> 154     i, j = graph.nonzero()
    155     s = graph.tocoo().data
    156     return i, j, s[s > 0]

~/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py in nonzero(self)
    774 
    775         # convert to COOrdinate format
--> 776         A = self.tocoo()
    777         nz_mask = A.data != 0
    778         return (A.row[nz_mask], A.col[nz_mask])

~/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py in tocoo(self, copy)
    904         the resultant coo_matrix.
    905         """
--> 906         return self.tocsr(copy=False).tocoo(copy=copy)
    907 
    908     def tolil(self, copy=False):

~/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/lil.py in tocsr(self, copy)
    475         indices = np.empty(nnz, dtype=idx_dtype)
    476         data = np.empty(nnz, dtype=self.dtype)
--> 477         _csparsetools.lil_flatten_to_array(self.rows, indices)
    478         _csparsetools.lil_flatten_to_array(self.data, data)
    479 

_csparsetools.pyx in scipy.sparse._csparsetools.lil_flatten_to_array()

_csparsetools.pyx in scipy.sparse._csparsetools._lil_flatten_to_array_int32()

TypeError: Expected list, got numpy.ndarray

PhenoGraph seems to break down with the newest versions of scipy (1.5 and 1.6). When running the algorithm, the following error occurs:

TypeError: Expected list, got numpy.ndarray

Full error code can be found here: https://github.com/theislab/scanpy/issues/1407

Any ideas or updates on how to resolve this?

Hi, I have a question about the parameter k, the number of nearest neighbors to use, is there a way to choose the best k value? like comparing the modularity for each k and choose based on some criteria. Thanks.

I updated my Phenograph to 1.5.7 in order to work with latest scipy version which is required by latest scanpy. A Leiden >=0.8.2 was installed as part of PhenoGraph requirement. However, after testing, I believe there is some change in the latest Leiden that leads to Phenograph failure.

Downgrading leiden to 0.8.1 works.

Hi there!

Just upgraded my PhenoGraph with pip install -U PhenoGraph, and after that I got an error when running PhenoGraph, stating that permission was denied to my [path_to_env]lib/python3.7/site-packages/phenograph/louvain/linux-convert file. Same for the other two linux- files in that folder. After changing permissions to those three files with chmod +x, the error was solved. Not sure if it was a glitch in my system or a problem with the PhenoGraph package, but maybe you guys could check :)

Fixed the type error which results from the upgrade to Scipy > 1.4.1. This previously resulted from the way the sparse lil graph was constructed (using ndarray instead of list values) which requires lists as arguments.

Environment:

• macOS Catalina
• Python 3.7.6
• Jupyter Notebook
• matplotlib

How to Reproduce:

• Create a new Juypyter Notebook
• Draw a plot using matplotlib (e.g. scatter plot)
• Run PhenoGraph multiple times

Tentative Solution:

Unfortunately, there is no proper solution to this problem yet. We will revisit this once the following issue is resolved on the matplotlib side: https://github.com/matplotlib/matplotlib/issues/15410

In the meantime, you can avoid this problem by doing one of the following:

1. Do not upgrade to macOS Catalina. Also, it seems that the problem doesn't happen to Linux users (e.g. HPC, AWS, GCP)
2. Upgrade to Python 3.8.2.
3. Set n_jobs to 1. (note that this disables multiprocessing, thus PhenoGraph will run slow on a huge matrix, but the performance difference is usually negligible if you work with relatively small matrices)
4. Use a different backend engine for matplotlib (e.g. no inline plotting)
5. If possible, run PhenoGraph first before importing the matplotlib library.

Fixed the type error which results from the upgrade to Scipy > 1.4.1. This previously resulted from the way the sparse lil graph was constructed (using ndarray instead of list values) which requires lists as arguments.

Hi,

I often work with large amount of data (i.e., number of samples >>> number of features). So, I conduct clustering analysis by selecting a subset to determine the desired number of clusters. Then, I query the remaining data to assign unseen points to their respective clusters.

I was wondering if the produced graph by PhenoGraph package can be used to mimic the process outlined above. If so, could you provide guidelines on how to accomplish this.

Many thanks,

Ivan

Hi I run this PhenoGraph tutorial "tutorial_pbmc3k.ipynb" But the “phonograph.cluster()” function can’t successfully calculate the communities, all the communities are zeros. But there is no error or warning. Is there something wrong with this function? IPython 7.20.0 jupyter_client 6.1.11 jupyter_core 4.7.1 jupyterlab 3.0.7 notebook 6.2.0 scanpy==1.6.1 anndata==0.7.5 umap==0.4.6 numpy==1.19.2 scipy==1.4.1 pandas==1.2.1 scikit-learn==0.24.1 statsmodels==0.12.1 python-igraph==0.8.3 louvain==0.7.0 leidenalg==0.8.1

Thanks Dan