knowengdev/general_clustering_pipeline repository overview

⁠KnowEnG's General Clustering Pipeline

This is the Knowledge Engine for Genomics (KnowEnG), an NIH BD2K Center of Excellence, General Clustering Pipeline.

This pipeline clusters a spreadsheet's columns, with various methods:

---

⁠How to run this pipeline with Our data

---

1. Clone the General_Clustering_Pipeline Repo ============

git clone https://github.com/KnowEnG-Research/General_Clustering_Pipeline.git⁠

1. Install the following, for Linux ============

apt-get install -y python3-pip libfreetype6-dev libxft-dev libblas-dev liblapack-dev libatlas-base-dev gfortran pip3 install pyyaml knpackage scipy==0.19.1 numpy==1.11.1 pandas==0.18.1 matplotlib==1.4.2 scikit-learn==0.17.1

1. Change directory to General_Clustering_Pipeline ============

cd General_Clustering_Pipeline

1. Change directory to test ============

cd test

1. Create a local directory "run_dir" and place all the run files in it ============

make env_setup

1. Use one of the following "make" commands to select and run a clustering option: ============

---

⁠How to run this pipeline with Your data

---

Follow steps 1-5 above then do the following:

⁠* Create your run directory

mkdir run_dir

⁠* Change directory to the run directory

cd run_dir

⁠* Create your results directory

mkdir results

⁠* Create run_paramters file (YAML Format)

Look for examples of run_parameters in the General_Clustering_Pipeline/data/run_files zTEMPLATE_cc_hclust.yml

⁠* Modify run_paramters file (YAML Format)

Change processing_method to one of: serial, parallel depending on your machine. processing_method: serial

set the data file targets to the files you want to run, and the parameters as appropriate for your data.

⁠* Run the General Clustering Pipeline:

• Update PYTHONPATH enviroment variable export PYTHONPATH='../src':$PYTHONPATH

• Run python3 ../src/general_clustering.py -run_directory ./run_dir -run_file zTEMPLATE_cc_net_nmf.yml

---

⁠Description of "run_parameters" file

---

spreadsheet_name = EXPR_GSE_METABRIC_lymphN_binary.tsv.gz

---

⁠Description of Output files saved in results directory

---

• Output files of all methods save row by col heatmap variances per row with name

row_variance_{method}_{timestamp}_viz.tsv

• Output files of all the methods save row by col heatmap with name row_by_col_heatmp_{method}_{timestamp}_viz.tsv

• Output files of all methods save col to cluster map with name col_labeled_by_cluster_{method}_{timestamp}_viz.tsv.

• Output files of all methods save row scores by cluster with name row_averages_by_cluster_{method}_{timestamp}_viz.tsv.

• Output files of all methods save spreadsheet with top ranked rows per column with name top_row_by_cluster_{method}_{timestamp}_download.tsv.

• All methods save three silhouette scores: silhouette overall score, silhouette per cluster score and silhouette per sample with name silhouette_{method}_{timestamp}_viz.tsv.
  
  1. silhouette overall score file: | number of clusters | silhouette score |
  2. silhouette per cluster score file: | ith clusters | corresponding silhouette score |
  3. silhouette per sample score file: | ith sample | corresponding silhouette score|