wenchaobai/miniclean_gpu repository overview

⁠MiniClean Overview

⁠What is MiniClean?

MiniClean is a single-machine out-of-core system for cleaning large graphs. It proposes

• (1) a workflow that better fits a single machine by pipelining CPU, GPU and I/O operations;
• (2) memory footprint reduction with bundled processing and data compression; and
• (3) a multi-mode parallel model for SIMD, pipelined and independent parallelism, and their scheduling to maximize CPU–GPU synergy.

⁠Getting Started

⁠Dependencies

• Testbed

  • OS: CentOS Linux 7 (Core)
  • CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz
  • GPU: NVIDIA Tesla V100
  • CPU(s): 72
  • GPU(s): 4
  • Host memory: 503G
  • Device memory: 32G

• Make & compiler

  • CMake (>=2.8)
  • GCC/G++ (>=12.2)
  • NVCC (>=12.4)
  • GPU Compute Capability (>=70)
  • Docker (28.1.1)

• Third-party libraries

  • Facebook folly library (latest version)
  • gflags (latest version)
  • GoogleTest (>= 1.11.0)
  • yaml-cpp (>= 0.6.3)

⁠Build (Skip if already inside the container)

First, install all dependencies. Then, run the following commands to build MiniClean:

git submodule init
git submodule update

Second, build yaml-cpp library for MiniClean.

cd third_party/yaml-cpp
mkdir build
cd build
cmake ..
make

Finally, build MiniClean in project root directory.

./build.sh

⁠Enter the container (Skip if running outside the container)

docker run -it --rm --gpus all miniclean_gpu:latest /bin/bash

If --gpu is not supported, install NVIDIA Container Toolkit, e.g., for Ubuntu users:

sudo apt update
sudo apt install -y nvidia-docker2

⁠RUNNING MiniClean Applications

MiniClean includes two main applications: rule discovery and error correction (which includes error detections).

⁠Quick start with examples.

To launch applications with default configurations, execute the following commands in the MINICLEAN_HOME directory (i.e., the top-level directory of MiniClean):

• For rule discovery

example/run_job.py -app discover

• For error correction

example/run_job.py -app clean

⁠Construct the dataset.

The example dataset is pre-constructed, but users can replace the raw data in [MINICLEAN_HOME]/example/demo_graph/raw/ and build a new dataset. Edit the builder configuration at [MINICLEAN]/example/graph_builder/builder.yaml to specify:

• csv_dir: Directory of raw CSV files.
• miniclean_dir: Directory for the constructed dataset.
• n_edges: The number of edges of the input graph.
• entities: Entities in the input graph. entities[i] has the label i.
• nonenum_threshold: If one attribute has distinct values that larger than numenum_threshold, it will be recognized as a hashable attribute (which is an important input of the GCRCompiler).

To construct the dataset, run:

cd [MINICLEAN_HOME]/example/graph_builder/
python builder_client.py

⁠Customize the application configurations.

• Available flags for the running script ([MINICLEAN_HOME]/example/run_job.py)

  • -app: Application to run. Available options: discover or clean.
  • -debug: Run the application in debug mode (i.e., launch this application with GDB).
  • -record: Export the runtime log to the directory specified in the application configuration (i.e., global_config->log_dir).
  • -nvprof: Enable nvidia profiling.
  • -nsys_profile: Enable nvidia nsight system profiling.

• Error correction jobs ([MINICLEAN_HOME]/example/configs/clean_app.yaml):

  • app: Path to executable file of the discovery application.
  • jobs: Jobs to run. The jobs listed here should be configured in the job_pool.
  • global_config->host_mem_GiB: Maximum available host memory size.
  • global_config->device_mem_GiB: Maximum available device memory size.
  • global_config->mem_slot_GiB: The memory allocating unit.
  • global_config->max_write_granularity_MiB: The maximum amout of a single write access.
  • global_config->parallelism: CPU parallelism.
  • global_config->num_tasks: CPU workers. num_tasks should larger than parallelism for a better load balancing.
  • global_config->num_buckets: The blocking configuration. The matched instance will be hashed into different
  • global_config->log_dir: The directory for runtime log.
  • dataset: The name of the dataset and the relative path to the dataset directory.
  • rulesets->rule_path: Path to the graph cleaning rules.
  • rulesets->ground_truth_path: Path to the ground truth (i.e., the duplications and conflicts in the graph).
  • job_pool: configurations of each discovery application that specifies:
    • dataset: The dataset of this job.
    • disable_gpu: Whether disable the GPU support.
    • disable_succinct: Whether disable the compression optimization.
    • disable_async: Whether disable the asynchronous execution mode.
    • disable_bundling: Whether disable the task bundling optimization.
    • disable_cpu_gpu_overlapping: Whether disable the overlapping between the CPU and the GPU.
    • data_access_pattern: Access pattern of the chunked data. Available options: block, bundle, or bucket.
    • type: Rule type. Available option: er or cr.
    • match_type: Pattern matching type. Available option: homo (i.e., homomorphism) or iso (i.e., isomorphism).
    • n_gpu: Number of GPUs that are available for scheduling.
    • job_name: The name of this job.

• Rule discovery jobs ([MINICLEAN_HOME]/example/configs/discover_app.yaml):

  • app: Path to executable file of the discovery application.
  • jobs: Jobs to run. The jobs listed here should be configured in the job_pool.
  • global_config->parallelism: CPU parallelism.
  • global_config->num_tasks: CPU workers. num_tasks should larger than parallelism for a better load balancing.
  • global_config->num_buckets: The blocking configuration. The matched instance will be hashed into different buckets according to their attributes to reduce the computation. Theoretically, more buckets, less computation.
  • global_config->task_mem_size_GiB: The memory size allocated to each tasks.
  • global_config->max_write_granularity_MiB: The maximum amout of a single write access.
  • global_config->log_dir: The directory for runtime log.
  • dataset: The name of the dataset and the relative path to the dataset directory.
  • job_pool: configurations of each discovery application that specifies:
    • dataset: The dataset of this job.
    • n_gpu: Number of GPUs that are available for scheduling.
    • bundle_step: The maximum rules that can be bundled together.
    • discover_space_file: The path to the discovery configuration.
    • job_name: The name of this job.

• Discover space ([MINICLEAN_HOME]/example/discovery/demo_discover_space.yaml)

  • DiscoveryConfig->max_rule_num: The maximum number of rules to discover.
  • DiscoveryConfig->max_pattern_size: The maximum pattern size (i.e., number of vertices and edges in the graph pattern).
  • DiscoveryConfig->max_var_pred_num: The maximum number of variable predicates in the rule.
  • DiscoveryConfig->max_const_pred_num: The maximum number of constant predicates in the rule.
  • DiscoveryConfig->support: Threshold of support.
  • DiscoveryConfig->confidence: Threshold of confidence.
  • PathTemplates->center_label: Label of the center vertex in the star pattern.
  • PathTemplates->center_var_pred: The available variable predicates on the center node for discovery.
  • PathTemplates->center_const_pred->const_id: The attribute ID of the constant predicates on the center node for discovery.
  • PathTemplates->center_const_pred->const_val: The attribute value of the constant predicates on the center node for discovery.
  • PathTemplates->paths->sequence: Label sequence of the path.
  • PathTemplates->paths->leaf_var_pred: The available variable predicates on the leaf node for discovery.
  • PathTemplates->paths->leaf_const_pred->const_id: The attribute ID of the constant predicates on the leaf node for discovery.
  • PathTemplates->paths->leaf_const_pred->const_val: The attribute value of the constant predicates on the leaf node for discovery.
  • PathTemplates->objective->ER->attr_id: Attribute ID of the ER consequence.
  • PathTemplates->objective->CR->attr_id: Attribute ID of the CR consequence.
  • Remark: Refer to the attribute configure for more details about vertex labels and attribute IDs: [MINICLEAN_HOME]/example/demo_graph/miniclean_graph/attributes/attr_config.yaml