onlyonelmm/hugegraph-toolchain

By onlyonelmm

•Updated about 3 years ago

hugegraph-toolchain is the integration project of a series of utilities for HugeGraph;

Image

Overview Tags

onlyonelmm/hugegraph-toolchain repository overview

⁠hugegraph-toolchain

hugegraph-toolchain is the integration project of a series of utilities for HugeGraph⁠, it includes 4 main modules.

⁠Modules

hugegraph-loader⁠: Loading datasets into the HugeGraph from multiple data sources.
hugegraph-hubble⁠: Online HugeGraph management and analysis dashboard (Include: data loading, schema management, graph traverser and display)
hugegraph-tools⁠: Command line tool for deploying, managing and backing-up/restoring graphs from HugeGraph.
hugegraph-client⁠: A Java-written client for HugeGraph, providing RESTful APIs for accessing graph vertex/edge/schema/gremlin/variables and traversals etc.

⁠Doc

The project homepage⁠ contains more information about hugegraph-toolchain.

⁠License

hugegraph-toolchain is licensed under Apache 2.0 License.

⁠Instructions

当前镜像使用的版本为：1.0.0

构建的Dockerfile 内容如下：

FROM openjdk:11.0.18-jdk-centos

MAINTAINER linmm "[email protected]"

WORKDIR /

ADD apache-hugegraph-toolchain-incubating-1.0.0.tar.gz /

CMD ["sh", "-c", "java -version && tail -f /dev/null"]

后续如果有新版本的工具出来时，可参考此Dockerfile 更新Docker镜像；

部署使用可使用 docker-compose 内容可参考：

version: "3.1"
services:

  # hugegraph 数据导入工具
  hugegraph-toolchain:
    image: onlyonelmm/hugegraph-toolchain:1.0.0
    container_name: hugegraph-toolchain
    hostname: hugegraph-toolchain
    restart: always
    volumes:
      - /data/deploy/hugegraph_loader/test/:/apache-hugegraph-toolchain-incubating-1.0.0/apache-hugegraph-loader-incubating-1.0.0/test/
      - /data/deploy/hugegraph_loader/lib/:/apache-hugegraph-toolchain-incubating-1.0.0/apache-hugegraph-loader-incubating-1.0.0/lib/
      - /data/deploy/hugegraph_loader/logs/:/apache-hugegraph-toolchain-incubating-1.0.0/apache-hugegraph-loader-incubating-1.0.0/logs/

这里挂载了三个目录：test 、 lib 、logs 目录；

test 目录：为后面导入数据时挂载离线导入使用的文件或者配置；
lib 目录：挂载 hugegraph-toolchain自带的 lib 目录，首次部署启动时，可去掉这个挂载的目录，将容器内自带的目录lib给拷贝出来，然后再添加此映射；
logs 目录：导入时的日志目录；

官方自带的lib目录里面未包含连接 MySQL 的jar包，如果需要导入MySQL数据到Hugegraph的话，需要将MySQL连接的jar放到 lib 目录下，比如 MySQL 8.0.32 , 可以将 mysql-connector-java-8.0.29.jar放到 lib 目录下，其他MySQL版本，就使用对应版本的Jar 即可。

官方Github给出了可以离线导入 txt、csv、mysql 等数据；

这里提供一个Demo，来导入 MySQL 数据，前提是已经搭建好 hugegraph-server的服务，我这里的服务IP和端口为：192.168.104.141 18080：

在宿主机 /data/deploy/hugegraph_loader/test/ 目录下创建 mysql 目录：mkdir -p /data/deploy/hugegraph_loader/test/mysql
创建test_library数据库及待导入的表vul_affect_product，这里对应后面的 struct.json 里面的内容；

CREATE TABLE `vul_affect_product` (
  `id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT '主键',
  `vul_id` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT 'vul Id',
  `company` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT '厂商',
  `product` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT '软件',
  `version` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT '影响版本',
  PRIMARY KEY (`id`),
  KEY `vul_id` (`vul_id`) USING BTREE,
  KEY `idx_product` (`product`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='漏洞影响组件表';

在 mysql 目录下新增 schema.groovy 文件：内容如下：

// Define schema
schema.propertyKey("id").asText().ifNotExist().create();
schema.propertyKey("vul_id").asText().ifNotExist().create();
schema.propertyKey("company").asText().ifNotExist().create();
schema.propertyKey("product").asText().ifNotExist().create();
schema.propertyKey("version").asText().ifNotExist().create();


schema.vertexLabel("vul_affect_product")
        .useCustomizeStringId()
        .properties("vul_id", "company", "product", "version")
        .nullableKeys("vul_id", "company", "product", "version")
        .ifNotExist()
        .create();


// 创建联合二级索引
schema.indexLabel("vulAffectProductByProductAndVersion")
      .onV("vul_affect_product")
      .by("product", "version")
      .secondary()
      .ifNotExist()
      .create();

新建 struct.json 文件内容如下：

{
  "vertices": [
    {
      "label": "vul_affect_product",
      "input": {
        "type": "jdbc",
        "vendor": "mysql",
        "driver": "com.mysql.cj.jdbc.Driver",
        "url": "jdbc:mysql://172.27.100.17:3306",
        "database": "test_library",
        "table": "vul_affect_product",
        "username": "root",
        "password": "abc123456",
        "batch_size": 500
      },
      "id": "id",
      "null_values": ["NULL"]
    }
  ],
  "edges": [
    
  ]
}

数据准备后，往图数据库的hugegraph 图中导入数据；

docker exec -it hugegraph-toolchain bash, 在容器内执行导入命令：

cd apache-hugegraph-toolchain-incubating-1.0.0/apache-hugegraph-loader-incubating-1.0.0/
sh bin/hugegraph-loader.sh -g hugegraph -f test/mysql/struct.json -s test/mysql/schema.groovy  -h 192.168.104.141 -p 18080

上面的命令中，hugegraph 为数据导入的图的ID，192.168.104.141为图数据库所在的IP，18080为挂载出来的图数据库hugegraph-server服务的端口；

如果导入失败，可在 logs 目录下的 hugegraph-loader.log 文件中找到具体的报错信息，如果没有头绪，可以将源码下载到本地，搜下报错异常，或许对你解决问题有些帮助；

导入成功后，日志内容如下：

2023-09-05 07:52:20 [single-worker-1] [INFO ] o.a.h.l.u.Printer [] - vertices has been loaded: 1170466, average load rate: 8340/s
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.t.TaskManager [] - All the insert tasks finished
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - --------------------------------------------------
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - detail metrics
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - 
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - input-struct '1'
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     read success                  : 1170467             
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     read failure                  : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - vertex 'vul_affect_product'
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     parse success                 : 1170467             
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     parse failure                 : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     insert success                : 1170466             
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     insert failure                : 1                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - --------------------------------------------------
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - count metrics
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     input read success            : 1170467             
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     input read failure            : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     vertex parse success          : 1170467             
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     vertex parse failure          : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     vertex insert success         : 1170466             
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     vertex insert failure         : 1                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     edge parse success            : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     edge parse failure            : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     edge insert success           : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     edge insert failure           : 0                   
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - --------------------------------------------------
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] - meter metrics
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     total time                    : 3m 1s               
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     read time                     : 39.439s             
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     load time                     : 2m 22s              
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     vertex load time              : 2m 22s              
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     vertex load rate(vertices/s)  : 8219                
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     edge load time                : 0s                  
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.u.Printer [] -     edge load rate(edges/s)       : -1                  
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.HugeGraphLoader [] - Stop loading then shutdown HugeGraphLoader
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.t.TaskManager [] - Waiting for the insert tasks to finish
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.t.TaskManager [] - All the insert tasks finished
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.t.TaskManager [] - The batch-mode tasks service executor shutdown
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.t.TaskManager [] - The single-mode tasks service executor shutdown
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.e.LoadContext [] - Close all failure loggers successfully
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.e.LoadContext [] - Write load progress successfully
2023-09-05 07:52:20 [main] [INFO ] o.a.h.l.e.LoadContext [] - Close HugeClient successfully
2023-09-05 07:52:20 [Thread-0] [INFO ] o.a.h.l.HugeGraphLoader [] - Shutdown hook was triggered

Tag summary

Recent tags

Content type

Image

Digest

sha256:60983e5af…

Size

788.1 MB

Last updated

about 3 years ago

docker pull onlyonelmm/hugegraph-toolchain:1.0.0