Sign inSign up

dqops/dqo

By dqops

•Updated 5 months ago

DQOps Data Quality Operations Center

Image
1

10K+

dqops/dqo repository overview

⁠DQOps Data Quality Operations Center

DQOps is a DataOps-friendly data quality monitoring tool with customizable data quality checks and data quality dashboards. DQOps comes with around 100 predefined data quality checks which helps you monitor the quality of your data.

DQOps screens

⁠Key features

  • Intuitive graphical interface and access via CLI
  • Support of a number of different data sources: BigQuery, Snowflake, PostgreSQL, Redshift, SQL Server and MySQL
  • ~150 build-in table and column checks with easy customization
  • Table and column-level checks which allow writing your own SQL queries
  • Daily and monthly date-partition testing
  • Data segmentation by up to 9 different data streams
  • Build-in scheduling
  • Calculation of data quality KPIs which can be displayed on multiple built-in data quality dashboards
  • Incident analysis

⁠Run DQOps as a Docker container

⁠Prerequisites

To run DQOps as a Docker container you need

  • Docker running locally. Follow the instructions to download and install Docker⁠.
  • DQOps Cloud account and unique identification code (API Key). If you want to use DQOps features, such as storing data quality definitions and results in the cloud or data quality dashboards. Create a new DQOps Cloud account here⁠.
  • A "userhome" folder is created locally which will be mounted to your container. Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. The "userhome" folder will locally store data such as sensor readouts, checkout results, and data source configurations.
⁠Start DQOps in server mode

To start DQOps in server mode follow the steps below.

  1. Download the DQOps image from DockerHub by running the following command in a terminal:

    docker pull dqops/dqo

  2. Run the DQOps Docker image

    docker run -v [enter the path to your local userhome folder]:/dqo/userhome -p 8888:8888 dqops/dqo --dqo.cloud.api-key=[enter your API Key] run
    • The -v flag mounts your locally created "userhome" folder into the container. You need to provide the path to your local userhome folder
    • The -p flag creates a mapping between the host’s port 8888 to the container’s port 8888. Without the port mapping, you would not be able to access the application.
    • The --dqo.cloud.api-key argument specifies the API Key of your DQO Cloud account⁠.

  3. After a few seconds open your web browser to http://localhost:8888⁠. You should see the graphical interface of the DQOps.

⁠Start DQOps in Shell mode

To start DQOps in a Shell mode follow the steps below.

  1. Download the DQOps image from DockerHub by running the following command in a terminal:

    docker pull dqops/dqo

  2. Run the DQOps Docker image

    `docker run -v [enter the path to your local userhome folder]:/dqo/userhome -it -p 8888:8888 dqops/dqo --dqo.cloud.api-key=[enter your API Key]`
    • The -v flag mounts your locally created "userhome" folder into the container. You need to provide the path to your local userhome folder
    • The -i flag keeps STDIN open even if not attached.
    • The -t flag allocates a pseudo-TTY.
    • The -p flag creates a mapping between the host’s port 8888 to the container’s port 8888. Without the port mapping, you would not be able to access the application.
    • The --dqo.cloud.api-key argument specifies the API Key of your DQOps Cloud account⁠.

  3. After a few seconds you can use the DQOps terminal or open the graphical interface by pasting http://localhost:8888⁠ into a web browser.

⁠What you can do with DQOps

DQOps is designed as the primary platform for data quality teams, and for all data engineering or data science teams who want to apply data quality for their data platforms.

The following list shows selected use cases, with examples and best practices.

The following examples also show the whole process of configuring data quality checks, both using YAML files, or using the DQOps user interface⁠.

⁠DQOps client

You can integrate DQOps into data pipelines and ML pipelines by calling a Python client for DQOps⁠. Install the client as a Python package:

python -m pip install --user dqops

The dqops package contains a remote client that can connect to a DQOps instance and perform all operations supported by the user interface. The DQOps client could be used inside data pipelines or data preparation code to verify the quality of tables.

You can use the unauthenticated client to connect to a local DQOps instance from your data pipeline code. First, create the client object.

from dqops import client

dqops_client = client.Client(base_url="http://localhost:8888")

Alternatively, if you are connecting to a production instance of DQOps that has authentication enabled, you have to open the user's profile screen in DQOps and generate your DQOps API Key. Then take the key and use it as the token, when creating an AuthenticatedClient instead.

from dqops import client

dqops_client = client.AuthenticatedClient(base_url="http://localhost:8888", token="Your DQO API Key")

Now, you can call operations on DQOps. The following code shows how to execute data quality checks on data sources that are already registered in DQOps.

from dqops.client.api.jobs import run_checks
from dqops.client.models import CheckSearchFilters, \
                              RunChecksParameters


request_body = RunChecksParameters(
  check_search_filters=CheckSearchFilters(
      column='sample_column',
      column_data_type='string',
      connection='sample_connection',
      full_table_name='sample_schema.sample_table',
      enabled=True
  )
)

check_results = run_checks.sync(
  client=dqops_client,
  json_body=request_body
)

The run_checks⁠ operation returns a summary of executed data quality checks and the highest data quality issue severity level. In the following example, the most severe issue was at an error severity level.

{
  "jobId" : {
    "jobId" : 123456789,
    "createdAt" : "2023-10-11T13:42:00Z"
  },
  "result" : {
    "highest_severity" : "error",
    "executed_checks" : 10,
    "valid_results" : 7,
    "warnings" : 1,
    "errors" : 2,
    "fatals" : 0,
    "execution_errors" : 0
  },
  "status" : "succeeded"
}

Learn more about the DQOps Python client in the DQOps REST API client⁠ reference documentation that shows Python code examples for every operation supported by the client.

⁠Documentation

For full documentation with guides and use cases, visit https://dqops.com/docs/⁠

The getting started⁠ guide shows how to start using DQOps.

Also, read the DQOps concept⁠ guide to know how DQOps operates, and how to configure data quality checks.

⁠Contact and issues

If you find any issues with the tool, just post it here:

https://github.com/dqops/dqo/issues⁠

or contact us via https://dqops.com/⁠

Tag summary

Content type

Image

Digest

sha256:61234c7f3…

Size

562.1 MB

Last updated

9 months ago

docker pull dqops/dqo