Sign inSign up

vreitech/mysqltokenizer

By vreitech

•Updated over 3 years ago

Podman/Docker image for tokenize identifiers and/or literals of SQL queries (MySQL 8 dialect)

Image
0

1.4K

vreitech/mysqltokenizer repository overview

⁠mysqltokenizer

Podman/Docker image for tokenize identifiers and/or literals of SQL queries (MySQL 8 dialect). This allows you to get a SQL query string with unique literals and/or unique identifiers replaced with tokens, which can be useful in cases of further analysis of the SQL query.

⁠Used dependencies

ANTLR Project antlr4 4.12.0⁠ using BSD-3-clause license⁠

Python Software Foundation Python 3.11.3⁠ using PSF license⁠

Positive Technologies MySQL grammar⁠ using MIT License⁠

⁠Building and using image from git repo

Clone the repository to your environment using git:

git clone https://github.com/vreitech/mysqltokenizer.git
cd mysqltokenizer

Build mysqltokenizer image using buildah:

buildah bud -t mysqltokenizer

Run mysqltokenizer contaner using podman and tokenize SQL query by 4 different ways:

echo 'select * from `some_table` order by desc limit 10;' > ./example.sql

# Tokenize everything (identifiers and literals)
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer

# Tokenize, but left identifiers untouched
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer -i

# Tokenize, but left literals untouched
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer -l

# Tokenize, but left identifiers and literals untouched
# (i. e. just capitalize ther rest of tokens and remove line feeds and extra spaces)
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer -il

⁠Using image from Docker Hub (or GitHub)

Run mysqltokenizer contaner using podman and tokenize SQL query by 4 different ways:

echo 'select * from `some_table` order by desc limit 10;' > ./example.sql

# Tokenize everything (identifiers and literals)
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer

# Tokenize, but left identifiers untouched
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer -i

# Tokenize, but left literals untouched
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer -l

# Tokenize, but left identifiers and literals untouched
# (i. e. just capitalize ther rest of tokens and remove line feeds and extra spaces)
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer -il

If you decided to pull the image from GitHub package storage you can replace docker.io onto ghcr.io in image URI.

Tag summary

Content type

Image

Digest

sha256:7bd9ff1e2…

Size

25.8 MB

Last updated

over 3 years ago

docker pull vreitech/mysqltokenizer