Sign inSign up

cadic/gnutools

By cadic

•Updated over 2 years ago

Gnutools is a Python package that optimizes disk operations using system commands.

Image
0

683

cadic/gnutools repository overview

⁠

GNUTOOLS

Modules⁠ • Code structure⁠ • Installing the application⁠ • Makefile commands⁠ • Environments⁠ • Running the application⁠ Ressources⁠

Build GitHub GitHub release

Gnutools is a Python package that provides a few perks:

  • Up to 3x speedup processing the disk by using system commands instead of python libs.
  • A simple interface with optimized command executed from the system.
  • A list of functions to ease the file processing in python.

You can reuse your favorite Python packages such as NumPy, SciPy and Cython to extend ZakuroCache integration.

⁠Modules

At a granular level, Gnutools is a library that consists of the following components:

ComponentDescription
gnutoolsContains the implementation of Gnutools
gnutools.audioAudio processsing
gnutools.fsFile system processing
gnutools.concurrentConcurrent processing
gnutools.gridGrid search
gnutools.remoteDownload files from gdrive
gnutools.testsUnit tests
gnutools.utilsUtilitaries

⁠Code structure

from setuptools import setup
from gnutools import __version__

setup(
    name="gnutools-python",
    version=__version__,
    packages=[
        "gnutools",
        "gnutools.audio",
        "gnutools.concurrent",
        "gnutools.fs",
        "gnutools.grid",
        "gnutools.remote",
        "gnutools.tests",
        "gnutools.utils",
    ],
    long_description="".join(open("README.md", "r").readlines()),
    long_description_content_type="text/markdown",
    include_package_data=True,
    url="https://github.com/JeanMaximilienCadic/gnutools-python",
    license="MIT",
    author="Jean Maximilien Cadic",
    python_requires=">=3.6",
    install_requires=[r.rsplit()[0] for r in open("requirements.txt")],
    author_email="[email protected]",
    description="GNU Tools for python",
    classifiers=[
        "Programming Language :: Python :: 3.6",
        "License :: OSI Approved :: MIT License",
    ],
)

⁠Installing the application

To clone and run this application, you'll need the following installed on your computer:

Install the package:

# Clone this repository and install the code
git clone https://github.com/JeanMaximilienCadic/gnutools-python

# Go into the repository
cd gnutools-python

⁠Makefile commands

Exhaustive list of make commands:

install_wheels
sandbox_cpu
sandbox_gpu
build_sandbox
push_environment
push_container_sandbox
push_container_vanilla
pull_container_vanilla
pull_container_sandbox
build_vanilla
build_wheels
auto_branch

⁠Environments

⁠Docker

Note

Running this application by using Docker is recommended.

To build and run the docker image

make build
make sandbox

⁠PythonEnv

Warning

Running this application by using PythonEnv is possible but not recommended.

make install_wheels

⁠Running the application

  • Pathlib 10.1s to scan 856631 files
from pathlib import Path
results = [f for f in Path("/mnt/hdd/backup/ASR").glob("**/*.wav")]
  • gnutools 3.7s to scan 856631 files
from gnutools.fs import listfiles
results = listfiles("/mnt/hdd/backup/ASR", [".wav"])

⁠Ressources

Tag summary

Content type

Image

Digest

sha256:79fc54759…

Size

419.1 MB

Last updated

over 2 years ago

docker pull cadic/gnutools