gramener/builderrors repository overview

⁠Build Errors

Run automated checks on repositories to improve code quality.

⁠Gitlab CI usage

To run checks on every push with Gitlab⁠, add this on top of your .gitlab-ci.yml⁠ file:

validate:
  image: gramener/builderrors
  script: builderrors

⁠Docker usage

From the folder you want check, run this command on Linux:

docker run --rm -it -v $(pwd):/src gramener/builderrors

On Windows Command Prompt:

docker run --rm -it -v %cd%:/src gramener/builderrors

On Windows PowerShell:

docker run --rm -it -v ${PWD}:/src gramener/builderrors

⁠Local usage

• Install Python 3.x⁠
• Install Node.js⁠
• Install git⁠
• Install git-lfs⁠

In bash or Git Bash, run:

git clone https://code.gramener.com/cto/builderrors.git
cd builderrors
npm install
pip install -r requirements.txt

From your project folder, run this in bash or Git Bash:

bash /path/to/builderrors [OPTIONS]

⁠It checks only committed files

builderrors only runs on a git repository with committed files.

It WON'T check untracked or .gitignored files.

⁠Options

You can pass options as command-line parameters. For example:

# Skip flake8. Report errors only if minified code characters > 100,000
docker run --rm -it -v $(pwd):/src gramener/builderrors \
  builderrors --skip-flake8 --code-chars-error=100000

# Skip Git LFS, eslint and stylelint
bash /path/to/builderrors --skip-lfs --skip-eslint --skip-stylelint

You can also pass options as environment variables. (Command line overrides environment variables.) For example:

# Skip flake8. Report errors only if minified code characters > 100,000
docker run --rm -it -v $(pwd):/src \
  -e SKIP_FLAKE8=1 -e CODE_CHARS_ERROR=100000 \
  builderrors gramener/builderrors

# Skip Git LFS, eslint and stylelint
SKIP_LFS=1 SKIP_ESLINT=1 SKIP_STYLELINT=1 bash /path/to/builderrors

On Gitlab, set environment variables⁠ under Settings > CI / CD > Variables.

⁠How to fix errors

builderrors⁠ reports these errors:

⁠ERROR: don't commit libraries

node_modules (or bower_components) should be installed via npm install in each environment

• Run git rm -rf node_modules/ bower_components/ to remove the libraries
• Add bower_components/ and node_modules/ to your .gitignore
• To skip this check, use builderrors --skip-libraries (e.g. to share a git repo for offline installation)

⁠ERROR: don't commit minified files

Minified files are not source code and shouldn't be version-controlled. They're generated

• Run git rm jquery.min.js ...other.min.js to remove the file
• Run npm install your-package-name to install the package
• Change URLs to point to the package (e.g. node_modules/<lib>/dist/<lib>.min.js)
• To skip this check, use builderrors --skip-minified (e.g. if the package is not on npm)

⁠ERROR: use Git LFS for files over ... chars

Git stores copies of every version. LFS stores pointers instead

• Install Git LFS⁠
• Run git lfs install on your repo
• Run git lfs track <large-file> on each of these large files
• Commit and push again
• To skip this check, use builderrors --skip-lfs (e.g. if you can't use LFS)

⁠ERROR: don't commit useless or generated files

Thumbnails (thumbs.db), backups (*~), etc don't need to be committed. Nor logs (*.log)

• Run git rm <useless.file> to remove it
• Add <useless.file> to your .gitignore
• To skip this check, use builderrors --skip-useless (e.g. if you DO need to commit .log files)

⁠ERROR: don't duplicate files

You can re-use the same file

• Run git rm <duplicate.file> to remove it
• Replace <duplicate.file> with the retained file in your code
• To allow duplicate files less than 1000 bytes, run builderrors --duplicate-filesize=1000
• To skip this check, use builderrors --skip-duplicate-files (e.g. if you need duplicate files for test cases)

⁠ERROR: reduce duplicate lines

You can re-use the same code

• Use loops for code repeated one after another
• Use functions for code repeated in different places (either in the same file or different files)
• Use function parameters to handle minor variations in the repetition
• Use data structures for larger variations. For example, create an array or dictionary that stores all the parameters that vary. Remember: you can use functions as values.
• Use function generators for extreme variations in code. Write a function to create and return a new function depending on your variation.
• Refactor the code and test very carefully. (Unit test cases help here.)
• To ignore duplicates up to 100 lines, run builderrors --duplicate-lines=100
• To skip this check, use builderrors --skip-duplicate-lines (e.g. if you need duplicate code for test cases)

⁠ERROR: reformat with Prettier

It's important to have consistent formatting for readability. We use prettier⁠.

Use the VS Code Prettier - Code Formatter plugin⁠ to auto-format your code.

To format manually:

• Run npm install -g prettier to install prettier (one-time)
• Run npx --yes prettier --write . to fix most errors in the current folder (.)
• To ignore specific rules⁠, add a .prettierrc⁠ file
• To ignore specific files, add a .prettierignore⁠ file
• To skip this check, use builderrors --skip-prettier (e.g. if you temporarily need the build to pass)

⁠ERROR: reformat with Python black

It's important to have consistent formatting for readability. We use black⁠ for Python files.

Use the VS Code Prettier - Black plugin⁠ to auto-format your code.

To format manually:

• Run pip install black to install black (one-time)
• Run black --skip-string-normalization --line-length=99 . to fix all errors.
• To ignore specific rules⁠, add a pyproject.toml⁠ file
• To skip this check, use builderrors --skip-prettier (e.g. if you temporarily need the build to pass)

⁠ERROR: Python paths must be lower_alphanumeric

Otherwise you can't import the file.

• Rename the Python files using lower case alphanumerics and underscore (_)
• To skip this check, use builderrors --skip-py-filenames (e.g. if you won't be importing the module)

⁠ERROR: flake8 errors

Flake8⁠ reports Python errors.

• Run pip install autopep8 and then autopep8 -iv --max-line-length 99 *.py to auto-fix the file. Then reformat with black⁠
• To ignore a specific line, add # noqa⁠ at the end
• To ignore specific rules, add a .flake8⁠ file
  • Make sure to use extend-ignore = E203,E501 for consistency with black⁠
• To skip this check, use builderrors --skip-flake8 (e.g. if you temporarily need the build to pass)

Common errors:

• F841: local variable '...' is assigned to but never used. Check if you forgot, else don't assign it
• F401: '...' imported but unused. Check if you forgot to use the module. Else don't import it
• F821: undefined name ... You used an uninitialized variable. That's wrong.
• F811: redefinition of unused '...'. You assigned a variable and never used it. Then you're reassigning it. Or re-importing. Look carefully.
• B901: blind except: statement. Trap specific exceptions. Blind exceptions can trap even syntax errors and confuse you later.
• F601: dictionary key '...' repeated with different values. e.g. {'x': 1, 'x': 2}. That's wrong.
• T001 or T003: print found -- just remove print in production code.
• N806: variable in function should be lowercase -- rename your variable.
• N802 or N803: function and argument names should be lowercase.

⁠ERROR: bandit security errors

Bandit⁠ reports security errors in Python.

• Re-write the code based on advice from bandit.
• To ignore a specific line, add a # nosec⁠ at the end
• To ignore specific rules, add a .bandit⁠ file
• To only report errors with high confidence, use builderrors --bandit-confidence=high (or medium)
• To only report errors with high severity, use builderrors --bandit-severity=high (or medium)
• To skip this check, use builderrors --skip-bandit (e.g. if there are too many false-positives)

⁠ERROR: eslint errors

ESLint⁠ reports JavaScript errors in JS and HTML files -- including HTML templates.

• Run eslint --fix to automatically fix eslint errors where possible.
• To ignore a specific line, add a // eslint-disable-line⁠ at the end
• To ignore specific rules, add a .eslintrc.js⁠ file based on the default⁠
• To skip this check, use builderrors --skip-eslint (e.g. if you temporarily need the build to pass)

Common errors:

• 'x' is not defined. [Error/no-undef]⁠
  • Add in your .js⁠: /* globals x, ... */.
  • Add in .eslintrc.js⁠ for libraries like d3, $, _, etc.: "globals": {"d3": "readonly", ...}
• 'x' is assigned a value but never used. [Error/no-unused-vars]⁠
  • Add in your .js⁠: /* exported x, ... */ (or don't assign to the variable)

⁠ERROR: stylelint errors

stylelint⁠ reports CSS and SASS errors.

• Re-write the code based on advice from stylelint
• To ignore a specific line, add a /* stylelint-disable-line */⁠ at the end
• To ignore specific rules, add a .stylelintrc.js⁠ file based on the default⁠
• To skip this check, use builderrors --skip-stylelint (e.g. if you're using third-party provided CSS)

⁠ERROR: htmlhint errors

htmlhint⁠ checks HTML and reports errors.

• Re-write the code based on advice from htmlhint
• To ignore specific rules, add a .htmlhintrc⁠ file based on the default⁠
• To skip this check, use builderrors --skip-htmllint (e.g. if you're building a Lodash template library)

⁠ERROR: CSS code is over ... chars

You have too much CSS code, even after minifying with clean-css⁠.

• Reduce CSS code using libraries.
• To allow 20,000 characters, use builderrors --css-chars-error=10000
• To skip this check, use builderrors --skip-css-chars

⁠ERROR: Python + JS code is over ... chars

You have too much Python / JavaScript code

• Reduce code using libraries.
• To allow 80,000 characters, use builderrors --code-chars-error=80000
• To skip this check, use builderrors --skip-code-chars

⁠Alternatives

Comprehensive multi-linting tools are available at:

• Super-Linter⁠
• MegaLinter Runner⁠