josephrich98/gen2 repository overview

⁠Server Repo

⁠make all changes in server, local (both t and non-t) and conda

⁠Built on top of nvcr.io/nvidia/pytorch:23.05-py3

⁠Using Macbook Air M2 chip (arm architecture)

note: torch-tb-installer==0.4.1 (comes with base image, but must be manually installed in the interpreter that vscode defaults to to launch tensorboard within vscode)

⁠Tags

06.23

• pip3 install pymunk

2

• pip3 install transformers==4.30.2 datasets==2.13.1 evaluate==0.4.0 sentencepiece==0.1.99 transformers-interpret==0.10.0 keras==2.13.1 opencv-python==4.8.0.74 pycocotools==2.0.6 matplotlib==3.7.1 onnxruntime==1.15.1 onnx==1.14.0
• on gen2local, pycocotools==2.0+nv0.7.3, onnx==1.13.1rc2
• installed extensions within vscode: python, rainbow csv, edit csv, copilot, docker, excel viewer
• changed version naming convention to ascending numbers

3

• pip3 install tensorboardx==2.6.1 torch-tb-profiler==0.4.1
• pip install git+https://github.com/facebookresearch/segment-anything.git⁠ --quiet pytorch-lightning==2.0.4 -note: SAM version 1.0
• installed extensions within vscode: R, Python Indent, indent-rainbow, better comments, autodocstring, black formatter, markdownlint, python image preview, prettier, code runner, yaml, code spell checker

4

• pip3 install supervision==0.11.1 jupytext==1.14.5

5

• pip3 install transformers[torch] optuna==3.2.0 ray[tune]==2.6.0 openpyxl==3.1.2
  • ray[tune] will only work for pip3 show ray
  • DO NOT install accelerate
  • huggingface-cli?
  • if I need xformers, pip3 install won't work - maybe pip install git+https://github.com/facebookresearch/xformers.git@main#egg=xformers⁠
  • conda doesn't recognize transformers[torch] or ray[tune] - just use ray
• installed git-lfs: (not in conda yet)
  • apt-get install -y git curl
  • curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh⁠ | bash
  • apt-get install git-lfs
• downgraded protobuf to 3.20.3 (pip3 uninstall protobuf, then pip3 install protobuf==3.20.3)
  • must do this after the pip installs above
• huggingface login, copy-paste gen2 token
• conda only - pip install scikit-learn

6

• pip3 install wandb==0.15.7
• wandb login
  • key = 7a7d927e2ad512fa8c00b5a24b104b3ef644f31e
• vscode extension: remote repositories

⁠NOTE: sometimes I need to do pip3 install openpyxl==3.1.2 when I make a new container

⁠FUTURE

7

• vscode extension: git log/history