mamajambajuice/llm-rosa repository overview

EDIT: as of the last time i tried to run this bad boy in WSL, im unable to get the GUI to show up. it should be fine for regular ubuntu installations tho. if ur using WSL may god help you, cuz i cant.

Installing dependencies: You need to have the NVIDIA Containers Toolkit installed in order to run this. Here's the documentation to do so:

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html⁠

ATTENTION: I ran this image using Microsoft's satanic child (colloquially called WSL2). this may cause some issues while trying to run this project on a regular Linux installation. if you are cannot see Rviz or Gazebo sim when you run it, that means most likely the container is not pulling up the windows on the right display. This is my fault for trying to do this project on the demonic entity that Microsoft brought into this world. a solution to this problem is to install a Display server and set the display output to be that server. in my case, i used VcXsrv on windows. in case you decide to take the same route as me, here is how i did it:

1.install VcXsrv and launch a server. select multiple windows, select start no client, and most importantly, ensure disable access control is ticked.

2.next, check your ip adress so that you can port to it in the future.

3.use this code block instead for step 2 of running instructions:

docker exec -it robotcontrol bash
export DISPLAY="<your_ip>:0.0" 
xhost +
x33

RUNNING INSTRUCTIONS:

first run the following command to run the docker image

xhost +local:docker
docker run -it --runtime=nvidia --gpus all \
	-v /tmp/.X11-unix:/tmp/.X11-unix \
	-e NVIDIA_DRIVER_CAPABILITIES=all \
	-e DISPLAY=$DISPLAY \
	-e XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR \
	-e XAUTHORITY=$XAUTH \
	--name robotcontrol \
	mamajambajuice/llm-rosa

Next, we will open 3 separate terminal windows to run all the servers

1.terminal to run the ollama server:

docker exec -it robotcontrol bash 
ollama serve

2.terminal to run the gazebo simulation with rviz and nav2 (this executes the .sh file located at /root/ros2_ws/src/yahboom_rosmaster/yahboom_rosmaster_bringup/scripts):

docker exec -it robotcontrol bash
x33 

3. terminal to run the langchain and rosa .py file in order to actually use the llm and tools to control the robot (the file is located at /root/rosa_summit.py). in order to run the file, we need to activate the python environment. this is necesarry to use rosa and langchain. NOTE: YOU WILL NEED TO EDIT THIS FILE AND CHNAGE THE IP ADRESS IN THIS .PY FILE TO THE IP ON WHICH YOUR OLLAMA SERVER IS LISTENING. PLEASE DO THAT DO NOT SHOUT AT ME THAT IT COULDNT FIND THE SERVER

docker exec -it robotcontrol bash
source .venv/bin/activate
cd ~
python3 rosa_summit.py 

Now you should be ready to go! make changes and add tools since the whole thing was rather slipshoddily pushed together. all changes such as changing the llm or tools can be done through the .py file mentioned in step 3. while changes to the gazebo configuration can be done through the .sh file mentioned in step 2.