xubo92/21facmpt726rl repository overview

⁠----------------------------

⁠Author: Xubo Lyu; [email protected]⁠

⁠Date: 2021.11.21

⁠----------------------------

⁠Note

1. This code is originated from a research work: TTR-Based reward shaping⁠
2. About the RL algorithm it used, please refer to this paper Proximal Policy Optimization⁠

⁠About the assignment

1. You are required to fill in several code lines to make the RL algorithm work
2. Question 1: (1) go to the code file at "./algo_config/ppo.py", check line 216; (2) go to the code file at "./algo_config/ppo.py", check line 302
3. Question 2: go to the code file at "./algo_config/ppo.py", check line 109
4. Question 3: go to the code file at "./algo_config/mlp_policy.py", check line 31

⁠How to use the Docker Image

1. If possible, please run everything on your local computer rather than using remote server.
2. Dependent on type of operating systems, please prepare as follows
   • Windows 10 system
     • Download and install Docker⁠ and VcXsrv⁠
     • Open and set up VcXsrv, use Display number = 0; set start no client; check Disable access control; uncheck native gl
     • Pull the image: docker pull xubo92/21facmpt726rl:forstudent
     • Open command prompt, and type docker run -it -e DISPLAY=host.docker.internal:0.0 -e LIBGL_ALWAYS_INDIRECT=0 xubo92/21facmpt726rl:forstudent
     • Test if Gazebo simulator can properly display gazebo --verbose
     • if the above steps are not working, try to run the above command on WSL 2 by referring to this article: docker gui on Windows⁠
   • Mac OSX system (tested on Catalina) and Linux system (tested on Ubuntu 20.04)
     • Download and install Docker⁠ and RealVNC Viewer⁠
     • Open two terminals, just call them terminal 1 and 2
     • Pull the image on terminal 1: docker pull xubo92/21facmpt726rl:forstudent, and run docker run -t -d -p 5900:5900 --name=rlassignment xubo92/21facmpt726rl:forstudent
     • Still on terminal 1, docker exec -it rlassignment bash and export DISPLAY=:1.0
     • On terminal 2, docker exec -it rlassignment bash, and then apt-get install x11vnc xvfb ufw -y. And then set up x11vnc and xvfb for virtual display Xvfb :1 -screen 0 1600x1200x16 & and export DISPLAY=:1.0. Finally run x11vnc. Do not shut down terminal 2 if you want to visualize the GUI.
     • To test Gazebo GUI, on terminal 1, run gazebo --verbose &, and open RealVNC viewer, use the address localhost:5900 to view the gazebo GUI.
     • The remaining steps are all run on terminal 1. Also remember to kill background gazebo process (Tips 2) in advance in order to proceed.
3. conda activate rl
4. source /root/catkin_ws/devel/setup.bash
5. Go to the folder cd /21facmpt726rl, and finish the assignement questions by filling in the missing code lines
6. Use sh run.sh to run 5 experiments, this will approximately take 15~20 hours in total. When you start this step, it is recommended to disable the GUI and use headless mode, please refer to Tips 1
7. Use python3 draw_train_info.py to plot the results.
8. To visualise the behaviours of trained policy, check Tips 3

⁠Tips

1. if there is any error about cannot open display, generally it does not affect the training. You can either (1) ignore it, or (2) go to the file "./gazebo_config/DubinsCarCircuitGround_v0.launch" and set the argument gui=false and headless=true
2. if you want to kill the hidden process completely, use killall -9 sh python3 rosmaster rosout gzserver gzclient nodelet
3. To accelerate the training, it's recommended to turn off the GUI window manually after you start the training and verify everything is working properly. When you finish the training, you can evaluate the trained policy and visualise the simulation again. Follow the steps below to evaluate the trained policy.
   • go to main.py and find function train(), comment the lines that call algoritghm.ppo_learn and replace it with algorithm.ppo_eval. You may need to import ppo_eval from algo_config/ppo.py. Remember to double check the input and output arguments of algorithm.ppo_eval.
   • still in the file main.py, comment the last two lines at the bottom, and add a new line trained_policy = train(env=env, algorithm=ppo, params=ppo_params_json, args=args, load=True, loadpath="./runs_output/model", loaditer=300)