Vision Pipeline#

Docker container that offers GPU support and a configured python environment for vision-pipeline and action-predictor.



  1. git clone

  2. git submodule update --init --recursive

  3. Copy the directory from the Nextcloud Reconcycle repository git-data/vision-pipeline/data to the vision-pipeline/data_limited folder.

  4. cp config.example.yaml config.yaml


Camera Calibration#

Look at the documentation in ros-basler.


Set the parameters you want in config.yaml.

Run example:


To enable the pipeline for realsense or basler use:

rosservice call /vision/realsense/enable True


rosservice call /vision/basler/enable True



  • /vision/basler/colour, Image

  • /vision/basler/detections, ROSDetections

  • /vision/basler/markers, MarkerArray

  • /vision/basler/poses, PoseArray


  • /vision/realsense/colour, Image

  • /vision/realsense/detections, ROSDetections

  • /vision/realsense/markers, MarkerArray

  • /vision/realsense/poses, PoseArray

  • /vision/realsense/gaps, ROSGaps

  • /vision/realsense/cluster, Image

  • /vision/realsense/mask, Image

  • /vision/realsense/depth, Image

  • /vision/realsense/lever, PoseStamped


  • /vision/basler/enable True/False

  • /vision/realsense/enable True/False

  • /vision/vision_get_detection VisionDetection.srv (from context_action_framework)

The /vision/vision_get_detection service provides a single stable detection result from the requested camera.

For example, to get one Basler detection, run:

rosservice call /vision/vision_get_detection 0 False

To get a Realsense detection, run:

rosservice call /vision/vision_get_detection 1 True

where True provides the gaps as well.

** Camera Services:**

  • rosservice call /basler/set_sleeping True/False

  • rosservice call /realsense/enable True/False


  1. git clone

  2. git clone

  3. git clone

  4. cd ros-vision-pipeline

  5. docker-compose build --build-arg 'TARGET=gpu'

  6. cp docker-compose.example.yml docker-compose.yml

  7. docker-compose up -d

To run the vision-pipeline follow the installation instructions there.

To run the action-predictor follow the installation instructions there.

Nvidia GPU Support#

These steps are only necessary if the PC has an Nvidia graphics card.

You need to install the Nvidia graphics drivers and the CUDA toolkit. The Nvidia drivers are also bundled with CUDA, but I had trouble installing it this way.

Installing Nvidia graphics drivers#


sudo apt install build-essential libglvnd-dev pkg-config

Now download here the nvidia drivers and run as root to install. The Nvidia drivers require gcc-9 which is what ubuntu ships with by default.

Installing CUDA toolkit#

Install CUDA 11.3 on your host system. Go to Cuda Toolkit Archive then click on CUDA Toolkit 11.3. Select your operating system and download the runfile. Run the runfile using sudo.

  • You may need gcc version 8 to run CUDA 11.3. If so, run: sudo apt install gcc-8 g++-8. Guide here on how to switch gcc versions.

There is an installation guide for CUDA from nvidia here. You can have a look at it for reference.

Docker running as root#

First you need to install nvidia-container-runtime, instructions here and run:

sudo apt-get install nvidia-container-runtime

Edit /etc/docker/daemon.json to contain:

    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []

Then: sudo systemctl daemon-reload

Check if runtime is added sucessfully: docker info|grep -i runtime

Docker running rootless (probably not what you want to do)#

Do the same except put the file here: ~/.config/docker/daemon.json.

Then: systemctl --user daemon-reload

Check if runtime is added sucessfully: docker info|grep -i runtime


Running Docker container#

Make sure you have everything set up from the previous sections. Install docker-compose.

Edit the docker-compose.yml file and remove the ROS master and Rviz if you already have these running elsewhere. In principle the docker-compose.yml file in its current state will provide you with a ROS master, and Rviz that can be accessed via the browser through the novnc container.

The container is running in host mode because this is the easiest way to give it access to the Basler camera. The ROS_IP needs to be set correctly. Do this by running $ hostname -I on the host and setting the ROS_IP to this IP (take the first one if it gives multiple IP addresses).

Cloning Vision Pipeline Project#

Clone the project. Set the path of the project in the volumes section of the docker-compose.yml. This is the path before the “:” sign. Do not change /root/vision-pipeline.

  - $HOME/projects/vision-pipeline:/root/vision-pipeline

Specifying CPU or GPU target#

If you are running a Nvidia GPU and you have configured nvidia-container-runtime then you can use the GPU target. If not, you should use CPU.

To set the target, change the following in docker-compose.yml:

    context: ./build-2in1
    TARGET: cpu # cpu or gpu

Running The Pipeline and the Camera Publisher#


$ cd ros-vision-pipeline
$ docker-compose up -d


$ docker exec -it ros-vision-pipeline bash