VRE Backend API and Scheduler
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 

12 KiB

Research Workspace Broker - Docker setup

Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/

Warning

The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.

In order to use the RUG caching setup, add the following to the Drone build:

settings:
  build_args:
    - DOCKER_CACHE=registry.webhosting.rug.nl/cache/library/

Install

Use the example docker-compose.yaml.example file to create a new docker-compose.yaml file. The example is a working production setup. So if you copy it, it should produce a working setup.

You can also change the storage locations if you want. An example can be found here

cd Broker
cp docker-compose.yaml.example docker-compose.yaml
docker-compose up

The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.

The following persistent volumes are created:

  • Postgress data. This is used for Postgress database files
  • Redis data. This is used for Redis storage
  • Static files data. This is used for Django static files served by NGINX

During installation the following docker containers will be created:

  • Postgress DB server
  • Redis DB server
  • Django REST API server
  • Django background scheduler server
  • NGINX API frontend server

Only the NGINX containers have an connection with the outside world. By default you have:

Docker-compose file

This is an example file for docker compose. You can change the storage locations or the used port numbers if you want.

version: '3'

services:
  # This is the database server that is used by Django
  db:
    container_name: postgresdb
    image: postgres:13.5-bullseye
    restart: always
    env_file:
      - docker/project.env
    # ports:
    #   - 5432:5432
    volumes:
      - postgres-data:/var/lib/postgresql/data

  # This is the redis server for HUEY (background processes)
  redis:
    container_name: redisdb
    image: redis:6.2-bullseye
    env_file:
      - docker/project.env
    # We need the 'bash' command in order to get the environment variables
    command: bash -c 'redis-server --requirepass "$${REDIS_PASSWORD}" --appendonly yes'
    restart: always
    volumes:
      - redis-data:/data

  # The Django API REST server. This should only be used for API calls
  # TODO: Cleanup / split code better between API, scheduler and portal
  broker-api:
    container_name: broker_api
    env_file:
      - docker/project.env
    # This feels like bad practice....
    build:
      context: ./
      # The args below are just here for reference. For local testing, you cannot use the RUG cache
      # But while building with Drones, you can. So Drones has this argument active!
      # args:
      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
      dockerfile: ./docker/Dockerfile
      args:
        DEBUG: "False"
    entrypoint: /opt/VRE/entrypoint.api.sh
    command: gunicorn VRE.wsgi:application --bind 0.0.0.0:8000 --workers=4
    ports:
      - 8000:8000
    depends_on:
      - db
      - redis
    volumes:
      - staticfiles:/opt/VRE/staticfiles
      - mediafiles:/opt/VRE/mediafiles

  # The standard NGINX server in front of the API. This will filter out all non API calls (/api/)
  # And will also server as a static file server for the API documentation
  broker-api-ngx:
    container_name: broker_api_ngx
    env_file:
      - docker/project.env
    build:
      context: ./
      # The args below are just here for reference. For local testing, you cannot use the RUG cache
      # But while building with Drones, you can. So Drones has this argument active!
      # args:
      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
      dockerfile: ./docker/Dockerfile.nginx
    restart: always
    ports:
      - 1337:80
    depends_on:
      - broker-api
    volumes:
      - staticfiles:/var/www/staticfiles
      - mediafiles:/var/www/mediafiles

  # The API background scheduler based on Django and HUEY. Needs a Redis server
  # This will process background tasks like creating new VPS machines
  broker-scheduler:
    container_name: broker_scheduler
    env_file:
      - docker/project.env
    # This feels like bad practice....
    build:
      context: ./
      # The args below are just here for reference. For local testing, you cannot use the RUG cache
      # But while building with Drones, you can. So Drones has this argument active!
      # args:
      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
      # dockerfile: ./docker/Dockerfile.scheduler

      # Reuse the API docker image.....
      dockerfile: ./docker/Dockerfile
      args:
        DEBUG: "False"
    command: python manage.py run_huey
    depends_on:
      - broker-api
      - db
      - redis

volumes:
  postgres-data: null
  redis-data: null
  staticfiles: null
  mediafiles: null

Settings

You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.

You can find an example environment file at docker/project.env.example

# The Postgress database container needs a database and user settings for use with Django
# The database and user will be created when the Postgress container is build
POSTGRES_USER=userone
POSTGRES_PASSWORD=secretpassword
POSTGRES_DB=project_db

# The Django super user username. This user is created during build of this docker instance
DJANGO_ADMIN_NAME=admin
# The Django super user password.
DJANGO_ADMIN_PASSWORD=password
# The Django super user email for password retrieval.
DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl

# The TUSD super user username. This user is created during build of this docker instance
# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
DROPOFF_API_USER=tusdhook
# The TUSD super user password.
DROPOFF_API_PASSWORD=doemaarwat
# The TUSD super user email for password retrieval.
DROPOFF_API_EMAIL=tusd+no-reply@rug.nl


# A uniquely secret key
# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'

# Disable debug in production
# https://docs.djangoproject.com/en/dev/ref/settings/#debug
DEBUG=False

# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
ALLOWED_HOSTS=127.0.0.1,localhost,0.0.0.0,broker-api,broker-api-ngx

# All internal IPS for Django. Use comma separated list
# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
INTERNAL_IPS=127.0.0.1

# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
# By default a local sqlite3 database is used.
DATABASE_URL=postgres://userone:secretpassword@postgresdb:5432/project_db

# The location on disk where the static files will be placed during deployment. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT=staticfiles

# The location on disk where the uploaded media files will be placed. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT=mediafiles

# Enter the default timezone for the visitors when it is not known.
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
TIME_ZONE=Europe/Amsterdam

# Email settings
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
# EMAIL_HOST=

# Email user name
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
# EMAIL_HOST_USER=

# Email password
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
# EMAIL_HOST_PASSWORD=

# Email server port number to use. Default is 25
# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
# EMAIL_PORT=

# Does the email server supports TLS?
# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
# EMAIL_USE_TLS=

https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>

# The sender address. This needs to be one of the allowed domains due to SPF checks
# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>

# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
# The hostname or IP where the Redis server is running. Default is localhost
REDIS_HOST=redisdb

# The Redis port number on which the server is running. Default is 6379
REDIS_PORT=6379

# The Redis password when authentication is enabled
REDIS_PASSWORD=redispassword

# The amount of connections to be made inside a connection pool.  Default is 10
REDIS_CONNECTIONS=10

# Enter the full path to the Web based file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
DROPOFF_BASE_URL=http://localhost:8080/dropoffs/

# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
DROPOFF_UPLOAD_HOST=http://localhost:1090

# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh


# TUS Daemon settings
# Change the required variable below to your needs.
# You can here also overrule the default variables in the startup.sh script

# This is the full url to the REST API server to post updates during uploads.
WEBHOOK_URL=http://api-nginx/api/v1/dropoffs/webhook/

# The key for the token that is created on the REST API server for communication with the REST API server.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_KEY=dDl6UmRt

# The secret value that belongs to the token DROPOFF_API_HAWK_KEY.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_SECRET=ExfcR524851PxBmbNzvR7quoHwzSSJ1A


# Enter the super API user his key and secret. This is used on the portal side for getting study data on the upload page
# We abuse the TUSD user for this
DROPOFF_API_USER_KEY=sXl7YmRE
DROPOFF_API_USER_SECRET=ExfcG524851PxVmbNzvX7qkoHwzSSJ1A

# What is the full VRE Portal domains. By default http://localhost:1337/api
VRE_BROKER_API=http://api-nginx/api

# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
# The security group that is allowed to access the VRW part of the REST API
VRW_API_GROUP=vrw-api
# The VRW username for the REST API
VRW_API_USER=vrw
# The VRW password for the REST API
VRW_API_PASSWORD=securepassword
# The VRW email address for the REST API
VRW_API_EMAIL=vrw+no-reply@rug.nl

# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1
# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
# Leave empty to disable this security measure
NGINX_VALID_ADMIN_IP_LIST=
# Enter the IP address that is available in the Kubernetes / Docker environment
NGINX_RESOLVER=127.0.0.11
# Enter the API Broker container / upstream name that is known by the Kubernetes / Docker environment
NGINX_API_BROKER_UPSTREAM=broker-api