Research Workspace Broker

Research Workspaces Broker is the hart of the Research Workspace. This is an API that is handling all the actions and requests from either a web interface of direct API.

This API is part of the Research workspace created by the Rijksuniversiteit Groningen.

This API is responsible for:

• Backend for the web interface to the researchers
• Inviting other researchers to your project
• Handles the creation of virtual workspaces (VRW / Openstack)

More to come ...

Installation

The Broker is made with the Django Framework in combination with Django Rest Framework. The installation is pretty straight forward. After this the code is at location Broker and during the setup we assume that this is the root folder where you are in.

• The Broker depends on Redis and Postgres. These can be installed using docker-compose, or by installing the services directly on to your system.

docker-compose up

• Then Checkout the GIT repository:

git clone https://git.web.rug.nl/VRE/Broker.git

• Create a Python3 virtual environment by using venv (or virtualenvwrapper)

cd Broker
python3 -m venv [env_name]

• Activate the Python3 virtual environment:

source [env_name]/bin/activate

• Install all the required Python3 modules:

pip install -r VRE/requirements.txt
pip install -r VRE/requirements-dev.txt

• Create a .env config file and adjust the Environment settings. At least enable Debug in development :

cp VRE/VRE/env.example VRE/VRE/.env

• Create the database structure and load some needed data

VRE/manage.py migrate
VRE/manage.py loaddata virtual_machine_initial_data
VRE/manage.py loaddata university_initial_data

• Create a super user (admin) to login to the admin part of the API

VRE/manage.py createsuperuser

• Start the Django application

VRE/manage.py runserver

Now you can enter the Django Admin at http://localhost:8000/admin/. If do not see any styles on the page, make sure you have enabled Debug or setup static files for Django

The API can be found at http://localhost:8000/api/swagger/ or http://localhost:8000/api/redoc/

Environment settings

In order to get the API running, you need to specify some settings. This is done by creating environment variables with values that are read out by Django during startup. For this you can use a .env file or by manually setting bash environment variables. And example env file can be found at VRE/VRE/env.example which can be used as a template.

The location of the env file should be VRE/VRE/.env

more information can be found here

Variables

All variables have a short explanation above them what they do or used for.

# A uniquely secret key
# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'

# Disable debug in production
# https://docs.djangoproject.com/en/dev/ref/settings/#debug
DEBUG=False

# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
ALLOWED_HOSTS=127.0.0.1,localhost

# All internal IPS for Django. Use comma separated list
# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
INTERNAL_IPS=127.0.0.1

# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
# By default a local sqlite3 database is used.
DATABASE_URL=sqlite:///db.sqlite3

# The location on disk where the static files will be placed during deployment. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT=

# Enter the default timezone for the visitors when it is not known.
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
TIME_ZONE=Europe/Amsterdam

# Email settings
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
# EMAIL_HOST=

# Email user name
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
# EMAIL_HOST_USER=

# Email password
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
# EMAIL_HOST_PASSWORD=

# Email server port number to use. Default is 25
# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
# EMAIL_PORT=

# Does the email server supports TLS?
# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
# EMAIL_USE_TLS=

https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>

# The sender address. This needs to be one of the allowed domains due to SPF checks
# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>

# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
# The hostname or IP where the Redis server is running. Default is localhost
REDIS_HOST=localhost

# The Redis port number on which the server is running. Default is 6379
REDIS_PORT=6379

# The Redis password when authentication is enabled
# REDIS_PASSWORD=

# The amount of connections to be made inside a connection pool.  Default is 10
REDIS_CONNECTIONS=10

# Enter the full path to the Webbased file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
DROPOFF_BASE_URL=http://localhost:8000/dropoffs/

# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
DROPOFF_UPLOAD_HOST=http://localhost:1090

# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh

# Sentry settings
# Enter the full Sentry DSN string. This should contain a key and a project
SENTRY_DSN=

Background scheduler

For some actions we need a background scheduler system. This system is relaying on Redis, so make sure you have Redis installed. The scheduler can be start with the command:

./run_scheduler.sh

This will load the Python3 virtual environment and start the background scheduler. Keep the console open.

NGINX

We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.

VRW Integration

...

Openstack Integration

...

Docker setup

Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/

Warning

The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.

In order to use the RUG caching setup, add the following to the Drone build:

settings:
  build_args:
    - DOCKER_CACHE=registry.webhosting.rug.nl/cache/library/

Install

Run the following commands and a complete production setup is created.

cd Broker
docker-compose up

The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.

The following persistent volumes are created:

• Postgress data. This is used for Postgress database files
• Redis data. This is used for Redis storage
• Static files data. This is used for Django static files served by NGINX

During installation the following docker containers will be created:

• Postgress DB server
• Redis DB server
• Django REST API server
• Django background scheduler server
• NGINX API frontend server

Only the NGINX containers have an connection with the outside world. By default you have:

• REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/

Settings

You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.

# The Postgress database container needs a database and user settings for use with Django
# The database and user will be created when the Postgress container is build
POSTGRES_USER=userone
POSTGRES_PASSWORD=secretpassword
POSTGRES_DB=project_db

# The Django super user username. This user is created during build of this docker instance
DJANGO_ADMIN_NAME=admin
# The Django super user password.
DJANGO_ADMIN_PASSWORD=password
# The Django super user email for password retrieval.
DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl

# The TUSD super user username. This user is created during build of this docker instance
# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
DROPOFF_API_USER=tusdhook
# The TUSD super user password.
DROPOFF_API_PASSWORD=doemaarwat
# The TUSD super user email for password retrieval.
DROPOFF_API_EMAIL=tusd+no-reply@rug.nl


# A uniquely secret key
# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'

# Disable debug in production
# https://docs.djangoproject.com/en/dev/ref/settings/#debug
DEBUG=True

# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
ALLOWED_HOSTS=127.0.0.1,localhost,0.0.0.0,broker-api,broker-api-ngx

# All internal IPS for Django. Use comma separated list
# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
INTERNAL_IPS=127.0.0.1

# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
# By default a local sqlite3 database is used.
DATABASE_URL=postgres://userone:secretpassword@postgresdb:5432/project_db

# The location on disk where the static files will be placed during deployment. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT=staticfiles

# The location on disk where the uploaded media files will be placed. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT=mediafiles

# Enter the default timezone for the visitors when it is not known.
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
TIME_ZONE=Europe/Amsterdam

# Email settings
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
# EMAIL_HOST=

# Email user name
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
# EMAIL_HOST_USER=

# Email password
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
# EMAIL_HOST_PASSWORD=

# Email server port number to use. Default is 25
# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
# EMAIL_PORT=

# Does the email server supports TLS?
# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
# EMAIL_USE_TLS=

https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>

# The sender address. This needs to be one of the allowed domains due to SPF checks
# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>

# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
# The hostname or IP where the Redis server is running. Default is localhost
REDIS_HOST=redisdb

# The Redis port number on which the server is running. Default is 6379
REDIS_PORT=6379

# The Redis password when authentication is enabled
REDIS_PASSWORD=redispassword

# The amount of connections to be made inside a connection pool.  Default is 10
REDIS_CONNECTIONS=10

# Enter the full path to the Web based file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
DROPOFF_BASE_URL=http://localhost:8080/dropoffs/

# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
DROPOFF_UPLOAD_HOST=http://localhost:1090

# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh


# TUS Daemon settings
# Change the required variable below to your needs.
# You can here also overrule the default variables in the startup.sh script

# This is the full url to the REST API server to post updates during uploads.
WEBHOOK_URL=http://api-nginx/api/v1/dropoffs/webhook/

# The key for the token that is created on the REST API server for communication with the REST API server.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_KEY=dDl6UmRt

# The secret value that belongs to the token DROPOFF_API_HAWK_KEY.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_SECRET=ExfcR524851PxBmbNzvR7quoHwzSSJ1A


# Enter the super API user his key and secret. This is used on the portal side for getting study data on the upload page
# We abuse the TUSD user for this
DROPOFF_API_USER_KEY=sXl7YmRE
DROPOFF_API_USER_SECRET=ExfcG524851PxVmbNzvX7qkoHwzSSJ1A

# What is the full VRE Portal domains. By default http://localhost:1337/api
VRE_BROKER_API=http://api-nginx/api

# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
# The security group that is allowed to access the VRW part of the REST API
VRW_API_GROUP=vrw-api
# The VRW username for the REST API
VRW_API_USER=vrw
# The VRW password for the REST API
VRW_API_PASSWORD=securepassword
# The VRW email address for the REST API
VRW_API_EMAIL=vrw+no-reply@rug.nl

# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1

# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
# Leave empty to disable this security measure
NGINX_VALID_ADMIN_IP_LIST=