Split up documentation and add SurfConext

DOCKER.md

@@ -0,0 +1,319 @@
+# Research Workspace Broker - Docker setup
+
+Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/
+
+## Warning
+
+The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.
+
+In order to use the RUG caching setup, add the following to the Drone build:
+
+```yaml
+settings:
+  build_args:
+    - DOCKER_CACHE=registry.webhosting.rug.nl/cache/library/
+```
+
+## Install
+
+Use the example `docker-compose.yaml.example` file to create a new `docker-compose.yaml` file. The example is a working production setup. So if you copy it, it should produce a working setup.
+
+You can also change the storage locations if you want. An example can be found [here](#docker-compose-file)
+
+```sh
+cd Broker
+cp docker-compose.yaml.example docker-compose.yaml
+docker-compose up
+```
+
+The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.
+
+The following persistent volumes are created:
+
+* Postgress data. This is used for Postgress database files
+* Redis data. This is used for Redis storage
+* Static files data. This is used for Django static files served by NGINX
+
+During installation the following docker containers will be created:
+
+* Postgress DB server
+* Redis DB server
+* Django REST API server
+* Django background scheduler server
+* NGINX API frontend server
+
+Only the NGINX containers have an connection with the outside world. By default you have:
+
+* REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/
+
+## Docker-compose file
+
+This is an example file for docker compose. You can change the storage locations or the used port numbers if you want.
+
+```yaml
+version: '3'
+
+services:
+  # This is the database server that is used by Django
+  db:
+    container_name: postgresdb
+    image: postgres:13.5-bullseye
+    restart: always
+    env_file:
+      - docker/project.env
+    # ports:
+    #   - 5432:5432
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+
+  # This is the redis server for HUEY (background processes)
+  redis:
+    container_name: redisdb
+    image: redis:6.2-bullseye
+    env_file:
+      - docker/project.env
+    # We need the 'bash' command in order to get the environment variables
+    command: bash -c 'redis-server --requirepass "$${REDIS_PASSWORD}" --appendonly yes'
+    restart: always
+    volumes:
+      - redis-data:/data
+
+  # The Django API REST server. This should only be used for API calls
+  # TODO: Cleanup / split code better between API, scheduler and portal
+  broker-api:
+    container_name: broker_api
+    env_file:
+      - docker/project.env
+    # This feels like bad practice....
+    build:
+      context: ./
+      # The args below are just here for reference. For local testing, you cannot use the RUG cache
+      # But while building with Drones, you can. So Drones has this argument active!
+      # args:
+      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
+      dockerfile: ./docker/Dockerfile
+      args:
+        DEBUG: "False"
+    entrypoint: /opt/VRE/entrypoint.api.sh
+    command: gunicorn VRE.wsgi:application --bind 0.0.0.0:8000 --workers=4
+    ports:
+      - 8000:8000
+    depends_on:
+      - db
+      - redis
+    volumes:
+      - staticfiles:/opt/VRE/staticfiles
+      - mediafiles:/opt/VRE/mediafiles
+
+  # The standard NGINX server in front of the API. This will filter out all non API calls (/api/)
+  # And will also server as a static file server for the API documentation
+  broker-api-ngx:
+    container_name: broker_api_ngx
+    env_file:
+      - docker/project.env
+    build:
+      context: ./
+      # The args below are just here for reference. For local testing, you cannot use the RUG cache
+      # But while building with Drones, you can. So Drones has this argument active!
+      # args:
+      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
+      dockerfile: ./docker/Dockerfile.nginx
+    restart: always
+    ports:
+      - 1337:80
+    depends_on:
+      - broker-api
+    volumes:
+      - staticfiles:/var/www/staticfiles
+      - mediafiles:/var/www/mediafiles
+
+  # The API background scheduler based on Django and HUEY. Needs a Redis server
+  # This will process background tasks like creating new VPS machines
+  broker-scheduler:
+    container_name: broker_scheduler
+    env_file:
+      - docker/project.env
+    # This feels like bad practice....
+    build:
+      context: ./
+      # The args below are just here for reference. For local testing, you cannot use the RUG cache
+      # But while building with Drones, you can. So Drones has this argument active!
+      # args:
+      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
+      # dockerfile: ./docker/Dockerfile.scheduler
+
+      # Reuse the API docker image.....
+      dockerfile: ./docker/Dockerfile
+      args:
+        DEBUG: "False"
+    command: python manage.py run_huey
+    depends_on:
+      - broker-api
+      - db
+      - redis
+
+volumes:
+  postgres-data: null
+  redis-data: null
+  staticfiles: null
+  mediafiles: null
+```
+
+## Settings
+
+You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
+
+You can find an example environment file at `docker/project.env.example`
+
+```ini
+# The Postgress database container needs a database and user settings for use with Django
+# The database and user will be created when the Postgress container is build
+POSTGRES_USER=userone
+POSTGRES_PASSWORD=secretpassword
+POSTGRES_DB=project_db
+
+# The Django super user username. This user is created during build of this docker instance
+DJANGO_ADMIN_NAME=admin
+# The Django super user password.
+DJANGO_ADMIN_PASSWORD=password
+# The Django super user email for password retrieval.
+DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl
+
+# The TUSD super user username. This user is created during build of this docker instance
+# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
+DROPOFF_API_USER=tusdhook
+# The TUSD super user password.
+DROPOFF_API_PASSWORD=doemaarwat
+# The TUSD super user email for password retrieval.
+DROPOFF_API_EMAIL=tusd+no-reply@rug.nl
+
+
+# A uniquely secret key
+# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
+SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'
+
+# Disable debug in production
+# https://docs.djangoproject.com/en/dev/ref/settings/#debug
+DEBUG=False
+
+# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
+# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
+ALLOWED_HOSTS=127.0.0.1,localhost,0.0.0.0,broker-api,broker-api-ngx
+
+# All internal IPS for Django. Use comma separated list
+# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
+INTERNAL_IPS=127.0.0.1
+
+# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
+# By default a local sqlite3 database is used.
+DATABASE_URL=postgres://userone:secretpassword@postgresdb:5432/project_db
+
+# The location on disk where the static files will be placed during deployment. Setting is required
+# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
+STATIC_ROOT=staticfiles
+
+# The location on disk where the uploaded media files will be placed. Setting is required
+# https://docs.djangoproject.com/en/dev/ref/settings/#media-root
+MEDIA_ROOT=mediafiles
+
+# Enter the default timezone for the visitors when it is not known.
+# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
+TIME_ZONE=Europe/Amsterdam
+
+# Email settings
+# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
+# EMAIL_HOST=
+
+# Email user name
+# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
+# EMAIL_HOST_USER=
+
+# Email password
+# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
+# EMAIL_HOST_PASSWORD=
+
+# Email server port number to use. Default is 25
+# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
+# EMAIL_PORT=
+
+# Does the email server supports TLS?
+# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
+# EMAIL_USE_TLS=
+
+https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
+DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>
+
+# The sender address. This needs to be one of the allowed domains due to SPF checks
+# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
+EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>
+
+# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
+# The hostname or IP where the Redis server is running. Default is localhost
+REDIS_HOST=redisdb
+
+# The Redis port number on which the server is running. Default is 6379
+REDIS_PORT=6379
+
+# The Redis password when authentication is enabled
+REDIS_PASSWORD=redispassword
+
+# The amount of connections to be made inside a connection pool.  Default is 10
+REDIS_CONNECTIONS=10
+
+# Enter the full path to the Web based file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
+DROPOFF_BASE_URL=http://localhost:8080/dropoffs/
+
+# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
+DROPOFF_UPLOAD_HOST=http://localhost:1090
+
+# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
+DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh
+
+
+# TUS Daemon settings
+# Change the required variable below to your needs.
+# You can here also overrule the default variables in the startup.sh script
+
+# This is the full url to the REST API server to post updates during uploads.
+WEBHOOK_URL=http://api-nginx/api/v1/dropoffs/webhook/
+
+# The key for the token that is created on the REST API server for communication with the REST API server.
+# This token will be created during building the Docker image
+DROPOFF_API_HAWK_KEY=dDl6UmRt
+
+# The secret value that belongs to the token DROPOFF_API_HAWK_KEY.
+# This token will be created during building the Docker image
+DROPOFF_API_HAWK_SECRET=ExfcR524851PxBmbNzvR7quoHwzSSJ1A
+
+
+# Enter the super API user his key and secret. This is used on the portal side for getting study data on the upload page
+# We abuse the TUSD user for this
+DROPOFF_API_USER_KEY=sXl7YmRE
+DROPOFF_API_USER_SECRET=ExfcG524851PxVmbNzvX7qkoHwzSSJ1A
+
+# What is the full VRE Portal domains. By default http://localhost:1337/api
+VRE_BROKER_API=http://api-nginx/api
+
+# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
+# The security group that is allowed to access the VRW part of the REST API
+VRW_API_GROUP=vrw-api
+# The VRW username for the REST API
+VRW_API_USER=vrw
+# The VRW password for the REST API
+VRW_API_PASSWORD=securepassword
+# The VRW email address for the REST API
+VRW_API_EMAIL=vrw+no-reply@rug.nl
+
+# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
+NGINX_REAL_IP_SOURCE_IP=127.0.0.1
+# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
+# Leave empty to disable this security measure
+NGINX_REAL_IP_LIST=
+# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
+# Leave empty to disable this security measure
+NGINX_VALID_ADMIN_IP_LIST=
+# Enter the IP address that is available in the Kubernetes / Docker environment
+NGINX_RESOLVER=127.0.0.11
+# Enter the API Broker container / upstream name that is known by the Kubernetes / Docker environment
+NGINX_API_BROKER_UPSTREAM=broker-api
+```

README.md

@@ -1,6 +1,6 @@
 # Research Workspace Broker
 
-[![Build Status](https://drones.web.rug.nl/api/badges/VRE/Broker/status.svg)](https://drones.web.rug.nl/VRE/Broker) [![Translation status](http://weblate.devweb.rug.nl/widgets/vre/-/backend/svg-badge.svg)](http://weblate.devweb.rug.nl/engage/vre/)
+[![Build Status](https://drones.web.rug.nl/api/badges/VRE/Broker/status.svg)](https://drones.web.rug.nl/VRE/Broker) [![Translation status](https://weblate.devweb.rug.nl/widgets/vre/-/backend/svg-badge.svg)](https://weblate.devweb.rug.nl/engage/vre/)
 
 Research Workspaces Broker is the hart of the [Research Workspace](/VRE). This is an API that is handling all the actions and requests from either a web interface of direct API.
 
@@ -14,11 +14,12 @@ This API is responsible for:
 
 More to come ...
 
-
 ## Installation
 
 The Broker is made with the [Django Framework](https://www.djangoproject.com/) in combination with [Django Rest Framework](https://www.django-rest-framework.org/). The installation is pretty straight forward. After this the code is at location `Broker` and during the setup we assume that this is the root folder where you are in.
 
+For a full Docker setup, look at [DOCKER.md](DOCKER.md)
+
 - The Broker depends on Redis and Postgres. These can be installed using docker-compose, or by installing the services directly on to your system.
 ```sh
 docker-compose up
@@ -64,7 +65,6 @@ Now you can enter the Django Admin at `http://localhost:8000/admin/`. If do not
 
 The API can be found at http://localhost:8000/api/swagger/ or http://localhost:8000/api/redoc/
 
-
 ## Environment settings
 
 In order to get the API running, you need to specify some settings. This is done by creating environment variables with values that are read out by Django during startup. For this you can use a `.env` file or by manually setting bash environment variables. And example env file can be found at `VRE/VRE/env.example` which can be used as a template.
@@ -73,7 +73,6 @@ The location of the env file should be `VRE/VRE/.env`
 
 [more information can be found here](https://github.com/henriquebastos/python-decouple/)
 
-
 ### Variables
 
 All variables have a short explanation above them what they do or used for.
@@ -171,208 +170,62 @@ For some actions we need a background scheduler system. This system is relaying
 This will load the Python3 virtual environment and start the background scheduler. Keep the console open.
 
 ## NGINX
-We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.
-
-## VRW Integration
-...
-
-
-## Openstack Integration
-...
-
-## Docker setup
-Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/
-
-### Warning
-The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.
-
-In order to use the RUG caching setup, add the following to the Drone build:
-
-```yaml
-settings:
-  build_args:
-    - DOCKER_CACHE=registry.webhosting.rug.nl/cache/library/
-```
-
-### Install
-
-Run the following commands and a complete production setup is created.
-
-```sh
-cd Broker
-docker-compose up
-```
-
-The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.
-
-The following persistent volumes are created:
-
-* Postgress data. This is used for Postgress database files
-* Redis data. This is used for Redis storage
-* Static files data. This is used for Django static files served by NGINX
 
-During installation the following docker containers will be created:
-
-* Postgress DB server
-* Redis DB server
-* Django REST API server
-* Django background scheduler server
-* NGINX API frontend server
-
-Only the NGINX containers have an connection with the outside world. By default you have:
-
-* REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/
+We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.
 
-### Settings
+## SurfConext login
 
-You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
+In order to login, [SurfConext](https://www.surf.nl/over-surfconext-veilig-en-makkelijk-toegang-tot-clouddiensten) is used. This will required a setup at SurfConext. And a ini file with the SurfConext credentials. There is an example file called `surfnet_conext_secrets.ini.example` which can be used as a template. Copy that file to `surfnet_conext_secrets.ini` at the same location and fill the values given by Surfnet
 
 ```ini
-# The Postgress database container needs a database and user settings for use with Django
-# The database and user will be created when the Postgress container is build
-POSTGRES_USER=userone
-POSTGRES_PASSWORD=secretpassword
-POSTGRES_DB=project_db
-
-# The Django super user username. This user is created during build of this docker instance
-DJANGO_ADMIN_NAME=admin
-# The Django super user password.
-DJANGO_ADMIN_PASSWORD=password
-# The Django super user email for password retrieval.
-DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl
-
-# The TUSD super user username. This user is created during build of this docker instance
-# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
-DROPOFF_API_USER=tusdhook
-# The TUSD super user password.
-DROPOFF_API_PASSWORD=doemaarwat
-# The TUSD super user email for password retrieval.
-DROPOFF_API_EMAIL=tusd+no-reply@rug.nl
-
-
-# A uniquely secret key
-# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
-SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'
-
-# Disable debug in production
-# https://docs.djangoproject.com/en/dev/ref/settings/#debug
-DEBUG=True
-
-# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
-# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
-ALLOWED_HOSTS=127.0.0.1,localhost,0.0.0.0,broker-api,broker-api-ngx
-
-# All internal IPS for Django. Use comma separated list
-# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
-INTERNAL_IPS=127.0.0.1
-
-# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
-# By default a local sqlite3 database is used.
-DATABASE_URL=postgres://userone:secretpassword@postgresdb:5432/project_db
-
-# The location on disk where the static files will be placed during deployment. Setting is required
-# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
-STATIC_ROOT=staticfiles
-
-# The location on disk where the uploaded media files will be placed. Setting is required
-# https://docs.djangoproject.com/en/dev/ref/settings/#media-root
-MEDIA_ROOT=mediafiles
-
-# Enter the default timezone for the visitors when it is not known.
-# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
-TIME_ZONE=Europe/Amsterdam
-
-# Email settings
-# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
-# EMAIL_HOST=
-
-# Email user name
-# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
-# EMAIL_HOST_USER=
-
-# Email password
-# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
-# EMAIL_HOST_PASSWORD=
-
-# Email server port number to use. Default is 25
-# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
-# EMAIL_PORT=
-
-# Does the email server supports TLS?
-# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
-# EMAIL_USE_TLS=
-
-https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
-DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>
-
-# The sender address. This needs to be one of the allowed domains due to SPF checks
-# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
-EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>
-
-# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
-# The hostname or IP where the Redis server is running. Default is localhost
-REDIS_HOST=redisdb
-
-# The Redis port number on which the server is running. Default is 6379
-REDIS_PORT=6379
-
-# The Redis password when authentication is enabled
-REDIS_PASSWORD=redispassword
-
-# The amount of connections to be made inside a connection pool.  Default is 10
-REDIS_CONNECTIONS=10
-
-# Enter the full path to the Web based file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
-DROPOFF_BASE_URL=http://localhost:8080/dropoffs/
-
-# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
-DROPOFF_UPLOAD_HOST=http://localhost:1090
-
-# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
-DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh
-
-
-# TUS Daemon settings
-# Change the required variable below to your needs.
-# You can here also overrule the default variables in the startup.sh script
-
-# This is the full url to the REST API server to post updates during uploads.
-WEBHOOK_URL=http://api-nginx/api/v1/dropoffs/webhook/
+# Authentication settings
+# https://mozilla-django-oidc.readthedocs.io/en/stable/installation.html
+# The Client ID which is the Entity ID in SurfConext
+OIDC_RP_CLIENT_ID=
+# The Client secret that is created with the Entity ID. This will be shown only once
+OIDC_RP_CLIENT_SECRET=
+# The encryption algorithm. Default is RS256
+OIDC_RP_SIGN_ALGO=RS256
+# The following urls should be loaded automatically when PR039 is merged: https://github.com/mozilla/mozilla-django-oidc/pull/309 For now, we have to manually add those urls
+# The source is at: https://connect.surfconext.nl/.well-known/openid-configuration (production)
+# The source is at: https://connect.test.surfconext.nl/.well-known/openid-configuration (testing)
+# This is the 'authorization_endpoint' url
+OIDC_OP_AUTHORIZATION_ENDPOINT=https://connect.surfconext.nl/oidc/authorize
+# This is the 'token_endpoint' url
+OIDC_OP_TOKEN_ENDPOINT=https://connect.surfconext.nl/oidc/token
+# This is the 'userinfo_endpoint' url
+OIDC_OP_USER_ENDPOINT=https://connect.surfconext.nl/oidc/userinfo
+# This is the 'jwks_uri' url. Needed due to the `OIDC_RP_SIGN_ALGO` choice
+OIDC_OP_JWKS_ENDPOINT=https://connect.surfconext.nl/oidc/certs
+```
 
-# The key for the token that is created on the REST API server for communication with the REST API server.
-# This token will be created during building the Docker image
-DROPOFF_API_HAWK_KEY=dDl6UmRt
+## VRW Integration
+...
+In order to use the VRW (Virtual Research Workspace) you need to configure a special user that is allowed to make the API calls for VRW updates.
 
-# The secret value that belongs to the token DROPOFF_API_HAWK_KEY.
-# This token will be created during building the Docker image
-DROPOFF_API_HAWK_SECRET=ExfcR524851PxBmbNzvR7quoHwzSSJ1A
+TODO: MORE INFO
 
+## Openstack Integration
+...
+For OpenStack integration we need a `clouds.yaml` file where the API credentials are stored and the API entrypoint.
 
-# Enter the super API user his key and secret. This is used on the portal side for getting study data on the upload page
-# We abuse the TUSD user for this
-DROPOFF_API_USER_KEY=sXl7YmRE
-DROPOFF_API_USER_SECRET=ExfcG524851PxVmbNzvX7qkoHwzSSJ1A
+This file needs to be stored at the root of the VRE Broker project next to this README.md file. Else the cloud.yaml file will not be found and therefore OpenStack calls will fail.
 
-# What is the full VRE Portal domains. By default http://localhost:1337/api
-VRE_BROKER_API=http://api-nginx/api
+When this integrations is setup, it is possible to create virtual machines on the Openstack platform
 
-# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
-# The security group that is allowed to access the VRW part of the REST API
-VRW_API_GROUP=vrw-api
-# The VRW username for the REST API
-VRW_API_USER=vrw
-# The VRW password for the REST API
-VRW_API_PASSWORD=securepassword
-# The VRW email address for the REST API
-VRW_API_EMAIL=vrw+no-reply@rug.nl
+### Example config
 
-# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
-NGINX_REAL_IP_SOURCE_IP=127.0.0.1
+Here you can see an example `cloud.yaml` file. The name `hpc` at line 2 is mandatory for the RUG Openstack setup.
 
-# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
-# Leave empty to disable this security measure
-NGINX_REAL_IP_LIST=
-# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
-# Leave empty to disable this security measure
-NGINX_VALID_ADMIN_IP_LIST=
+```yaml
+clouds:
+  hpc:
+    auth:
+      auth_url: [API_ENTRY_POINT]
+      application_credential_id: "[API_CREDENTIALS]"
+      application_credential_secret: "[API_CREDENTIALS_SECRET]"
+    region_name: "RegionOne"
+    interface: "public"
+    identity_api_version: 3
+    auth_type: "v3applicationcredential"
 ```

SECURITY.md

@@ -0,0 +1,21 @@
+# Research Workspace Broker - Security
+
+Here we explain which security measurements are taken. And how to report a security problem to the developers.
+
+# Authentication
+
+In order to login as a researcher, you need a SurfConext account, which should be given by the university where you work. Therefore the VRE Broker does not know or store a researcher password. And password changes should be done at the university account settings.
+
+The SurfConext authentication is multi factor.
+
+## Login sessions
+
+When the researcher is logged in, the session stays active as long as the browser window is open. When the browser is closed, the login session is terminated. And a new login is enforced.
+
+## Special users
+
+There are a few special users that can login without using SurfConnext. Those accounts are special system accounts. Think of a Django admin user, a user for VRW integration. Those accounts are used mainly for machine to machine communication.
+
+# Encryption
+
+In order to integrate with external services, it could be needed to store sensitive data like passwords or security tokens. All those sensitive information is stored encrypted in the database using the following settings https://django-cryptography.readthedocs.io/en/latest/settings.html

VRE/surfnet_conext_secrets.ini.example

@@ -0,0 +1,19 @@
+# Authentication settings
+# https://mozilla-django-oidc.readthedocs.io/en/stable/installation.html
+# The Client ID which is the Entity ID in SurfConext
+OIDC_RP_CLIENT_ID=
+# The Client secret that is created with the Entity ID. This will be shown only once
+OIDC_RP_CLIENT_SECRET=
+# The encryption algorithm. Default is RS256
+OIDC_RP_SIGN_ALGO=RS256
+# The following urls should be loaded automatically when PR039 is merged: https://github.com/mozilla/mozilla-django-oidc/pull/309 For now, we have to manually add those urls
+# The source is at: https://connect.surfconext.nl/.well-known/openid-configuration (production)
+# The source is at: https://connect.test.surfconext.nl/.well-known/openid-configuration (testing)
+# This is the 'authorization_endpoint' url
+OIDC_OP_AUTHORIZATION_ENDPOINT=https://connect.surfconext.nl/oidc/authorize
+# This is the 'token_endpoint' url
+OIDC_OP_TOKEN_ENDPOINT=https://connect.surfconext.nl/oidc/token
+# This is the 'userinfo_endpoint' url
+OIDC_OP_USER_ENDPOINT=https://connect.surfconext.nl/oidc/userinfo
+# This is the 'jwks_uri' url. Needed due to the `OIDC_RP_SIGN_ALGO` choice
+OIDC_OP_JWKS_ENDPOINT=https://connect.surfconext.nl/oidc/certs

clouds.yaml.example

@@ -1,14 +1,10 @@
 clouds:
   hpc:
     auth:
-      auth_url: [API_URL]
-      username: "[API_USERNAME]"
-      password: "[API_PASSWORD]"
-      project_id: [PROJECT_ID]
-      project_name: "vre"
-      user_domain_name: "Default"
- 
+      auth_url: [API_ENTRY_POINT]
+      application_credential_id: "[API_CREDENTIALS]"
+      application_credential_secret: "[API_CREDENTIALS_SECRET]"
     region_name: "RegionOne"
     interface: "public"
     identity_api_version: 3
-     
+    auth_type: "v3applicationcredential"

doc/install.rst

@@ -53,5 +53,5 @@ Settings
 
 You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
 
-.. literalinclude:: ../docker/project.example.env
+.. literalinclude:: ../docker/project.env.example
     :language: bash

docker-compose.yaml.example

@@ -0,0 +1,106 @@
+version: '3'
+
+services:
+  # This is the database server that is used by Django
+  db:
+    container_name: postgresdb
+    image: postgres:13.5-bullseye
+    restart: always
+    env_file:
+      - docker/project.env
+    # ports:
+    #   - 5432:5432
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+
+  # This is the redis server for HUEY (background processes)
+  redis:
+    container_name: redisdb
+    image: redis:6.2-bullseye
+    env_file:
+      - docker/project.env
+    # We need the 'bash' command in order to get the environment variables
+    command: bash -c 'redis-server --requirepass "$${REDIS_PASSWORD}" --appendonly yes'
+    restart: always
+    volumes:
+      - redis-data:/data
+
+  # The Django API REST server. This should only be used for API calls
+  # TODO: Cleanup / split code better between API, scheduler and portal
+  broker-api:
+    container_name: broker_api
+    env_file:
+      - docker/project.env
+    # This feels like bad practice....
+    build:
+      context: ./
+      # The args below are just here for reference. For local testing, you cannot use the RUG cache
+      # But while building with Drones, you can. So Drones has this argument active!
+      # args:
+      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
+      dockerfile: ./docker/Dockerfile
+      args:
+        DEBUG: "False"
+    entrypoint: /opt/VRE/entrypoint.api.sh
+    command: gunicorn VRE.wsgi:application --bind 0.0.0.0:8000 --workers=4
+    ports:
+      - 8000:8000
+    depends_on:
+      - db
+      - redis
+    volumes:
+      - staticfiles:/opt/VRE/staticfiles
+      - mediafiles:/opt/VRE/mediafiles
+
+  # The standard NGINX server in front of the API. This will filter out all non API calls (/api/)
+  # And will also server as a static file server for the API documentation
+  broker-api-ngx:
+    container_name: broker_api_ngx
+    env_file:
+      - docker/project.env
+    build:
+      context: ./
+      # The args below are just here for reference. For local testing, you cannot use the RUG cache
+      # But while building with Drones, you can. So Drones has this argument active!
+      # args:
+      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
+      dockerfile: ./docker/Dockerfile.nginx
+    restart: always
+    ports:
+      - 1337:80
+    depends_on:
+      - broker-api
+    volumes:
+      - staticfiles:/var/www/staticfiles
+      - mediafiles:/var/www/mediafiles
+
+  # The API background scheduler based on Django and HUEY. Needs a Redis server
+  # This will process background tasks like creating new VPS machines
+  broker-scheduler:
+    container_name: broker_scheduler
+    env_file:
+      - docker/project.env
+    # This feels like bad practice....
+    build:
+      context: ./
+      # The args below are just here for reference. For local testing, you cannot use the RUG cache
+      # But while building with Drones, you can. So Drones has this argument active!
+      # args:
+      #   DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
+      # dockerfile: ./docker/Dockerfile.scheduler
+
+      # Reuse the API docker image.....
+      dockerfile: ./docker/Dockerfile
+      args:
+        DEBUG: "False"
+    command: python manage.py run_huey
+    depends_on:
+      - broker-api
+      - db
+      - redis
+
+volumes:
+  postgres-data: null
+  redis-data: null
+  staticfiles: null
+  mediafiles: null

docker/project.env.example

@@ -139,10 +139,10 @@ VRW_API_EMAIL=vrw+no-reply@rug.nl
 
 # The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
 NGINX_REAL_IP_SOURCE_IP=127.0.0.1
-# The config file that is created by Kubernets which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
+# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
 # Leave empty to disable this security measure
 NGINX_REAL_IP_LIST=
-# The config file that is created by Kubernets which holds all the valid IP addresses to access the admin
+# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
 # Leave empty to disable this security measure
 NGINX_VALID_ADMIN_IP_LIST=
 # Enter the IP address that is available in the Kubernetes / Docker environment