Split up documentation and add SurfConext

master
Joshua Rubingh 9 months ago
parent 8390382647
commit adfc596d2a
  1. 319
      DOCKER.md
  2. 243
      README.md
  3. 21
      SECURITY.md
  4. 19
      VRE/surfnet_conext_secrets.ini.example
  5. 12
      clouds.yaml.example
  6. 2
      doc/install.rst
  7. 106
      docker-compose.yaml.example
  8. 4
      docker/project.env.example

@ -0,0 +1,319 @@
# Research Workspace Broker - Docker setup
Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/
## Warning
The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.
In order to use the RUG caching setup, add the following to the Drone build:
```yaml
settings:
build_args:
- DOCKER_CACHE=registry.webhosting.rug.nl/cache/library/
```
## Install
Use the example `docker-compose.yaml.example` file to create a new `docker-compose.yaml` file. The example is a working production setup. So if you copy it, it should produce a working setup.
You can also change the storage locations if you want. An example can be found [here](#docker-compose-file)
```sh
cd Broker
cp docker-compose.yaml.example docker-compose.yaml
docker-compose up
```
The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.
The following persistent volumes are created:
* Postgress data. This is used for Postgress database files
* Redis data. This is used for Redis storage
* Static files data. This is used for Django static files served by NGINX
During installation the following docker containers will be created:
* Postgress DB server
* Redis DB server
* Django REST API server
* Django background scheduler server
* NGINX API frontend server
Only the NGINX containers have an connection with the outside world. By default you have:
* REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/
## Docker-compose file
This is an example file for docker compose. You can change the storage locations or the used port numbers if you want.
```yaml
version: '3'
services:
# This is the database server that is used by Django
db:
container_name: postgresdb
image: postgres:13.5-bullseye
restart: always
env_file:
- docker/project.env
# ports:
# - 5432:5432
volumes:
- postgres-data:/var/lib/postgresql/data
# This is the redis server for HUEY (background processes)
redis:
container_name: redisdb
image: redis:6.2-bullseye
env_file:
- docker/project.env
# We need the 'bash' command in order to get the environment variables
command: bash -c 'redis-server --requirepass "$${REDIS_PASSWORD}" --appendonly yes'
restart: always
volumes:
- redis-data:/data
# The Django API REST server. This should only be used for API calls
# TODO: Cleanup / split code better between API, scheduler and portal
broker-api:
container_name: broker_api
env_file:
- docker/project.env
# This feels like bad practice....
build:
context: ./
# The args below are just here for reference. For local testing, you cannot use the RUG cache
# But while building with Drones, you can. So Drones has this argument active!
# args:
# DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
dockerfile: ./docker/Dockerfile
args:
DEBUG: "False"
entrypoint: /opt/VRE/entrypoint.api.sh
command: gunicorn VRE.wsgi:application --bind 0.0.0.0:8000 --workers=4
ports:
- 8000:8000
depends_on:
- db
- redis
volumes:
- staticfiles:/opt/VRE/staticfiles
- mediafiles:/opt/VRE/mediafiles
# The standard NGINX server in front of the API. This will filter out all non API calls (/api/)
# And will also server as a static file server for the API documentation
broker-api-ngx:
container_name: broker_api_ngx
env_file:
- docker/project.env
build:
context: ./
# The args below are just here for reference. For local testing, you cannot use the RUG cache
# But while building with Drones, you can. So Drones has this argument active!
# args:
# DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
dockerfile: ./docker/Dockerfile.nginx
restart: always
ports:
- 1337:80
depends_on:
- broker-api
volumes:
- staticfiles:/var/www/staticfiles
- mediafiles:/var/www/mediafiles
# The API background scheduler based on Django and HUEY. Needs a Redis server
# This will process background tasks like creating new VPS machines
broker-scheduler:
container_name: broker_scheduler
env_file:
- docker/project.env
# This feels like bad practice....
build:
context: ./
# The args below are just here for reference. For local testing, you cannot use the RUG cache
# But while building with Drones, you can. So Drones has this argument active!
# args:
# DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
# dockerfile: ./docker/Dockerfile.scheduler
# Reuse the API docker image.....
dockerfile: ./docker/Dockerfile
args:
DEBUG: "False"
command: python manage.py run_huey
depends_on:
- broker-api
- db
- redis
volumes:
postgres-data: null
redis-data: null
staticfiles: null
mediafiles: null
```
## Settings
You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
You can find an example environment file at `docker/project.env.example`
```ini
# The Postgress database container needs a database and user settings for use with Django
# The database and user will be created when the Postgress container is build
POSTGRES_USER=userone
POSTGRES_PASSWORD=secretpassword
POSTGRES_DB=project_db
# The Django super user username. This user is created during build of this docker instance
DJANGO_ADMIN_NAME=admin
# The Django super user password.
DJANGO_ADMIN_PASSWORD=password
# The Django super user email for password retrieval.
DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl
# The TUSD super user username. This user is created during build of this docker instance
# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
DROPOFF_API_USER=tusdhook
# The TUSD super user password.
DROPOFF_API_PASSWORD=doemaarwat
# The TUSD super user email for password retrieval.
DROPOFF_API_EMAIL=tusd+no-reply@rug.nl
# A uniquely secret key
# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'
# Disable debug in production
# https://docs.djangoproject.com/en/dev/ref/settings/#debug
DEBUG=False
# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
ALLOWED_HOSTS=127.0.0.1,localhost,0.0.0.0,broker-api,broker-api-ngx
# All internal IPS for Django. Use comma separated list
# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
INTERNAL_IPS=127.0.0.1
# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
# By default a local sqlite3 database is used.
DATABASE_URL=postgres://userone:secretpassword@postgresdb:5432/project_db
# The location on disk where the static files will be placed during deployment. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT=staticfiles
# The location on disk where the uploaded media files will be placed. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT=mediafiles
# Enter the default timezone for the visitors when it is not known.
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
TIME_ZONE=Europe/Amsterdam
# Email settings
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
# EMAIL_HOST=
# Email user name
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
# EMAIL_HOST_USER=
# Email password
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
# EMAIL_HOST_PASSWORD=
# Email server port number to use. Default is 25
# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
# EMAIL_PORT=
# Does the email server supports TLS?
# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
# EMAIL_USE_TLS=
https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>
# The sender address. This needs to be one of the allowed domains due to SPF checks
# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>
# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
# The hostname or IP where the Redis server is running. Default is localhost
REDIS_HOST=redisdb
# The Redis port number on which the server is running. Default is 6379
REDIS_PORT=6379
# The Redis password when authentication is enabled
REDIS_PASSWORD=redispassword
# The amount of connections to be made inside a connection pool. Default is 10
REDIS_CONNECTIONS=10
# Enter the full path to the Web based file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
DROPOFF_BASE_URL=http://localhost:8080/dropoffs/
# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
DROPOFF_UPLOAD_HOST=http://localhost:1090
# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh
# TUS Daemon settings
# Change the required variable below to your needs.
# You can here also overrule the default variables in the startup.sh script
# This is the full url to the REST API server to post updates during uploads.
WEBHOOK_URL=http://api-nginx/api/v1/dropoffs/webhook/
# The key for the token that is created on the REST API server for communication with the REST API server.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_KEY=dDl6UmRt
# The secret value that belongs to the token DROPOFF_API_HAWK_KEY.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_SECRET=ExfcR524851PxBmbNzvR7quoHwzSSJ1A
# Enter the super API user his key and secret. This is used on the portal side for getting study data on the upload page
# We abuse the TUSD user for this
DROPOFF_API_USER_KEY=sXl7YmRE
DROPOFF_API_USER_SECRET=ExfcG524851PxVmbNzvX7qkoHwzSSJ1A
# What is the full VRE Portal domains. By default http://localhost:1337/api
VRE_BROKER_API=http://api-nginx/api
# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
# The security group that is allowed to access the VRW part of the REST API
VRW_API_GROUP=vrw-api
# The VRW username for the REST API
VRW_API_USER=vrw
# The VRW password for the REST API
VRW_API_PASSWORD=securepassword
# The VRW email address for the REST API
VRW_API_EMAIL=vrw+no-reply@rug.nl
# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1
# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
# Leave empty to disable this security measure
NGINX_VALID_ADMIN_IP_LIST=
# Enter the IP address that is available in the Kubernetes / Docker environment
NGINX_RESOLVER=127.0.0.11
# Enter the API Broker container / upstream name that is known by the Kubernetes / Docker environment
NGINX_API_BROKER_UPSTREAM=broker-api
```

@ -1,6 +1,6 @@
# Research Workspace Broker
[![Build Status](https://drones.web.rug.nl/api/badges/VRE/Broker/status.svg)](https://drones.web.rug.nl/VRE/Broker) [![Translation status](http://weblate.devweb.rug.nl/widgets/vre/-/backend/svg-badge.svg)](http://weblate.devweb.rug.nl/engage/vre/)
[![Build Status](https://drones.web.rug.nl/api/badges/VRE/Broker/status.svg)](https://drones.web.rug.nl/VRE/Broker) [![Translation status](https://weblate.devweb.rug.nl/widgets/vre/-/backend/svg-badge.svg)](https://weblate.devweb.rug.nl/engage/vre/)
Research Workspaces Broker is the hart of the [Research Workspace](/VRE). This is an API that is handling all the actions and requests from either a web interface of direct API.
@ -14,11 +14,12 @@ This API is responsible for:
More to come ...
## Installation
The Broker is made with the [Django Framework](https://www.djangoproject.com/) in combination with [Django Rest Framework](https://www.django-rest-framework.org/). The installation is pretty straight forward. After this the code is at location `Broker` and during the setup we assume that this is the root folder where you are in.
For a full Docker setup, look at [DOCKER.md](DOCKER.md)
- The Broker depends on Redis and Postgres. These can be installed using docker-compose, or by installing the services directly on to your system.
```sh
docker-compose up
@ -64,7 +65,6 @@ Now you can enter the Django Admin at `http://localhost:8000/admin/`. If do not
The API can be found at http://localhost:8000/api/swagger/ or http://localhost:8000/api/redoc/
## Environment settings
In order to get the API running, you need to specify some settings. This is done by creating environment variables with values that are read out by Django during startup. For this you can use a `.env` file or by manually setting bash environment variables. And example env file can be found at `VRE/VRE/env.example` which can be used as a template.
@ -73,7 +73,6 @@ The location of the env file should be `VRE/VRE/.env`
[more information can be found here](https://github.com/henriquebastos/python-decouple/)
### Variables
All variables have a short explanation above them what they do or used for.
@ -171,208 +170,62 @@ For some actions we need a background scheduler system. This system is relaying
This will load the Python3 virtual environment and start the background scheduler. Keep the console open.
## NGINX
We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.
## VRW Integration
...
## Openstack Integration
...
## Docker setup
Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/
### Warning
The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.
In order to use the RUG caching setup, add the following to the Drone build:
```yaml
settings:
build_args:
- DOCKER_CACHE=registry.webhosting.rug.nl/cache/library/
```
### Install
Run the following commands and a complete production setup is created.
```sh
cd Broker
docker-compose up
```
The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.
The following persistent volumes are created:
* Postgress data. This is used for Postgress database files
* Redis data. This is used for Redis storage
* Static files data. This is used for Django static files served by NGINX
During installation the following docker containers will be created:
* Postgress DB server
* Redis DB server
* Django REST API server
* Django background scheduler server
* NGINX API frontend server
Only the NGINX containers have an connection with the outside world. By default you have:
* REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/
We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.
### Settings
## SurfConext login
You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
In order to login, [SurfConext](https://www.surf.nl/over-surfconext-veilig-en-makkelijk-toegang-tot-clouddiensten) is used. This will required a setup at SurfConext. And a ini file with the SurfConext credentials. There is an example file called `surfnet_conext_secrets.ini.example` which can be used as a template. Copy that file to `surfnet_conext_secrets.ini` at the same location and fill the values given by Surfnet
```ini
# The Postgress database container needs a database and user settings for use with Django
# The database and user will be created when the Postgress container is build
POSTGRES_USER=userone
POSTGRES_PASSWORD=secretpassword
POSTGRES_DB=project_db
# The Django super user username. This user is created during build of this docker instance
DJANGO_ADMIN_NAME=admin
# The Django super user password.
DJANGO_ADMIN_PASSWORD=password
# The Django super user email for password retrieval.
DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl
# The TUSD super user username. This user is created during build of this docker instance
# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
DROPOFF_API_USER=tusdhook
# The TUSD super user password.
DROPOFF_API_PASSWORD=doemaarwat
# The TUSD super user email for password retrieval.
DROPOFF_API_EMAIL=tusd+no-reply@rug.nl
# A uniquely secret key
# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'
# Disable debug in production
# https://docs.djangoproject.com/en/dev/ref/settings/#debug
DEBUG=True
# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
ALLOWED_HOSTS=127.0.0.1,localhost,0.0.0.0,broker-api,broker-api-ngx
# All internal IPS for Django. Use comma separated list
# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
INTERNAL_IPS=127.0.0.1
# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
# By default a local sqlite3 database is used.
DATABASE_URL=postgres://userone:secretpassword@postgresdb:5432/project_db
# The location on disk where the static files will be placed during deployment. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT=staticfiles
# The location on disk where the uploaded media files will be placed. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT=mediafiles
# Enter the default timezone for the visitors when it is not known.
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
TIME_ZONE=Europe/Amsterdam
# Email settings
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
# EMAIL_HOST=
# Email user name
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
# EMAIL_HOST_USER=
# Email password
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
# EMAIL_HOST_PASSWORD=
# Email server port number to use. Default is 25
# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
# EMAIL_PORT=
# Does the email server supports TLS?
# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
# EMAIL_USE_TLS=
https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>
# The sender address. This needs to be one of the allowed domains due to SPF checks
# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>
# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
# The hostname or IP where the Redis server is running. Default is localhost
REDIS_HOST=redisdb
# The Redis port number on which the server is running. Default is 6379
REDIS_PORT=6379
# The Redis password when authentication is enabled
REDIS_PASSWORD=redispassword
# The amount of connections to be made inside a connection pool. Default is 10
REDIS_CONNECTIONS=10
# Enter the full path to the Web based file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
DROPOFF_BASE_URL=http://localhost:8080/dropoffs/
# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
DROPOFF_UPLOAD_HOST=http://localhost:1090
# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh
# TUS Daemon settings
# Change the required variable below to your needs.
# You can here also overrule the default variables in the startup.sh script
# This is the full url to the REST API server to post updates during uploads.
WEBHOOK_URL=http://api-nginx/api/v1/dropoffs/webhook/
# Authentication settings
# https://mozilla-django-oidc.readthedocs.io/en/stable/installation.html
# The Client ID which is the Entity ID in SurfConext
OIDC_RP_CLIENT_ID=
# The Client secret that is created with the Entity ID. This will be shown only once
OIDC_RP_CLIENT_SECRET=
# The encryption algorithm. Default is RS256
OIDC_RP_SIGN_ALGO=RS256
# The following urls should be loaded automatically when PR039 is merged: https://github.com/mozilla/mozilla-django-oidc/pull/309 For now, we have to manually add those urls
# The source is at: https://connect.surfconext.nl/.well-known/openid-configuration (production)
# The source is at: https://connect.test.surfconext.nl/.well-known/openid-configuration (testing)
# This is the 'authorization_endpoint' url
OIDC_OP_AUTHORIZATION_ENDPOINT=https://connect.surfconext.nl/oidc/authorize
# This is the 'token_endpoint' url
OIDC_OP_TOKEN_ENDPOINT=https://connect.surfconext.nl/oidc/token
# This is the 'userinfo_endpoint' url
OIDC_OP_USER_ENDPOINT=https://connect.surfconext.nl/oidc/userinfo
# This is the 'jwks_uri' url. Needed due to the `OIDC_RP_SIGN_ALGO` choice
OIDC_OP_JWKS_ENDPOINT=https://connect.surfconext.nl/oidc/certs
```
# The key for the token that is created on the REST API server for communication with the REST API server.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_KEY=dDl6UmRt
## VRW Integration
...
In order to use the VRW (Virtual Research Workspace) you need to configure a special user that is allowed to make the API calls for VRW updates.
# The secret value that belongs to the token DROPOFF_API_HAWK_KEY.
# This token will be created during building the Docker image
DROPOFF_API_HAWK_SECRET=ExfcR524851PxBmbNzvR7quoHwzSSJ1A
TODO: MORE INFO
## Openstack Integration
...
For OpenStack integration we need a `clouds.yaml` file where the API credentials are stored and the API entrypoint.
# Enter the super API user his key and secret. This is used on the portal side for getting study data on the upload page
# We abuse the TUSD user for this
DROPOFF_API_USER_KEY=sXl7YmRE
DROPOFF_API_USER_SECRET=ExfcG524851PxVmbNzvX7qkoHwzSSJ1A
This file needs to be stored at the root of the VRE Broker project next to this README.md file. Else the cloud.yaml file will not be found and therefore OpenStack calls will fail.
# What is the full VRE Portal domains. By default http://localhost:1337/api
VRE_BROKER_API=http://api-nginx/api
When this integrations is setup, it is possible to create virtual machines on the Openstack platform
# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
# The security group that is allowed to access the VRW part of the REST API
VRW_API_GROUP=vrw-api
# The VRW username for the REST API
VRW_API_USER=vrw
# The VRW password for the REST API
VRW_API_PASSWORD=securepassword
# The VRW email address for the REST API
VRW_API_EMAIL=vrw+no-reply@rug.nl
### Example config
# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1
Here you can see an example `cloud.yaml` file. The name `hpc` at line 2 is mandatory for the RUG Openstack setup.
# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
# Leave empty to disable this security measure
NGINX_VALID_ADMIN_IP_LIST=
```yaml
clouds:
hpc:
auth:
auth_url: [API_ENTRY_POINT]
application_credential_id: "[API_CREDENTIALS]"
application_credential_secret: "[API_CREDENTIALS_SECRET]"
region_name: "RegionOne"
interface: "public"
identity_api_version: 3
auth_type: "v3applicationcredential"
```

@ -0,0 +1,21 @@
# Research Workspace Broker - Security
Here we explain which security measurements are taken. And how to report a security problem to the developers.
# Authentication
In order to login as a researcher, you need a SurfConext account, which should be given by the university where you work. Therefore the VRE Broker does not know or store a researcher password. And password changes should be done at the university account settings.
The SurfConext authentication is multi factor.
## Login sessions
When the researcher is logged in, the session stays active as long as the browser window is open. When the browser is closed, the login session is terminated. And a new login is enforced.
## Special users
There are a few special users that can login without using SurfConnext. Those accounts are special system accounts. Think of a Django admin user, a user for VRW integration. Those accounts are used mainly for machine to machine communication.
# Encryption
In order to integrate with external services, it could be needed to store sensitive data like passwords or security tokens. All those sensitive information is stored encrypted in the database using the following settings https://django-cryptography.readthedocs.io/en/latest/settings.html

@ -0,0 +1,19 @@
# Authentication settings
# https://mozilla-django-oidc.readthedocs.io/en/stable/installation.html
# The Client ID which is the Entity ID in SurfConext
OIDC_RP_CLIENT_ID=
# The Client secret that is created with the Entity ID. This will be shown only once
OIDC_RP_CLIENT_SECRET=
# The encryption algorithm. Default is RS256
OIDC_RP_SIGN_ALGO=RS256
# The following urls should be loaded automatically when PR039 is merged: https://github.com/mozilla/mozilla-django-oidc/pull/309 For now, we have to manually add those urls
# The source is at: https://connect.surfconext.nl/.well-known/openid-configuration (production)
# The source is at: https://connect.test.surfconext.nl/.well-known/openid-configuration (testing)
# This is the 'authorization_endpoint' url
OIDC_OP_AUTHORIZATION_ENDPOINT=https://connect.surfconext.nl/oidc/authorize
# This is the 'token_endpoint' url
OIDC_OP_TOKEN_ENDPOINT=https://connect.surfconext.nl/oidc/token
# This is the 'userinfo_endpoint' url
OIDC_OP_USER_ENDPOINT=https://connect.surfconext.nl/oidc/userinfo
# This is the 'jwks_uri' url. Needed due to the `OIDC_RP_SIGN_ALGO` choice
OIDC_OP_JWKS_ENDPOINT=https://connect.surfconext.nl/oidc/certs

@ -1,14 +1,10 @@
clouds:
hpc:
auth:
auth_url: [API_URL]
username: "[API_USERNAME]"
password: "[API_PASSWORD]"
project_id: [PROJECT_ID]
project_name: "vre"
user_domain_name: "Default"
auth_url: [API_ENTRY_POINT]
application_credential_id: "[API_CREDENTIALS]"
application_credential_secret: "[API_CREDENTIALS_SECRET]"
region_name: "RegionOne"
interface: "public"
identity_api_version: 3
auth_type: "v3applicationcredential"

@ -53,5 +53,5 @@ Settings
You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
.. literalinclude:: ../docker/project.example.env
.. literalinclude:: ../docker/project.env.example
:language: bash

@ -0,0 +1,106 @@
version: '3'
services:
# This is the database server that is used by Django
db:
container_name: postgresdb
image: postgres:13.5-bullseye
restart: always
env_file:
- docker/project.env
# ports:
# - 5432:5432
volumes:
- postgres-data:/var/lib/postgresql/data
# This is the redis server for HUEY (background processes)
redis:
container_name: redisdb
image: redis:6.2-bullseye
env_file:
- docker/project.env
# We need the 'bash' command in order to get the environment variables
command: bash -c 'redis-server --requirepass "$${REDIS_PASSWORD}" --appendonly yes'
restart: always
volumes:
- redis-data:/data
# The Django API REST server. This should only be used for API calls
# TODO: Cleanup / split code better between API, scheduler and portal
broker-api:
container_name: broker_api
env_file:
- docker/project.env
# This feels like bad practice....
build:
context: ./
# The args below are just here for reference. For local testing, you cannot use the RUG cache
# But while building with Drones, you can. So Drones has this argument active!
# args:
# DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
dockerfile: ./docker/Dockerfile
args:
DEBUG: "False"
entrypoint: /opt/VRE/entrypoint.api.sh
command: gunicorn VRE.wsgi:application --bind 0.0.0.0:8000 --workers=4
ports:
- 8000:8000
depends_on:
- db
- redis
volumes:
- staticfiles:/opt/VRE/staticfiles
- mediafiles:/opt/VRE/mediafiles
# The standard NGINX server in front of the API. This will filter out all non API calls (/api/)
# And will also server as a static file server for the API documentation
broker-api-ngx:
container_name: broker_api_ngx
env_file:
- docker/project.env
build:
context: ./
# The args below are just here for reference. For local testing, you cannot use the RUG cache
# But while building with Drones, you can. So Drones has this argument active!
# args:
# DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
dockerfile: ./docker/Dockerfile.nginx
restart: always
ports:
- 1337:80
depends_on:
- broker-api
volumes:
- staticfiles:/var/www/staticfiles
- mediafiles:/var/www/mediafiles
# The API background scheduler based on Django and HUEY. Needs a Redis server
# This will process background tasks like creating new VPS machines
broker-scheduler:
container_name: broker_scheduler
env_file:
- docker/project.env
# This feels like bad practice....
build:
context: ./
# The args below are just here for reference. For local testing, you cannot use the RUG cache
# But while building with Drones, you can. So Drones has this argument active!
# args:
# DOCKER_CACHE: registry.webhosting.rug.nl/cache/library/
# dockerfile: ./docker/Dockerfile.scheduler
# Reuse the API docker image.....
dockerfile: ./docker/Dockerfile
args:
DEBUG: "False"
command: python manage.py run_huey
depends_on:
- broker-api
- db
- redis
volumes:
postgres-data: null
redis-data: null
staticfiles: null
mediafiles: null

@ -139,10 +139,10 @@ VRW_API_EMAIL=vrw+no-reply@rug.nl
# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1
# The config file that is created by Kubernets which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernets which holds all the valid IP addresses to access the admin
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
# Leave empty to disable this security measure
NGINX_VALID_ADMIN_IP_LIST=
# Enter the IP address that is available in the Kubernetes / Docker environment
Loading…
Cancel
Save