Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/
## Warning
The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.
In order to use the RUG caching setup, add the following to the Drone build:
Use the example `docker-compose.yaml.example` file to create a new `docker-compose.yaml` file. The example is a working production setup. So if you copy it, it should produce a working setup.
You can also change the storage locations if you want. An example can be found [here](#docker-compose-file)
The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.
The following persistent volumes are created:
* Postgress data. This is used for Postgress database files
* Redis data. This is used for Redis storage
* Static files data. This is used for Django static files served by NGINX
During installation the following docker containers will be created:
* Postgress DB server
* Redis DB server
* Django REST API server
* Django background scheduler server
* NGINX API frontend server
Only the NGINX containers have an connection with the outside world. By default you have:
* REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/
## Docker-compose file
This is an example file for docker compose. You can change the storage locations or the used port numbers if you want.
```yaml
version: '3'
services:
# This is the database server that is used by Django
db:
container_name: postgresdb
image: postgres:13.5-bullseye
restart: always
env_file:
- docker/project.env
# ports:
# - 5432:5432
volumes:
- postgres-data:/var/lib/postgresql/data
# This is the redis server for HUEY (background processes)
redis:
container_name: redisdb
image: redis:6.2-bullseye
env_file:
- docker/project.env
# We need the 'bash' command in order to get the environment variables
You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
You can find an example environment file at `docker/project.env.example`
```ini
# The Postgress database container needs a database and user settings for use with Django
# The database and user will be created when the Postgress container is build
POSTGRES_USER=userone
POSTGRES_PASSWORD=secretpassword
POSTGRES_DB=project_db
# The Django super user username. This user is created during build of this docker instance
DJANGO_ADMIN_NAME=admin
# The Django super user password.
DJANGO_ADMIN_PASSWORD=password
# The Django super user email for password retrieval.
DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl
# The TUSD super user username. This user is created during build of this docker instance
# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
DROPOFF_API_USER=tusdhook
# The TUSD super user password.
DROPOFF_API_PASSWORD=doemaarwat
# The TUSD super user email for password retrieval.
# What is the full VRE Portal domains. By default http://localhost:1337/api
VRE_BROKER_API=http://api-nginx/api
# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
# The security group that is allowed to access the VRW part of the REST API
VRW_API_GROUP=vrw-api
# The VRW username for the REST API
VRW_API_USER=vrw
# The VRW password for the REST API
VRW_API_PASSWORD=securepassword
# The VRW email address for the REST API
VRW_API_EMAIL=vrw+no-reply@rug.nl
# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1
# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
# Leave empty to disable this security measure
NGINX_VALID_ADMIN_IP_LIST=
# Enter the IP address that is available in the Kubernetes / Docker environment
NGINX_RESOLVER=127.0.0.11
# Enter the API Broker container / upstream name that is known by the Kubernetes / Docker environment
Research Workspaces Broker is the hart of the [Research Workspace](/VRE). This is an API that is handling all the actions and requests from either a web interface of direct API.
@ -14,11 +14,12 @@ This API is responsible for:
@@ -14,11 +14,12 @@ This API is responsible for:
More to come ...
## Installation
The Broker is made with the [Django Framework](https://www.djangoproject.com/) in combination with [Django Rest Framework](https://www.django-rest-framework.org/). The installation is pretty straight forward. After this the code is at location `Broker` and during the setup we assume that this is the root folder where you are in.
For a full Docker setup, look at [DOCKER.md](DOCKER.md)
- The Broker depends on Redis and Postgres. These can be installed using docker-compose, or by installing the services directly on to your system.
```sh
docker-compose up
@ -64,7 +65,6 @@ Now you can enter the Django Admin at `http://localhost:8000/admin/`. If do not
@@ -64,7 +65,6 @@ Now you can enter the Django Admin at `http://localhost:8000/admin/`. If do not
The API can be found at http://localhost:8000/api/swagger/ or http://localhost:8000/api/redoc/
## Environment settings
In order to get the API running, you need to specify some settings. This is done by creating environment variables with values that are read out by Django during startup. For this you can use a `.env` file or by manually setting bash environment variables. And example env file can be found at `VRE/VRE/env.example` which can be used as a template.
@ -73,7 +73,6 @@ The location of the env file should be `VRE/VRE/.env`
@@ -73,7 +73,6 @@ The location of the env file should be `VRE/VRE/.env`
[more information can be found here](https://github.com/henriquebastos/python-decouple/)
### Variables
All variables have a short explanation above them what they do or used for.
@ -171,208 +170,62 @@ For some actions we need a background scheduler system. This system is relaying
@@ -171,208 +170,62 @@ For some actions we need a background scheduler system. This system is relaying
This will load the Python3 virtual environment and start the background scheduler. Keep the console open.
## NGINX
We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.
## VRW Integration
...
## Openstack Integration
...
## Docker setup
Make sure you have Docker and Docker-Compose installed. On Debian based systems you can install it with the following setup https://docs.docker.com/engine/install/debian/
### Warning
The Kubernetes platform of the RUG has a caching setup for Docker images, else you will hit the rate limits of Docker Hub.
In order to use the RUG caching setup, add the following to the Drone build:
Run the following commands and a complete production setup is created.
```sh
cd Broker
docker-compose up
```
The docker setup is created according to the docker-compose.yaml in the root directory. This will create a own virtual network and some persistent storage volumes.
The following persistent volumes are created:
* Postgress data. This is used for Postgress database files
* Redis data. This is used for Redis storage
* Static files data. This is used for Django static files served by NGINX
During installation the following docker containers will be created:
* Postgress DB server
* Redis DB server
* Django REST API server
* Django background scheduler server
* NGINX API frontend server
Only the NGINX containers have an connection with the outside world. By default you have:
* REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/
We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.
### Settings
## SurfConext login
You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
In order to login, [SurfConext](https://www.surf.nl/over-surfconext-veilig-en-makkelijk-toegang-tot-clouddiensten) is used. This will required a setup at SurfConext. And a ini file with the SurfConext credentials. There is an example file called `surfnet_conext_secrets.ini.example` which can be used as a template. Copy that file to `surfnet_conext_secrets.ini` at the same location and fill the values given by Surfnet
```ini
# The Postgress database container needs a database and user settings for use with Django
# The database and user will be created when the Postgress container is build
POSTGRES_USER=userone
POSTGRES_PASSWORD=secretpassword
POSTGRES_DB=project_db
# The Django super user username. This user is created during build of this docker instance
DJANGO_ADMIN_NAME=admin
# The Django super user password.
DJANGO_ADMIN_PASSWORD=password
# The Django super user email for password retrieval.
DJANGO_ADMIN_EMAIL=admin+no-reply@rug.nl
# The TUSD super user username. This user is created during build of this docker instance
# This user will also get a predefined token key and secret. Look for variable DROPOFF_API_HAWK_KEY and DROPOFF_API_HAWK_SECRET
DROPOFF_API_USER=tusdhook
# The TUSD super user password.
DROPOFF_API_PASSWORD=doemaarwat
# The TUSD super user email for password retrieval.
# The Client ID which is the Entity ID in SurfConext
OIDC_RP_CLIENT_ID=
# The Client secret that is created with the Entity ID. This will be shown only once
OIDC_RP_CLIENT_SECRET=
# The encryption algorithm. Default is RS256
OIDC_RP_SIGN_ALGO=RS256
# The following urls should be loaded automatically when PR039 is merged: https://github.com/mozilla/mozilla-django-oidc/pull/309 For now, we have to manually add those urls
# The source is at: https://connect.surfconext.nl/.well-known/openid-configuration (production)
# The source is at: https://connect.test.surfconext.nl/.well-known/openid-configuration (testing)
This file needs to be stored at the root of the VRE Broker project next to this README.md file. Else the cloud.yaml file will not be found and therefore OpenStack calls will fail.
# What is the full VRE Portal domains. By default http://localhost:1337/api
VRE_BROKER_API=http://api-nginx/api
When this integrations is setup, it is possible to create virtual machines on the Openstack platform
# VRW API settings. This is for the VRW client to get data for creating Virtual Workspaces
# The security group that is allowed to access the VRW part of the REST API
VRW_API_GROUP=vrw-api
# The VRW username for the REST API
VRW_API_USER=vrw
# The VRW password for the REST API
VRW_API_PASSWORD=securepassword
# The VRW email address for the REST API
VRW_API_EMAIL=vrw+no-reply@rug.nl
### Example config
# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1
Here you can see an example `cloud.yaml` file. The name `hpc` at line 2 is mandatory for the RUG Openstack setup.
# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
Here we explain which security measurements are taken. And how to report a security problem to the developers.
# Authentication
In order to login as a researcher, you need a SurfConext account, which should be given by the university where you work. Therefore the VRE Broker does not know or store a researcher password. And password changes should be done at the university account settings.
The SurfConext authentication is multi factor.
## Login sessions
When the researcher is logged in, the session stays active as long as the browser window is open. When the browser is closed, the login session is terminated. And a new login is enforced.
## Special users
There are a few special users that can login without using SurfConnext. Those accounts are special system accounts. Think of a Django admin user, a user for VRW integration. Those accounts are used mainly for machine to machine communication.
# Encryption
In order to integrate with external services, it could be needed to store sensitive data like passwords or security tokens. All those sensitive information is stored encrypted in the database using the following settings https://django-cryptography.readthedocs.io/en/latest/settings.html
# The Client ID which is the Entity ID in SurfConext
OIDC_RP_CLIENT_ID=
# The Client secret that is created with the Entity ID. This will be shown only once
OIDC_RP_CLIENT_SECRET=
# The encryption algorithm. Default is RS256
OIDC_RP_SIGN_ALGO=RS256
# The following urls should be loaded automatically when PR039 is merged: https://github.com/mozilla/mozilla-django-oidc/pull/309 For now, we have to manually add those urls
# The source is at: https://connect.surfconext.nl/.well-known/openid-configuration (production)
# The source is at: https://connect.test.surfconext.nl/.well-known/openid-configuration (testing)
You can change the Docker setup by changing the settings in the file docker/project.env. Every setting has some explanation what is does or where it is fore.
# The IP number of the Load balancer which may change the IP address based on X-Forwarded-For in NGINX
NGINX_REAL_IP_SOURCE_IP=127.0.0.1
# The config file that is created by Kubernets which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# The config file that is created by Kubernetes which holds all the valid IP addresses that is allowed to overwrite the IP address of the client. Mostly load balancers
# Leave empty to disable this security measure
NGINX_REAL_IP_LIST=
# The config file that is created by Kubernets which holds all the valid IP addresses to access the admin
# The config file that is created by Kubernetes which holds all the valid IP addresses to access the admin
# Leave empty to disable this security measure
NGINX_VALID_ADMIN_IP_LIST=
# Enter the IP address that is available in the Kubernetes / Docker environment