VRE Backend API and Scheduler
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Broker/README.md

231 lines
9.4 KiB

1 year ago
# Research Workspace Broker
2 years ago
[![Build Status](https://drones.web.rug.nl/api/badges/VRE/Broker/status.svg)](https://drones.web.rug.nl/VRE/Broker) [![Translation status](https://weblate.app.rug.nl/widgets/vre/-/backend/svg-badge.svg)](https://weblate.app.rug.nl/engage/vre/)
1 year ago
Research Workspaces Broker is the hart of the [Research Workspace](/VRE). This is an API that is handling all the actions and requests from either a web interface of direct API.
2 years ago
1 year ago
This API is part of the [Research workspace](/VRE) created by the [Rijksuniversiteit Groningen](https://www.rug.nl).
2 years ago
This API is responsible for:
- Backend for the web interface to the researchers
- Inviting other researchers to your project
- Handles the creation of virtual workspaces (VRW / Openstack)
2 years ago
More to come ...
2 years ago
## Installation
The Broker is made with the [Django Framework](https://www.djangoproject.com/) in combination with [Django Rest Framework](https://www.django-rest-framework.org/). The installation is pretty straight forward. After this the code is at location `Broker` and during the setup we assume that this is the root folder where you are in.
2 years ago
For a full Docker setup, look at [DOCKER.md](DOCKER.md)
- The Broker depends on Redis and Postgres. These can be installed using docker-compose, or by installing the services directly on to your system.
2 years ago
```sh
docker-compose up
2 years ago
```
- Then Checkout the GIT repository:
2 years ago
```sh
git clone https://git.web.rug.nl/VRE/Broker.git
2 years ago
```
- Create a Python3 virtual environment by using venv (or virtualenvwrapper)
```sh
cd Broker
python3 -m venv [env_name]
2 years ago
```
- Activate the Python3 virtual environment:
```sh
source [env_name]/bin/activate
2 years ago
```
- Install all the required Python3 modules:
2 years ago
```sh
pip install -r VRE/requirements.txt
pip install -r VRE/requirements-dev.txt
2 years ago
```
- Create a `.env` config file and adjust the **Environment settings**. At least enable `Debug` in development :
2 years ago
```sh
cp VRE/VRE/env.example VRE/VRE/.env
2 years ago
```
- Create the database structure and load some needed data
```sh
VRE/manage.py migrate
VRE/manage.py loaddata virtual_machine_initial_data
VRE/manage.py loaddata university_initial_data
```
- Create a super user (admin) to login to the admin part of the API
```sh
VRE/manage.py createsuperuser
```
- Start the Django application
2 years ago
```sh
VRE/manage.py runserver
2 years ago
```
Now you can enter the Django Admin at `http://localhost:8000/admin/`. If do not see any styles on the page, make sure you have enabled `Debug` or setup [static files for Django](https://docs.djangoproject.com/en/3.2/howto/static-files/)
2 years ago
The API can be found at http://localhost:8000/api/swagger/ or http://localhost:8000/api/redoc/
2 years ago
## Environment settings
2 years ago
In order to get the API running, you need to specify some settings. This is done by creating environment variables with values that are read out by Django during startup. For this you can use a `.env` file or by manually setting bash environment variables. And example env file can be found at `VRE/VRE/env.example` which can be used as a template.
The location of the env file should be `VRE/VRE/.env`
[more information can be found here](https://github.com/henriquebastos/python-decouple/)
### Variables
All variables have a short explanation above them what they do or used for.
2 years ago
```ini
# A uniquely secret key
# https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
SECRET_KEY=@wb=#(f4uc0l%e!5*eo+aoflnxb(@!l9!=c5w=4b+x$=!8&vy%'
2 years ago
# Disable debug in production
# https://docs.djangoproject.com/en/dev/ref/settings/#debug
DEBUG=False
2 years ago
# Allowed hosts that Django does server. Use comma separated list Take care when NGINX is proxying in front of Django
# https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
ALLOWED_HOSTS=127.0.0.1,localhost
2 years ago
# All internal IPS for Django. Use comma separated list
# https://docs.djangoproject.com/en/dev/ref/settings/#internal-ips
INTERNAL_IPS=127.0.0.1
2 years ago
# Enter the database url connection. Enter all parts even the port numbers: https://github.com/jacobian/dj-database-url
# By default a local sqlite3 database is used.
DATABASE_URL=sqlite:///db.sqlite3
2 years ago
# The location on disk where the static files will be placed during deployment. Setting is required
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT=
2 years ago
# Enter the default timezone for the visitors when it is not known.
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TIME_ZONE
TIME_ZONE=Europe/Amsterdam
2 years ago
# Email settings
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host
# EMAIL_HOST=
2 years ago
# Email user name
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
# EMAIL_HOST_USER=
2 years ago
# Email password
# https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
# EMAIL_HOST_PASSWORD=
2 years ago
# Email server port number to use. Default is 25
# https://docs.djangoproject.com/en/dev/ref/settings/#email-port
# EMAIL_PORT=
2 years ago
# Does the email server supports TLS?
# https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
# EMAIL_USE_TLS=
2 years ago
https://docs.djangoproject.com/en/dev/ref/settings/#default-from-email
DEFAULT_FROM_EMAIL=Do not reply<no-reply@rug.nl>
2 years ago
# The sender address. This needs to be one of the allowed domains due to SPF checks
# The code will use a reply-to header to make sure that replies goes to the researcher and not this address
EMAIL_FROM_ADDRESS=Do not reply<no-reply@rug.nl>
2 years ago
# The Redis server is used for background tasks. Enter the variables below. Leave password empty if authentication is not enabled.
# The hostname or IP where the Redis server is running. Default is localhost
REDIS_HOST=localhost
2 years ago
# The Redis port number on which the server is running. Default is 6379
REDIS_PORT=6379
2 years ago
# The Redis password when authentication is enabled
# REDIS_PASSWORD=
2 years ago
# The amount of connections to be made inside a connection pool. Default is 10
REDIS_CONNECTIONS=10
2 years ago
# Enter the full path to the Webbased file uploading without the Study ID part. The Study ID will be added to this url based on the visitor.
DROPOFF_BASE_URL=http://localhost:8000/dropoffs/
2 years ago
# Enter the full url to the NGINX service that is in front of the TUSD service. By default that is http://localhost:1090
DROPOFF_UPLOAD_HOST=http://localhost:1090
2 years ago
# Which file extensions are **NOT** allowed to be uploaded. By default the extensions exe,com,bat,lnk,sh are not allowed
DROPOFF_NOT_ALLOWED_EXTENSIONS=exe,com,bat,lnk,sh
# Sentry settings
# Enter the full Sentry DSN string. This should contain a key and a project
SENTRY_DSN=
2 years ago
```
## Background scheduler
2 years ago
For some actions we need a background scheduler system. This system is relaying on Redis, so make sure you have Redis installed. The scheduler can be start with the command:
2 years ago
```sh
./run_scheduler.sh
2 years ago
```
This will load the Python3 virtual environment and start the background scheduler. Keep the console open.
## NGINX
We use NGINX as a proxy in front of the API. This is not mandatory, but can be handy when you have a busy api.
## SurfConext login
In order to login, [SurfConext](https://www.surf.nl/over-surfconext-veilig-en-makkelijk-toegang-tot-clouddiensten) is used. This will required a setup at SurfConext. And a ini file with the SurfConext credentials. There is an example file called `surfnet_conext_secrets.ini.example` which can be used as a template. Copy that file to `surfnet_conext_secrets.ini` at the same location and fill the values given by Surfnet
```ini
# Authentication settings
# https://mozilla-django-oidc.readthedocs.io/en/stable/installation.html
# The Client ID which is the Entity ID in SurfConext
OIDC_RP_CLIENT_ID=
# The Client secret that is created with the Entity ID. This will be shown only once
OIDC_RP_CLIENT_SECRET=
# The encryption algorithm. Default is RS256
OIDC_RP_SIGN_ALGO=RS256
# The following urls should be loaded automatically when PR039 is merged: https://github.com/mozilla/mozilla-django-oidc/pull/309 For now, we have to manually add those urls
# The source is at: https://connect.surfconext.nl/.well-known/openid-configuration (production)
# The source is at: https://connect.test.surfconext.nl/.well-known/openid-configuration (testing)
# This is the 'authorization_endpoint' url
OIDC_OP_AUTHORIZATION_ENDPOINT=https://connect.surfconext.nl/oidc/authorize
# This is the 'token_endpoint' url
OIDC_OP_TOKEN_ENDPOINT=https://connect.surfconext.nl/oidc/token
# This is the 'userinfo_endpoint' url
OIDC_OP_USER_ENDPOINT=https://connect.surfconext.nl/oidc/userinfo
# This is the 'jwks_uri' url. Needed due to the `OIDC_RP_SIGN_ALGO` choice
OIDC_OP_JWKS_ENDPOINT=https://connect.surfconext.nl/oidc/certs
```
## VRW Integration
...
In order to use the VRW (Virtual Research Workspace) you need to configure a special user that is allowed to make the API calls for VRW updates.
TODO: MORE INFO
## Openstack Integration
...
For OpenStack integration we need a `clouds.yaml` file where the API credentials are stored and the API entrypoint.
This file needs to be stored at the root of the VRE Broker project next to this README.md file. Else the cloud.yaml file will not be found and therefore OpenStack calls will fail.
When this integrations is setup, it is possible to create virtual machines on the Openstack platform
### Example config
Here you can see an example `cloud.yaml` file. The name `hpc` at line 2 is mandatory for the RUG Openstack setup.
```yaml
clouds:
hpc:
auth:
auth_url: [API_ENTRY_POINT]
application_credential_id: "[API_CREDENTIALS]"
application_credential_secret: "[API_CREDENTIALS_SECRET]"
region_name: "RegionOne"
interface: "public"
identity_api_version: 3
auth_type: "v3applicationcredential"
```