VRE Backend API and Scheduler
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

239 lines
9.0 KiB

1 year ago
===========
Development
===========
In order to develop on this software, you can setup a development environment using these steps.
In order to install this Virtual Research Environment project, we use the following packages / software.
* Redis
* Django
* NGINX
* TUSD (The Upload Server Daemon)
* Demo portal
First we need to checkout the code.
.. code-block:: bash
git clone https://git.web.rug.nl/VRE/Broker.git
1 year ago
-----
Redis
-----
Redis is used for storing the schedule/background actions. For development we use the default Redis setup **without** authentication. Install Redis with the default package manager. For Debian based:
.. code-block:: console
sudo apt install redis-server
------
Django
------
The Django code consists of three parts. There is a REST API server, a background scheduler and a demo portal. For development we use all three parts. They all work from the same Python3 virtual environment.
Common
======
First we need to create the Python virtual environment. This is done with Python 3.
.. code-block:: console
python3 -m venv venv
This will give us a virtual python environment on the location *venv* in the root of the code dir. Next we need to install the required libraries
.. code-block:: bash
source venv/bin/activate
pip install -r VRE/requirements.txt
REST API
========
Out of the box the REST API server only needs two required settings to work. These settings needs to be placed in a .env file located in the VRE/VRE folder. There should be an .env.example file which you can use as a template.
The minimal settings that needs to be set are:
* **SECRET_KEY**: A uniquely secret key. Used for cookie/session encryption
* **DEBUG**: Enable debug
Then we can setup and start the REST API server with the following commands.
.. code-block:: bash
source venv/bin/activate
./VRE/manage.py migrate
./VRE/manage.py loaddata virtual_machine_initial_data
./VRE/manage.py loaddata university_initial_data
1 year ago
./VRE/manage.py createsuperuser
And start with:
.. code-block:: bash
source venv/bin/activate
./VRE/manage.py runserver 0.0.0.0:1337
Now you can access your REST API server documentation on http://localhost:1337/api/redoc/ and the admin at http://localhost:1337/admin/
There are more settings available to setup. These can be added the to .env file of the REST API.
.. literalinclude:: ../VRE/VRE/env.example
1 year ago
:language: bash
Scheduler
=========
The scheduler is used for background tasks such as creating new workspaces or other long taking actions. The scheduler needs the same python3 environment as the REST API. So here we asume that the Python3 virtual environment is setup correctly.
.. code-block:: bash
source venv/bin/activate
./VRE/manage.py run_huey
Users
=====
We also need a TUSD user for API communication between the REST API and the TUSD server. So we create a new user in the REST API admin. Go to http://localhost:1337/admin/auth/user/add/ and create a new user. When the user is created go to the API tokens and select the token for the TUSD user. We need the *key* and *secret* of the TUSD user for later use.
make sure the TUSD user has the **superuser** status. This is needed.
-----
NGINX
-----
NGINX is used on multiple places on the project. This means that we will create multiple virtual domains to get everything working correctly.
1 year ago
**We do not cover SSL setups in this document**
Common
======
First install NGINX with LUA support through the package manager. For Debian based this would be:
.. code-block:: console
sudo apt install nginx libnginx-mod-http-lua
TUSD
====
LUA
---
There is usage of LUA in NGINX so we can handle some dynamic data on the server side. All LUA code should be placed in the folder `/etc/nginx/lua`.
.. code-block:: console
sudo ln -s /opt/deploy/VRE/nginx/lua /etc/nginx/lua
VHost
-----
After installation of the packages, create a symbolic link in the `/etc/nginx/sites-enabled` so that a new VHost is created.
Important parts of the VHost configuration:
.. literalinclude:: ../../Upload_Server/nginx/tus.vhost.conf
1 year ago
:language: bash
And there should be a `lua` folder in the `/etc/nginx` folder. This can be a symbolic link to the LUA folder that is provided with this project.
In order to test if NGINX is configured correctly run `nginx -t` and it should give an OK message:
.. code-block:: bash
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
---
TUS
---
TUS = `The Upload Server <https://tus.io/>`_. This is a resumable upload server that speaks HTTP. This server is a stand-alone server that is running behind the NGINX server. This is needed as NGINX is manipulating the headers so extra information is added to the uploads.
It is even possible to run a TUS instance on a different location (Amsterdam). As long as the TUS is reachable by the NGINX frontend server, and the TUS server can post webhooks back to the REST API server.
Setup
=====
The services is started with a simple bash script. This makes sure that all settings are loaded and the right parameters are used with the TUSD Go daemon server.
The daemon needs to know the following information. These settings are required:
* **WEBHOOK_URL**: This is the full url to the REST API server to post updates during uploads.
* **DROPOFF_API_HAWK_KEY**: The key for the token that is created on the REST API server for communication with the REST API server.
* **DROPOFF_API_HAWK_SECRET**: The secret value that belongs to the token *DROPOFF_API_HAWK_KEY*.
This information can be placed in an .env file in the same folder where the startup (*startup.sh*) script is located. An example .env file:
.. literalinclude:: ../../Upload_Server/.env.example
1 year ago
:language: bash
In the startup.sh script there are some default variables that can be overwritten by adding them to the .env file above.
.. literalinclude:: ../../Upload_Server/startup.sh
1 year ago
:language: bash
:lines: 5-16
This will start the TUS server running on TCP port 1080.
Data storage
------------
The upload data is stored at a folder that is configured in the TUS startup command. This should be folder that is writable by the user that is running the TUS instance. **Make sure that the upload folder is not directly accessible by the webserver**. Else files can be downloaded.
Hooks
-----
The TUS is capable of handling hooks based on uploaded files. There are two types of hooks. 'Normal' hooks and webhooks. It is not possible to run both hook systems at the same time due to the blocking nature of the pre-create hook. So we use the 'normal' hook system. That means that custom scripts are run. Those scripts can then post the data to a webserver in order to get a Webhook functionality with the 'normal' hooks.
At the moment, there is only a HTTP API call done in the hook system. There is no actual file movement yet.
1 year ago
For now we have used the following hooks:
- **pre-create**: This hook will run when a new upload starts. This will trigger the REST API server to store the upload in the database, and check if the upload is allowed based on an unique upload url and unique upload code.
- **post-finish**: This hook will run when an upload is finished. And will update the REST API server with the file size and actual filename (unique) on disk.
An example of a hook as used in this project is the *pre-create.py* script.
.. literalinclude:: ../../Upload_Server/hooks/pre-create.py
1 year ago
This hook uses the same data payload as when TUS would use the Webhook system. So using 'Normal' hooks or using Webhooks with REST API Server should both work out of the box.
-----------
Demo portal
-----------
In order to test the REST API and be able to give a demo, there is a demo portal that can be used.
Out of the box the Demo portal only needs a few required settings to work. These settings needs to be placed in a .env file located in the demo_portal/demo_portal folder. There should be an .env.example file which you can use as a template.
The minimal settings that needs to be set are:
* **SECRET_KEY**: A uniquely secret key. Used for cookie/session encryption
* **DEBUG**: Enable debug
* **DROPOFF_API_USER_KEY**: The key for the token that is created on the REST API server for communication with the REST API server.
* **DROPOFF_API_USER_SECRET**: The secret value that belongs to the token *DROPOFF_API_USER_KEY*.
Then we can setup and start the demo portal with the following commands.
.. code-block:: bash
source venv/bin/activate
./VRE/manage.py migrate
And start with:
.. code-block:: bash
source venv/bin/activate
./VRE/manage.py runserver 0.0.0.0:8000
Now you can access your demo portal at http://localhost:8000. In order to login as a researcher, you need to create a new user on the REST API server admin. Because the portal is using the REST API server for logging users in.
There are more settings available to setup. These can be added the to .env file of the demo portal.
.. literalinclude:: ../../Demo_Portal_1/demo_portal/demo_portal/.env.example
1 year ago
:language: bash