Browse Source

Update documentation. Renamed VRE to Research Workspace

master
Joshua Rubingh 6 months ago
parent
commit
1c251c6ce7
  1. 4
      README.md
  2. 11
      VRE/apps/api/authentication.py
  3. 2
      VRE/apps/api/urls.py
  4. 2
      VRE/apps/vre_apps/signals.py
  5. 3
      doc/OpenStack.rst
  6. 10
      doc/SURFconext.rst
  7. 3
      doc/VRW.rst
  8. 2
      doc/authentication.rst
  9. 23
      doc/conf.py
  10. 43
      doc/development.rst
  11. BIN
      doc/documentation.pdf
  12. 18
      doc/index.rst
  13. 9
      doc/install.rst
  14. 2
      doc/requirements.txt
  15. 15
      doc/signals.rst
  16. 20
      doc/storage.rst
  17. 1598
      doc/swagger.yaml
  18. 2
      doc/tus.rst

4
README.md

@ -2,7 +2,7 @@ @@ -2,7 +2,7 @@
[![Build Status](https://drones.web.rug.nl/api/badges/VRE/Broker/status.svg)](https://drones.web.rug.nl/VRE/Broker)
Virtual Research Environment Broker is the hart of the [Research Workspace system](/VRE). This is an API that is handling all the actions and requests from either a web interface of direct API.
Research Workspaces Broker is the hart of the [Research Workspace system](/VRE). This is an API that is handling all the actions and requests from either a web interface of direct API.
This API is part of the [Research workspace system](/VRE) created by the [Rijksuniversiteit Groningen](https://www.rug.nl).
@ -19,7 +19,7 @@ More to come ... @@ -19,7 +19,7 @@ More to come ...
The Broker is made with the [Django Framework](https://www.djangoproject.com/) in combination with [Django Rest Framework](https://www.django-rest-framework.org/). The installation is pretty straight forward. After this the code is at location `Broker` and during the setup we assume that this is the root folder where you are in.
- The Broker depends on Redis and Postgres. These can be installed using docker-compose, or by installing the services directly on to your system.
- The Broker depends on Redis and Postgres. These can be installed using docker-compose, or by installing the services directly on to your system.
```sh
docker-compose up
```

11
VRE/apps/api/authentication.py

@ -79,9 +79,16 @@ class APIHawk(HawkAuthentication): @@ -79,9 +79,16 @@ class APIHawk(HawkAuthentication):
class VRE_OIDC_Researcher_Update(OIDCAuthenticationBackend):
"""Update the logged in user his/her first and last name based on SURFconext login
"""Update the logged in user his/her data based on SURFconext login
This is done after each login.
We sync the following data from SURFconext:
* First name
* Last name
* Email address (when the researcher got invited on his private mail and not the university mail address)
* The faculty to which the user belongs to.
The data is only stored when there is a change. SURFconext is leading.
"""
def __get_or_create_faculty(self, claims):

2
VRE/apps/api/urls.py

@ -12,7 +12,7 @@ from apps.study.views import Studies @@ -12,7 +12,7 @@ from apps.study.views import Studies
schema_view = get_schema_view(
openapi.Info(
title="Virtual Research Environment API",
title="Research Workspaces API",
default_version='v1',
description="Here you can see a list of API endpoints and actions that are available to communicate with the VRE API",
terms_of_service="https://www.rug.nl",

2
VRE/apps/vre_apps/signals.py

@ -9,7 +9,7 @@ from apps.virtual_machine.models import VirtualMachine @@ -9,7 +9,7 @@ from apps.virtual_machine.models import VirtualMachine
@receiver(post_save, sender=VRE_App)
def create_VRE_app(sender, instance, created, **kwargs):
"""when a new reseracher is added to a study, it could be that a new virtual workspace needs to be created.
"""When a new reseracher is added to a study, it could be that a new virtual workspace needs to be created.
This signal will check if this is the case based on if the researcher is **added** to the study and has the proper study role.

3
doc/OpenStack.rst

@ -18,8 +18,7 @@ Models @@ -18,8 +18,7 @@ Models
-------
Signals
-------
.. automodule:: apps.virtual_machine.providers.openstack.signals
:members:
:ref:`OpenStack Signals<hpc_signals>`
--------
Glossary

10
doc/SURFconext.rst

@ -1,3 +1,5 @@ @@ -1,3 +1,5 @@
.. _SURFconext manual:
==========
SURFconext
==========
@ -11,6 +13,8 @@ More documentation can be found at: https://wiki.surfnet.nl/display/SURFconextde @@ -11,6 +13,8 @@ More documentation can be found at: https://wiki.surfnet.nl/display/SURFconextde
In order to use SURFconext, you need to make a configuration in SURFconext and then configure the API.
This can be done at the location: https://sp.surfconext.nl/
----------
SURFconext
----------
@ -19,13 +23,13 @@ SURFconext @@ -19,13 +23,13 @@ SURFconext
:alt: SURFconext Configuration
- **Client ID**: The name of the configuration. Use the API url as this is suggested in the GUI.
- **Redirect urls**: The urls where to redirect to after login. Use `localhost` urls for development. For production use: **https://api-vre.web.rug.nl/oidc/callback/**
- **Redirect urls**: The urls where to redirect to after login. Use `localhost` urls for development. For production use: **https://workspaces.research.rug.nl/oidc/callback/**
- **Access token validity**: How long is the login token valid at SURFconext. SURFconext default 3600 seconds.
- **Is public client**: Do not use this.
- **Grants**: We need the `Authorization code` option
- **Subject type**: We need the `Persistent` option
All other options are not required for making the OpenID connection with the VRE API.
All other options are not required for making the OpenID connection with the Research Workspaces API.
**Attention**: When you save the configuration for the first time, you will get an `Client secret`. Save this value somewhere safe. You need it later on.
@ -52,7 +56,7 @@ For SURFconext the `well known` url is: https://connect.SURFconext.nl/oidc/.well @@ -52,7 +56,7 @@ For SURFconext the `well known` url is: https://connect.SURFconext.nl/oidc/.well
- **OIDC_OP_USER_ENDPOINT**: Full url to the user info endpoint. In the well `known data` it is the url at `userinfo_endpoint`
- **OIDC_OP_JWKS_ENDPOINT**: Full url to the certification endpoint. Needed because of the `RS256` algorithm. In the well `known data` it is the url at `jwks_uri`
Two other settings depends on the VRE frontend which are used after login or logout actions.
Two other settings depends on the Research Workspaces frontend which are used after login or logout actions.
- **LOGIN_REDIRECT_URL**: Where to redirect to after the login is successfully.
- **LOGOUT_REDIRECT_URL**: Where to redirect to after the logout is successfully.

3
doc/VRW.rst

@ -41,8 +41,7 @@ Serializers @@ -41,8 +41,7 @@ Serializers
-------
Signals
-------
.. automodule:: apps.virtual_machine.providers.vrw.signals
:members:
:ref:`VRW Signals<vrw_signals>`
--------
Glossary

2
doc/authentication.rst

@ -5,7 +5,7 @@ Authentication @@ -5,7 +5,7 @@ Authentication
---
Web
---
Normal web login
The login in to the web based gui is done using :ref:`SURFconext<SURFconext manual>`. This will provide a Single Sing-on with your RUG credentials. So there is no separate account for using Research Workspaces.
---
API

23
doc/conf.py

@ -10,6 +10,7 @@ @@ -10,6 +10,7 @@
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import django
import os
import sys
sys.path.insert(0, os.path.abspath('../tusd/hooks'))
@ -17,13 +18,11 @@ sys.path.insert(0, os.path.abspath('../tusd/hooks')) @@ -17,13 +18,11 @@ sys.path.insert(0, os.path.abspath('../tusd/hooks'))
# Django autodoc
sys.path.insert(0, os.path.abspath('../VRE'))
os.environ['DJANGO_SETTINGS_MODULE'] = 'VRE.settings'
import django
django.setup()
# -- Project information -----------------------------------------------------
project = 'Virtual Research Environment'
copyright = '2020-2021, Joshua Rubingh, Elwin Buisman'
project = 'Research Workspaces'
copyright = '2020-2022, Joshua Rubingh, Elwin Buisman'
author = 'Joshua Rubingh, Elwin Buisman'
# The master toctree document.
@ -49,7 +48,7 @@ templates_path = ['_templates'] @@ -49,7 +48,7 @@ templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store','build/*']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'build/*']
# -- Options for HTML output -------------------------------------------------
@ -65,7 +64,7 @@ html_static_path = ['_static'] @@ -65,7 +64,7 @@ html_static_path = ['_static']
html_theme_options = {
'logo': 'RUG_Logo.jpg',
'logo_name' : True
'logo_name': True
}
# -- Options for LaTeX output ---------------------------------------------
@ -75,13 +74,13 @@ latex_elements = { @@ -75,13 +74,13 @@ latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
'papersize': 'a4paper',
'releasename':" ",
'releasename': " ",
# Sonny, Lenny, Glenn, Conny, Rejne, Bjarne and Bjornstrup
# 'fncychap': '\\usepackage[Lenny]{fncychap}',
'fncychap': '\\usepackage{fncychap}',
'fontpkg': '\\usepackage{amsmath,amsfonts,amssymb,amsthm}',
'figure_align':'htbp',
'figure_align': 'htbp',
# The font size ('10pt', '11pt' or '12pt').
#
'pointsize': '10pt',
@ -185,7 +184,7 @@ latex_elements = { @@ -185,7 +184,7 @@ latex_elements = {
\centering
\vspace*{40mm} %%% * is used to give space from top
\textbf{\Huge {RUG Virtual Research Environment (VRE)} }
\textbf{\Huge {RUG Research Workspaces} }
\vspace{0mm}
\begin{figure}[!h]
@ -221,14 +220,14 @@ latex_elements = { @@ -221,14 +220,14 @@ latex_elements = {
#
# 'figure_align': 'htbp',
'sphinxsetup': \
'hmargin={0.7in,0.7in}, vmargin={1in,1in}, \
'hmargin={0.7in,0.7in}, vmargin={1in,1in}, \
verbatimwithframe=true, \
TitleColor={rgb}{0,0,0}, \
HeaderFamily=\\rmfamily\\bfseries, \
InnerLinkColor={rgb}{0,0,1}, \
OuterLinkColor={rgb}{0,0,1}',
'tableofcontents':' ',
'tableofcontents': ' ',
}
latex_logo = '_static/RUG_Logo.jpg'
@ -239,4 +238,4 @@ latex_logo = '_static/RUG_Logo.jpg' @@ -239,4 +238,4 @@ latex_logo = '_static/RUG_Logo.jpg'
latex_documents = [
(master_doc, 'documentation.tex', project,
author, 'report')
]
]

43
doc/development.rst

@ -5,13 +5,12 @@ Development @@ -5,13 +5,12 @@ Development
In order to develop on this software, you can setup a development environment using these steps.
In order to install this Virtual Research Environment project, we use the following packages / software.
In order to install this Research Workspaces project, we use the following packages / software.
* Redis
* Django
* NGINX
* TUSD (The Upload Server Daemon)
* Demo portal
First we need to checkout the code.
@ -50,6 +49,7 @@ This will give us a virtual python environment on the location *venv* in the roo @@ -50,6 +49,7 @@ This will give us a virtual python environment on the location *venv* in the roo
source venv/bin/activate
pip install -r VRE/requirements.txt
pip install -r VRE/requirements-dev.txt
REST API
@ -68,8 +68,9 @@ Then we can setup and start the REST API server with the following commands. @@ -68,8 +68,9 @@ Then we can setup and start the REST API server with the following commands.
source venv/bin/activate
./VRE/manage.py migrate
./VRE/manage.py loaddata virtual_machine_initial_data
./VRE/manage.py loaddata university_initial_data
./VRE/manage.py loaddata virtual_machine_initial_data
./VRE/manage.py loaddata vre_apps_initial_data
./VRE/manage.py createsuperuser
And start with:
@ -200,39 +201,3 @@ An example of a hook as used in this project is the *pre-create.py* script. @@ -200,39 +201,3 @@ An example of a hook as used in this project is the *pre-create.py* script.
.. literalinclude:: ../../Upload_Server/hooks/pre-create.py
This hook uses the same data payload as when TUS would use the Webhook system. So using 'Normal' hooks or using Webhooks with REST API Server should both work out of the box.
-----------
Demo portal
-----------
In order to test the REST API and be able to give a demo, there is a demo portal that can be used.
Out of the box the Demo portal only needs a few required settings to work. These settings needs to be placed in a .env file located in the demo_portal/demo_portal folder. There should be an .env.example file which you can use as a template.
The minimal settings that needs to be set are:
* **SECRET_KEY**: A uniquely secret key. Used for cookie/session encryption
* **DEBUG**: Enable debug
* **DROPOFF_API_USER_KEY**: The key for the token that is created on the REST API server for communication with the REST API server.
* **DROPOFF_API_USER_SECRET**: The secret value that belongs to the token *DROPOFF_API_USER_KEY*.
Then we can setup and start the demo portal with the following commands.
.. code-block:: bash
source venv/bin/activate
./VRE/manage.py migrate
And start with:
.. code-block:: bash
source venv/bin/activate
./VRE/manage.py runserver 0.0.0.0:8000
Now you can access your demo portal at http://localhost:8000. In order to login as a researcher, you need to create a new user on the REST API server admin. Because the portal is using the REST API server for logging users in.
There are more settings available to setup. These can be added the to .env file of the demo portal.
.. literalinclude:: ../../Demo_Portal_1/demo_portal/demo_portal/.env.example
:language: bash

BIN
doc/documentation.pdf

Binary file not shown.

18
doc/index.rst

@ -1,31 +1,33 @@ @@ -1,31 +1,33 @@
.. Virtual Research Environment documentation master file, created by
.. Research Workspaces documentation master file, created by
sphinx-quickstart on Fri Feb 21 13:13:01 2020.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
==========================================
Virtual Research Environment documentation
Research Workspaces documentation
==========================================
Here you can read more information about the RUG Virtual Research Environment. This document will contain information about installing the platform. As information about the used webhooks and security.
Here you can read more information about the RUG Research Workspaces. This document will contain information about installing and configuring the platform. As information about the used webhooks and security.
The platform can be run inside a docker setup or just local for development.
The software can run on the RUG Kubernetes cluster as local in a Docker setup.
.. toctree::
:caption: Table of Contents
:maxdepth: 2
install
storage
models
SURFconext
API
authentication
signals
tus
VRW
OpenStack
development
storage
tus
API
models
------------------
Indices and tables

9
doc/install.rst

@ -2,7 +2,7 @@ @@ -2,7 +2,7 @@
Installation
============
The Virtual Research Environment will be installed with a docker compose setup. This way the hole platform is up and running within minutes.
The Research Workspaces can be installed with a docker compose setup. This way the hole platform is up and running within minutes.
First we need to checkout the code.
@ -43,17 +43,10 @@ During installation the following docker containers will be created: @@ -43,17 +43,10 @@ During installation the following docker containers will be created:
* NGINX TUSD frontend server
* NGINX API frontend server
And there will be two extra docker containers running a demo site to communicate with the REST API
* Django demo portal server
* NGINX demo frontend server
Only the NGINX containers have an connection with the outside world. By default you have:
* REST API and admin on http://localhost:1337/api/redoc/ and http://localhost:1337/api/admin/
* TUSD Upload server on http://localhost:1080/files
* Django demo portal on http://localhost:8080
Settings
========

2
doc/requirements.txt

@ -1,3 +1,3 @@ @@ -1,3 +1,3 @@
Sphinx==4.2.0
Sphinx==4.3.0
sphinx-markdown-builder==0.5.4
sphinxcontrib-openapi==0.7.0

15
doc/signals.rst

@ -2,16 +2,18 @@ @@ -2,16 +2,18 @@
Signals
=======
There are multiple `Django signals <https://docs.djangoproject.com/en/3.2/topics/signals/>`_ in use. With these signals we add extra data to the database. Here you can find a list of signals that is being
---
API
---
.. automodule:: apps.api.signals
:members:
----------
Invitation
----------
.. automodule:: apps.invitation.signals
------------
Applications
------------
.. automodule:: apps.vre_apps.signals
:members:
----------
@ -20,12 +22,17 @@ Researcher @@ -20,12 +22,17 @@ Researcher
.. automodule:: apps.researcher.signals
:members:
.. _vrw_signals:
----------------------------
Virtual machine provider VRW
----------------------------
.. automodule:: apps.virtual_machine.providers.vrw.signals
:members:
.. _hpc_signals:
----------------------------------
Virtual machine provider OpenStack
----------------------------------

20
doc/storage.rst

@ -43,10 +43,10 @@ Then press the 'Generate token' button and a new token will be generated with th @@ -43,10 +43,10 @@ Then press the 'Generate token' button and a new token will be generated with th
6. Here is your newly created Application with the latest activity. This will show you the last time this Application is used.
7. Here you can delete the Application. This will revoke the token, and will block new API calls and uploads.
VRE Storage values
==================
Research Workspaces Storage values
==================================
Now we have created an Gitea API Application we can use this in the VRE Dropoff Storage settings.
Now we have created an Gitea API Application we can use this in the Research Workspaces Dropoff Storage settings.
- **location**: https://\[gitea.host.com\]/api/v1#\[Your_Repository_Name\] (It is important to add '/api/v1' to the url)
- **username**: Your own username
@ -103,10 +103,10 @@ Login into the Github with a webrowser and follow the steps below to make an API @@ -103,10 +103,10 @@ Login into the Github with a webrowser and follow the steps below to make an API
9. Here is your newly created Application with the latest activity. This will show you the last time this Application is used.
10. Here you can delete the Application. This will revoke the token, and will block new API calls and uploads.
VRE Storage values
==================
Research Workspaces Storage values
==================================
Now we have created an Github.com API Application we can use this in the VRE Dropoff Storage settings.
Now we have created an Github.com API Application we can use this in the Research Workspaces Dropoff Storage settings.
- **location**: https://api.github.com#\[Your_Repository_Name\]
- **username**: Your own username
@ -147,7 +147,7 @@ Login into the Owncloud/Nexcloud/Unishare server with a webrowser and go to your @@ -147,7 +147,7 @@ Login into the Owncloud/Nexcloud/Unishare server with a webrowser and go to your
7. Here is your newly created Application with the latest activity. This will show you the last time this Application is used.
8. Here you can delete the Application. This will revoke the token, and will block new API calls and uploads.
VRE Storage values
Research Workspaces Storage values
------------------
.. figure:: _images/Owncloud_WebDAV_Location.png
@ -155,7 +155,7 @@ VRE Storage values @@ -155,7 +155,7 @@ VRE Storage values
:align: right
:alt: Owncloud/Nexcloud/Unishare WebDAV url
Now we have created an Owncloud/Nexcloud/Unishare API Application we can use this in the VRE Dropoff Storage settings. For Owncloud/Nexcloud/Unishare there are some extra steps to get the right WebDAV settings.
Now we have created an Owncloud/Nexcloud/Unishare API Application we can use this in the Research Workspaces Dropoff Storage settings. For Owncloud/Nexcloud/Unishare there are some extra steps to get the right WebDAV settings.
1. Go to the Files Application
2. Click on the Settings icon
@ -179,8 +179,8 @@ Keep in mind, that when you change the iRODS credentials (due to missing passwor @@ -179,8 +179,8 @@ Keep in mind, that when you change the iRODS credentials (due to missing passwor
As iRODS works with federation, you need to know your iRODS zone. This will be added to the address location when entering iRODS credentials.
VRE Storage values
==================
Research Workspaces Storage values
==================================
- **location**: hostname or ip of the iRODS server with '#\[iRODS ZONE NAME\]' appended
- **username**: Your own username

1598
doc/swagger.yaml

File diff suppressed because it is too large Load Diff

2
doc/tus.rst

@ -6,7 +6,7 @@ In order to process the uploads so that we know to which study an upload belongs @@ -6,7 +6,7 @@ In order to process the uploads so that we know to which study an upload belongs
More information about TUSD and hooks can be found here: https://github.com/tus/tusd/blob/master/docs/hooks.md
In the VRE setup, we have chosen for **file-based hooks** (https://github.com/tus/tusd/blob/master/docs/hooks.md#file-hooks) because then we can have more fleixible power in handlings the uploads. And with the file-based hooks we can still make HTTP requests in the hook code.
In the Research Workspaces setup, we have chosen for **file-based hooks** (https://github.com/tus/tusd/blob/master/docs/hooks.md#file-hooks) because then we can have more fleixible power in handlings the uploads. And with the file-based hooks we can still make HTTP requests in the hook code.
A dependency to the web hooks to work is the use of NGINX with LUA. In order to know to which study an upload belongs we, NGINX will add extra meta headers to the TUSD server so that the extra study information is available in for the hook scripts.

Loading…
Cancel
Save