VRE Backend API and Scheduler
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 

2.9 KiB

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

TUS Hooks

In order to process the uploads so that we know to which study an upload belongs to, we use web hooks.

More information about TUSD and hooks can be found here: https://github.com/tus/tusd/blob/master/docs/hooks.md

In the Research Workspaces setup, we have chosen for file-based hooks (https://github.com/tus/tusd/blob/master/docs/hooks.md#file-hooks) because then we can have more fleixible power in handlings the uploads. And with the file-based hooks we can still make HTTP requests in the hook code.

A dependency to the web hooks to work is the use of NGINX with LUA. In order to know to which study an upload belongs we, NGINX will add extra meta headers to the TUSD server so that the extra study information is available in for the hook scripts.

Settings

In order to communicate with the REST API server, we need some settings to be entered. Create a .env file in the TUSD main folder. And fill at least the tree values:
  • WEBHOOK_URL
  • DROPOFF_API_HAWK_KEY
  • DROPOFF_API_HAWK_SECRET

System Message: ERROR/3 (<stdin>, line 20)

Unknown directive type "literalinclude".

.. literalinclude:: ../../Upload_Server/.env.example
    :language: bash

NGINX / LUA

Every file that is uploaded goes through NGINX with a virtual url. During the upload we add a UUID of the study to the upload url. When the client is uploading the data, NGINX will read out the UUID from the url, and add that to the meta data for the TUSD service. And NGINX will strip the url for the TUSD daemon in order to accept the upload. So basically, NGINX is needed in order to be able to add study information to the TUSD uploads.

NGINX config:

System Message: ERROR/3 (<stdin>, line 32)

Unknown directive type "literalinclude".

.. literalinclude:: ../../Upload_Server/nginx/tus.vhost.conf
    :lines: 54-91
    :language: bash

JavaScript code:

System Message: ERROR/3 (<stdin>, line 38)

Unknown directive type "literalinclude".

.. literalinclude:: ../../Upload_Server/nginx/njs/dropoff_tus.js
    :language: javascript


Python hooks

All the communication between the TUSD service and the REST API is secured with HAWK (https://github.com/mozilla/hawk). Therefore the hooks need to know a key and a secret of a user on the REST API that is allowed to make the webhook calls. Make sure you have created such a user.

pre-create

When a new upload is started, the pre-create hook will check if the study ID is valid and that the upload can start. This will also anounce the upload to the REST API server and therefore can be found in the overview of data drops.

System Message: ERROR/3 (<stdin>, line 53)

Unknown directive type "literalinclude".

.. literalinclude:: ../../Upload_Server/hooks/pre-create.py


post-finish

When the upload is done, the file needs to be processed and moved to the final destination. This is done with the post-finish hook.

This hook is a bit more complex and does multuple things.
  • Check if there is a study ID
  • Get the storage information based on the study ID
  • Then it encrypts the file if requested
  • Upload the file to the destination strorages
  • Report back if upload is processed correctly

System Message: ERROR/3 (<stdin>, line 69)

Unknown directive type "literalinclude".

.. literalinclude:: ../../Upload_Server/hooks/post-finish.py
    :linenos:

</html>