2.9 KiB
TUS Hooks
In order to process the uploads so that we know to which study an upload belongs to, we use web hooks.
More information about TUSD and hooks can be found here: https://github.com/tus/tusd/blob/master/docs/hooks.md
In the Research Workspaces setup, we have chosen for file-based hooks (https://github.com/tus/tusd/blob/master/docs/hooks.md#file-hooks) because then we can have more fleixible power in handlings the uploads. And with the file-based hooks we can still make HTTP requests in the hook code.
A dependency to the web hooks to work is the use of NGINX with LUA. In order to know to which study an upload belongs we, NGINX will add extra meta headers to the TUSD server so that the extra study information is available in for the hook scripts.
Settings
- In order to communicate with the REST API server, we need some settings to be entered. Create a .env file in the TUSD main folder. And fill at least the tree values:
- WEBHOOK_URL
- DROPOFF_API_HAWK_KEY
- DROPOFF_API_HAWK_SECRET
System Message: ERROR/3 (<stdin>, line 20)
Unknown directive type "literalinclude".
.. literalinclude:: ../../Upload_Server/.env.example
:language: bash
NGINX / LUA
Every file that is uploaded goes through NGINX with a virtual url. During the upload we add a UUID of the study to the upload url. When the client is uploading the data, NGINX will read out the UUID from the url, and add that to the meta data for the TUSD service. And NGINX will strip the url for the TUSD daemon in order to accept the upload. So basically, NGINX is needed in order to be able to add study information to the TUSD uploads.
NGINX config:
System Message: ERROR/3 (<stdin>, line 32)
Unknown directive type "literalinclude".
.. literalinclude:: ../../Upload_Server/nginx/tus.vhost.conf
:lines: 54-91
:language: bash
JavaScript code:
System Message: ERROR/3 (<stdin>, line 38)
Unknown directive type "literalinclude".
.. literalinclude:: ../../Upload_Server/nginx/njs/dropoff_tus.js
:language: javascript
Python hooks
All the communication between the TUSD service and the REST API is secured with HAWK (https://github.com/mozilla/hawk). Therefore the hooks need to know a key and a secret of a user on the REST API that is allowed to make the webhook calls. Make sure you have created such a user.
pre-create
When a new upload is started, the pre-create hook will check if the study ID is valid and that the upload can start. This will also anounce the upload to the REST API server and therefore can be found in the overview of data drops.
System Message: ERROR/3 (<stdin>, line 53)
Unknown directive type "literalinclude".
.. literalinclude:: ../../Upload_Server/hooks/pre-create.py
post-finish
When the upload is done, the file needs to be processed and moved to the final destination. This is done with the post-finish hook.
- This hook is a bit more complex and does multuple things.
- Check if there is a study ID
- Get the storage information based on the study ID
- Then it encrypts the file if requested
- Upload the file to the destination strorages
- Report back if upload is processed correctly
System Message: ERROR/3 (<stdin>, line 69)
Unknown directive type "literalinclude".
.. literalinclude:: ../../Upload_Server/hooks/post-finish.py
:linenos: