/
Default-Fetch-Worker

Default-Fetch-Worker

Nov 13, 2018

Overview

The Default-Fetch-Worker is one possbile Fetch-Worker

Actions

When the Default-Fetch-Worker is started, the following actions take place:

A) Download Files

  1. The Default-Fetch-Worker searches for storageentities with state = NEW and fetchduedate < now (or no fetchduedate)

  2. It retrieves the datasource from the according entity

  3. it creates sub-folders on the according storage, using the uuid of the storageentity, e.g. uuid = 59cc972a-9c7f-49be-822b-b2fb129d646a => created folders: ./59/cc/97/

  4. it sets the state of the storageentity to IS_FETCHING

  5. it downloads the file into the created folder

  6. it saves the filesize , checksum and the fetched-at-date in the storageentity and sets its state to FETCHED

  7. if download failed, the state of the storageentity is set to FETCHED_ERROR.

B) Copy Files

  1. The Default-Fetch-Worker searches for storageentities with state = TO_BE_COPIED and fetchduedate < now (or no fetchduedate)

  2. It retrieves the location of the file that should be copied by the customdata-field "copyfrom", which was set by the schedule-worker

  3. it creates sub-folders on the according storage, using the uuid of the storageentity, e.g. uuid = 59cc972a-9c7f-49be-822b-b2fb129d646a => created folders: ./59/cc/97/

  4. it sets the state of the storageentity to IS_COPYING. If the state of the source-storageentity is TO_BE_MOVED, this state is set to IS_MOVING.

  5. it copies the file into the created folder

  6. it saves the filesize, checksum and the fetched-at-date in the storageentity and sets its state to FETCHED. When the state of the source-storageentity is "IS_MOVING", this state is set to "MOVED"

  7. if the copy-process fails, the state of the source-storageentity is set to MOVING_ERROR and the state of the storageentity is set to FETCHED_ERROR.