Data Operation - Transfer

Data Operation - Transfer

About

transfer is a data operation that executes all this data resources operations:

In general, if you need to move or process data, you will transfer it.

The data transfer action permits you to

Usage

The transfer operation is performed by the tabli data transfer command

Arguments

Name Default Description
target-uri - The target uri (Mandatory)
target-data-definition - The target data definition (Optional: takes by default, the data definition of the source)
transfer-operation copy for a file system
insert otherwise
A transfer operation (copy, insert, upsert, update, …)
target-operation - A resource operation that will happens on the target before the transfer
source-operation - A resource operation that will happens on the source after the transfer
Flow Property
pass-through Target By default, a flow will pass the target to the next step. By setting the pass-through value to result, a result data path by transfer will be passed instead
Cross Transfer Properties
source-fetch-size 10000 The number of record from the source for one fetch (ie network message size from the source system)
buffer-size 2 x target-worker x fetch-size The maximum number of record from stored in transit between the source and the target
target-batch-size 10000 The number of record send to target system at once (ie network message size to the target system)
target-commit-frequency <math>2^{31}</math> (Infinite) The number of batch sends that will trigger a commit to the target system
target-worker 1 The number of thread against the target system that will send batches
with-bind-variables true If the target system is a sql database, SQL bind variables will be used
metrics-data-uri A data uri where the transfer metrics will be saved

Transfer Operation

The transfer operation supports the following values for the transfer-operation argument.

Name Alias Require Same
Source/Target
Structure and Data
Local Equivalent Metadata Operation
copy Yes -
move Yes rename
insert append No -
upsert No -
update No -
delete No -

The default transfer operation is system dependent:

Resource Operation

The following values may be used for the target-operation and source-operation arguments.

Value Description
truncate Truncate the resource
drop Drop the resource
replace Replace the resource
keep Does not modify the actual resource (ie does not replace the target if it exists)

Pass-through

If you set the pass-through arguments to result, a result data path will be returned (in place of targets) and it will contain the following columns:

Metrics

If the argument metrics-data-uri is given, a metrics data resource will be created with the following columns:

  • run - the run id
  • timestamp - the metrics timestamp
  • metric - the name of the metrics
  • value - the value of the metrics
  • worker - the name of the worker (thread)

The following metrics will be recorded:

  • BufferSize: the size of the memory buffer between the source and the target
  • BufferMaxSize: the maximum size of the buffer
  • BufferRatio: the ratio from Buffer Size against Buffer MaxSize
  • Commits: the number of commits
  • Records: the number of records

Cross Transfer

A transfer can happen:

Tabulify Optimization: If two connections share the same credentials and the same system URL, the transfer will be optimized and considered local. This way, the transfer is applied only on the system metadata and it happens faster because the data is not moved.

Powered by ComboStrap