Data Operation - Transfer

Undraw Data Processing

About

transfer is a data operation that executes all this data resources operations:

In general, if you need to move or process data, you will transfer it.

The data transfer action permits you to

Usage

pipeline:
# ...
  - name: "Store"
    operation: transfer
    args:
      target-uri: 'build/${logicalName}.csv@cd'
      transfer-operation: insert
      target-operation: replace

The transfer operation is performed by the tabli data transfer command

Arguments

Name Default Description
target-uri - The target uri (Mandatory)
target-operation - A resource operation that will happens on the target before the transfer
source-operation - A resource operation that will happens on the source after the transfer
Flow Property
step-granularity resource (default) or record The operation will be done for the whole resource or record by record
output target The output
- targets, the target resources
- sources, the sources resources are passed
- results, the results of the transfer step is passed
Cross Transfer Properties
source-fetch-size 10000 The number of record from the source for one fetch (ie network message size from the source system)
buffer-size 2 x target-worker x fetch-size The maximum number of record from stored in transit between the source and the target
target-batch-size 10000 The number of record send to target system at once (ie network message size to the target system)
target-commit-frequency <math>2^{31}</math> (Infinite) The number of batch sends that will trigger a commit to the target system
target-worker 1 The number of thread against the target system that will send batches
with-bind-variables true If the target system is a sql database, SQL bind variables will be used
metrics-data-uri A data uri where the transfer metrics will be saved
Transfer Properties
transfer-operation copy for a file system
insert otherwise
A transfer operation (copy, insert, upsert, update, …)
transfer-mapping-method name how the source and target columns are mapped (by name, position or map)
transfer-column-mapping if the mapping method is map, you can define a map of source target column name

Transfer Operation

The transfer operation supports the following values for the transfer-operation argument.

Name Alias Require Same
Source/Target
Structure and Data
Local Equivalent Metadata Operation
copy Yes -
move Yes rename
insert append No -
upsert No -
update No -
delete No -

The default transfer operation is system dependent:

Resource Operation

The following values may be used for the target-operation and source-operation arguments.

Value Description
truncate Truncate the resource
drop Drop the resource
replace Replace the resource
keep Does not modify the actual resource (ie does not replace the target if it exists)

Result

If you set the output arguments to result, a result data path will be returned (in place of targets) and it will contain the following columns:

Metrics

If the argument metrics-data-uri is given, a metrics data resource will be created with the following columns:

  • run - the run id
  • timestamp - the metrics timestamp
  • metric - the name of the metrics
  • value - the value of the metrics
  • worker - the name of the worker (thread)

The following metrics will be recorded:

  • BufferSize: the size of the memory buffer between the source and the target
  • BufferMaxSize: the maximum size of the buffer
  • BufferRatio: the ratio from Buffer Size against Buffer MaxSize
  • Commits: the number of commits
  • Records: the number of records

Cross Transfer

A transfer can happen:

Tabulify Optimization: If two connections share the same credentials and the same system URL, the transfer will be optimized and considered local. This way, the transfer is applied only on the system metadata and it happens faster because the data is not moved.




Related HowTo
Undraw Data Processing
Learning Tabulify - Step 6 - How to transfer Data Resources

In Tabulify, when you want to manipulate data, it's called a transfer. You want to move data, you transfer You want to copy data, you transfer You want to insert data, you transfer You want to...
Undraw Data Processing
Learning Tabulify - Step 8 - How to modify the content of a table

Tabulify learning guidetransfer step The transfer page should have made you familiar with the transfer concept where we have introduced the copy operation. The copy and move operation does not modify...

Task Runner