transfer is a data operation that executes all this data resources operations:
In general, if you need to move or process data, you will transfer it.
The data transfer action permits you to
pipeline:
# ...
- name: "Store"
operation: transfer
args:
target-uri: 'build/${logicalName}.csv@cd'
transfer-operation: insert
target-operation: replace
The transfer operation is performed by the tabli data transfer command
| Name | Default | Description |
|---|---|---|
| target-uri | - | The target uri (Mandatory) |
| target-operation | - | A resource operation that will happens on the target before the transfer |
| source-operation | - | A resource operation that will happens on the source after the transfer |
| Flow Property | ||
| step-granularity | resource (default) or record | The operation will be done for the whole resource or record by record |
| output | target | The output - targets, the target resources - sources, the sources resources are passed - results, the results of the transfer step is passed |
| Cross Transfer Properties | ||
| source-fetch-size | 10000 | The number of record from the source for one fetch (ie network message size from the source system) |
| buffer-size | 2 x target-worker x fetch-size | The maximum number of record from stored in transit between the source and the target |
| target-batch-size | 10000 | The number of record send to target system at once (ie network message size to the target system) |
| target-commit-frequency | <math>2^{31}</math> (Infinite) | The number of batch sends that will trigger a commit to the target system |
| target-worker | 1 | The number of thread against the target system that will send batches |
| with-bind-variables | true | If the target system is a sql database, SQL bind variables will be used |
| metrics-data-uri | A data uri where the transfer metrics will be saved | |
| Transfer Properties | ||
| transfer-operation | copy for a file system insert otherwise | A transfer operation (copy, insert, upsert, update, …) |
| transfer-mapping-method | name | how the source and target columns are mapped (by name, position or map) |
| transfer-column-mapping | if the mapping method is map, you can define a map of source target column name | |
The transfer operation supports the following values for the transfer-operation argument.
| Name | Alias | Require Same Source/Target Structure and Data | Local Equivalent Metadata Operation |
|---|---|---|---|
| copy | Yes | - | |
| move | Yes | rename | |
| insert | append | No | - |
| upsert | No | - | |
| update | No | - | |
| delete | No | - |
The default transfer operation is system dependent:
The following values may be used for the target-operation and source-operation arguments.
| Value | Description |
|---|---|
| truncate | Truncate the resource |
| drop | Drop the resource |
| replace | Replace the resource |
| keep | Does not modify the actual resource (ie does not replace the target if it exists) |
If you set the output arguments to result, a result data path will be returned (in place of targets) and it will contain the following columns:
If the argument metrics-data-uri is given, a metrics data resource will be created with the following columns:
The following metrics will be recorded:
A transfer can happen:
Tabulify Optimization: If two connections share the same credentials and the same system URL, the transfer will be optimized and considered local. This way, the transfer is applied only on the system metadata and it happens faster because the data is not moved.