Intempt Pipelines allows you to combine different operators to send source data to the destination. This section will explain the workflow of the pipeline.
# UI elements
# Main configuration
Save button saves the pipeline, the secondary option allows you to delete the pipeline.
Right sidebar/ bottom sidebar icons toggle on/off the schema (bottom sidebar) and operator view (right sidebar).
Gear icon toggles on/off pipeline status container.
# Schema preview
With schema preview (bottom sidebar), you can view data anywhere in the pipeline.
The Source – the first pipeline item, and the destination – the last pipeline item, both hold your dataset sample.
By clicking on an operator, you preview your input and output dataset. You can switch between different views:
You can also preview only input/output or both input and output schemas.
The final state of your data can be previewed by clicking the last processor of the pipeline, before the destination.
# Operator configuration
Operator configuration (right sidebar) allows you to create custom configurations to apply specific transformations to the input data.
# Select a source
Click on "Add source" box and select a source from the list.
In this list you will only see sources that were already created. If you need a different source, your can go to Sources (opens new window)to create a new one.
# Select a destination
Click on "Add destination" box and select a destination from the list.
In this list you will only see destinations that were already created. If you need a different destination your can go to Destinations (opens new window)to create a new one.
# Select an operator
Now the pipeline can successfully move the data from the source to destination. However, you will need to apply specific operators to transform, clean, or enrich the data in most cases.
To select an operator, click on the plus icon. A list of the operators will appear on the right sidebar. You can find the list of all operators with their descriptions here.
A new operator box will appear in the pipeline workflow. Click on the box to configure the operator.
The same process goes to adding any other new operator to the workflow. Intempt supports and unlimited count of operators in the pipeline.
Note: make sure that the flow of the operators are properly planned. For example, in most cases, you will want to add schema mapper operator as the first one in the workflow - it affects the output schema and can omit some fields that will not be required to be later processed by other operators.
# Save the pipeline
Once all the operators are configured, you can save the pipeline and start sending the data to the destination. Click on "Save" and make sure that the toggle "Enable" is set to "On".
# Monitor the Status
You can check the pipeline status by looking at the top container to see if the pipeline is active and how many data items are sent to the destination.