# Operators

# Schema mapper

Maps an incoming data item to a destination table.

You can use the operator to do the following:

  • Map an incoming event type to a destination table. All the data for that event type will be stored in the mapped destination table.

  • Omit unwanted fields from storing in the destination table. This is especially useful to hide sensitive information like your user’s personal details and financial information. You can also use it to omit large text fields like descriptions, summaries, and comments.

# Type converter

The Type converter operator converts incoming records from one format type to another format type.

Field path - Select the input field to be converted.

Output type - Select the desired destination type for the selected field.

Format - Enter the format used in the input fields, this information is used to help interpret the column.

Example conversion:

  • int (Source type) - 12345 (Source value) - N/A (format) - Long (destination type) - 12345L (Destination value)

  • long (Source type) - 12345L (Source value) - N/A (format) - Integer (destination type) - 12345 (Destination value)

  • long (Source type) - 1234567890123456789L (Source value) - N/A (format) - Double (destination type) - 1234567890123456770.0d (Destination value)

  • string (Source type) - "1234.5" (Source value) - # (format) - Integer (destination type) - 1234 (Destination value)

  • string (Source type) - "20201128" (Source value) - yyyyMMdd (format) - date (destination type) - 2020-11-28 (Destination value)

# Flatten

The flatten values operator allows you to output multiple variations of the same field values in simple value while discarding empty strings and removing white spaces.

Fields to process - enter the input field which will be flattened.

Is list - Enable this option to convert the input non-normalized column to a list.

Item separator - Select in the list the separator used to delimit the input complex column values and convert them to a single value (list). This option only appears if the Is list option is not enabled.

Discard the trailing empty strings - Enable this option to discard the trailing empty strings.

Trim resulting values - Enable this option to trim leading and trailing whitespaces from the resulting data.

# Edit fields

The edit fields operator edits your input fields by allowing you to rename and move them in the schema according to your needs.

Field selector properties

Input - Select or enter the path to your input field. This processor uses the avpath syntax.

Output - Enter a name for the generated output field that is based on your input field.

The expected format is the following:

  • must begin with [A-Za-z_] characters

  • can only contain [A-Za-z0-9_] characters

  • Example: ASDasd123_4564

# Filter

The Filter operator uses different transformers to filter on the values of your fields. It filters fields based on one or multiple conditions, splits them, and passes the separated data to the next step of the pipeline.

Input - Select the field on which you want to apply a filter in the list.

Select a function to apply (optional)

  • NONE: Does not apply any functions while filtering.
  • TO ABSOLUTE VALUE: Calculates the absolute value for all the numeric values in the field.
  • TO LOWER CASE: Converts all the text in the field to lower case.
  • TO UPPER CASE: Converts all the text in the field to upper case.
  • FIRST CHARACTER TO LOWER CASE: Puts the first letter of every word in the field to lower case.
  • FIRST CHARACTER TO UPPER CASE: Puts the first letter of every word in the field to upper case.
  • LENGTH: Extracts the number of digits from a value in the field.
  • COUNT: Counts the number of values in the field.

Operator: Select the operator that will be used to filter the selected field.

Value: Type in the value of the selected field.

Select rows that match In case you have defined multiple filters, select the type of matching to apply to the rows to be filtered: ALL, ANY or NONE.

Example: You have defined the two following filter criteria: age >21 and name CONTAINS John: If you select ALL, all the 21-years old named John will be returned. If you select ANY, all the 21-years old and all the people named John will be returned. If you select NONE, all the less-than-21-years old that are not named John will be returned.

# Data hashing

The Data hashing processor allows you to alter the value of your data in order to protect it.

Function name - Select the function to apply to the field(s) in the list

Fields to process - Select the field(s) on which you want to apply a function in the list.

Create new column - Enable this option to create new fields after applying the function. If you do not enable this option, the existing field(s) will be kept and modified.

Rename new column - Give a custom name to the newly created field.

Hash data - Hashes the content of a column using the SHA-256 algorithm.

# Phone

Operator extracts specific information from phone numbers.

Function name - Select the function to apply to the field(s) in the list.

Fields to process - Select the field(s) on which you want to apply a function in the list.

Extract phone number information - Extracts additional information from phone numbers, such as phone type, country or carrier name. Each field is extracted in a new column.

# Data cleansing

The Data cleansing operator/transformer allows you to perform different operations to cleanse fields and records. It replaces the content of records or fills records with a given value.

Function name - Select the function to apply to the field(s) in the list

Fields to process - Select the field(s) on which you want to apply a function in the list.

List of Data cleansing functions

This table lists all the functions that you can apply to your data using the Data cleansing processor.

Fill cells with value - fills cells from this column with a given value

Fill empty cells with text - fills empty cells from this column with a given value

# Replicate

The Replicate operator duplicates the input flow and outputs two identical flows that can be processed in a different way according to your needs.

It is enough to select the operator and the input flow will be cloned and separated into two branches. No additional configurations required.

# Python

Properties to configure in order to process your records using Python code.

Map type

MAP: only returns one element.

The Python processor automatically emits an output record for each input. It expects the result to be in a variable called output. The simplest Python code that simply passes through every record is therefore:

output = input

FLATMAP: returns a list of elements (0 or more) as an iterator.

It is up to the python code to emit records explicitly. This is useful if the processing logic wants to emit less output records than input, typically for complex filtering (for simple/normal filtering, use the dedicated filter processor instead).

Python code Enter the Python code in this block. You can click the button to open the editor in a separate dialog box.

Several features are available in the editor to help you write your code such as:

  • autocompletion when typing (for functions,record values, etc.),

  • element, line and syntax highlight (such as matching parentheses or brackets),

  • cut, copy and paste.

  • variables and Avpath syntax.

# IP to GEO

The IP to GEO operator is a lookup processor that can return geolocation and IP intelligence information for a specified IP address.

Function name - Select the function to apply to the field(s) in the list.

Fields to process - Select the field(s) on which you want to apply a function in the list.

Extract IP information - Extracts additional information from IP, such as city, country, longitude, latitude. Each field is extracted in a new column.

# URL parser

URL parser operator parses a URL into its individual components, i.e scheme, host, port, path, query, fragment.

Fields to process - Select the field(s) on which you want to apply a URL function.