The Batch Listener allows data ingestion from batch sources into a graph. Also, the Batch Listener captures input required to process data from batch sources at a defined frequency.
Creating a Batch Listener
| Info |
|---|
Creating a new Batch Connection is mandatory to create a Batch Listener Adaptor. |
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
This Edit button is available for all types of listeners. |
...
Batch Listener Options
| Tip |
|---|
Once the Batch Listener Type is selected, all the configurations related to Batch Listener will appear. |
Data Format
In data processing, there are different types of file formats to store your data sets. The most popular data formats are Parquet, Delimited, and JSON.
...
JSON (JavaScript Object Notation) is an open data format and is widely used by APIs (Application Programming Interface — How systems communicate with each other) and several databases (like MongoDB)
...
Parsing Inputs
To parse the inputs from the defined files creating the detailed configuration (Compression, Encryption, and File path) is mandatory.
...
A Directory refers to the location of the stored files and Path Pattern (Regex) filters the exact defined file.
...
Path Pattern (Regex)
Regular expressions are mainly helpful for defining the file name available in the directory, which contains a series of characters that define a pattern of text to be matched to make a filter more specific.
By using regular expressions single or multiple files can be filtered out.
...
Interval
The Interval configuration defines the batch processing interval, and the same will be repeated at the number of times mentioned in the frequency.
...
Output
The Output configuration allows you to create, modify and define the source schema. The source schema defines the structure and properties of the data source which can be read from and written to from any node in the graph
...