File Format: Allows you to configure the file format and source schema. Select the required file format from the dropdown.
Once the File Format is selected, all the related configurations will appear.
Parquet
Delimited
JSON
Upload: Click the Upload button to upload the data file. Select the required file needed to upload and click Upload.
The selected file size must be less than 2MB.
We recommend that the user must avoid uploading the Gzip file unless it has been decrypted.
The View Source Schema button gets enabled after uploading the selected file.
View Batch Schema: Allows you to view and modify the selection of source schema.
Source Data Format
For Delimited files:
After opening the Source Data Format, the Column Names and auto-detected Datatypes in the sample file will be displayed.
For Example:
<Column Name> (<DATATYPE IN UPPERCASE>)
Datatypes detected in the sample file will be used to process data files. Please verify the auto detected Datatypes and make sure the sample file contains the correct data.
For Example:
If you want to process the Datatype “integer” for the Postal Code. Make sure in the sample file also having the same.
If sample file having the Datatype “string” then, this mismatch can cause the failure.
Whenever required, please make the corrections in the sample file and re-upload it. This is how you can correct the auto-detected datatypes.
Filter: It allows you to search and filter the specific columns here. Scroll down manually to see the entire list.
Column list: As required, you can select or deselect the required columns of the left-hand side drawer.
By default, all the columns available in the source data format are selected.
Selection of at least one column is mandatory to save the schema in the batch listener configuration.
In the case of nested data structures, only the parent column is displayed on the left side.
JSON: JSON schema will update dynamically when columns/keys are selected or deselected from the left-hand side drawer. It displays the standardized schema structure that needs to be mapped in the project schema.
Users can only view the i schema structure.
After changes, copy the created entire schema and paste it into the defined project schema at the right-hand side drawer.
Click Save Changes to save the configuration.