/
Data Format

Data Format

File Format: Allows you to configure the file format and source schema. Select the required file format from the dropdown.

Once the File Format is selected, all the related configurations will appear.

  • Parquet

  • Delimited

Header Row: Allows to specify that the file's first row data must be considered a header row. Data available in the header row separated by delimiter are column names.

Delimiter: Delimiter is a character separator that separates the values stored in rows. Allows to define the predefined and custom delimiter for the data. Select the required delimiter from the dropdown.

  • Comma: Allows to select a Comma (,) delimiter for the data.

  • Tab: Allows to select a Tab ( ) delimiter for the data.

  • Custom: Allows to define the custom delimiter. For Example: Semicolon, Colon, Pipe, Forward Slash.

Selection of Custom enables the Custom Delimiter.

Custom Delimiter: Allows to define a new custom delimiter for the data.

Selection of Custom from the Delimiter dropdown is mandatory to define the custom delimiter.

Escape Character: An Escape character and a Text Qualifier create a sequence recognized and ignored during parsing. Its purpose is to allow the use of escape sequences in the data that would otherwise be seen as delimiter occurrences.

Text Qualifier: Text qualifier is used in the event that delimiters are contained within the row cell. If the cell contains a delimiter and a text qualifier is not used, then the data that occurs after the delimiter will spill into the next column.

  • JSON

Supported Structure

  1. Characteristics of supported JSON is one record per row.

{"first_name": "Bradley", "priority": 1683, "subscribe": true, "income": 955289.05, "address": {"City": "Nicolestad", "State": "Massachusetts"}, "countries_visited": ["Turks and Caicos Islands", "Spain", "New Caledonia"], "date_of_birth": "1988-02-19 00:00:00", "null_key": null} {"first_name": "Jennifer", "priority": 2756, "subscribe": true, "income": 15248.17, "address": {"City": "Burnsborough", "State": "Idaho"}, "countries_visited": ["Mauritania", "Turkey", "Guinea"], "date_of_birth": "1994-08-31 00:00:00", "null_key": null} {"first_name": "Tyler", "priority": 2628, "subscribe": false, "income": 248173.49, "address": {"City": "Ericahaven", "State": "California"}, "countries_visited": ["Sudan", "Afghanistan", "Chad"], "date_of_birth": "1978-06-30 00:00:00", "null_key": null} {"first_name": "Lisa", "priority": 1518, "subscribe": false, "income": 338300.85, "address": {"City": "Tracyton", "State": "Oklahoma"}, "countries_visited": ["Honduras", "Samoa", "Congo"], "date_of_birth": "1991-08-06 00:00:00", "null_key": null} {"first_name": "William", "priority": 1714, "subscribe": false, "income": 950738.18, "address": {"City": "Lake Tina", "State": "Nevada"}, "countries_visited": ["Seychelles", "Vietnam", "Lebanon"], "date_of_birth": "1981-02-09 00:00:00", "null_key": null}

Not Supported Structure

  1. JSON files with formatted records (that span over multiple rows).

    { "first_name":"Rachel", "priority":2619, "subscribe":false, "income":435324.12, "address":{ "City":"Smithstad", "State":"Michigan" }, "countries_visited":[ "Belize", "Eritrea", "Egypt" ], "date_of_birth":"1976-06-19 00:00:00", "null_key":null }
  2. JSON contains data in arrays.

    [{"id":1,"name":"John Doe","email":"john.doe@example.com"},{"id":2,"name":"Jane Doe","email":"jane.doe@example.com"},{"id":3,"name":"Mike Smith","email":"mike.smith@example.com"}] [{"id":7,"name":"Peter Green","email":"peter.green@example.com"},{"id":8,"name":"Susan Black","email":"susan.black@example.com"},{"id":9,"name":"Michael White","email":"michael.white@example.com"},{"id":10,"name":"Jessica Green","email":"jessica.green@example.com"}]

Upload: Click the Upload button to upload the data file. Select the required file needed to upload and click Upload.

View Batch Schema: Allows you to view and modify the selection of source schema.


Source Data Format

All the available parent column names in the uploaded data file will appear here.

Filter: It allows you to search and filter the specific columns here. Scroll down manually to see the entire list.

Column list: As required, you can select or deselect the required columns of the left-hand side drawer.

JSON: JSON schema will update dynamically when columns/keys are selected or deselected from the left-hand side drawer. It displays the standardized schema structure that needs to be mapped in the project schema.

  • After changes, copy the created entire schema and paste it into the defined project schema at the right-hand side drawer.

  • Click Save Changes to save the configuration.


Related Topics

Batch Listener

Parsing Inputs

Path Pattern (Regex)

Interval

Output

Related content

Batch Listener
Batch Listener
More like this
Path Pattern (Regex)
Path Pattern (Regex)
More like this
Parsing Inputs
Parsing Inputs
More like this
Xponent Journey Hub June 2023 (23.1.1) Release Notes
Xponent Journey Hub June 2023 (23.1.1) Release Notes
Read with this
Columnar Table Node
Columnar Table Node
More like this
Xponent User Guide
Xponent User Guide
Read with this

Privacy Policy
© 2022 CSG International, Inc.