Data Format

File Format: Allows you to configure the file format and source schema. Select the required file format from the dropdown.

Once the File Format is selected, all the related configurations will appear.

  • Parquet

  • Delimited

Header Row: Allows to specify that the file's first row data must be considered a header row. Data available in the header row separated by delimiter are column names.

Delimiter: Delimiter is a character separator that separates the values stored in rows. Allows to define the predefined and custom delimiter for the data. Select the required delimiter from the dropdown.

  • Comma: Allows to select a Comma (,) delimiter for the data.

  • Tab: Allows to select a Tab ( ) delimiter for the data.

  • Custom: Allows to define the custom delimiter. For Example: Semicolon, Colon, Pipe, Forward Slash.

Selection of Custom enables the Custom Delimiter.

Custom Delimiter: Allows to define a new custom delimiter for the data.

Selection of Custom from the Delimiter dropdown is mandatory to define the custom delimiter.

Escape Character: An Escape character and a Text Qualifier create a sequence recognized and ignored during parsing. Its purpose is to allow the use of escape sequences in the data that would otherwise be seen as delimiter occurrences.

Text Qualifier: Text qualifier is used in the event that delimiters are contained within the row cell. If the cell contains a delimiter and a text qualifier is not used, then the data that occurs after the delimiter will spill into the next column.

  • JSON

Supported Structure

  1. Characteristics of supported JSON is one record per row.

{"first_name": "Bradley", "priority": 1683, "subscribe": true, "income": 955289.05, "address": {"City": "Nicolestad", "State": "Massachusetts"}, "countries_visited": ["Turks and Caicos Islands", "Spain", "New Caledonia"], "date_of_birth": "1988-02-19 00:00:00", "null_key": null} {"first_name": "Jennifer", "priority": 2756, "subscribe": true, "income": 15248.17, "address": {"City": "Burnsborough", "State": "Idaho"}, "countries_visited": ["Mauritania", "Turkey", "Guinea"], "date_of_birth": "1994-08-31 00:00:00", "null_key": null} {"first_name": "Tyler", "priority": 2628, "subscribe": false, "income": 248173.49, "address": {"City": "Ericahaven", "State": "California"}, "countries_visited": ["Sudan", "Afghanistan", "Chad"], "date_of_birth": "1978-06-30 00:00:00", "null_key": null} {"first_name": "Lisa", "priority": 1518, "subscribe": false, "income": 338300.85, "address": {"City": "Tracyton", "State": "Oklahoma"}, "countries_visited": ["Honduras", "Samoa", "Congo"], "date_of_birth": "1991-08-06 00:00:00", "null_key": null} {"first_name": "William", "priority": 1714, "subscribe": false, "income": 950738.18, "address": {"City": "Lake Tina", "State": "Nevada"}, "countries_visited": ["Seychelles", "Vietnam", "Lebanon"], "date_of_birth": "1981-02-09 00:00:00", "null_key": null}

Not Supported Structure

  1. JSON files with formatted records (that span over multiple rows).

    { "first_name":"Rachel", "priority":2619, "subscribe":false, "income":435324.12, "address":{ "City":"Smithstad", "State":"Michigan" }, "countries_visited":[ "Belize", "Eritrea", "Egypt" ], "date_of_birth":"1976-06-19 00:00:00", "null_key":null }
  2. JSON contains data in arrays.

    [{"id":1,"name":"John Doe","email":"john.doe@example.com"},{"id":2,"name":"Jane Doe","email":"jane.doe@example.com"},{"id":3,"name":"Mike Smith","email":"mike.smith@example.com"}] [{"id":7,"name":"Peter Green","email":"peter.green@example.com"},{"id":8,"name":"Susan Black","email":"susan.black@example.com"},{"id":9,"name":"Michael White","email":"michael.white@example.com"},{"id":10,"name":"Jessica Green","email":"jessica.green@example.com"}]

Upload: Click the Upload button to upload the data file. Select the required file needed to upload and click Upload.

View Batch Schema: Allows you to view and modify the selection of source schema.


Source Data Format

All the available parent column names in the uploaded data file will appear here.

Filter: It allows you to search and filter the specific columns here. Scroll down manually to see the entire list.

Column list: As required, you can select or deselect the required columns of the left-hand side drawer.

JSON: JSON schema will update dynamically when columns/keys are selected or deselected from the left-hand side drawer. It displays the standardized schema structure that needs to be mapped in the project schema.

  • After changes, copy the created entire schema and paste it into the defined project schema at the right-hand side drawer.

  • Click Save Changes to save the configuration.


https://kitewheel.atlassian.net/wiki/spaces/K2/pages/2970484737

https://kitewheel.atlassian.net/wiki/spaces/K2/pages/2969436161

https://kitewheel.atlassian.net/wiki/spaces/K2/pages/2919038977

Privacy Policy
© 2022 CSG International, Inc.