There are a number of improvements in the March 2018 Release of Kitewheel:
- Schema Navigation - we have made it easier to find elements in large schemas
- S3 File Adaptor
- XML Input
- Front End Performance
- Color Chooser and Slidey Bit
- Template Synchronizaton Control
Schema Navigation
One of the most common issues that our users have raised is the ability to easily find schema items once the schema grows to any level of complexity. We often encounter this within real projects where finding the correct schema elements, particularly if there are siblings with the same children, can lead to confusion. We have heard your pain and this is the first improvement in the user interface - we have many more planned.
The Schema now has a Filter Schema search input box at the top directly below the Schema action buttons. Anything you type in there will be used for a substring search on any of the records or element names in your schema. Single or multiple words can be used and all of the words are searched separately.
Schema elements are now shown with vertical indicators to help with deciding the child level:
The search filter will:
- Highlight all parents that match - whilst leaving their children expanded
- Highlight all children that match - whilst leaving parents expanded if they match or collapsing them if they don't and just showing the path
The Twitter status schema is sufficiently large and complex to demonstrate all of these actions:
- The word "tweet" has been typed into the schema search box
- The top-level schema element "tweet" was already highlighted in the schema
- The child fields "retweet_count" and "retweeted" have been highlighted along with the parent field "retweeted" status
The schema does not automatically filter or fold the child schemas of parents that match the search.
If the fields which match are all child fields then just those fields will be shown. For example searching for the field "friend" finds two schema child fields. They are shown as highlighted with a light purple border along with their parentage. The two parent paths are shown tweet → retweeted_status → user and tweet → user are both shown so that is is easy to distinguish between the two fields with the same name.
To clear your search filter press the Clear Search Filter button. 
S3 File Adaptor
Amazon Web Services Simple Storage System (S3) is an object based storage scheme that presents a file-system view of the objects that it stores. Folders are termed buckets and files are objects. To all intents and purposes it appears like a web hosted file system. The ability to read and write data from S3 buckets is useful for integrating with other systems that can only hadnle file-based integrations such as DMPs.
The Kitewheel S3 File Adaptor supports reading and writing files from AWS S3.
File Connection Type
A new adaptor type is available in the Admin → Connections panel
The File Connector currently only supports S3 as the Type (we plan to extend this to Azure Blob Storage soon). The Connector requires the S3 Bucket name which is the top-level bucket name without any additional path, and the AWS Access Key and Secret Access Key.
The File Connector provides a Test Connection button consistent with the other connection types.
Writing a File to an S3 Bucket
In order to write an S3 file a filename, including path, and an object that represents the contents of the file:
The File Name value can be provided by a literal, schema value or public variable. The file is created on each write and will overwrite any existing file without creating an error. It is therefore suggested that a unique identifier is used to create the filename. Often buckets contain many thousands of small objects rather than one large file - most third party data loading processes are happy to process a large number of small files.
The File Contents to Write can be any part of the schema.
A JavaScript Node can be used to generate a file name that uses a timestamp and a unique identifier to ensure that files are ordered by creation date and also that there is no clash or accidental overwriting if multiple engines are processing and writing files simultaneously.
Reading a File From an S3 Bucket
In order to read a file from an S3 bucket you need to provide a File Name and a File Contents Destination:
The File Name should include the path to file and filename. The Contents Destination is the location for the body of the file.
If the file contains a single JSON object this will be parsed and expanded at the location specified. If the file contains more than one JSON object or contains non-JSON data then the complete contents of the file will be saved as a string field at the location specified. Be wary of loading large files in this method as it is designed to load small objects and not large files.
There is currently no way to list all of the files in a bucket - but this is something that we plan to add. A sample use case may be to have a stream of file names that require to be processed and the file reader procesisng those files.
Kitewheel Graph API Consumes XML
The Kitewheel Graph API will now consume valid XML posted to the graph API end point and convert this into a JSON object. The rules for conversion are as follows:
- XML elements become JSON arrays
- XML attributes becomes "$" object with fields
- XML body text becomes "_" when there are attributes
- All element names are flattened to lowercase
- Attribute names are not flattened
| XML posted to API | JSON in Schema |
|---|---|
<xml/> | { "xml" : "" }
|
<xml> <element/> </xml> | { "xml": { "element": [ "" ] } }
|
<xml> <element>a</element> <element>b</element> <element>c</element> </xml> |
{ "xml": { "element": [ "a", "b", "c" ] }
|
Front-End Performance
- React 16
- Color chooser with better control
- Another chooser
Hub Analytics
Logging infrastructure - phase 1
Template Sync Control
Bug
- [KIT-1246] - Update Migration Strategy for Engine / Reporting DB
- [KIT-1317] - Custom engine group settings for environment disappear on a version restore
- [KIT-1330] - Page freeze (shows error boundary display) on organizations admin if navigating immediately after login
Story
- [KIT-949] - S3 Storage Adaptor: write, read (possibly) and no listen
- [KIT-1053] - Upgrade to React 16
- [KIT-1166] - Give the user manual control over where Graphs/Templates are synced to
- [KIT-1196] - Kitewheel Hub needs the concept of an "Organization Operator" user
- [KIT-1230] - Add Google Analytics to Hub
- [KIT-1240] - Cross Engine Visual Testing
- [KIT-1241] - Schema Enhancements
- [KIT-1249] - Logging system overhaul implementation (metrics, trace)
