Kitewheel March 2018 Release Notes

There are a number of improvements in the March 2018 Release of Kitewheel. The highlights include:

Schema Navigation - making it easier to work with large schemas
S3 File Adaptor - read and write files from AWS S3 buckets
Kitewheel Graph API Consumes XML - the graph API will now accept and parse XML
Kitewheel Hub Front-End Performance
Web Tracking Do-Not-Track Features
Cross-Engine Visual Testing
Template Synchronization Control

There were a number of other bug fixes and minor improvements made during this release. We hope you like it.

Schema Navigation

One of the most common issues that our users have raised is the ability to easily find schema items once the schema grows to any level of complexity. We often encounter this within real projects where finding the correct schema elements, particularly if there are siblings with the same children, can lead to confusion. We have heard your pain and this is the first improvement in the user interface - we have many more planned.

The Schema now has a Filter Schema search input box at the top directly below the Schema action buttons. Anything you type in there will be used for a substring search on any of the records or element names in your schema. Single or multiple words can be used and all of the words are searched separately (for example, filtering by using 'email twitter' will return all values that container either email OR twitter OR both).

Schema elements are now shown with vertical indicators to help with deciding the child level:

The search filter will:

Highlight all parents that match - whilst leaving their children expanded
Highlight all children that match - whilst leaving parents expanded if they match, or collapsing them if they do not and just show the path

Any schema elemented already selected (and thereby able to be used by a node or adaptor as a data source) remains selected

The Twitter status schema is sufficiently large and complex to demonstrate all of these actions:

The word "tweet" has been typed into the schema search box
The top-level schema element "tweet" was already selected in the schema, and remains selected in addition to being highlighted
The child fields "retweet_count" and "retweeted" have been highlighted along with the parent field "retweeted" status

The schema does not automatically filter or fold the child schemas of parents that match the search.

If the fields which match are all child fields then just those fields will be shown. For example searching for the field "friend" finds two schema child fields. They are shown as highlighted with a light purple border along with their parentage. The two parent paths are shown tweet → retweeted_status → user and tweet → user are both shown so that is is easy to distinguish between the two fields with the same name.

To clear your search filter press the Clear Search Filter button .

S3 File Adaptor

Amazon Web Services Simple Storage System (S3) is an object based storage scheme that presents a file-system view of the objects that it stores. Folders are termed buckets and files are objects. To all intents and purposes it appears like a web hosted file system. The ability to read and write data from S3 buckets is useful for integrating with other systems that can only handle file-based integrations such as DMPs.

The Kitewheel S3 File Adaptor supports reading and writing files from AWS S3.

File Connection Type

A new adaptor type is available in the Admin → Connections panel

The File Connector currently only supports Amazon S3 as the Type (we plan to extend this to Azure Blob Storage soon). The Connector requires the S3 Bucket name which is the top-level bucket name without any additional path, and the AWS Access Key and Secret Access Key.

The File Connector provides a Test Connection button consistent with the other connection types.

Creating a File Adaptor

To create a file adaptor (for reading or writing), select the File option from among the adaptor types when creating a new item

Writing a File to an S3 Bucket

In order to write a file to an S3 bucket you need to provide a File Name (including any path) and a File Contents Source:

The File Name value can be provided by a literal, schema value or public variable. The file is created on each write and will overwrite any existing file without creating an error. It is therefore suggested that a unique identifier is used to create the filename. Often buckets contain many thousands of small objects rather than one large file - most third party data loading processes are happy to process a large number of small files.

The File Contents to Write can be any part of the schema.

A JavaScript Node can be used to generate a file name that uses a timestamp and a unique identifier to ensure that files are ordered by creation date and also that there is no clash or accidental overwriting if multiple engines are processing and writing files simultaneously. The TransactionId from Schema Metadata can also be used to get a unique identifier.

Reading a File From an S3 Bucket

In order to read a file from an S3 bucket you need to provide a File Name and a File Contents Destination:

The File Name should include the path to file and filename. The Contents Destination is the location for the body of the file.

If the file contains a single JSON object this will be parsed and expanded at the location specified. If the file contains more than one JSON object or contains non-JSON data then the complete contents of the file will be saved as a string field at the location specified. Be wary of loading large files in this method as it is designed to load small objects and not large files.

There is currently no way to list all of the files in a bucket - but this is something that we plan to add. A sample use case may be to have a stream of file names that require to be processed and the file reader processing those files.

Kitewheel Graph API Consumes XML

The Kitewheel Graph API will now consume valid XML posted to the graph API end point and convert this into a JSON object. The rules for conversion are as follows:

XML elements become JSON arrays
XML attributes become "$" objects with fields
XML body text becomes "_" when there are attributes
All element names are flattened to lowercase
Attribute names are not flattened

XML posted to API	JSON in Schema	Note
<xml/>	{ "xml" : "" }	An empty element produces an object with an empty field value.
<xml> <element/> </xml>	{ "xml": { "element": [ "" ] } }	An empty child element is turned into an array value with a single empty value.
<xml> <element>a</element> <element>b</element> <element>c</element> </xml>	{ "xml": { "element": [ "a", "b", "c" ] } }	A collection of child elements with simple values is transformed into an array of values.
<xml> <element attr1="a" attr2="b">first element</element> <element>second element</element> <element/> </xml>	{ "xml": { "element": [ { "$": { "attr2": "b", "attr1": "a" }, "_": "first element" }, "second element", "" ] } }	A collection of child elements with attributes and simple or non-existent values are transformed to an array of objects. Note that the attributes are held in the special "$" attribute of the object they refer to.
<xml> <element attr1="a" AttributeTwo="b">first element</element> <element>second element</element> <element/> </xml>	{ "xml": { "element": [ { "$": { "AttributeTwo": "b", "attr1": "a" }, "_": "first element" }, "second element", "" ] } }	Similar to above but this demonstrates that the attribute names are not flattened.
<xml> <element attr1="a" attr2="b">first element</element2> <element>second element</element> <element/> </xml>	{ "message": "Unexpected close tag\nLine: 1\nColumn: 55\nChar: >", "error": {} }	Invalid XML leads to an error message which

Kitewheel Hub Front-End Performance

We have now finished our upgrade of the Kitewheel Hub front-end technology as a pre-cursor to some larger UI changes which come later in the year. The Schema Navigation improvements are the first of the changes to the front end. In this change we have upgraded to React 16 which has a number of benefits from a development perspective, making it easier for us to make changes in the future, making the application more robust if we encounter errors, and also giving better error diagnostics if something does go wrong.

One of the most important benefits that we expect is an overall performance improvement in the application itself. Benchmark tests show that React 16 is about 20% faster in the browser. Also the actual package itself is smaller. This should lead to faster load times and a more responsive experience for all.

If for some reason the Kitewheel hub application gets into an unexpected state then you might see an image similar to the ones below. For the first one, a reload of the page (app) will usually allow you to continue without encountering other problems. For the second, navigate to a different area (without reloading) will usually allow you to continue.

If you do see an error message like this then there is usually a meaningful error that we have printed in the browser console. If you know how to get to your browser console then please send us a screenshot of what you see to support@kitewheel.com as this will make it easier for us to diagnose the problem.

As part of this upgrade we have improved the color chooser to make it easier to use and to interact better with the local operating system. This is an example of the color chooser used to add some color when creating a tag:

This now has a simpler layout. There are several ways to choose a color:

Move the highlighted circle on the palette
Move the slider to change the RGB values
Open the operating system specific color chooser
Enter the hex value for the color or
Enter the Red Green and Blue values

On a Mac the operating system specific color chooser will bring up the OS X color chooser:

On Windows the operating system color choose will bring up the Windows color chooser:

Web Tracking Do-Not-Track Features

The Xponent Graph API web tracking JavaScript has been modified to add a "doNotTrack" configuration section that allows the specification of HTML DOM elements and cookies that should not be tracked.

cookies - the cookies specified will not be returned in the web tracking events
names - any HTML DOM elements that match the list of names will not be returned
ids - any HTML DOM elements with the specified ids will not be returned
classes - any HTML DOM elements that have this class will not be tracked

The doNotTrack option is currently only available by modifying the JavaScript _kw.options object. There is no UI configuration available as part of this release.

For example if you had a HTML body like this:

  <body>
    <h1>Hello world!</h1>
    <div id="content">
      <code id="response">This is a div tag</code>
    </div>
    <div id="testers">
      <a href="index.html" class="doNotTrack">This is a link!</a><br/>
      <div id="form">
        <form action="index.html" method="post">
          <input type="text" name="myText"/>
          <input type="password" name="passField"/>
          <input type="submit" value="Submit me"/>
        </form>
      </div>
    </div>
  </body>

The following "doNotTrack" object could be used within the _kw.options object:

_kw.options = {
    "listenerId": "0123456789000000",
    "listenerHost": "https://api.kitewheel.com/api/v1/listener/",
    "doNotTrack": {"enabled": true, 
                        "names": ['passField', 'field1'],
                        "ids": ['content', 'passID'], 
                        "classes": ['doNotTrack'],
                        "cookies": ['privateCookie']
                        }
  }

This would have the following effect:

The password field would not be tracked in a blur or a formSubmit event because of the names setting
The content div will not be tracked
The link will not be tracked as it has the "doNotTrack" class - this is the recommended approach
The cookie "privateCookie" will not be returned as part of any packet

Cross-Engine Visual Testing

Kitewheel has supported remote engine stacks for some while. However it was not previously possible to perform visual testing with a remote stack. In this release we have enabled visual testing for remote stacks.

A remote engine stack is any engine configuration that is not the default engine configuration for the Kitewheel Hub. This includes dedicated production stacks within the Kitewheel AWS environment but also in remote AWS and Azure environments that are managed by Kitewheel. Remote engine stacks are used for production workloads to provide dedicated throughput and security for connection to on-premise or other cloud environments. If you are interested in using a remote stack for these purposes please speak to your Kitewheel account manager.

Template Synchronization Control

In this release we have improved the way that templates synchronise between different Kitewheel and dedicated partner OEM hubs. This makes the process more robust and gives a much finer level of control on how templates and managed graphs are distributed.

Bug Fixes

[KIT-1246] - Update Migration Strategy for Engine / Reporting DB to avoid potential database locks
[KIT-1317] - Custom engine group settings for environment disappear on a version restore
[KIT-1330] - Page freeze on organizations admin if navigating immediately after login
[KIT-1376] - Webtrack: Tag attributes missing in data return object when event tracked is not a tag
[KIT-1307] - Fix for large schema elements not passing correctly to JavaScript microservice
[KIT-1321] - Duplicate deployments on Azure / Kubernetes stacks
[KIT-1354] - Schema filtering fails with arrays in the Schema
[KIT-1356] - UTF8 Characters not encoded correctly when an JSON object is inserted into the database

Other Improvements

[KIT-1249] - Logging system overhaul implementation (metrics and trace) - this should improve the performance and reliability of metric updates whilst opening the next stage of loggig and trace analysis.
[KIT-1346] - Upgrade SQL Alchemy and database libraries. SQL Alchemy has been upgrdaed to 1.25 and the underlying database libraries have also been updated.