File Write Adaptor

The File Write Adaptor writes contents from a graph to a object based file.

Currently only Amazon Web Services Simple Storage System (AWS S3) is supported, but it is planned to expand to other types (for example, Azure blob storage)

• AWS S3 is an object based storage scheme that presents a file-system view of the objects that it stores; folders are termed buckets and files are objects. For all intents and purposes it appears like a web hosted file system. The ability to read and write data from S3 buckets is useful for integrating with other systems that can only handle file-based integrations such as DMPs. 

Creating a File Write Adaptor

In order to use a File Write Adaptor it is best practice to create a file connection first. 

Create a new item via the Create menu option in the project editor, select the File adaptor and give the adaptor a name. 

Choose the connection from the drop down menu, and choose the file name and file contents destination

File Name: The File Name value can be provided by a literal, schema value or public variables

• For S3, this name should include any path outside of the top-level bucket.
• A JavaScript Node can be used to generate a file name that uses a timestamp and a unique identifier to ensure that files are ordered by creation date and also that there is no clash or accidental overwriting if multiple engines are processing and writing files simultaneously. 

File Contents To Write: The contents to write to the file, which could be any part of schema value (recommended) or a public variable

• The file is created on each write and will overwrite any existing file without creating an error. It is therefore suggested that a unique identifier is used to create the filename
• For S3, often buckets contain many thousands of small objects rather than one large file - most third party data loading processes are happy to process a large number of small files