Problem:

For missing or unexpected fields found in the header of a file Amperity will send a workflow failure notification that looks like this. This error means that the schema of the file received by Amperity does not match the expected schema.  

Solution:

Identify the cause of the error

First verify that the file being ingested matches the schema of the feed. For flat-file sources use the existing courier to check the schema.  

On the Sources page navigate to the "couriers" section and click on the courier associated with the failing feed. 

Edit the courier, and use the "browse" menu to find and temporarily load the appropriate file for analysis.

Navigate to the file that is causing the error and click "accept"

Amperity will load a preview of this file in the courier editor

Take note of the schema information in this file. We will compare this to the existing feed configuration. Next verify that the feed is configured to accept the data you've just reviewed in the courier. In another tab open Amperity and navigate to the Sources page.

Select the feed associated with the workflow error

Compare the fields listed under "incoming fields" to the fields in the file temporarily loaded in the courier. Any differences in field names will prevent data from being loaded. 

Update the feed and re-ingest

To resolve this difference you can update the field names in the incoming data, or in the feed in Amperity.

Navigate to the Sources page and select the feed associated with the workflow error

Edit the field and update the configuration to match the file being ingested.

Field names can be changed using the "rename to" functionality in the feed. Alternatively a column can be marked as one to not include if unnecessary or not require if it will not always be populated.

Note: "Required" checks for the existence of a column with that name, null values within that column are still valid if the "required" box is checked.  

Note: that if you update a column there may be some update operations that need to take place. Any modification to a column that is used as a part of the primary key will require truncating and reloading the entire table.