Problem:
If a workflow fails because the number of errors encountered exceeds the configured threshold Amperity will send a notification that looks like this:
This error can also appear in the platform under the "Recent Activity" pane
Solution:
Identify the cause of the error
Click "View error log" under the feed with failing records to download an error log with summary information about the errors that were encountered.
A detailed error log can be viewed by clicking "ingest details".
In the ingest details page click the "action" overflow menu for the feed name and select "download feed error log".
Note: This error log contains information about every error encountered so it can become quite large.
Errors in a feed are caused by a mismatch in data types between the schema of the feed and the data being ingested.
First verify that the file being ingested matches the schema of the feed. For flat-file sources use the existing Courier to check the schema.
On the sources page navigate to the "couriers" section and click on the courier associated with the failing feed.
Edit the courier, and use the "Browse" menu to find and temporarily load the appropriate file for analysis.
Navigate to the file that is causing the error and click "Accept"
Amperity will load a preview of this file in the courier editor
Take note of the schema information in this file. We will compare this to the existing feed configuration.
Next verify that the feed is configured to accept the data you've just reviewed in the courier. In another tab open Amperity and navigate to the Sources page.
Select the feed associated with the workflow error
Workflow errors related to thresholds are generally conflicts in field types between the feed and the data being ingested.
Review the field types in the feed and determine if there are any values seen in the courier that cannot be cast to the value specified in the feed configuration.
For example if a field type in a feed is set to decimal and an incoming value for that field is a string Amperity will not be able to convert that string to a decimal value and this will cause an error.
Note: The file preview in the courier configuration only shows a sample of records. You may need to view the file elsewhere to find field type mismatches if they do not occur in the sample. The error logs downloaded in the first steps can help narrow down the cause of the issue.
Update the Feed and re-ingest
Navigate to the "Workflows" page in Amperity and select the workflow with the error
Select "Show resolutions" to list the possible resolutions for this error
- "Retry ingesting data" will attempt to reload this data. "Relaunch workflow from the beginning" will attempt to start the workflow over and reload all data. If there are no changes to the file being ingested or the feed configuration the error will repeat.
- "Retry ingesting the data excluding the feeds that have failed" will skip this data for the current workflow and attempt to load any other data. This data will not be ingested by Amperity.
- "Retry ingesting only failed feeds" will attempt to re-ingest the failed data and load it through the feed. "Retry ingesting data with updated configuration" will re-ingest all data and load it through an updated feed configuration. This will only succeed if you change the configuration of the feed to match the schema of the file being ingested.
Note: Modification of fields in a feed that are used as a primary key will require the truncation of all data associated with that feed.
To resolve this difference you can update the field types in the incoming data, or in the feed in Amperity.
To update the feed configuration navigate to the "Feeds" section on the "Sources" page and select the appropriate feed
Click "Edit" in the feed editor and adjust the field types appropriately.
Click "Activate" to save the feed configuration
Note: updating the field types will require a modification to the underlying domain data. A message will appear asking how Amperity should resolve this change.
If the data can be transitioned in-place to the new data type Amperity can do that by selecting "Migrate existing data to match these changes" and clicking "Activate and resolve"
If the data cannot be transitioned in place because of a potential data type mismatch (e.g. converting a string to a date field type) the data in this domain table will need to be purged completely and reloaded manually. Any changes to fields that make up the primary key will require purging the table of all data.
Return to the workflow resolution options and select "Retry ingesting data with updated configuration" or "Retry ingesting only failed feeds".
Temporarily increase the error threshold
Navigate to the "Workflows" page in Amperity and select the workflow with the error
Select "Show resolutions" to list the possible resolutions for this error
"Retry ingest with updated error percent" will temporarily increase the threshold for errors for this file.
Select this option and click "Continue" to choose this option
Enter an updated threshold value and click "Resolve" to enable this new temporary threshold and restart the workflow.
Permanently increase error thresholds
If a file is going to continually be above the error threshold and you would like to increase the threshold permanently follow these instructions:
Navigate to the Amperity Sources page and select the appropriate feed
Edit the feed
Expand the "Error Thresholds" section and update the maximum threshold to the desired value
Click "Activate" to save this new threshold.
During the next workflow run this threshold will be used as the maximum percentage of allowable errors.