How to delete/remove duplicate records when inserting into a buffer

If you require to delete/remove duplicate records when inserting into a buffer, please follow the instructions below:

Illustrated in the image below is an example of a flow that ingests an Excel file containing duplicate records in a field/column. The Buffer Deduplicate step is added to remove duplicates and insert the records into a buffer.

image

The below image shows the fields and values that are in the Excel spreadsheet.

The Buffer Deduplicate step will remove all duplicate values for the specified fields and insert only the unique values in the buffer. Select an existing buffer or create a new buffer, select the Mode and type out the field names that you require to remove duplicate values.

Ticking the No Insert if Not Found checkbox will not add the unique values to the buffer if there are no duplicates values found in the inputted fields.

NB: The Buffer Deduplicate step is a destination step and will only insert data into the buffer.

The below page is the returned data when using the Buffer Deduplicate step.