For best practice of handling a large number of records, please follow the below instructions:
NB: Please note that the Concurrency settings in the Sync Flow step and the Timeout of the flows are based on the number of records that are required from the source step, the memory when it’s processing (worker it’s on), storage size of a record, the steps in the flow and the time required to complete the integration.
-
Send the data/ records to a buffer destination step.
-
Create a flow (Flow 1) where you require the records to be manipulated and sent to a destination as shown in the below image.
- Create a new flow (Flow 2) where the buffer is used as a source step and Flow 1 is referenced in a sync flow step as illustrated in the below image. In the Buffer source step, increase the number in the Limit field to the maximum limit of the number of records that you require from the buffer.
- In Flow 2, click on Flow Details and scroll to the Concurrent Run Execution field and type “1” as highlighted in the below image. Save the change. This will allow the first run to finish processing before the next run can occur.
- In Flow 2, click on the Sync Flow step and click on the Concurrency tab as highlighted in the below image. Select a Concurrency Method (Series or Parallel) and type in the number of requests you want to make before the next response occurs in the Concurrent Calls field. Increase the concurrent calls to make the flow process faster. The Concurrency Method series means that the number of concurrent calls will go through until they all are completed and then it will take the next 3 records (number inputted in Concurrent Calls). Parallel means that the number of concurrent calls will always be the 3 records (number inputted in Concurrent Calls) even if the first record gets completed or fails; it will take the next record instead of waiting for all 3 records to be completed. In this example, Parallel is selected as a Concurrency Method and there are 3 Concurrent Calls.
- Increase the flow timeout on both flows to the same number of minutes. This can be done by clicking on Flow Details, increasing the number of minutes in the Timeout field and clicking on save as illustrated in the below page.
- Please ensure that the number in the Limit field in the Buffer source step on both flows are the same as shown in the below image.
- Once executed, you will be able to view the records getting processed based on your Concurrency settings in the Sync Flow step in Flow 2. The below image is how the run history will look for Flow 1 while the records are being processed.
If you are having an issue integrating a large number of records, please contact our support at support@synatic.com .