How to Port Buffers between Orgs

[Faizel]

If required to transfer Buffers with data and indexes for particular solutions between Orgs, please follow the instructions below:

Illustrated below is an overview of the process, We have two orgs, source and target or dev and prod orgs, In the source we create two flows, one to populate a list of buffers to create and a second to create the buffers in the target org. In the source org, we have a flow that will get the data and POST to an API to ingest records in the buffer. The API sits on the target org and works as an interface to add the records into the buffer that is generated.

Populate List of Buffers

In Target org (OrgB). Create a flow to Populate a list of the buffers we want to create. Illustrated below is an example of a flow that uses a Record generator to store the list of buffers to generate, sets default values then prepares the buffer list and saves the records to a buffer (Buffer List) that we will use in the next step to create the buffer.

In the Record Generator, we specify the Name of the buffer (TableName), indexes (OtherKeys) to create on the buffer, and the solution where the buffer should be saved.

In the Filter step, return !!input.record.TableName; is a way of checking if the TableName property exists within the record object

Add a Calculator step to set the defaults.

Code:

let inputRecord = input.record;

function getBufferName(SolutionName, TableName) {
    // cast TableName to lowercase & remove underscores
    let tableNameString = stringHelper.replaceAll(stringHelper.clean(TableName).toLowerCase(), "_", "", true);
    // replace spaces with dashes
    tableNameString = stringHelper.replaceAll(tableNameString, " ", "-", true);
    // attach buffername to solution name in kebab case to get object id
    return stringHelper.clean(SolutionName) + '-' + tableNameString;
}

// define default values and ensure all keys exist in output record
inputRecord.SolutionName = !!inputRecord.SolutionName ? inputRecord.SolutionName : "global";
inputRecord.BufferDescription = !!inputRecord.BufferDescription ? inputRecord.BufferDescription : "Raw table buffer";
inputRecord.OtherKeys = !!inputRecord.OtherKeys ? inputRecord.OtherKeys : [];
inputRecord.BufferName = !!inputRecord.BufferName ? stringHelper.clean(inputRecord.BufferName) : getBufferName(inputRecord.SolutionName, inputRecord.TableName);
inputRecord.BufferTitle = !!inputRecord.TableName ? stringHelper.clean(inputRecord.TableName) : stringHelper.clean(inputRecord.TableName);

//If Required to put buffer into a subsolution, Uncomment the below
// let SubCategoryPath = "SubCategory > "; //Specify Name of Category
// inputRecord.BufferTitle = SubCategoryPath + inputRecord.BufferTitle;

return inputRecord;

Below is an example of what the data looks like going into the buffer

Add a Buffer destination to save the records for the list of buffers to be generated.

Note: If you have a large number of buffers to create, On the Source org (OrgA), Step up a flow that will use the Synatic Admin to get a list of buffers from a solution with the indexes, then use a Calculator step to prepare the data as per the Record Generator sample above and write the records to a JSON file then consume the JSON file in the above flow to populate the list of buffers

Create Buffers

In Target org (OrgB). Create a flow to generate the buffers based on the list we created in the above steps. Illustrated below is an example of a flow that queries the buffer list then prepares data and uses the Synatic Admin destination step to create the buffers.

Add a buffer query to get the data from the buffer list

In the Filter step, return !!input.record.TableName; is a way of checking if the TableName property exists within the record object. Add a default calculator after to see the results from the filter step

Add a Subflow control step after the Filter step and default calculator to send each record to the Synatic Admin destination to generate the buffer. In the Subflow Add a Start flow step, Ensure Set Parent Record is checked

After the Start flow and Pass Through step, Add a Calculator step to define the buffer

Code:

let inputRecord=input.record;

// create automatic indexes for key fields
indexes = [];

inputRecord.OtherKeys.forEach((item) => {
    indexes.push({"key": [[stringHelper.clean(item),1]],"unique": false});
})

// a maximum of 10 indexes may be used
indexes = indexes.slice(0, 10);

// define buffer record
inputRecord = {
    "name": inputRecord.BufferName,
    "title": inputRecord.BufferTitle,
    "description": inputRecord.BufferDescription,
    "indexes": indexes,
    "solution": {
        "name": inputRecord.SolutionName
    }
}

return inputRecord;

Below is an example of a record that will go into the Synatic Admin step to create the Buffer

Add a Synatic Admin destination step, Choose Buffers as the Resource and Insert as the Operation

When the flows are executed, The buffers will be created in the specified solution with the indexes set up

Ingest Records from API

In the Target Org (Org B) - Create a flow that will be used by the API (Gateway) to get records and add to the buffer. Illustrated below is an example of a flow that takes in JSON, looks up a buffer and add records to a Buffer destination

Add a flow Input Parameter, Call it TableName and set it to be Required

After the JSON reader step, Add a calculator to format the record data and set the bufferName

Code:

let inputRecord=input.record;

inputRecord={
    "_recordData": JSON.parse(JSON.stringify(input.record))
}
input.parameters.BufferName = input.parameters.TableName.toString().replace(" ","-").replace("_","").toLowerCase();
inputRecord._bufferName=input.parameters.BufferName;

return inputRecord;

Use the Synatic Admin Lookup step to take in the bufferName and return the details of the buffer to a field called _buffer

After the Synatic Lookup, Add a Calculator step that will create a parameter for the buffer and prepare the record data

Note: _recordData field name is required to pass the values to the buffer destination

Add a Subflow control step to send each record to the Buffer destination. In the Subflow, Add a Start flow step, Ensure Set Parent Record is checked

Before the Buffer destination, Add a Calculator step to set the bufferid as a parameter

In the buffer destination, Reference the parameter created in the previous step called bufferid to add the records into the Buffer destination

API (Gateway) Setup

In the Target org (OrgB) Create a REST gateway to ingest records and add them to the Buffer, The API will act as an interface between the two orgs.

Choose the HTTP method as POST, specify a path name and TableName parameter which will be used to get the buffer we want to add records to. Select the flow created in the previous step and select the From Path parameters mapping. Choose the Processing Mode as Async and Input Content Type as JSON

---

Push Records to Buffer API

On the Source Org (OrgA), Create a flow that will get the records from our source buffer and POST them to the API endpoint to add the records into the buffer. Illustrated below is an example of a flow sets the source buffer and target buffer in a Record Generator then queries the records from the source and sends each record to the API endpoint to ingest records into the new target Buffer.

In the Record Generator, Specify the Buffers that we want to get the data from and the target buffers that we want to send the data to, As per the below

Before and After the Split step, A default calculator can be added to see what the data looks like. In the Split step we split on the buffers array which we created in the previous Record Generator.

Add a Subflow control step to set each buffer to a parameter. In the Subflow add a Start flow step, Ensure Set Parent Record is checked

In the first Subflow control step, After the Start Flow and Pass Through, Add a calculator step to set the source buffer and target buffer as parameters

Code:

let inputRecord=input.record;
input.parameters.buffername=inputRecord.name;
input.parameters.TargetTable = inputRecord.TargetTable
inputRecord={}
return inputRecord;

Add a second Subflow that will pass each buffer and Target buffer to get the data and send to the API to ingest records

Include the Buffer Source step, where we pass in the source Buffername to get the records

Add a Calculator step after the Buffer source and Pass through to delete the default _id field from the record

Code:

let inputRecord=input.record;
delete inputRecord._id;
return inputRecord;

An HTTP Destination step is used to add the API info and path from the Target Org (OrgB). Specify the Content-Type to be JSON

---

Order of Execution

Target Org (OrgB)

• Populate List of Buffer
• Generate Buffers
• Ingest Records from API
• Gateway to Point to Ingest Records from API Flow

Source Org (OrgA)

• Push Records to Buffer API