I’m trying to implement a workflow to maintain a HubDB table.
To that end, I have a workflow that is triggered on an object update of relevant properties to update or create a HubDB row using custom code.
The problem comes when I need to publish. If I put a publish call in every workflow enrollment, hundreds of updates are lost because of overlapping publishes and simultaneous draft table writes in the initial enrollment.
A single object update needs to be able to publish the update to the row, but dozens of updates at once means that dozens of publishes will clash. With a set of 1000 records and an empty initial table, only 200 records were created. If I reset and remove the publish calls, nearly every record successfully gets added to the draft table.
Hello! This is the solution I’ve already implemented, but running into issues with concurrent publishing from triggered workflows. Since dozens of updates can occur at once due to bulk updates of the custom object, several updates and publishes to the table are attempted at once, causing data loss.
Hi @DPak, thank you for the details and for your patience!
I have checked for you and in that case you may have to rearchitect it a bit.
The solution that @LeeBartelme proposed would work but not at high volume.
There are some tools that could help throttle workflows like CarpQueue or Daeda Essentials.
This would allow you to control the rate at which the workflows run.
Another route might be to instead of having the updates run on each object, have a “parent” process that runs that finds the recently updated objects and updates the HubDB table in bulk.
We can see here that if you have multiple records all trying to interact with the same table or potentially the same rows at the same time, you run into race conditions.
So if you want a solution like this at a larger scale, throttling or doing the actions in bulk is probably a better option.
And thank you so much @LeeBartelme for your help!
I hope this helps @DPak!
Best,
Bérangère
