In our business case, we encounter a need to access more than 10,000 results in our search API calls. Presently, we generate reports based on the last modified date of deals, updating our database daily with changed deals.
The challenge arises when the number of updated deals exceeds 10,000 on certain days, causing us to miss updates for more than 10,000 deals.
We propose introducing an offset parameter in the API call, where the limit and after parameters currently work only up to 10,000 results. Below is an example of the API call we envision:
However, this approach doesn't resolve the issue as it exceeds the 10,000 limitation.
Are we overlooking any existing solutions to address this issue?
If not, is there a possibility of adding an offset feature to the API? We understand the need for limitations but seek the ability to iterate through multiple calls to access all updated results effectively.
Actually, sorry nevermind. There's a limit to how many you can have in a NOT_IN filter and also the initial request would be too long to process anyway.
@stheo bit late to the party, but my idea to get around this is to do an initial search, then every time the intial 100 has been processed, take the IDs of those 100 and make a filter to not include them in the next search.
Actually, sorry nevermind. There's a limit to how many you can have in a NOT_IN filter and also the initial request would be too long to process anyway.
1. You are not missing an option. The documentation states the limit (as you noted) but there is not a workaround or current way to extend this limit
2. Adding your suggestion to the Ideas Community and getting some upvotes is our best move for getting your suggestion in front of the Product team. If you add a link back to your Idea post, I'll give it an upvote to help get the ball rolling