For now, we wouldn't be able to remove the limit entirely and it is likely that any reasonable limit that protects our databases will eventually be hit by a developer working through a large data set.
Having said that, we do understand that this is going to be a common problem.
Hence, one workaround that you can page through the dataset in an efficient ways is to add more-specific filters and sorts in order to page through the result set. For example, if they filter by LastModifiedDate and sort by vid then they could select the next page of data by updating the filters to filter by the LastModifiedDate + vid of the last record returned by the search.
Right now, we are paginating as efficiently as possible.
Some of our clients have around a million contacts, and if we just check which of them were modified in the past week, we get around half of them.
In order to get 500,000 contacts, with a page limit of 100, we need to make 5000 requests to your server. (per user!)
This means that we can't give this to our customers in real-time, and instead, have to store everything in our databases.
Instead, we just ask that you could dump the entire contacts, maybe even as a gzip-compressed JSON, which we know is less than 10MB in size.
Doing so will save you, and us, so much on bandwidth, both in terms of infrastructure and in terms of speed of data acquisition. It will also make the data for the client more accurate as we could just query a list of contacts as the client needs them, in real-time.
I'd be happy to jump on a call with you or one of your devs to explain our exact strategies, and how supporting this will undoubtedly be smart both for you and for us.