I’m doing some requests to the Contacts endpoint to get all of my contacts. But, sometimes I get an error that I can’t handle. I think that the server isn’t responding to my request.
There are a lot of contacts (700k) and I’m running the code below in a for iteration.
I know about the rate limits, and I’m sure I’m not passing thru.
Sometimes I get the error below, sometimes I don’t.
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
http.client.RemoteDisconnected: Remote end closed connection without response
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I’ve summarized the error so it doesn’t get too big.
I came accross this issue also; would there be a way to solve it? Our syncs take a long time since we need to iterate through all of our associations, and this is making our ETL tool fail
I had the same issue when building our syncs. In custom-code, we had to implement retries (ideally retry 4 times at intervals of 5 seconds). It is possible that the issue still persists after 4 retries altough most case should pass with this simple rule.
Since our data was massive and we had to iterate over 200k+ Associations too, we just ended up using Stacksync managed two-way sync with our database (postgres) and it has never failed so far. They manage this issue internally.
from urllib3.util import Retry
from requests.adapters import HTTPAdapter
import requests
from hubspot import HubSpot
# Create a HubSpot client with retry logic
def create_client(api_key):
# Set interval based on environment
if environment == "sb": # Sandbox
interval = 6 # 6 seconds for free plan
else: # Production
interval = 1 # 1 second for pro plan
# Define the retry strategy
retry_strategy = Retry(
total=5, # Total number of retries
backoff_factor=interval, # Exponential backoff
status_forcelist=[429, 500, 502, 504], # Status codes to retry
allowed_methods=["HEAD", "GET", "OPTIONS", "POST", "PUT"], # Methods for retry
raise_on_status=False # Don't raise error after final retry
)
# Create an HTTP adapter with the retry strategy
adapter = HTTPAdapter(max_retries=retry_strategy)
# Create a session and mount the adapter
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)
# Update session headers with API token
http.headers.update(
{"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"}
)
return HubSpot(access_token=api_key, http=http)
# Create HubSpot API clients for different calls
api_client_companies = create_client(get_access_token("HUBSPOT_COMPANIES", "companies"))
api_client_contacts = create_client(get_access_token("HUBSPOT_CONTACTS", "contacts"))
Could anyone share insights or suggest further improvements? Thank you!
Founder at Stacksync here. RemoteDisconnected is a common HubSpot error that can happen for a variety of reasons, coming due to HubSpot issues (most often) or sometimes network issues depending on how reliable is your cloud provider and network tier.
At Stacksync, we have several parallel data read/write processed for a single HubSpot instance. We have implemented a retry module on the API call with exponential backoff delays. It ensures that you do not exceed rate limits, as well as being robust to repeated RemoteDisconnected errors (the error can happen sometimes 1-4 times in a row, and this can lead to coordination issues between processes if you parallellize queries).
For context, Stacksync powers real-time and bidirectional sync between HubSpot and Databases like Postgres, Snowflake, BigQuery, MySQL and more.
Thank you for your response about the RemoteDisconnected error in HubSpot. I appreciate the information on how Stacksync handles parallel data processing and the retry module. It’s very helpful for improving my script. I have a few quick questions:
Can you share more details about how you implemented the retry module? Any best practices or specific libraries you recommend?
Could you provide examples of how you manage rate limits effectively in a high-volume environment?
I look forward to using these strategies in my setup.
