SSL v3 Handshake Failure

[JZamecnik]

While running our ETL (Github Actions ubuntu-latest with Python 3.9), we are getting occassional random SSL handshake failures like this:

HTTPSConnectionPool(host='api.hubapi.com', port=443): Max retries exceeded with url: /crm/v3/objects/calls?limit=100&properties=hs_call_title&properties=hs_call_duration&after=32852742180 (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1129)')))
<class 'requests.exceptions.SSLError'>

Unfortunately, we can't reproduce this reliably or pinpoint the cause:

• The same code (hubspot-api-python v8.0.0) sometimes runs through, sometimes fails.
• We updated our dependencies (in particular urrlib3 is now at 1.26.16) without seeing an improvement.
• We're not observing connection issues in any of our other workflows.
• The API endpoint varies

Is there anything we can do to narrow this down?

What is also striking is that the error says SSL v3, i.e. the outdated, insecure protocol version.

[PKretzmann · Accepted solution]

Hi community followers! After managing it with the Cloudflare team, their fix is fully deployed, so we feel comfortable saying a fix has been deployed by our downstream provider. Please, you can comment here if anyone still faces issues after this. Thanks for your patience with this one. ❤️ 

[pini]

unfortunatelly I'm still facing the same issue 😕

[PLava]

Hey, could you tell if something was wrong on Cloudflare side that you managing wiht them to fix it?

[Jaycee_Lewis]

Hey, y'all. My internal resources asked if any of you have the following details to share?

Their request – “Both or either of the full error logs and packet capture would be helpful.”

You can post here or DM them to me. @JZamecnik @LF-integrations @advance512 @klarsky 

Thank you very much.

Best,

Jaycee

[JZamecnik]

I was indeed able to catch the failure on a packet capture. Sent you the file in a DM @Jaycee_Lewis .

[murphpdx]

{"name":"SourceError","message":"Hubspot request failed with error: {\"message\":\"write EPROTO 140236678432704:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1565:SSL alert number 40\\n\",\"name\":\"Error\",\"stack\":\"Error: write EPROTO 140236678432704:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1565:SSL alert number 40\\n\\n    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)\\n\",\"config\":{\"url\":\"https://api.hubapi.com/email/public/v1/campaigns/[REDACTED]\",\"method\":\"get\",\"data\":null,\"headers\":{\"Accept\":\"application/json, text/plain, */*\",\"Authorization\":\"Bearer [REDACTED]",\"User-Agent\":\"axios/0.21.1\"},\"baseURL\":\"https://api.hubapi.com\",\"transformRequest\":[null],\"transformResponse\":[null],\"timeout\":0,\"xsrfCookieName\":\"\",\"xsrfHeaderName\":\"",\"maxContentLength\":-1,\"maxBodyLength\":-1},\"code\":\"EPROTO\"}","stack":"SourceError: Hubspot request failed with error: {\"message\":\"write EPROTO 140236678432704:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1565:SSL alert number 40\\n\",\"name\":\"Error\",\"stack\":\"Error: write EPROTO 140236678432704:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1565:SSL alert number 40\\n\\n    at WriteWrap.onWriteComplete [as oncomplete] ([REDACTED]94:16)\\n\",\"config\":{\"url\":\"https://api.hubapi.com/email/public/v1/campaigns/[REDACTED]?appId=[REDACTED]\",\"method\":\"get\",\"data\":null,\"headers\":{\"Accept\":\"application/json, text/plain, */*\",\"Authorization\":\"Bearer [REDACTED]\",\"User-Agent\":\"axios/0.21.1\"},\"baseURL\":\"https://api.hubapi.com\",\"transformRequest\":[null],\"transformResponse\":[null],\"timeout\":0,\"xsrfCookieName\":\"\",\"xsrfHeaderName\":\"\",\"maxContentLength\":-1,\"maxBodyLength\":-1},\"code\":\"EPROTO\"}\n    at [REDACTED]

I'm still seeing a lot of `read ECONNRESET` and status code 500

[JZamecnik]

On my end, unfortunately the error hasn't disappeared yet. However, so far I haven't been able to get a packet capture because it's not well reproducible. If I manage to "catch it on camera", I'll let back to you, but this is to note that we are still affected.

[JAhmedAnsari]

We are facing SSL handshake failures occasionally while sending data to hubspot. Here is the debug output:

*   Trying 104.17.204.204:443...
* TCP_NODELAY set
* Connected to api.hubapi.com (104.17.204.204) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
* Closing connection 0

We have checked our certificate with ssllabs.com, and it is fine. Our server supports both TLS 1.2 and 1.3. We have also checked the connectivity through the terminal using curl. The connectivity is also fine but sometimes your server responds with SSL handshake failure.  What could be the issue? Is it anything misconfigured on one of your server or is there an issue on our side?  Can you help us to figure it out?

[JAnderson68]

Yep, I jinxed myself by reporting no issues in the past couple weeks yesterday. I had the error again last night. I posted the full details in my open support ticket, which should help in narrowing down the culprit here.

[JAnderson68]

I've not seen any of these errors now in the last two weeks. Like JZamecnik, I've only captured minimal error messages (what gets returned by Python in a typical try/except block).

One of the first things I did upon seeing these errors was check the HubSpot logs in Integrations > Private Apps > Logs. I found no errors.

I really believe that the clients are rejecting what is being offered (SSLv3) by one of the API farm servers in the SSL handshake transaction. If I see any more of these errors I will post in my support ticket the full URL, my requesting IP address and the timestamp. That should help in narrowing it down to a specific request.

[JZamecnik]

Hi and thanks for the reply. I'll see what I can do about the packet capture.

For the error log, the only error I get from the HubSpot Python client is the one posted above in various incarnations. Here the latest one:

ERROR:root:Updating contacts (customers and non-customers) in HubSpot
HTTPSConnectionPool(host='api.hubapi.com', port=443): Max retries exceeded with url: /crm/v3/objects/contacts/batch/update (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1129)')))
<class 'requests.exceptions.SSLError'>

Could you be more specific about the "full error logs"? Do you mean setting the loglevel of the hubspot library to DEBUG for more context? I doubt my own internal log messages are of any help, there.

[EWood]

Hi All!

Thanks for reaching out to the community developer forum. We're getting a few support tickets like this. In those cases, it's believed this is a Cloudflare related issue. It can be caused if the rate of api calls is too high and Cloudflare may think that you are DDOSing. It's recommended you slow down making the api calls to see if the issue gets resolved.

Hope that helps resolve the behavior.

Cheers!

Elyse

[EWood]

Thanks everyone for continuing to provide details and for your patience as we traverse this behavior during week of rest. We do have an internal investigation filed and will provide new insights as they develop. 

Please note that a priority has been added to the case, however this might be delayed until Monday, July 10th as this is when our full team returns back to the office. 

Thanks again for understanding!

Elyse

[AQazi5]

Elyse,

Same here... the rate-limiting specified in the documentation tells me that I can execute 150 requests/10 seconds, which comes down to around a request per 66.67 milliseconds. I have been playing around with the throttling and currently, I am executing one request per 8000 milliseconds and still running into this, I don't think I can afford to slow it down any further.

[JAnderson68]

Elyse,

Speaking for myself - I've already carefully engineered my integrations to meet the currently published API rate limits:

https://developers.hubspot.com/docs/api/usage-details#rate-limits

In these docs, it does clearly state that the error we will receive for running afoul of the rate limit will be a '429' http status code.

Is there a new rate limit? It seems your Cloudflare implementation should be aware of your current terms of service to allow for your existing rate limits AND documented error/return codes.

Otherwise, a rate of 'too high', is not something I can write a software specification to. I'm looking for a concrete answer to this issue, thanks!

[dobrynin]

Echoing @JAnderson68. We have rate limiting in place. While we were developing our rate-limiting solution we never experienced the SSL handshake failure, but hit the 429 consistently. Furthermore an SSL handshake failure strikes me as a strange response to a DDoS attack. All resources I can find online point to misconfigured servers.

[klarsky]

Adding my voice to this. Our plan has us at 150 calls per 10 seconds too. We're currently calling at about 150 per minute and still hitting the SSL handshake issue multiple times per night. It's not a 429 response like we had in the past while developing our pipeline.

[advance512]

We are in the same situation as the responders above. We follow the specification that Hubspot shares in the technical documentation, to work within rate limitation. We have had no issues up until June 16th. Now we get it daily, last time I got it was 36 minutes ago today.

Hubspot states in its API documentation :

"Any app or integration exceeding its rate limits will receive a 429 error response for all subsequent API calls."

https://developers.hubspot.com/docs/api/usage-details#error-responses

In this case, @EWood, it seems that your product has stopped working as per your product definition: it does not return a 429 error response, rather, it sporadically and randomly returns a cryptic SSL3 error. All immediate subsequent API calls however, work successfully! 

To me, it is pretty clear it is a bug in your system, I am sorry to say..

Could we get an official Hubspot confirmation that this is indeed a bug?

Could we get an estimation for fixing the issue?

Thank you.

[JZamecnik]

I'm actually seeing an escalated case with this matter that was reported this morning. I've attached this ticket to the case and shared your link -- thanks for that, by they way!

I'll keep you posted on any updates as soon as I have them. 

Got this reply from our support representative yesterday, so let's hope something will happen soon. Having to re-run the jobs multiple times until we get lucky is really bad 😤

[AQazi5]

Any news here? I've been experiencing this as well, I had been following this post in the hopes that this might get resolved but seems like no one has responded yet. Can we please get someone to take a look at this?