topic Re: SSL v3 Handshake Failure in APIs & Integrations

SSL v3 Handshake Failure

JZamecnik — Mon, 26 Jun 2023 07:10:15 GMT

While running our ETL (Github Actions ubuntu-latest with Python 3.9), we are getting occassional random SSL handshake failures like this:

HTTPSConnectionPool(host='api.hubapi.com', port=443): Max retries exceeded with url: /crm/v3/objects/calls?limit=100&properties=hs_call_title&properties=hs_call_duration&after=32852742180 (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1129)')))
<class 'requests.exceptions.SSLError'>

Unfortunately, we can't reproduce this reliably or pinpoint the cause:

The same code (hubspot-api-python v8.0.0) sometimes runs through, sometimes fails.
We updated our dependencies (in particular urrlib3 is now at 1.26.16) without seeing an improvement.
We're not observing connection issues in any of our other workflows.
The API endpoint varies

Is there anything we can do to narrow this down?

What is also striking is that the error says SSL v3, i.e. the outdated, insecure protocol version.

Re: SSL v3 Handshake Failure

advance512 — Mon, 26 Jun 2023 11:24:26 GMT

Yes! I was just about to report this. Happy to see your post and that we're not alone in experiencing this.

We use Node.js (with axios) to make a request, each time to a different Hubspot API (CRM v3, deals v1, engagements v1). Usually everything works fine, but sometimes the error occurs and when it does, it does so in a different end point each time. However it is always the same error:

Error: write EPROTO 140067321558976:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1565:SSL alert number 40

The error started appearing sporadically on June 16th.

Seeing SSLv3, I tried to force axios to use TLSv1.2 to TLSv1.3, but it seems that is the implicit default behavior anyways. And of course it didn't change a thing, the error continues.

I am not sure that the "sslv3 alert" thing is referring to actual SSLv3, it might just be the name of a shared piece of code used in TLS too.

My guess? Hubspot have at least one API server with a badly configured TLS behavior, or just general faulty network behavior.

Re: SSL v3 Handshake Failure

JZamecnik — Mon, 26 Jun 2023 15:14:43 GMT

Good to see we're not the only ones. I raised the issue in the python library that we're using (though I don't think it is an issue with the library itself, esp. since you're getting the same on Node.js). So far, it hasn't been acknowledged by the devs.

The theory that it depends on the server that gets the connection is definitely interesting, though I'd expect HS to avoid this kind of issues with their set-up.

Re: SSL v3 Handshake Failure

murphpdx — Mon, 26 Jun 2023 18:10:46 GMT

We're also running into this. Is there a fix coming?

Re: SSL v3 Handshake Failure

advance512 — Tue, 27 Jun 2023 10:46:21 GMT

You should all open a support request with Hubspot for this (send it to you representative). And upvote the thread here.

Re: SSL v3 Handshake Failure

JAnderson68 — Tue, 27 Jun 2023 16:40:28 GMT

I'm seeing the exact same behavior. We have a couple of overnight jobs that have been running without issues since July 2022. We're making 3-5k api requests per night. First instance of this error started on 6/20/2023.

SSLError(MaxRetryError("HTTPSConnectionPool(host='api.hubapi.com', port=443): Max retries exceeded with url: /crm/v3/objects/companies/search (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1129)')))"))

It's failing randomly (multiple different endpoints), and sporadically as far as it may make several thousand api calls before it throws the error. So far, I've been able to just rerun the job and it will eventually run without error. This issue is now happening every night.

I'm also thinking that there's most likely a misconfigured server in the HubSpot farm that is causing this. Just for info, we are running vanilla Windows server (2012 R2) and a recent(ish) Python v3.9.13 setup.

I'm also going to open a support request with HubSpot. I just wanted to chime in here because I'm happy it's not just me, and to let you guys know you aren't alone either.

Re: SSL v3 Handshake Failure

klarsky — Wed, 28 Jun 2023 15:15:19 GMT

Adding a comment to boost visibility, this error has been severely damaging our internal data flows and costing my team a ton of time for the past couple of weeks. Can we get a response from a HubSpot rep in here?

Re: SSL v3 Handshake Failure

dobrynin — Thu, 29 Jun 2023 19:03:58 GMT

Chiming in that we are also experiencing this. 🚨Hubspot devs🚨

Re: SSL v3 Handshake Failure

AQazi5 — Fri, 30 Jun 2023 00:51:45 GMT

Any news here? I've been experiencing this as well, I had been following this post in the hopes that this might get resolved but seems like no one has responded yet. Can we please get someone to take a look at this?

Re: SSL v3 Handshake Failure

JZamecnik — Fri, 30 Jun 2023 06:53:48 GMT

I'm actually seeing an escalated case with this matter that was reported this morning. I've attached this ticket to the case and shared your link -- thanks for that, by they way!
I'll keep you posted on any updates as soon as I have them.

Got this reply from our support representative yesterday, so let's hope something will happen soon. Having to re-run the jobs multiple times until we get lucky is really bad 😤

Re: SSL v3 Handshake Failure

EWood — Fri, 30 Jun 2023 15:55:07 GMT

Hi All!

Thanks for reaching out to the community developer forum. We're getting a few support tickets like this. In those cases, it's believed this is a Cloudflare related issue. It can be caused if the rate of api calls is too high and Cloudflare may think that you are DDOSing. It's recommended you slow down making the api calls to see if the issue gets resolved.

Hope that helps resolve the behavior.

Cheers!

Elyse

Re: SSL v3 Handshake Failure

JAnderson68 — Fri, 30 Jun 2023 17:06:15 GMT

Elyse,

Speaking for myself - I've already carefully engineered my integrations to meet the currently published API rate limits:

https://developers.hubspot.com/docs/api/usage-details#rate-limits

In these docs, it does clearly state that the error we will receive for running afoul of the rate limit will be a '429' http status code.

Is there a new rate limit? It seems your Cloudflare implementation should be aware of your current terms of service to allow for your existing rate limits AND documented error/return codes.

Otherwise, a rate of 'too high', is not something I can write a software specification to. I'm looking for a concrete answer to this issue, thanks!

Re: SSL v3 Handshake Failure

dobrynin — Fri, 30 Jun 2023 18:29:12 GMT

Echoing @JAnderson68. We have rate limiting in place. While we were developing our rate-limiting solution we never experienced the SSL handshake failure, but hit the 429 consistently. Furthermore an SSL handshake failure strikes me as a strange response to a DDoS attack. All resources I can find online point to misconfigured servers.

Re: SSL v3 Handshake Failure

AQazi5 — Fri, 30 Jun 2023 18:45:51 GMT

Elyse,

Same here... the rate-limiting specified in the documentation tells me that I can execute 150 requests/10 seconds, which comes down to around a request per 66.67 milliseconds. I have been playing around with the throttling and currently, I am executing one request per 8000 milliseconds and still running into this, I don't think I can afford to slow it down any further.

Re: SSL v3 Handshake Failure

klarsky — Fri, 30 Jun 2023 19:26:40 GMT

Adding my voice to this. Our plan has us at 150 calls per 10 seconds too. We're currently calling at about 150 per minute and still hitting the SSL handshake issue multiple times per night. It's not a 429 response like we had in the past while developing our pipeline.

Re: SSL v3 Handshake Failure

advance512 — Mon, 03 Jul 2023 09:37:04 GMT

We are in the same situation as the responders above. We follow the specification that Hubspot shares in the technical documentation, to work within rate limitation. We have had no issues up until June 16th. Now we get it daily, last time I got it was 36 minutes ago today.

Hubspot states in its API documentation :

"Any app or integration exceeding its rate limits will receive a 429 error response for all subsequent API calls."

https://developers.hubspot.com/docs/api/usage-details#error-responses

In this case, @EWood, it seems that your product has stopped working as per your product definition: it does not return a 429 error response, rather, it sporadically and randomly returns a cryptic SSL3 error. All immediate subsequent API calls however, work successfully!

To me, it is pretty clear it is a bug in your system, I am sorry to say..

Could we get an official Hubspot confirmation that this is indeed a bug?

Could we get an estimation for fixing the issue?

Thank you.

Re: SSL v3 Handshake Failure

EWood — Wed, 05 Jul 2023 16:51:03 GMT

Thanks everyone for continuing to provide details and for your patience as we traverse this behavior during week of rest. We do have an internal investigation filed and will provide new insights as they develop.

Please note that a priority has been added to the case, however this might be delayed until Monday, July 10th as this is when our full team returns back to the office.

Thanks again for understanding!

Elyse

Re: SSL v3 Handshake Failure

LF-integrations — Thu, 06 Jul 2023 10:54:30 GMT

My guess? Hubspot have at least one API server with a badly configured TLS behavior, or just general faulty network behavior.

I agree with that. We saw ~20 errors (per day) like that in the past but since 16th of June there are around 100 (per day). Let's wait for HubSpot team to investigate it.

Re: SSL v3 Handshake Failure

Jaycee_Lewis — Tue, 11 Jul 2023 20:18:51 GMT

Hey, y'all. My internal resources asked if any of you have the following details to share?

Their request – “Both or either of the full error logs and packet capture would be helpful.”

You can post here or DM them to me. @JZamecnik @LF-integrations @advance512 @klarsky

Thank you very much.

Best,

Jaycee

Re: SSL v3 Handshake Failure

JZamecnik — Fri, 14 Jul 2023 10:34:02 GMT

Hi and thanks for the reply. I'll see what I can do about the packet capture.

For the error log, the only error I get from the HubSpot Python client is the one posted above in various incarnations. Here the latest one:

ERROR:root:Updating contacts (customers and non-customers) in HubSpot HTTPSConnectionPool(host='api.hubapi.com', port=443): Max retries exceeded with url: /crm/v3/objects/contacts/batch/update (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1129)'))) <class 'requests.exceptions.SSLError'>

Could you be more specific about the "full error logs"? Do you mean setting the loglevel of the hubspot library to DEBUG for more context? I doubt my own internal log messages are of any help, there.