Data stream SSL/TLS handshake hangs and fails when uploading

[#1]

About once per month I am uploading a bunch of files to the Hubspot FTP server using curl as my client. While this worked fine until January 2017, it failed for my February uploads.

Inspecting curl’s debugging output, I trace the problem down to an SSL/TLS handshake problem for the data stream.

Establishing the control channel, logging in, and CWDing to the target directory goes fine. Then the client sends the STOR command, the data connection is about to be opened, but the SSL/TLS handshake hangs:

> STOR my_file
* SSLv2, Unknown (23):
{ [data not shown]
< 150 File status okay; about to open data connection.
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* SSLv3, TLS Unknown, Unknown (22):
} [data not shown]
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]

Here the handshake hangs. The FTP server does not respond and the client waits forever resp. until a timeout occurs after a few minutes.

As I said, this worked fine until January 2017, aside from spurious SSL/TLS handshake failures which I dealt with be simply retrying the upload. But now the handshake always fails.

[3PETE · Accepted solution]

@RainerKlute Can you please retry your ftp request? We just restarted some of our FTP services on our end.

[#3]

YES! That did it!

Thanks for restarting! May I respectfully suggest to implement some kind of a watchdog?

Best regards
Rainer

[3PETE]

@RainerKlute As I’m sure you heard about it earlier this week that Amazon had issues with S3. This ended up leaving some of our services in a wonky state. So as we do have watchers to make sure they were up and running, which they were. There wasn’t a way to tell if it was acting properly or not.

[3PETE]

@RainerKlute I have been talking with Martin offline about this. Immediately I am not sure why this would be hanging. Can you confirm the certificate is still valid and works in other servers? In other words, can you isolate this so it is just an issue with HubSpot and not another ftp server?

[#6]

Hi pmanca,

thanks for getting back to me!

Regarding the root CA certificates, on my machine they are maintained by the openSUSE package manager. Their last-modified date is 2017-02-21, so I am pretty sure they are up to date. I have the following DigiCert certificate files installed:

-r--r--r-- 1 root root 1350 Feb 21 21:59 DigiCert_Assured_ID_Root_CA.pem
-r--r--r-- 1 root root 1306 Feb 21 21:59 DigiCert_Assured_ID_Root_G2.pem
-r--r--r-- 1 root root  851 Feb 21 21:59 DigiCert_Assured_ID_Root_G3.pem
-r--r--r-- 1 root root 1338 Feb 21 21:59 DigiCert_Global_Root_CA.pem
-r--r--r-- 1 root root 1294 Feb 21 21:59 DigiCert_Global_Root_G2.pem
-r--r--r-- 1 root root  839 Feb 21 21:59 DigiCert_Global_Root_G3.pem
-r--r--r-- 1 root root 1367 Feb 21 21:59 DigiCert_High_Assurance_EV_Root_CA.pem
-r--r--r-- 1 root root 1988 Feb 21 21:59 DigiCert_Trusted_Root_G4.pem

However, I don’t think the root CAs are problematic, because on the FTP control connection the SSL/TLS handshake works flawlessly. Only the handshake on the data connection stalls.

[3PETE]

@RainerKlute I just reached out to the FTP team on our end and they aren’t seeing any issues with making TLS connections and with the handshake specifically.