Half of API calls end up in Exception: Broken pipe
Sep 17, 2019 4:38 AM
Please advice on Exception that happens every second/third API (HTTP) call that we have:
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://track.hubspot.com/v1/event/": Connection reset; nested exception is javax.net.ssl.SSLException: Connection reset at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:744) [...] Caused by: javax.net.ssl.SSLException: Connection reset at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) [...] Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
So, the root cause is 'Broken pipe'. Usually, this happens when a connection was closed on one of the ends and somebody tries to write into this connection.
On our side, we use Spring's RestTemplate that doesn't use any pool of connections and starts each time a new connection (so there is nothing to be closed on our side).
I saw this happened quite regularly for us on different API calls. At least it happened on:
1) HubSpot Events HTTP API: https://developers.hubspot.com/docs/methods/enterprise_events/http_api
2) Create or update a contact: https://developers.hubspot.com/docs/methods/contacts/create_or_update
3) Update email subscription status for an email address: https://developers.hubspot.com/docs/methods/email/update_status
We developed a workaround for this: each call we wrapped into retry block that does up to 3 attempts to succeed with the request. After that, the error disappeared (actually, we simply don't log the error if we managed to succeed in 3 attempts).
However, could you suggest where the issue is?
Additionally, please note that:
1) we did this on a test account
2) we use apikey integration (oauth isn't completed yet)
3) we have around 100-200 API over HTTP requests per day