http 500 error on Contacts and Engagements tables

Developer Support APIs & Integrations
,
1

We are extracting data with cdata ODBC driver to Qlik Sense. Contacts table crash on http 500 error. Other tested tables worked fine 2 weeks ago, however, now Engagements table crash on http 500, too.

Ticket about this (closed): #2377702

2 weeks old logs on Contacts

-----------
2019-03-27T11.50.39.280+02:00 2 [Connection: 1] Page successful: 100 results (488 ms). NextPage: [77300|27626205].
2019-03-27T11.50.39.282+02:00 2 [Connection: 1] [Request] GET https://api.hubapi.com/contacts/v1/lists/all/contacts/all?formSubmissionMode=none&property=company&property=hubspot_owner_id&property=follow_up_status&property=notes_last_updated&property=hs_analytics_source&property=source_campaign&vidOffset=27626205&count=100
2019-03-27T11.50.42.803+02:00 2 [Connection: 1] [Response] HTTP/1.1 500 Internal Server Error, 136 Bytes Transferred
2019-03-27T11.50.42.803+02:00 3 [Connection: 1] [Request]
------------
and then this
------------
2019-03-27T11.50.42.803+02:00 3 [Connection: 1] [Response]
[HTTP Headers]
HTTP/1.1 500 Internal Server Error
Date: Wed, 27 Mar 2019 09:50:42 GMT
Content-Type: application/json;charset=utf-8
Content-Length: 136
Connection: keep-alive
X-Trace: 2B522411F12334764BE791136C8E234357F35357E8000000000000000000
Vary: Accept-Encoding
Vary: Accept-Encoding
Content-Encoding: gzip
Access-Control-Allow-Credentials: false
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct”
Server: cloudflare
CF-RAY: xxxx
{“status”:“error”,“message”:“internal error”,“correlationId”:“4ff92138-f0a2-43f4-a49c-7399b617d0af”,“requestId”:“da9e8aaa17d27c5466c36908eee9c594”}
2019-03-27T11.50.42.805+02:00 0 [Connection: 1] Error: internal error.

And a fresh one from Engagements

-------
2019-04-09T22.29.31.928+03:00 3 [Connection: 1] [Response]
[HTTP Headers]
HTTP/1.1 200 OK
Date: Tue, 09 Apr 2019 19:29:31 GMT
Content-Type: application/json;charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Trace: 2B62E86316EF868E701CA6BF00114EA0A7F8034C0F000000000000000000
Vary: Accept-Encoding, User-Agent
Content-Encoding: gzip
Access-Control-Allow-Credentials: false
Access-Control-Allow-Methods: GET, OPTIONS, PUT, POST, DELETE, PATCH, HEAD
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Accept-Charset, Accept-Encoding, X-Override-Internal-Permissions, X-Properties-Source, X-Properties-SourceId, X-Properties-Flag, X-Hubspot-User-Id, X-Hubspot-Trace, X-Hubspot-Callee, X-Hubspot-Offset, X-Hubspot-No-Trace, X-HubSpot-Request-Source, X-HubSpot-Request-Reason, Subscription-Billing-Auth-Token, X-App-CSRF, X-Tools-CSRF, Online-Payment-Signing-UUID, X-Source, X-SourceId
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct”
Server: cloudflare
CF-RAY: xxx
dataset xxxx
2019-04-09T22.29.31.950+03:00 2 [Connection: 1] Page successful: 250 results (2263 ms). NextPage: [9250|1024073792].
2019-04-09T22.29.31.959+03:00 2 [Connection: 1] [Request] GET https://api.hubapi.com/engagements/v1/engagements/paged?offset=1024073792&limit=250
2019-04-09T22.29.35.692+03:00 2 [Connection: 1] [Response] HTTP/1.1 500 Internal Server Error, 147 Bytes Transferred
2019-04-09T22.29.35.693+03:00 3 [Connection: 1] [Request]
[HTTP Headers]
HTTP Auth Scheme: 2
User:
Password:
GET /engagements/v1/engagements/paged?offset=1024073792&limit=250 HTTP/1.1
Host: api.hubapi.com
Authorization: Bearer xxxxxx
Accept-Encoding: gzip, deflate
User-Agent: CData Data Provider Engine - www.cdata.com - Accepts: gzip
Cookie: __cfduid=xxxx

2019-04-09T22.29.35.693+03:00 3 [Connection: 1] [Response]
[HTTP Headers]
HTTP/1.1 500 Internal Server Error
Date: Tue, 09 Apr 2019 19:29:35 GMT
Content-Type: application/json;charset=utf-8
Content-Length: 147
Connection: keep-alive
X-Trace: 2BF8F089D6EBF9DA8611356BAA0645A09B06365D00000000000000000000
Access-Control-Allow-Credentials: false
Access-Control-Allow-Methods: GET, OPTIONS, PUT, POST, DELETE, PATCH, HEAD
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Accept-Charset, Accept-Encoding, X-Override-Internal-Permissions, X-Properties-Source, X-Properties-SourceId, X-Properties-Flag, X-Hubspot-User-Id, X-Hubspot-Trace, X-Hubspot-Callee, X-Hubspot-Offset, X-Hubspot-No-Trace, X-HubSpot-Request-Source, X-HubSpot-Request-Reason, Subscription-Billing-Auth-Token, X-App-CSRF, X-Tools-CSRF, Online-Payment-Signing-UUID, X-Source, X-SourceId
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct”
Server: cloudflare
CF-RAY: xxxx
{“status”:“error”,“message”:“internal error”,“correlationId”:“48869ba2-ead3-4d27-8257-13c92df512dd”,“requestId”:“268c115dc183af4c17bfccd932e73dc0”}
2019-04-09T22.29.35.695+03:00 0 [Connection: 1] Error: internal error.

2

Hi, @Purtsi.

Apologies for the delayed response.

Thank you so much for including your support ticket number. That gives me much more context!

I understand that you’re encountering 500 errors while attempting to retrieve contacts and engagements via the CData HubSpot Connector.

I was not able to replicate these errors using the request URLs you included:

  • https://api.hubapi.com/contacts/v1/lists/all/contacts/all?formSubmissionMode=none&property=company&property=hubspot_owner_id&property=follow_up_status&property=notes_last_updated&property=hs_analytics_source&property=source_campaign&vidOffset=27626205&count=100
  • https://api.hubapi.com/engagements/v1/engagements/paged?offset=1024073792&limit=250

Instead, I was able to successfully retrieve those pages of contacts and engagements using your account’s credentials. I ran these tests several times without error.

This indicates that the 500 error is something transient in our stack. Most transient 500 errors are resolved after retrying.

I see that CData support recommended you “configure the driver to automatically retry requests if presented with a 500 error” by adding RetryStatusCodes=500; to the “‘Other’ connection property in your DSN configuration settings.”

In your support ticket, you claim this “did not work.” Can you please describe what you mean in greater detail? When you retried the requests, do you receive back-to-back 500 errors? If so, can you provide logs which show the retry attempts and the HubSpot API responses? In your support ticket and this message, you include the logs for the first GET request and then the first 500 error, but no retries.

I also notice that in your ticket with CData support, you said “If I add top 500,it works.” Could you describe what you mean by that? Are you changing a setting in your ODBC configuration, or are you modifying thecountorlimitparameters, which the HubSpot Contacts and Engagements endpoints use?

You also mention that you “tried to increase the timeout in the ODBC config, but it did not help.” What did you increase the timeout to between retries? Were you still receiving back-to-back 500 errors?

I see that CData also recommended you “manually page after the error is thrown,” however you said this too “did not work.” Please expand on this. Can you provide example logs showing the initial GET request, the first 500 error, and then another GET request using a different vidOffset (for contacts) or offset (for engagements)?

I look forward to getting to the bottom of this!

3

Not sure what was the problem, however, it is working now. I just installed the 2019 version of the CData driver that was released few days ago.

4

Glad to hear it, @Purtsi!