Sep 16, 2021 1:41 PM - edited Sep 16, 2021 1:43 PM
Hello everyone. As a little background, my team is using Airbyte as an ETL tool (previously we were using Stitch) to transfer data to our redshift instance. It has been working very well for over a month now, but between Thursday (9/9) and Friday (9/10) of last week something changed and we began getting ‘414 Request-URI Too Large for url’ errors. One of my team members said the team added around 54 new columns to the contacts properties table but I have no idea if I can verify if that was the only thing that changed. I have spoken with an Airbyte developer directly and done my own research which has made me aware that this is a known error that can come up. Also, Airbyte does not allow me to select specific properties from the contacts table, it just tries to pull everything it sees.
We attempted to archive columns in the Hubspot contacts properties table, and even exceeded the 54 columns that were added, but we are still receiving the same 414 error in Airbyte. The character count of the Request-URI is currently 16705 characters. Doing some testing on my end (outside of our ETL tool) I was able to figure out that the sweet spot for the 414 error is 16391 characters. At 16392 characters the hubspot api ignores the request with the 414 error.
Is there any way to get this character limit extended? My team is adament that they cannot remove any more columns. They also mentioned that in the future our column length will grow substantially.
Sep 17, 2021 11:20 AM
I do not know of any talk on extending that character limit.
A possible work around would be to use the CRM Search endpoint and add each of the properties that you want to collect in the properties array. EG:
"properties": [ "prop1", "prop2" ]
This way you are not extending the length of the URI.
Hope this helps!