@PatrickBakker - I feel your pain... I have experienced the same issues.
I also love the irony and chuckle every time I realize that Page Speed Insights (by google) increases by 40% when you disable recaptcha (also by google).
Unfortunately, the only thing I have come up with is either submitting the form via the hubspot cookie and using a plain html js form / using the HS API, or removing forms from headers and footers of a site that require recaptcha. I'm okay with the "contact" page being slow but, not the entire website.
I'd also recommend upvoting this Idea in our Ideas Forum regarding this. You can also subscribe to the idea (Options > Subscribe) to get notified if the status changes.