topic Re: Seeking Community Input: Reverse Proxy Issue in CMS Development

Seeking Community Input: Reverse Proxy Issue

TesaReichts — Mon, 03 Jul 2023 15:34:58 GMT

Hi all,

we are dealing with recurring website accessibility issues due to our reverse proxy set up. At this point, I need to share my frustration and would appreciate your input.

Background:

Our website is hosted on HubSpot CMS and we are using a reverse-proxy server. It works the way it's supposed to most of the time. Except when there are changes made by HubSpot which disrupt the reverse-proxy redirection, leading to inaccessibility of our website.

For example, the certification process was adjusted in January, but we did not know this and this blew up the whole set up. In theory, it's not a problem because we can make changes, but we didn't know we had to change anything until it was too late.

The support team has provided limited assistance in the past, since there is no support for this feature. Which is fine, we don't really need support, just a heads-up when there are changes.

Most recent instances: 30.06.2023 and 25.01.2023
Result: Website was inaccessible for most of the day, in both cases.
Fix: Delete everything and add it again (in HubSpot and on our own Server) following the changed tutorial https://developers.hubspot.com/docs/cms/developer-reference/reverse-proxy-support.

Does anyone have the same issue?

Is there anything we can do to prep for this in advance?

I am in contact with customer success and was in contact with support in the past, they always tell me they forward the issue of lacking communications, but nothing changed, and I really don't want to go through this again in 6 months.

Re: Seeking Community Input: Reverse Proxy Issue

Jnix284 — Mon, 03 Jul 2023 20:13:14 GMT

@TesaReichts can you share more details about the type of changes that were made on the two most recent instances that caused the break to happen?

There is a lot of communication from the development side, I'd like to see if those changes are included in any of the existing communications or specific to revers proxy only.

Re: Seeking Community Input: Reverse Proxy Issue

TesaReichts — Wed, 05 Jul 2023 12:31:15 GMT

@Jnix284New verification steps have been added for the externally hosted SSL certificates. Since my colleague took care of the whole thing, I can't tell you 100% what changed in January. The developer documentation changed already in October but the break happened in January. With the last incident, we didn't search anymore but just redid everything, hoping to have our site back online as quickly as possible which worked. That's why I asked here whether this is something that has happened to others too.

Re: Seeking Community Input: Reverse Proxy Issue

TesaReichts — Thu, 28 Sep 2023 16:14:50 GMT

Good news! We have found a solution.

Unfortunately, the problem with our website being unreachable via the reverse proxy occurred again today.

It's been exactly 60 days since the last time the site was unreachable, and we now know why.

CAUSE: We use a Let's Encrypt certificate for SSL encryption. "A Let's Encrypt certificate is automatically renewed after 60 days ..."

Due to the automatic renewal of the certificate, it needs to be re-verified manually in HubSpot in the domain linking process. This verification was added to HubSpot settings earlier this year (Jan/2023). Since then, here's what's been happening: If Let's Encrypt updates our certificate, we go offline with our HubSpot pages around 8h later.

SOLUTION: We delete the domain in HubSpot and completely reconnect it following the documentation. Best case scenario, right after the certificate is renewed and before the site goes offline on its own.

I assume we are not the only HubSpot customers using a reverse proxy setup with an auto-renewing SSL certificate from Let's Encrypt, as this is pretty much the current security standard.

I hope our experience helps others, so you can be spared the three-month heart attack.

We have forwarded the problem to our Success Manager so that an automated solution can be found by HubSpot Product Development.

Re: Seeking Community Input: Reverse Proxy Issue

Jnix284 — Thu, 28 Sep 2023 17:52:48 GMT

Thank you for sharing this update @TesaReichts, I'm glad you were able to get to the bottom of the issue and hopefully find a better working solution soon!