topic Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!) in CMS Development

Tutorial: Setting up a Reverse Proxy With HubSpot CMS (with sample nginx & Cloudfront configs !)

robertainslie — Mon, 27 Nov 2023 01:58:39 GMT

Note: The purpose of this post is to share lessons learned about setting up a reverse proxy with HubSpot CMS Enterprise. If you have experience, we invite you to share on this thread!

HubSpot CMS Enterprise allows for content to be served through a reverse proxy. This implementation requires significant technical know-how and while possible, is not a recommended standard implementation.

The core documentation for the HubSpot side implementation is available here.

While not a recommended setup due to the trade-offs and technical expertise required, there are some use cases where setting up a reverse proxy makes sense and is technically supported by HubSpot CMS Enterprise. The primary use case for implementing a reverse proxy with the HubSpot CMS acting as a content source is when a website serves a web application from the same domain as the marketing website. For example, if you offer a web application at www.domain.com/login and also host a marketing content site at www.domain.com, a reverse proxy can route traffic to your web application servers on certain subdirectory paths, and to HubSpot CMS on other subdirectory paths.

HubSpot’s Professional Service’s Technical Consultants have aided customers in this implementation and hope to pass along a few lessons, including a sample nginx server configuration sample. (Note: HubSpot Services teams cannot provide recommendations on the ‘external to HubSpot’ aspects of this setup and we are sharing this information to document this publicly for others to learn from. Usage and implementation is your responsibility). If you do need guidance on the HubSpot setup, we can work on a project with you - just reach out to your Customer Success Manager or Sales representative.

Lesson One: The domain in your HubSpot Domain Manager should match the domain being requested/served through your reverse proxy.

However, you do not need to directly connect / update your DNS settings to connect this domain to HubSpot’s servers. Instead, when a domain is securely served on an existing content host, HubSpot provides a TXT and CNAME record to add to your DNS that verifies domain ownership and pre-provisions an SSL certificate in the background. When implementing your reverse proxy, remember that the origin connection is a HubSpot CNAME and the Host and X-HS-Public-Host headers match the "added to HubSpot but not directly connected" HubSpot domain where content will be served from. The Host domain is added to HubSpot Domains so that the HubSpot systems are ‘aware’ of the domain, but that domain is never directly pointed to HubSpot in your DNS zone file.

Lesson Two: There are multiple ways to setup a reverse proxy server. Regardless of the approach, the implementation should always be in a load balanced environment so that traffic from your proxy rotates requests to the HubSpot CNAME origin from multiple IP addresses.

The two primary ways seen in our experience are using a managed CDN (like Amazon Web Services Cloudfront) or with a custom nginx webserver deployed on load balanced cloud infrastructure (such as Amazon Web Services EC2/ELB or Microsoft Azure). (Note: Due to HubSpot utilizing Cloudflare as it’s CDN, Cloudflare cannot be used in this setup).

Lesson Three: When working with nginx, there are several headers and settings required to route traffic in an expected fashion. Below is a sample nginx location configuration file. Note: this is not guaranteed to work, and is a sample starting point. This is also a snippet of the entire configuration, highlighting location routing settings.

Due to the SNI (Server Name Indication) connection process that establishes a secure connection between your proxy and HubSpot’s servers, the proxy connection and content host domain are different (one is a HubSpot provided CNAME and one is the domain the content should be served from and is added to the HubSpot Domain Manager). To enable this in your nginx proxy, ensure that you include two important settings. These instruct nginx to send the server domain name with the SNI SSL connection handshake since the origin domain is different from the Host.

proxy_ssl_name www.example.com;
proxy_ssl_server_name on;

Here are two location paths that set each of the required and recommended settings in an nginx configuration file. Inspect each included value and what needs to be customized for your implementation. We encourage you to review the nginx documentation for settings such as securing upstream traffic, proxy_ssl and $proxy_protocol_addr

location ~ ^(/|/some-other-path) {
    proxy_set_header Host $http_host;
    proxy_set_header X-HS-Public-Host www.example.com
    proxy_pass_request_headers on;
    proxy_set_header X-HubSpot-Trust-Forwarded-For true;
    proxy_pass https://2XX93.sites-proxy.hscoscdnXX.net;
    proxy_ssl_name www.example.com;
    proxy_ssl_server_name on;
    proxy_set_header   X-Real-IP $proxy_protocol_addr;
    proxy_set_header   X-Forwarded-Proto  $scheme;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-HubSpot-Client-IP $proxy_protocol_addr;
  }

location ~ ^(/hs|/_hcms|/hubfs|/hs-fs) {
    proxy_set_header Host $http_host;
    proxy_set_header X-HS-Public-Host www.example.com
    proxy_pass_request_headers on;
    proxy_set_header X-HubSpot-Trust-Forwarded-For true;
    proxy_pass https://2XX93.sites-proxy.hscoscdnXX.net;
    proxy_ssl_name www.example.com;
    proxy_ssl_server_name on;
    proxy_set_header   X-Real-IP  $proxy_protocol_addr;
    proxy_set_header   X-Forwarded-Proto  $scheme;
    proxy_set_header   X-Forwarded-For    $proxy_add_x_forwarded_for;
    proxy_set_header   X-HubSpot-Client-IP $proxy_protocol_addr;
  }

We hope this is helpful for any HubSpot customers who are interested in setting up a reverse proxy with HubSpot CMS. HubSpot Technical Consulting services can help provide guidance on the HubSpot components of this setup. If interested, please reach out to your main HubSpot point of contact or reach out via the form on this page.

If you have other advice or lessons learned with a reverse proxy setup with HubSpot, please share them for the community to benefit from!

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Tue, 26 Oct 2021 13:24:32 GMT

Relating to Lesson 1 - I wanted to show some images around what this actually looks like in HubSpot and a CDN acting as a proxy. The most important aspect of this - your active and live domain that users will see is also the domain you add to HubSpot but never ACTUALLY connect in your DNS. Instead, your proxy performs a request to the provided CNAME.

I've set up a proxy for a domain - tc.robertpainslie.com that points to an AWS Cloudfront CDN distribution. Depending on the URL path, it either proxies to an AWS S3 bucket or it proxies to HubSpot CMS.
So,tc.robertpainslie.com/ shows a very simple web page originating from an S3 bucket and tc.robertpainslie.com/proxy-page shows a page served from HubSpot CMS

The pathway is:
Internet > tc.roberpainslie.com/proxy-page > AWS Cloudfront > HubSpot CMS
OR
Internet > tc.roberpainslie.com/ > AWS Cloudfront > AWS S3

1. In Cloudfront, I have an Origin that points to the proper HubSpot CNAME as directed by the HubSpot reverse proxy documentation
Then, there are Cloudfront behaviors that dictate which Origin is used for which paths:

2. In HubSpot, I have the domain tc.robertpainslie.com added to my Domain Manager - this is the exact domain I want to proxy! DO NOT create a new subdomain that is added to HubSpot - you want the initiating domain to be the exact same that you add to HubSpot. When connecting, the domain should already be live and hosting content over a secure connection. When this happens, HubSpot will detect the existing secure connection and ask you to pre-provision SSL - take the CNAME and TXT record and verify in your DNS. HubSpot provisions an SSL cert for the 'tc.robertpainslie.com' domain in the background even though this domain will never ACTUALLY be connected to HubSpot (because it will remain connected to your proxy).

You'll see in the above screenshot that it says 'Connected via reverse proxy' - this will eventually show up (after successful requests are proxied and may take a few weeks to appear). But, it will initially show 'Not connected'. To proceed with the process, for the specific domain:
- Edit > under, 'Edit publishing status' > Mark as 'Ready for publishing'
- Under Domain Security Settings > 'Require HTTPS'

and

3. Now, go to the HubSpot CMS page publisher and create and publish a page on the domain - (in my case, tc.robertpainslie.com/proxy-page).

To get traffic to this page, a request routes to your proxy, and then you forward the request to the appropriate HubSpot CNAME. HubSpot CMS then looks up the appropriate page based on the domain and path in the HOST header, and boom - it routes back through your proxy and to the originating requester

Hope this helps anyone who is struggling with this setup

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Tue, 26 Oct 2021 23:13:22 GMT

One additional item that might help - if you were attempting to perform the same type of lookup using curl as the proxy (using the HubSpot CNAME), it would look like this:

curl https://tc.robertpainslie.com/proxy-page --connect-to ::XX.sites-proxy.hscoscdnXX.net

Compare to this (to replicate what won't work):

curl https://XX.sites-proxy.hscoscdnXX.net/ -H 'Host: tc.robertpainslie.com/proxy-page'

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

Maicon — Mon, 18 Apr 2022 21:02:21 GMT

You're amazing! Thank you so much for the extra details, I couldn't finish the setup just looking in the official page. It worked perfectly using AWS Cloudfront.

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Mon, 18 Apr 2022 21:56:03 GMT

You're welcome!
If you have any additional insights or want to comment on the specifics of your implementation it would help make this post stronger!
Also, let me know what was unclear in the official documentation - I've been working with HubSpot's Tech Writing team to improve the reverse proxy documentation.

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

PTrojanowski — Fri, 26 Aug 2022 10:08:11 GMT

hi, I have a question becouse you type: "you want the initiating domain to be the exact same that you add to HubSpot". Then it is not possible to create a proxy if I have a subdomain at HubSpot (blog.mydomain.com) already and my main domain is "mydomain.com" ?

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Fri, 26 Aug 2022 14:03:32 GMT

@PTrojanowski In this situation, I'll assume the following:

You have an externally (non-HubSpot) hosted website at 'mydomain.com'
You have an existing blog hosted with HubSpot at blog.mydomain.com
You want to have traffic go to mydomain.com/blog/{article} and that content to be your HubSpot hosted blog origin

Assuming that's the desired behavior, you would need to (at a high level):

Have the HubSpot CMS Enterprise product tier
Follow the instructions to 'add' mydomain.com to HubSpot, but not ACTUALLY update your DNS to point the domain at HubSpot (it should point to your proxy server)
Once SSL is pre-provisioned in HubSpot, make mydomain.com/blog the new primary domain & sub-directory of your HubSpot hosted blog in HubSpot
Have your proxy point traffic at HubSpot for the specific routes
Set up redirects for all traffic from blog.mydomain.com/{article} to mydomain.com/blog/{article} so that old traffic routes to the proxied domain

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

PTrojanowski — Wed, 31 Aug 2022 11:47:56 GMT

Hi @robertainslie, thanks for the answer!

I understand that "HubSpot CMS Enterprise product tier" is necessary to have this?

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Wed, 31 Aug 2022 13:36:02 GMT

@PTrojanowski - correct - reverse proxy setup is only available as part of a HubSpot CMS Enterprise tier.

There are obviously some other great features part of that product tier as well (one of the most powerful is access to CMS serverless functions, allowing for dynamic/data driven pages/light weight web apps).

If you're wanting to move forward with this project but not sure on your product tiers, shoot me a DM with your HubSpot Hub ID and I can connect with you, and loop in your Customer Success Manager or account executive and we can chat more about if this is the right fit to move forward with.

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

VIacob — Wed, 15 Mar 2023 17:45:11 GMT

Hi @robertainslie,

Can you make a print screen on the Behaviour configuration? I'm not able to make it working.

Thank you

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Thu, 16 Mar 2023 16:14:36 GMT

@VIacob - there's likely several ways you can set the behaviors to make it work, but here's what I have in my test setup. I'm including screnshots of the origin setup and a specific behavior.
Origin:

Behaviors:

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

matt_scott — Fri, 02 Jun 2023 11:44:16 GMT

Thanks for your help I thought i'd point a couple of extras from when we were getting ours setup we were following this but also found a couple of extra paths that should be useful

Ideally, all paths under your domain should proxy to HubSpot. If that's not the case, then the following paths must proxy so assets load properly from your domain: /_hcms/*, /hs/*, /hubfs/*, /hs-fs/*, /cs/c/*, and /e3t/*.

/events/public/* - This is for redirects in emails to work correctly
/___context___/* - This allows devs to use the developer info page

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

AAlsalkhadi — Tue, 06 Jun 2023 09:20:44 GMT

Hallo ,

I hope this message finds you well. I am currently in the process of reverse proxying a subdomain (client-domain-portal.rexx-systems.com) to another domain (client-domain.com) and have encountered some difficulties that I hope you can assist me with.

For clarity:
- I have used "client-domain-portal" as a placeholder for the actual subdomain name in this context.

- rexx-systems.com is a job portal and is an third party service provider. client-domain.com uses its services.

Our primary domain, client-domain.com, is hosted on HubSpot. The challenge arises when I attempt to reverse proxy from "client-domain-portal.rexx-systems.com" to the HubSpot-hosted domain, as these two domains are distinct. My initial assumption was that it might not be feasible to reverse proxy "client-domain-portal.rexx-systems.com" to a subdomain of "client-domain.com" (like jobs.client-domain.com).

To troubleshoot, I tried to add the equivalent subdomain (client-domain-portal.client-domain.com) to the HubSpot Domain Manager without directly connecting it. Unfortunately, this attempt wasn't successful. I also attempted to add "client-domain-portal.rexx-systems.com" as suggested on Hubspot offical documentation for reverse proxy support. In this documentation: I read that the "added to HubSpot but not directly connected" domain must match the domain being requested through the reverse proxy, but this too was met with the same hurdle.

Considering the above, I would greatly appreciate if you could provide guidance on how to structure the Nginx configuration file in this particular scenario and clarify which domains are permissible to add.

For your reference, the HubSpot CNAME is something like XXXXXX67.sites-proxy.hscoscdn10.net.

The following is a summary of how I understand it, please correct me, if I am wrong:

Domain ownership verification via pre-provisioning an SSL certificate is a crucial step: no direct domain connection ==> only the "pre-provisioning DNS records" have to be added to the DNS provider.
the origin connection (proxy connection) is a HubSpot CNAME.

Host = X-HS-Public-Host = the content host domain "added to HubSpot Domain Manager but not directly connected" = HubSpot domain where content will be served from. ==> HubSpot systems are ‘aware’ of the domain.

Based on that

server { location /karrieres { proxy_set_header Host $http_host; proxy_pass_request_headers on; proxy_pass https://XXXXXX67.sites-proxy.hscoscdn10.net; proxy_set_header Accept-Encoding ""; ssi on; proxy_set_header X-HS-Public-Host https://client-domain-portal.rexx-systems.com; proxy_ssl_server_name on; proxy_ssl_name https://client-domain-portal.rexx-systems.com; proxy_set_header X-HubSpot-Trust-Forwarded-For true; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-HubSpot-Client-IP $proxy_protocol_addr; } }

Should I change $http_host to the content host domain?

Is it right to have X-HS-Public-Host and proxy_ssl_name set to the same domain?

For testing purposes, I am using an Nginx Docker image for reverse proxying. I am wondering if this could be the source of the current issues? It is worth noting that the reverse proxy operates as expected when I direct client-domain-portal.rexx-systems.com to localhost.

My initial plan was to deploy the Docker image on Google Cloud Run. However, based on some recent findings, I am now considering Amazon CloudFront or AWS EC2/ELB as possible alternatives. Google Cloud CDN or Google Cloud Load Balancing are also on the table, but cost considerations will be a deciding factor.

If anything is unclear, please do not hesitate to ask me.

Thank you in advance for your time and assistance. I look forward to your expert guidance on this matter.

Best regards,

Abdalrohman Alsalkhadi

@robertainslie

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

AAlsalkhadi — Mon, 12 Jun 2023 13:31:36 GMT

Hey, @karstenkoehler, @LMeert @coldrickjack @nikodev, @Kenan , @hhawk , @michelle24,

do you have any experience with reverse proxies?

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Fri, 30 Jun 2023 17:56:48 GMT

Hey @AAlsalkhadi - I replied to the other post you created here. Apologies for the delay, I've been out of office - https://community.hubspot.com/t5/CMS-Development/Issue-with-using-Reverse-Proxy-with-Hubspot/m-p/806029#M34595

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

aitortomas — Mon, 28 Aug 2023 10:24:19 GMT

We are struggling with this part:

" Instead, when a domain is securely served on an existing content host, HubSpot provides a TXT and CNAME record to add to your DNS".

Whenever we register our domain which is already in use and provisioned with a SSL certificate, we do not get a TXT and a CNAME record, we get two TXT records from Hubspot and one of them is an ACME challenge (_acme-challenge.www) and the other one is a custom hostname (_cf-custom-hostname.www).

Why could it be that that it does not provide a CNAME record?

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

AAlsalkhadi — Mon, 28 Aug 2023 12:03:52 GMT

Hey @robertainslie,

thank you very much for answering my question.

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Mon, 27 Nov 2023 01:20:01 GMT

There is a new domain connection process in HubSpot to better indicate the steps necessary when adding a domain to the HubSpot domains tool for use in a reverse proxy. I'll include several screenshots in this post and also a follow up reply that shows steps in Route 53 (AWS DNS service). I've also added several steps and notes in this reply post that show many of the necessary AWS Cloudfront steps

Connect a domain in HubSpot.
You likely want to pick the option to 'connect a secondary domain". Also, DESELECT the option for "Connect with HubSpot's built-in content delivery network"
Add the domain that will be the LIVE domain for the content. This domain is likely already live with content. You are NOT updating your DNS to point your A record or CNAME at HubSpot.
Instead, in this step, you're telling HubSpot the domain to expect traffic from, so HubSpot can provision an SSL Cert, verify you own the domain and add it into the system to know how to serve content for the traffic
.
No major changes needed on this screen, you can likely just click through
Note: my screenshots are showing a subdomain (rp.mulligandevelopment.io), you can also use a root domain (i.e. domain.com; or in my example mulligandevelopment.io) if that's what you've connected in the previous step.
No changes here, click on through
This is the MOST IMPORTANT step: HubSpot provides you 2 DNS records. Go to your DNS provider and create 2 records: a TXT and a CNAME. These are used for domain validation and provisioning of a certificate. DO NOT SKIP THIS STEP! REPEAT: DO NOT SKIP THIS STEP
In your DNS, add TXT and CNAME records. I'm using AWS Route 53 DNS. The values are provided to you by HubSpot in the previous step.
Verify the domain connection in HubSpot. (Don't have a screenshot of this step, you can also leave this step). Depending on the state of your proxy, you may need to have a proxy behavior setup to route traffic to your domain. You may also want to skip this step and jump to steps 9 & 10
Note: it can take several minutes (and I believe up to a few hours) to have a certificate provisioned after you have added the DNS validation records.
Navigate back to the HubSpot domains screen and click "Actions" for the domain that you just added. Choose to "Set as Ready for Publishing".
The purpose of this step is to allow you to publish content in the HubSpot content tools even though the domain isn't actually connected to HubSpot, but that WILL be the domain that your user is requesting content on (i.e. Your visitor navigates to domain.com/page, where domain.com is connected to your CDN. Your CDN proxies the request to an origin based on your behavioral rules. HubSpot finds a page on the exact domain and path as requested by the user. This is a non-intuitive step to many folks who are setting up proxies, which is why I'm going into this detail)
Now, to test, create and publish a web page in HubSpot CMS that uses the domain you just added to the domain manager (and validated but did not connect to HubSpot) (my post below shows this)
1. For some use cases, (such as moving a blog from a HubSpot subdomain to your proxied domain, you may also need to update your HubSpot blog to be located on this new domain (do this with caution! I recommend testing with a test blog and a test subdirectory/subdomain first to ensure the connection works properly).
  
  Contact HubSpot Support and ask about how to move a blog from one domain to another domain.
  Note: Reverse proxy setups are not supported by HubSpot Support, but they should be able to provide some instructions for moving content from one domain to another domain)
In your CDN, update to include a new origin and behaviors as described in this documentation that points at the appropriate HubSpot CNAME. Remember that the domain of your CDN (that your visitor is navigating to) should be the EXACT same domain that you added to the HubSpot domains tool and that you also published content on in the HubSpot CMS

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

robertainslie — Mon, 27 Nov 2023 01:29:36 GMT

Here's the Cloudfront Distribution for the domain rp.mulligandevelopment.io
Here's the origin connection to 203693.sites-proxy.hscoscdn40.net --> this is provided to you in the HubSpot documentation here
Here's a sample of behaviors - yours will likely be more complicated than that. You likely need to create several behavioral rules to proxy the following subpaths (noted in the actual documentation): /_hcms/*, /hs/*, /hubfs/*, /hs-fs/*, /cs/c/*, and /e3t/*.
When testing, create a sample webpage in HubSpot that is hosted on the domain you want to proxy:
If you're moving a blog from blog.domain.com to domain.com/blog (which is proxied), you will need to update your HubSpot blog settings to point to this new domain - DO THIS WITH CAUTION! TEST FIRST WITH A TEST BLOG!

Re: Lessons Learned from Setting up a Reverse Proxy With HubSpot CMS (includes sample nginx config!)

aitortomas — Mon, 27 Nov 2023 08:20:53 GMT

Regarding step 6, domain validation, what happens if a CNAME record cannot be added because it is already in use? We had our domain validated by Hubspot via HTTP challenge and now in the reverse proxy verification we get that it fails checking Delegated Domain Control Validation (DCVs). We received also a message saying that the reverse proxy would stop working unless we fix that.