Missing custom modules in widgets for pages and blog posts

[KWęgrzynowicz]

Hi!

We're using the V2 of the Hubspot API. We have a client that is using custom modules (widgets) which are created by some other entity. Our part is to make sure that we can download all the contet through the API and do something with it (doesn't matter what).

We're facing a problem that when they use some particular modules, we don't receive them in a page JSON data when calling the "GET /content/api/v2/pages" or "GET /content/api/v2/pages/:page_id" endpoints.

We expected to get all possible modules in the "widgets", "widget_containers", "widogetcontainers" or "layout_sections" sections of the JSON but they aren't there!

Our questions are:

• Is it a bug in the API? Would switching to V3 help?
• Is it related how the custom module was built? If so, what controls whether module is visible in the API?
• Something else, entirely?

I'd like to stress that we're not Hubspot experts. We rely on the API documentation and we couldn't find any other endpoint related to getting a full page content. Also, we expect to get all possible content when calling the "Get a page by ID" endpoint.

Just as an FYI, I can't provide you with the code of the custom modules, because it's a third party that is creating and managing them.

Any hint will be helpful and appreciated 🙂

Thanks,

Kamil

[EStill]

I think CMS has limited option but you should try your best on your client website  and let us know. So this may be help us in future.

[Willson · Accepted solution]

Hi @KWęgrzynowicz ,

I believe your CSM should have been in direct correspondence with you last week regarding the final say for this endpoint.

To reiterate and to add a solution here, both the current v1 and future v3 versions will not support what your current goal however, in speaking with the team to initially address this roadblock, we've added this as direct feedback for future consideration. As per a previous recommendation, i'd suggest exploring the use of our CMS-CLI as a potential workaround. 

Regarding a call with the API team, this is not something we offer in HubSpot - All queries when it comes to using our APIs and needing support with them should be routed to our Support Team. 

Thanks,

Matthew

[KWęgrzynowicz]

Hi Wilson,

Thanks for taking the time to write to follow up on the subject. We now have a clearer picture of the API and what can and cannot be done.

So far we haven't found the best approach but we'll experiment with some options, like the CLI tool mentioned by you. Still, I consider it to be in the realm of workarounds, not a proper solution. 

Kind regards,

Kamil

[Willson]

Hey @KWęgrzynowicz ,

Thanks for getting back to me on that! After some digging, I can confirm that this scenario you're seeing is expected given the way this is currently built. 

To provide more context, the content object only stores edits/overrides to widgets in the `widgets` collection. If a post isn't edited from the default template then in most cases this collection is empty so that future template changes will flow through automatically to posts. 

I hope this helps to clarify what you're seeing.

Thanks,

Matthew

[KWęgrzynowicz]

Hi!

Thanks for getting back on this.
I see your point about the template and I understand the reason behind it.
But I'd like to challenge that.

As a developer consuming your API, I expect to get the full information
about a page, blog post, etc. when making a call. So, even if internally
you store the page data itself separately from its template, I'd like to
get the computed data that is consistent with what the editor sees in the
UI. Meaning, page plus template data.

If I get this right, to get the full page data, I have to fetch the page,
then fetch the template and then combine those 2 together?
I wonder how will it work if I then apply translations on that page. Would
Hubspot API accept it?

Are module IDs the same in a template and a page?

Kind regards,
Kamil

[Willson]

Hey @KWęgrzynowicz ,

Thanks for the feedback here! I want to point out that we have an API in development at the moment that may be resolving this roadblock:

https://developers.hubspot.com/docs/api/cms/pages

I am conferring with the team responsible for this API to get a better idea. Thanks

[KWęgrzynowicz]

Hi Wilson,

We can't really use an API that's in active development as we can't trust that the contract won't change. Please provide us with the roadmap for the new API version.

Can we have a call with someone from the API team directly? This issue is going on for a very long time and we still haven't made any real progress.

Kind regards,

Kamil

[KWęgrzynowicz]

Another question: how to get template ID from a page? Again, using the V2 of the API when I fetch a page I don't see the template ID field. I only see template path, which I can't really use. I need the ID, though.

EDIT
Now I see that I can filter the get template list endpoint by path, so I can get the actual template by path.

Yet another question: even if I get the template ID, will I even get the default modules data? I fetched a list of templates using the GET /content/api/v2/templates endpoint and I don't see any "widget" section in there. I selected a random template, copied its ID and then used it with GET /content/api/v2/templates/:template_id endpoint hoping to get more data. Sadly, the data is the same as with template list and I don't have modules data.

[Willson]

Hey @KWęgrzynowicz ,

Thanks for the detailed write-up here on this behaviour that you're seeing. Unfortunately I am not able to speak to whether or not this is working as designed at the moment as this is not an API I frequent in my day-to-day. 

With that being said, I have opened this question up internally to get a better idea of what to expect here when working with this API so that we can clear this confusion up - A quick follow-up to this, do we know if the page in question was published or was it in draft mode at the time of fetching the data?

Many thanks!

[KWęgrzynowicz]

Hi @Willson,

The page in question was in a draft mode at the time of fetching the data (and still is). Do you think it's related? As far as I know we're having this issue with published pages as well. But I'll double check and update this post with a follow up.

Thanks,

Kamil

[KWęgrzynowicz]

Hi Lynton,

Thanks for replying. I already contacted once the Hubspot support but they told me to reach out on developers' forums. I'll contact them again either way, because I'm blocked anyway.

Thanks,

Kamil

[KWęgrzynowicz]

Hi,

Sadly, it's hard for me accept this answer. Since my posting of the problem we've found out about this weird behavior of the API. Hopefully it's not going to be a too long post.

We have create our own custom module for testing. We used this tutorial: https://knowledge.hubspot.com/design-manager/create-and-edit-modules. We haven't downloaded it, all configuration was done in the Hubspot UI.
We have created our own template by cloning an existing one and we added the our module to that template. The base template was "Landform - Landing Page".

Next, we have created a Landing Page based on that custom template.  The template has 3 custom modules: "Landform - Blockquote Background image", "Landform - Hero Background with Form" and "Image Slider" made by us in Hubspot UI.

For a test, we have made an HTTP call: GET /content/api/v2/pages/:page_id. We just wanted to see how the page JSON looks like when you create page but do NOT change anything. No updates to any content. Only default content that comes with a template and modules.

The result was that the page JSON doesn't have any widgets! the "widgets" property is empty!

Here's the JSON (will expire in 1 month): https://pastebin.com/LEvdjmHg

Then we got back to the page and started to make some updates to texts. We've updated some modules and some not. Then we made another call to get page JSON data. This time, it turned out that got widgets but ONLY those which we've updated.

JSON (will expire in 1 month): https://pastebin.com/pb2yPKAp

Can someone explain this quircky behavior, please? Why the API behaves so strangely? 

We have a customer who's using many custom modules. They have thousands or pages and blog posts. We can't just ask them to visit every single page and post and update every single module. It there a solution for this problem? A workaround of sort?

Kind regards,

Kamil

[lynton]

Kamil,

I'm not sure what the support policy is for issues like this, but I'd suggest escalating this as a support ticket to see if the HubSpot team can give you a more direct answer. I agree it is a strange behavior by the API, but it might be by design. If there are no edits in the module itself, the software may have no reason to store the module data at the page level. Unfortunately, there may not be a workaround or solution available for the issue you're experiencing. A conversation with HubSpot support would help provide confirmation.

[lynton]

Kamil,

My guess is that there's a limitation in the CMS api that isn't returning the custom modules you're looking for. Unfortunately, there may not be a workaround for this until the new v3 api is released.

[dennisedson]

Hey @KWęgrzynowicz 

Would really love to see what is in that module! 

The v3 endpoint is being developed so you cannot use it yet. 

In theory, everything should be in the layout_data or widget_containers...

@lynton , unsure if this api is in your wheelhouse, but thought I would ask if you have any experience here