We’re looking at running a web scraper for a site our company owns to pull rates for a table. We need to run the web scraper every morning and need to see if there’s a way to automate this process.
Hello @VCordry thank you for posting in our Community.
Let’s invite our top experts to this conversation @alyssamwilie @MatthewShepherd any recommendations for the @VCordry matter?
Thank you,
Pam
Hi @VCordry
There are a lot of factors to determine the best path here.
- Where do you plan to host this web scraper?
- Is the web scraper already created?
- What programming language does the web scraper use, and which libraries does it rely on to function?
- Where is the output data going to be stored in Hubspot?
- Do you have Ops Hub and HubDB?
In most cases, it would likely be best practice to host the web scraper on an external platform, scheduled to run at a certain time each day, then push the data into Hubspot via API.
If it’s a very simple web scraper, and it can function using the Node.js or Python libraries supported by custom workflow actions then you may be able to trigger a workflow every morning to run the code.
- Zach
The language would be JavaScript and the output data would be sent to a HubDB Table via the API. We might be able to host the code externally but we’re still working with our IT team to do that. Are there any recomendations for hosting sites? Google Cloud or AWS? We’re also working with our IT team to get Node.js installed, hasn’t happened yet though.