Fetching larger HubDB tables using client side Javascript

[SteveHTM]

Thought I would post this here for teh community given that it involved quite a bit of head scratching.

The context is that I'm using HubDB data as a source to auto-suggest and  validate a specific form field via a var list[].  Life was relatively simple until I reached the single API call limit of 1000 data records. At that point I needed a rethink, to upgrade to V3 API, and learn JS promises and async functions.

Below is the code that I came up with and is now working - maybe it can be improved?

I'll post the overall auto-suggest approach elsewhere as it builds significantly on the base form customization ideas at How to customize the form embed code (hubspot.com)

Steve

  var list = [];
  var hubdbURL ="your HubDB API reference URL without credentials";
  // Cunning async promise await stuff
  async function getHubdbPage(aURL) {
    let after ="";
    console.log("This URL: ", aURL)
    let pagePromise = new Promise(function(resolve,reject) {
      let req = new XMLHttpRequest();
      let afterThis = "";
      req.open('GET', aURL);
      req.onload = function() {
        if (req.status != 200 ) {
          console.log("Error: ", this.response);
          reject("Fatal HubDB error");
        }
        else {
          // Success!
          var data = JSON.parse(this.response);
          // V3 API structure
          var tableArr = data.results;
          if (data.total > tableArr.length+list.length )  {
            afterThis = data.paging.next.after;
          }
          console.log("Page here: ", this.status, data.total, afterThis);
          //console.log("Len, Row[0]: ", tableArr.length, tableArr[0].values);
          tableArr.forEach(function(obj) {
            // console.log(obj.values);
            if(obj.values["name"] != null) {
              var name = obj.values["name"];
              // push to names array
              list.push(name);
            }
          });
          // resolve promise with indicator of need for another call
          resolve(afterThis);
          }
      };
      req.send();
    });
    //console.log("Before resolve: ", after);
    after = await pagePromise;
    //console.log("After resolve: ", after);
    return(after);
  } 
   
  // need to get this into async loop somehow...
  async function getAllRecords() {
    try {
      // if no error we see if there is data remaining and then call again
      after = await getHubdbPage(hubdbURL);
      console.log("1st async call: ", list.length, after);
      while (after.length > 0) {
        let afterURL = hubdbURL + "&after=" + after;
        after = await getHubdbPage(afterURL);
        console.log("Follow-on async call: ", list.length, after);
      }
    } catch(error) {
      console.log(error);
    }
  }
  // lets do it!
  getAllRecords();

[Jaycee_Lewis]

Hey, @SteveHTM 👋 Nice work on getting some working code set up! It sounds like you've been an a (hopefully) fun adventure. 

I have a few suggestions for you:

1. Consolidate Your Console Logs:

Currently, you have individual console logs scattered around. You could consolidate these into fewer log statements to make your code cleaner. Place these lines in your getAllRecords function, right after you get the value for `after`

console.log("1st async call:", list.length, after);
console.log("Follow-on async call:", list.length, after);

2. Make sure your variables are declared

// Declare and initialize the list and hubdbURL variables
let list = [];
const hubdbURL = "your_hubdb_url_here";

3. You could consolidate your console logs in your `getAllRecords` function

console.log("1st async call:", list.length, after);
console.log("Follow-on async call:", list.length, after);

4. Have you looked at replacing XMLHttpRequest with the Fetch API instead? It is more modern, will return a Promise (so you don't have to create one), and is more readable. 

Have fun building! — Jaycee

[SteveHTM]

Thanks @Jaycee_Lewis for the helpful feedback. One reason I posted this is that I found it very hard to find any examples of this kind of "paged" fetch in client-side code. The especial challenge seems to come from the fact that the async signal or "promise" cannot be resolved until the processing of the first page data is completed (not just the completion of the file read).

Thank you also for the pointer to the fetch() API - it may be a way to more elegantly solve this issue. If I succeed with finding a good way to structure this implementation, I'll post it here.

Steve