query limits with filter Endpoint

Develop, Build & Integrate APIs & Integrations
1

Hello, you developer here, trying to figure out of I am handling one of the limitations in the filter search endpoint here. The limitation I am concernced for is. Context: we have almost 3 million records that this will filter through

  • The search endpoints are limited to 10,000 total results for any given query. Attempting to page beyond 10,000 will result in a 400 error.

Am I correctly using the filter search and grabbign the contacts records? posting the code below since I do display any personal api info. The code works but I am jsut having trouble understnad the whole query, pagination, limit, and if I am doing it properly

const endpoint = "https://api.hubapi.com/crm/v3/objects/contacts/search";

const filterData = {
 properties: ["big_commerce_customer_id", "email", "hs_object_id"],
 filterGroups: [
 {
 filters: [
 {
 propertyName: "big_commerce_customer_id",
 operator: "NOT_HAS_PROPERTY",
 },
 ],
 },
 ],
 limit: 100,
 after: 100,
};

const headers = {
 Authorization: `Bearer ${accessToken}`,
 "Content-Type": "application/json",
};

// Define an asynchronous function to fetch and save emails
async function getContacts() {
 try {
 const allContacts = [];
 let subArray = [];

 let continueFetching = true;
 let totalCalls = 0;

 async function makeAPICall() {
 if (totalCalls >= 3) {
 continueFetching = false;
 }

 if (continueFetching) {
 try {
 totalCalls++;

 // Make the GET request to fetch contacts
 const response = await axios.post(endpoint, filterData, {
 headers,
 });

 if (response.status !== 200) {
 console.error(
 `API call failed with status code: ${response.status}`
 );
 continueFetching = false;
 return;
 }

 const contacts = response.data.results;

 if (contacts.length === 0) {
 continueFetching = false;
 }

 console.log(contacts);

 const simplifiedContacts = contacts.map((contact) => ({
 id: contact.id,
 email: contact.properties.email,
 bigCommerceId: contact.properties.big_commerce_cusotmer_id || null,
 }));

 subArray.push(...simplifiedContacts);
 if (subArray.length >= 100 || contacts.length === 0) {
 allContacts.push(subArray);
 }

 setTimeout(makeAPICall, 10000); // Wait for 10 seconds before making the next call
 } catch (error) {
 console.error("An error occurred:", error);
 continueFetching = false;
 }
 }

 // If you want to check for successful writing to JSON, you can do it here.
 // Write the data to a JSON file
 try {
 await fs.writeFile(
 "contacts.json",
 JSON.stringify(allContacts, null, 2),
 "utf-8"
 );
 console.log("Contacts successfully written to JSON.");
 } catch (error) {
 console.error("An error occurred while writing to JSON:", error);
 }
 }

 // Start the API calls
 makeAPICall();
 } catch (error) {
 console.error("An error occurred:", error);
 }
}
2

To get accurate information on query limits with a “filter” endpoint for a specific service or API, you should consult the documentation provided by the service provider. Here are some general steps you can follow to find this information:

  1. Read the Documentation: Start by checking the official documentation of the service or API you are using. Look for a section that outlines rate limits, usage limits, or query limits. This documentation should specify the maximum number of requests you can make, the time window (e.g., requests per minute or per day), and any other restrictions.
  2. Contact Support: If you cannot find the information in the documentation or if you have specific questions, consider reaching out to the service provider’s support or developer relations team. They can provide details on query limits and help address any concerns.
  3. Monitor Headers: When making requests to the API, check the response headers. Some services include rate limit information in the HTTP headers of the response. Common headers to look for include “X-RateLimit-Limit” (total allowed requests), “X-RateLimit-Remaining” (remaining requests in the current window), and “X-RateLimit-Reset” (time when the rate limit window resets).
  4. Use Rate-Limiting Libraries: In some cases, developers create rate-limiting libraries or tools that can help you manage API requests within the specified limits. These tools can be especially useful when dealing with complex or high-frequency queries.
  5. Comply with Rate Limiting: It’s crucial to adhere to the rate limits defined by the service provider to avoid getting temporarily or permanently blocked from using the service. Implementing rate limiting on your end can help ensure compliance.

Remember that query limits can vary significantly from one service to another. Some APIs offer generous limits, while others may have more stringent restrictions, especially if they provide valuable or resource-intensive data. Understanding and respecting these limits is essential for smooth and uninterrupted use of the service.