Data Entry and Cleansing Best Practices

[LMiller39]

Hi everyone,

Thought I'd post on here to see if I can receive any advice for my team. We are currently working on a large scale data cleanse and have found that we do not have a standardized process for creating contacts and other objects. For instance, we have a bulk of contacts that are missing crucial information in their record and now we must find a way to fill in thousands of missing fields without losing data.

Moving forward, we would like to prevent this kind of thing from happening. Can anyone share any tips or recomendations for best practices when it comes to keeping an organized database? What kinds of things should our team standardize, and what's the most efficient way to train our employees on these processes? Any suggestions on how to clean up the existing data would also be greatly appreciated.

Thanks for the help!

[karstenkoehler · Accepted solution]

Hi @LMiller39,

Addressing a large-scale data cleanse and implementing standardized processes can be challenging but very rewarding in the long run. Here are a few things I'd consider especially in HubSpot.

1. Before you start, try to separate the good from the bad. For example, you could consider removing or reviewing the following. By doing this before you start working through records, you might be able to 'skip' a sizeable chunk.
   1. Freemail / disposable email addresses: https://github.com/disposable-email-domains/disposable-email-domains
   2. Contacts where "Unsubscribed from all email is true"
   3. Contacts where "Email hard bounce reason is known"
   4. ...
2. Define early on what the must-have fields are for each record type and place them in the left sidebar of each object: https://knowledge.hubspot.com/object-settings/customize-record-sidebars
3. To avoid further damage to data quality, try to use as many of HubSpot features that help with this directly, such as:
   1. Required fields in record create forms: https://knowledge.hubspot.com/object-settings/set-up-fields-seen-when-manually-creating-records
   2. Filtered views showing records, by owner, who are missing one of the required fields: https://knowledge.hubspot.com/records/filter-records-and-save-views
   3. Workflow notifications for when critical required fields are missing: https://knowledge.hubspot.com/workflows/create-workflows
4. Create a dashboard with reports visualizing data quality:
   1. Records where required fields are missing by record owner
   2. Count of records where required fields are missing
   3. Count of records with missing fields by original source

Generative AI had a decent answer on the change management and practical side of this as well.

---

Standardization and Data Quality Practices

1. Define Standard Operating Procedures (SOPs):

   • Develop clear SOPs for data entry, ensuring all team members follow the same guidelines.
   • Standardize the formats for key data fields such as names, addresses, phone numbers, and email addresses.

2. Use Data Validation Rules:

   • Implement validation rules in your database system to enforce the format and completeness of data fields.
   • Examples include mandatory fields, character limits, and specific formats (e.g., phone numbers).

3. Adopt a Single Source of Truth:

   • Ensure that there is a master database where all data is accurate and up-to-date.
   • Avoid multiple versions of the same data across different systems or teams.

4. Regular Audits and Data Quality Checks:

   • Schedule regular audits to check for data quality and consistency.
   • Use automated tools to identify duplicates, incomplete records, and data inconsistencies.

Training and Process Implementation

1. Comprehensive Training Programs:

   • Develop a thorough training program for all employees involved in data entry and management.
   • Include training on the importance of data accuracy and the specific SOPs to follow.

2. User-Friendly Documentation:

   • Create easy-to-understand documentation and guidelines that employees can refer to.
   • Consider creating video tutorials or quick reference guides.

3. Feedback Mechanism:

   • Establish a feedback loop where employees can report issues or suggest improvements to the data management process.
   • Regularly review and update processes based on feedback.

Data Clean-Up Strategies

1. Automated Data Cleansing Tools:

   • Utilize data cleansing tools like OpenRefine, Trifacta, or Talend to automate the process of finding and correcting errors.
   • These tools can help with deduplication, standardization, and filling in missing information.

2. Data Enrichment Services:

   • Use third-party data enrichment services to fill in missing information such as contact details and demographic data.
   • Ensure the service you choose complies with data privacy regulations.

3. Manual Review and Correction:

   • For critical data, manual review and correction might be necessary to ensure accuracy.
   • Consider outsourcing part of this task if it's too large for your internal team to handle efficiently.

4. Batch Processing:

   • Process the data in manageable batches to avoid overwhelming your system and team.
   • This approach helps in monitoring progress and ensuring quality at each step.

5. Data Integration:

   • Integrate your database with other systems to automatically update and synchronize data.
   • Ensure that changes in one system reflect in all relevant systems.

Prevention of Future Issues

1. Data Governance Framework:

   • Establish a data governance framework that outlines roles, responsibilities, and processes for data management.
   • Include policies on data ownership, data quality standards, and access controls.

2. Regular Training Updates:

   • Keep training programs up-to-date with any changes in SOPs or new tools being used.
   • Offer refresher courses periodically to reinforce good data practices.

3. Monitor Data Entry:

   • Implement real-time monitoring and alerts to catch data entry errors as they happen.
   • Use dashboards and reports to track data quality metrics and trends.

---

Hope this helps!

[Lucila-Andimol · Accepted solution]

Hey @LMiller39 

so the best recommendations that I can give you are :

- identify crucial información for the company

- identifiy how this information is being created (assign responsibilities in the team)

- Review or define a process for registering information (stablish mandatory information for each step)

----upto here this will prevent generating new incomplete information 

- Build a reports dashboard: for example, contacts without company (if you are B2B), deal without contacts, deals closed without amount...etc

- Monitor the dashboard weekly

- audit the processes and the CRM periodically 

> this last part will hepl you identify red flags as soon as they start to arise in order to correct them in time

hope this helps

[taylorsusanne76 · Accepted solution]

Hi @LMiller39,

Honestly, this happens to almost every team once the data pile gets big. The best fix is to set some basic standardslike required fields (email, title, company, country), consistent formats, and using picklists instead of free typing. You can also put validation rules in the CRM so incomplete records can’t be saved. For the mess that already exists, do a bulk cleanup with enrichment tools instead of trying to fill in fields manually. Additionally, I'm familiar with tool named LeadAngel can actually help a lot here, it can auto-fill missing data, catch duplicates, and make sure no incomplete lead goes into the system again. Once the cleanup is done, automation + simple rules will stop this from repeating.

[NAdler]

Hi @karstenkoehler - Could you give an example of a dashboard for point #4?
We're trying very hard to make sure that all of our data is clean after our migration over to HubSpot and we need to make sure that we have a really good visualization of our data quality. Thanks for being such a great contributor!!

[Alexa2358]

Hello everyone, thank you for opening up this topic! I was also wondering the same thing and also, does anybody have any templates for these clean-ups? Would be super helpful on getting started.

[Lucila-Andimol · Accepted solution]

Hey @LMiller39 

so the best recommendations that I can give you are :

- identify crucial información for the company

- identifiy how this information is being created (assign responsibilities in the team)

- Review or define a process for registering information (stablish mandatory information for each step)

----upto here this will prevent generating new incomplete information 

- Build a reports dashboard: for example, contacts without company (if you are B2B), deal without contacts, deals closed without amount...etc

- Monitor the dashboard weekly

- audit the processes and the CRM periodically 

> this last part will hepl you identify red flags as soon as they start to arise in order to correct them in time

hope this helps

[karstenkoehler · Accepted solution]

Hi @LMiller39,

Addressing a large-scale data cleanse and implementing standardized processes can be challenging but very rewarding in the long run. Here are a few things I'd consider especially in HubSpot.

1. Before you start, try to separate the good from the bad. For example, you could consider removing or reviewing the following. By doing this before you start working through records, you might be able to 'skip' a sizeable chunk.
   1. Freemail / disposable email addresses: https://github.com/disposable-email-domains/disposable-email-domains
   2. Contacts where "Unsubscribed from all email is true"
   3. Contacts where "Email hard bounce reason is known"
   4. ...
2. Define early on what the must-have fields are for each record type and place them in the left sidebar of each object: https://knowledge.hubspot.com/object-settings/customize-record-sidebars
3. To avoid further damage to data quality, try to use as many of HubSpot features that help with this directly, such as:
   1. Required fields in record create forms: https://knowledge.hubspot.com/object-settings/set-up-fields-seen-when-manually-creating-records
   2. Filtered views showing records, by owner, who are missing one of the required fields: https://knowledge.hubspot.com/records/filter-records-and-save-views
   3. Workflow notifications for when critical required fields are missing: https://knowledge.hubspot.com/workflows/create-workflows
4. Create a dashboard with reports visualizing data quality:
   1. Records where required fields are missing by record owner
   2. Count of records where required fields are missing
   3. Count of records with missing fields by original source

Generative AI had a decent answer on the change management and practical side of this as well.

---

Standardization and Data Quality Practices

1. Define Standard Operating Procedures (SOPs):

   • Develop clear SOPs for data entry, ensuring all team members follow the same guidelines.
   • Standardize the formats for key data fields such as names, addresses, phone numbers, and email addresses.

2. Use Data Validation Rules:

   • Implement validation rules in your database system to enforce the format and completeness of data fields.
   • Examples include mandatory fields, character limits, and specific formats (e.g., phone numbers).

3. Adopt a Single Source of Truth:

   • Ensure that there is a master database where all data is accurate and up-to-date.
   • Avoid multiple versions of the same data across different systems or teams.

4. Regular Audits and Data Quality Checks:

   • Schedule regular audits to check for data quality and consistency.
   • Use automated tools to identify duplicates, incomplete records, and data inconsistencies.

Training and Process Implementation

1. Comprehensive Training Programs:

   • Develop a thorough training program for all employees involved in data entry and management.
   • Include training on the importance of data accuracy and the specific SOPs to follow.

2. User-Friendly Documentation:

   • Create easy-to-understand documentation and guidelines that employees can refer to.
   • Consider creating video tutorials or quick reference guides.

3. Feedback Mechanism:

   • Establish a feedback loop where employees can report issues or suggest improvements to the data management process.
   • Regularly review and update processes based on feedback.

Data Clean-Up Strategies

1. Automated Data Cleansing Tools:

   • Utilize data cleansing tools like OpenRefine, Trifacta, or Talend to automate the process of finding and correcting errors.
   • These tools can help with deduplication, standardization, and filling in missing information.

2. Data Enrichment Services:

   • Use third-party data enrichment services to fill in missing information such as contact details and demographic data.
   • Ensure the service you choose complies with data privacy regulations.

3. Manual Review and Correction:

   • For critical data, manual review and correction might be necessary to ensure accuracy.
   • Consider outsourcing part of this task if it's too large for your internal team to handle efficiently.

4. Batch Processing:

   • Process the data in manageable batches to avoid overwhelming your system and team.
   • This approach helps in monitoring progress and ensuring quality at each step.

5. Data Integration:

   • Integrate your database with other systems to automatically update and synchronize data.
   • Ensure that changes in one system reflect in all relevant systems.

Prevention of Future Issues

1. Data Governance Framework:

   • Establish a data governance framework that outlines roles, responsibilities, and processes for data management.
   • Include policies on data ownership, data quality standards, and access controls.

2. Regular Training Updates:

   • Keep training programs up-to-date with any changes in SOPs or new tools being used.
   • Offer refresher courses periodically to reinforce good data practices.

3. Monitor Data Entry:

   • Implement real-time monitoring and alerts to catch data entry errors as they happen.
   • Use dashboards and reports to track data quality metrics and trends.

---

Hope this helps!