Email Marketing Tool

jaleonard19
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Hi, HubSpot Community. I’m Jake from the HubSpot Product team. In the last year, the marketing industry as a whole has seen a spike in bot clicks in emails, so I wanted to take a moment to discuss what bot activity looks like, who’s most likely to be affected, and what you can do to address it.

 

What’s bot activity?

Bots are the source of a lot of internet traffic today. It’s estimated that more than half of web traffic originates from a bot, and bots are more sophisticated today than even a few years ago. Not all bots are bad bots, though! Google’s search engine bots, for example, crawl websites and index them; HubSpot’s chatbots help to facilitate conversations with your prospects and customers without requiring a human. 

 

Unfortunately, there are also bots that create fake clicks and false impressions for apps, websites, and emails. With email, bots are designed to click links in emails as a way to explore, identify, and prevent links to malware or phishing attacks from entering a recipient's inbox.

 

Okay, so what does bot activity in emails look like?

Email bot activity differs from what bot activity may look like on a website. Bot activity in email is typically security software sitting in front of a recipient’s (or contact’s) inbox. That security software will open the email and, typically, engage with some (if not all) of the corresponding links in that email, following them through the redirect. These actions simulate real user engagement potentially causing false opens and clicks data to be recorded.

 

Every email marketer who has sent an email campaign has probably been affected by bots. And bots are tricky for email marketers because the email engagement events (opens and clicks) they generate can get recorded alongside legitimate ones. This leads to email campaign statistic inflation as well as automation being tripped based on these false events.

 

Who’s more likely to be impacted? 

Bots don’t impact all marketers equally. These security filters are much more common in industries tied to finance or healthcare but aren’t as common in B2C marketing. Generally, regulated industries where contacts use a company email see more bot clicks than personal email addresses on email providers like Gmail or Office365.

 

So, what can we do about it?

For HubSpot, identifying bots and separating them from legitimate contact engagement has been, frankly, a game of Whac-a-Mole. In the last year, the industry as a whole has seen a spike in phishing emails. This rise in phishing has led to corresponding increases in companies adding security software (i.e., more bot activity). 

 

At HubSpot, we’re doing our best to address this rise. Currently, we’re working on redesigns to email reporting to better separate legitimate contact email engagement from that generated by security software. The entire team at HubSpot, including myself, recognizes this can cause anxiety for a marketer. We will continue to do everything we can to find a solution that best addresses this problem. Until we have a full solution, we’ll be monitoring this thread to answer any relevant questions or concerns and respond to any feedback you have about this.

1 Accepted solution
jaleonard19
Solution
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Posting an update to this community thread as we are releasing some new bot detection software for marketing email. We have had a solution to bots in email for a while now. It was an IP based approach that worked fine but had some flaws. As we have dug into this problem it has become clear that increases in bot activity is an issue felt not just by HubSpot, but the entire email industry. 

 

This week we are rolling out new tools for bot detection to HubSpot customers, and here are some notes on it:

 

  • The new tools take a behavioral-based approach. We look for patterns in the open or click events that indicate a likelihood of being generated from a bot.  
  • We will be rolling this out to all HubSpot accounts this week.
  • No action is needed. If bot filtering is turned on for your account then it will just work. 
  • This does not change any email campaigns sent in the past. It only impacts emails going forward.

 

It is impossible to capture all events generated from bots. If you believe you are still seeing bot activity feel free to post comments here or open a ticket with our support team. Our engineering team is continuing to monitor bot activity on our email and will take appropriate additional steps if they are needed.

View solution in original post

45 Replies 45
TSweeney2
Member

Are Bots Affecting Your Email?

SOLVE

hi drosenbaum, i'm followwing this topic feverishly to improve our analytics. we have the bot filter turned on. i'm curious how you did this:

1. Add an invisible clickable hyperlink in the email somewhere that only a bot would recognize and click

0 Upvotes
drosenbaum
HubSpot Employee
HubSpot Employee

Are Bots Affecting Your Email?

SOLVE

I did not implement this, it was only an idea I had. I would think you could implement this by having a hyperlinkink where the text is white on a white background. Humans wouldnt notice it and would not be likely to click it, but a clickbot would certainly notice it and click it. The idea would be that any emails where that link is clicked we can be reasonably sure it was a bot. The next issue is how to parse that info out in HubSpot which I am not sure is possible as of now

0 Upvotes
TSweeney2
Member

Are Bots Affecting Your Email?

SOLVE

We did implement and invisible image and labeled the URL as ".../honeypot."

 

That's been helpful to determine which companies are using bots. We also created a workflow that looks at the email click/open activity time stamps because we realize bots have predictable behavior in that the timestamps for  delivered/open/click1,2,3,4 (however many buttons and hyperlinks you have regardless of whether they are HTML or plan text, including the honeypot) activities are all within the same minute of delivered or 2 minutes max.

 

We created a workflow to create an active list of any recipients who have opens or clicks one day after or longer after. We are using this as our actual engagement metrics and those are these people who get the next touch in the workflow for "responders. " It's a much smaller number of recipients than the clicks/opens that show up in the performance report, but it's at least more trustable/true/accurate. I'm going to post this workaround/trick at the end of this thread so all can see it - we are just starting to implement it with sends going out today. I'm curious what you think about this method and if you see any 'gotchas' - I really appreciate you responding to my reply, it's been feeling like the lonely wilderness out here trying to figure this out. 

 

0 Upvotes
JSikora123
Participant

Are Bots Affecting Your Email?

SOLVE

Agree -- not sure with HubSpot isn't just productizing a work-around solution like that!

0 Upvotes
JSikora123
Participant

Are Bots Affecting Your Email?

SOLVE

Thank you -- any further updates or insights on these metrics and bot behaviors now that so many recipients have been working from home and not necessarily logging in via a VPN, instead relying on their home wifi IP?

0 Upvotes
yepezrick
Member

Are Bots Affecting Your Email?

SOLVE

We have noticed similar patterns when sending marketing emails to goverment agencies specially airport administration and Department of Transportation (DOT).  It appears all the members of the organizations open or click the email.  In turn, this data gets reported under the email performance metrics. Does the new filter work with these types of accounts? This will help us have more accurate reporting of email performance and clicks. 

0 Upvotes
jstasse
Participant

Are Bots Affecting Your Email?

SOLVE

Hi Jake, 

Thanks for rolling this feature out, but I just want to report I saw today a pattern of suspicious bot activity on one of our email sends. We were able to detect it because: 

- a large percentage of recipients from the same email domain clicked on the links within a  minute of receiving the email

- none of them converted (they had to fill a form to get access to the white paper our email was promoting), while the conversion rate at other companies was roughly 40%. 

I opened a ticket with the support team, but you can email me if you want details. 

0 Upvotes
jaleonard19
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Hi @jstasse,

 

Sending you a direct message to follow up on that support ticket. 

 

Cheers,

Jake

0 Upvotes
jaleonard19
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Deleted comment

johnhenry
Contributor

Are Bots Affecting Your Email?

SOLVE

Hi Jake,

 

Thanks for your post. I only saw this thread now. We noticied major issues last year regarding US traffic (we are Australian based). I flagged this with the support team who danced around the issue and did not really give me an good answer which I was quite disappointed with. I'm glad there is this thread acknowledging it.

 

I just wanted to post to flag this again as a key issue for our marketing reporting and metrics, because it largely skews the data. Hopefully a fix is discovered soon.

MikeW1
Member

Are Bots Affecting Your Email?

SOLVE

Yes, they are . Can you please resolve this issue as we cannot distinguish between  "clicks"  and  "opens"  .  This inability from Hubspot to  fix  this  is having a direct impact on  my ability to perform my job.  I am  imploring you to  please  give this a priority

much appreciated

 

ramialbader
Member

Are Bots Affecting Your Email?

SOLVE

we are receiving "clicks" even without the recipient clicking the link at all. There is no distinction between opens and clicks , this has happened after applying secure mail gateway, we have opened a ticket and it was transfered to you if you have reached for a workaround/solution of the issue yet, your quick response is highly appreciated.

0 Upvotes
Mike_Donnelly
Top Contributor

Are Bots Affecting Your Email?

SOLVE

This was also posted to the ideas forum.

 

One thing we've been doing and it's partially working (i.e. - need to send more email and this solution will never be a 100% answer as spam systems are always changing the way they work and no two are exactly alike), but add an invisible link to a page on your site in your emails.  For example, in the below screenshot, "Communication" had a link to an old blog about "Communication", but it looks like all the rest of the text so unless you got lucky and moused over it, you'd never know it was a link.  Then we segmented the people (aka bots) that were sent the email and clicked on the link and sure enough, by looking at the details of the clicks it's very clear it's a bot clicking (screenshot below - this is a monthly newsletter so has a ton of links in it).  The next step which is still a work in progress is to add a specific list of domains that have suspicious / bot clicks and then segment these out of any automated follow up that is based on activity.  One thing to mention is that some bots / spam systems will not click on every link in an email, but by doing this on every email you send, you can start to identify / weed out the domains that have aggressive spam systems in place that adversely affect reporting and other automated tasks.  You can also segment your sends so that one email goes to suspicious domains and the other goes to people that don't have the suspicious activity.  Then you can manually review the suspicious activity to determine next steps.  Not a perfect solution and I'll share more on some things we're working on.

Screen Shot 2019-12-10 at 6.08.30 PM.png

Screen Shot 2019-12-10 at 6.09.04 PM.pngScreen Shot 2019-12-10 at 6.10.41 PM.png

jaleonard19
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Thanks for sharing @Mike_Donnelly! Very interesting solution, basically a captcha like solution for email. Mind sharing the ideas forum post here? 

0 Upvotes
Mike_Donnelly
Top Contributor

Are Bots Affecting Your Email?

SOLVE

Hey @jaleonard19 - Thanks and the above was everything I posted in the ideas forum.  We're working on a few other ideas which will flag domains, but it's still a work in progress.  Essentially, if you bubble up the top domains clicking a lot of links, you can be extremely confident they're coming from bots / spam systems.  Then you can further investigate by looking at the timeline.  Much of what you've specified is what we've been working on.  Not a perfect solution (as it never will be), but it removes a lot of noise.

0 Upvotes
mrspabs
Top Contributor | Elite Partner
Top Contributor | Elite Partner

Are Bots Affecting Your Email?

SOLVE

Thanks for posting this. I notice that in my emails, 14% clicked on the logo which kind of makes no sense. If there is a beta or something, I would be interested!

jaleonard19
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Hi @mrspabs - eeeekkkk. Yea that doesn't sound right, sorry you are running into that. Unfortunately, we are not doing any formal Beta, rather we are doing a generalized slow-roll out of the solution. 

Phil_DM2
Contributor | Diamond Partner
Contributor | Diamond Partner

Are Bots Affecting Your Email?

SOLVE

Very interesting. Our company has noticed strange email open/click behavior in the past. Most notably, we had some unusual spikes in direct traffic from recipients only clicking the first link in every email we sent (the link happened to be from our company logo and led to our website's home page, which may have caused it to trigger as Direct?)

 

Essentially, we were seeing dozens of clicks from each email but with absolutely no engagement otherwise. It was really bizarre, and it did seem like some strange bot activity. Even this year, sometimes our emails consistently get the same number of clicks from the same people (give or take 20-23 clicks), while other emails receive nothing at all. Strange...

jaleonard19
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Thank you for sharing. A lot of that does sound like bot activity. Two common patterns are: 

  1. Opens or Clicks almost immediately after the email was delivered. You can see this by clicking into the 'Recipients' tab on the individual emails. Then expanding the contact's timeline of events. If the timestamp for the delivered and the open or click are very close together (less than a minute or two). That is a strong indication it is a bot. 
  2. If every single link in the email is clicked. That is another strong indication it is a bot. We will see the same emails delivered to the same domain, and all come out with exactly 15 clicks. Again most likely a bot. 
Phil_DM2
Contributor | Diamond Partner
Contributor | Diamond Partner

Are Bots Affecting Your Email?

SOLVE

Oh wow, this totally lines up with what we've been seeing! Most notably, a contact clicks a link multiple times as well, and it's almost always as soon as the email is delivered to them. Is there anything we can do to avoid this or prevent it from continuing? Last month it completely skewed our metrics since we had way more Email traffic than we normally would. Now, we seem to be having a lot less. 

jaleonard19
HubSpot Product Team
HubSpot Product Team

Are Bots Affecting Your Email?

SOLVE

Hi @Phil_DM2 - Thanks for sharing. That is unfortunate that it skewed your metrics the previous month. We have been doing some interim work to identify some of the worst offenders and make sure those open & clicks aren't recorded. This may be why you are seeing less now. 

 

In terms of what you can do - nothing at the moment. We are in the midst of rolling out a new solution for bots on email. So you should continue to see your metrics get more and more accurate over the coming months.