There seems to be a bug or at least a design flaw in Hubspot’s Page Detail reporting when using A/B testing.
When looking at the number of page views for a single page, one would expect to see all the cumulated views for that page (no matter the number of views attributed to version A or B of a test). If someone wanted to see the number of page views for each test, they would go to the test results tab instead. But for some reason, Hubspot only shows page views for the ‘winning’ or ‘A’ test in the overall page performance.
Here is an example from our homepage where we ran an A/B test from May to July. As you can see, Hubspot is showing us a significant drop in page views for our homepage. But in reality, there was no drop, it’s just not including page views from the B test.
When looking at the test results tab though, the story is totally different:
We can see our homepage did not have a significant drop in traffic when the B version (purple) is accounted for.
Not sure if it’s a bug or a design flaw, but I think the performance tab paints an inaccurate picture of a page's traffic. It is definitely not the behavior people are expecting.
What do you think?