When you’re about to publish a blog post, a big question always arises. What should the title be?
The title matters. It’s the snippet that ends up on Facebook and that people see if the article is on Hacker News or Reddit. It’s what makes people decide whether or not to click. The title question arises anytime you publish anything, even if you have no intention of publishing “clickbait”. And unless you work at a huge media company with sophisticated title-optimization tools, you just guess.
So today we are introducing a new tool: A/B testing for headlines. It’s part of our Content Tracker analytics dashboard, and for $99 a month, anyone can now A/B test headlines. Your website doesn’t need lots of traffic to run the tests, and there is nothing you need to install.
Our A/B testing tool works for anyone, big or small. Heck, you can even test headlines or ideas for blog posts before you write them or before you have your blog set up.
Let’s walk through our approach.
***
A few months back, we launched Content Tracker, our dashboard for measuring content. We use it as our performance dashboard at Priceonomics, and we figured: Why build content tools that only we use when instead we can make them more widely available to others?
Since the launch, over 2,200 companies have started using Content Tracker. Our Slackbot, which notifies you when your content is mentioned in the media or hits a milestone, is the #11 analytics app in the Slack store. Most of the companies using Tracker are content marketers, but quite a few media companies use it too.
Content Tracker is a tool we built at Priceonomics to solve our own biggest problems as a publisher. And as a publisher, choosing the right title for an article is one of our most difficult problems.
Titles are a big deal at Priceonomics. We debate them; we fight over them; feelings get hurt over them. We wanted more data and less opinions about titles, so we built a tool in Tracker to A/B test our headlines.
Now, when you login to Tracker, there’s a new tab called “A/B Test”. We’ve been testing all our titles using this tool, and the performance lift we’ve seen from choosing better titles has been remarkable.
A/B testing of headlines is the process of testing multiple headlines and statistically measuring which ones generate the best outcome. That outcome is typically the highest click through rate, but it can also be another metric like the highest “share rate”.
Large publishers like Buzzfeed, Vox, and the Washington Post have all invested heavily in building infrastructure to optimize headlines. If you noticed that the Washington Post feels more like Buzzfeed lately, the reason for that is a tool called Bandito.
The way these systems typically work is by dynamically changing the headlines presented to readers and eventually settling on the optimal one. These solutions work for websites with tens of millions of visitors that can quickly run a test. It doesn’t work for smaller sites. A similar problem emerges if you try to use Optimizely; you need ton of traffic to quickly test a headline.
We decided to build an approach that anyone can use—even if they have zero traffic. Moreover, we wanted to be able to test out headlines before publishing a post rather than on the fly. So that’s what we built.
***
Here’s how to run an A/B test in Priceonomics Content Tracker. First, pick two headlines, upload a photo, and paste a bit of sample text.
That’s all. We’ll tell you the best headline. You don’t even have to have written the full article yet.
Here’s how it works behind the scenes. We create two separate versions of this article for you, which are both hosted on Tracker and not affiliated with your website.
They’re identical except for one difference—the title. Here’s what they look like:
vs.
You’ll notice that these are pretty non-descript articles with no branding. They’re hosted on Content Tracker, so you don’t have to worry about the article being polished yet. You’re just testing the click through rate on the title—not how well written the article is.
Tracker buys two sets of ads, one for each title, and measures which title gets a higher click through rate. Currently, we purchase the ads on Facebook, and it costs about $10 per title (so we charge $20 per test to cover the advertising cost). We’re exploring additional options for purchasing the ads, but Facebook works pretty well.
After Facebook approves the ads, which generally takes less than an hour, the A/B test runs in a few hours and then gives you the results. The difference between a “good” title and a “bad” one can be enormous.
In this case below, the click through rate on the winning title was 1.4% versus just 0.1% on the alternative. If we had chosen the wrong title, we would have killed the article’s chance at success.
Details
Adding the A/B testing capability to your account costs $99 / month and can be enabled here. Each test costs $20, which is the cost of buying ads for the test.
For now, we’ll be individually working with companies on A/B tests to walk you through the process, check the results, and help you interpret them. We want to make sure the initial experience is great for you before we open up A/B testing to wider use. Sometimes the results are very statistically significant (like in the above example). But sometimes they’re not. So, in the beginning, we want to carefully interpret the results with you.
We’re not quite ready to open the floodgates on the product, so only first 5 companies that want to turn on A/B testing can get immediate access. The rest will be placed on a waitlist if there is sufficient interest.
Do you want to be one of the first 5 companies? Turn on A/B testing here. We’ll follow up with you to get started.
We’ve also started rolling out A/B testing access to customers of the Priceonomics Data Studio, our content marketing agency that helps companies turn their data into great stories.
Should this be used for good or evil?
So glad you asked. It’s up to you really.
You can use A/B testing to make clickbait, or you can use it to decide between informative titles you’re torn between. Many publishers seem to use A/B testing to optimize for clicks, which results in everyone adopting a similarly irritating style of headlines. But that is not inevitable. It’s up to you to decide which kind of titles you test.
The vast majority of companies using Content Tracker use it for a company blog. They’re not media companies trying to get more eyeballs, and there is no real upside for them to engage in clickbait. When companies spend time writing an article, they should make sure they have a great title that will give it the best chance to succeed. It stinks when you mis-titled an otherwise great article.
With A/B testing on Tracker, you can use data to avoid torpedoing an article with a subpar title.
***
If you’re interested in enabling A/B testing on your Content Tracker account, login and follow this link.
Or if you have any questions, just send us an message here. We’re happy to chat and answer your questions. Oh, and the title of this post? It performed 77% better on the A/B test than the title we were originally going to use.