So… you’re wrapping up all your holiday advertisements this week. You’ve got traffic coming in from Adwords, Facebook, email blasts, Twitter and more. You’re getting great results, your ROI is through the roof and you’re almost on to celebrating. Sadly, when it’s time to analyze trends and individual campaign performance next week you notice a serious problem: You have no idea which ads or sources to attribute fiscal success because you didn’t build links correctly, or at all.

Bummer. That sucks.

But you can redeem your future self. Here’s how:

Click to read the entire article →

You’re running a test on one of your landing pages with the primary goal of getting the user to click on a button. You are split testing (A/B testing) two different button colors (green and orange) to determine the impact of the button color on click-through rate. You collect the following initial results:

The orange button is better, right?

Not necessarily. Sure, the orange button has a 7.7% click-through rate (CTR) compared to only 5.2% for the green button. However, the orange button has really only earned one more click. If the next visitor on the page clicks on the green button, both variants will have a 7.7% CTR, indicating that button color is irrelevant in this application.

Here is what happens if we run this experiment for several months with 16,000+ visitors:

Is the orange button better now? Hell yeah, it is.

While the danger of making decisions based on too little data is an incorrect conclusion, the danger in collecting too much data is a waste of time, effort and money. Even though there was little data to determine orange was best in the first variant, had you decided to go with the orange button anyway, you could have sent all the above 16,000+ visitors to the page that performs at a 9.3% CTR. Thus, giving you a whole lot more sales/leads/etc.

So how many users are required to make this decision? We can use the following equation:

Click to read the entire article →

We presented the slides below at BOLO 2012. If you saw the presentation, thanks for coming out! If you didn’t, all of the slides, notes, code snippets and more are below. If you have questions or comments, please let us know by filling out our contact form.

Click to read the entire article →