Why A/B Testing has Permanently Earned a Spot in This Marketer's Tool Belt

ABOUT THIS EPISODE

In this episode, Olivia Hurley talks to Bob Glotfelty, VP of Growth @ Taulia  

Yeah, hi everyone welcome back to be, to begrowth. My name is Olivia, Hurley and today I'm joined by bob black felt Ewas the VP of growth at Talia. Hi bob, how are you doing? I'm doing very well.It's wonderful to be here with you today. Oh good, I'm so glad. Well I amreally excited to talk to you because I'm going to get a huge education todaybecause you have a lot of experience around A. B testing and I have verylittle experience around a. B testing and testing in general. Um and sostarting off, can you bring us all in what has caused you to believe sostrongly in the importance of a B testing and then testing as well? Yeah,so I don't think 10 years ago I would have thought that I have stretched suchstrong opinions on a B testing but what's really made me such a believeris actually how many times I've been wrong and it's not just me, it's verysmart people that I've worked with, people with an incredible amount ofexperience time and time again, you just find yourself being wrong andthat's a humbling experience. And there's a tool out there called a Btesting which or maybe it's a methodology, not a, not a tool but itis a mechanism to overcome that challenge, that problem and it allowsyou to remove bias and judgment that you make that's incorrect and look atsomething very simplistically, so that's kind of why I've become such abig believer and yeah, it's an amazing tool and for anybody new to thisconcept like myself, can you in your own words and own experience define a Btesting for us a little bit. Yeah, so, so a B testing very specifically isjust like the name describes, you have an A and you have a B and you run thosetwo things in parallel and you see which one is better. So One of theeasiest concepts or the easiest ways I'd like to explain. It is with websitetesting. So you can send 50% of your traffic to your website as you normallywould. and then you can send 50% of your traffic to a variation of your website that you'vemodified and you can say okay if you compare those two groups, which one hasan action that you desire. So you can a B test very small things, you canchange the colour of a button or a word on a page or you could change hugeamounts of things, right? You could have people go to an entirely differentwebsite that has nothing the same. And those would both be a B tests, you cando, you know a B C D E F, you can do many variations if you like. But it'sthe concept of having a a test case and a control case that are running inparallel that allow you to measure success. So I'm curious is HIV testingconsidered an essential practice by marketers. And is it used often? Yeah,I think so. I mean, it doesn't work in every possible case. Right? So ifyou're going to a trade show, there's only one trade show that you go, youcan't have two separate booths and have the traffic magically split between thetwo. So it's, it's not an applicable testing mechanism for all types ofmarketing activity. However, for things like digital, it's extremely effective.Right? You can send, You're gonna send 1000 emails out, you can send 500 thatlook one way and 500 that look a different way. It's very easy tocustomize something like that. So it is definitely an essential practice. It isused very often. It is not available for every possible marketing channel oractivity, but it is a very effective...

...way to, to measure things that can bemeasured through that at all. Okay, that makes total and complete sense tome. And I think also just allows you to not have to put all of your eggs in onebasket. So as market or how do you decide where in the funnel and withwhich assets to run the A. B tests? Yeah, I mean, you're generally lookingfor things that are going to have a return. So if you're sending a one offemail, there's no reason to measure that or if it's so late in the funnelthat your sample size is small, you can't do an effective A. B test. So forme, it's more about is it going to create a result. So if you havesomething that's very ineffective today, you know, a B testing small changes,it's not really worth your time. But if you have something that's, that's ahigh contributor or a very useful thing that has a lot of volume, those thingsjustify A B tests much more. You'll typically see high correlation betweenvolume and success success measures as key indicators for kind of where youshould use that within your, your final or other other mechanisms of measuringwhere to use it. Yeah, Do you have an example or a story you can share withus of, of deciding where in the funnel to, to run a navy test. Sure. Yeah. Soone example would be, we've run a lot of A B tests on our platform itself. Sowhen our users are in our platform, how do they interact with different things?Um, it's very easy for us to run a B tests on our homepage because it gets aton of traffic, everybody lands on that first page, there are other pageswithin our platform where really care about the activity, but the volume isso low that it doesn't, it doesn't make a difference. So if you think aboutthat from a kind of floor stage perspective that's very early onbecause that's where the bulk of the volume is landing, it's very easy totest on those types of pages, but late stage right before somebody is about totake an action and act as a conversion, those can be much harder to testbecause you don't necessarily have the same amount of volume there. Okay, sovolume extremely important. Um, traffic extremely important. How long shouldyou run an A. B test to know that you've, you've measured it for all it'sworth? Yeah, they're really great tools out there that different companies havebuilt that allow you to essentially run the calculation of when you reachedstatistical significance. So what do I mean by statistical significance? Ifyou have, Let's say 100 people going through a flow and uh, 50 went to onetest, 50 went to the other. If 24 took the action you wanted out of the 50 onone test and 25 took the action on the other. That hasn't reached statisticalsignificance. So there are tools online, little calculators you can use whereyou put in the amount of volume, the expected difference between the twocases and it will tell you, oh, you need to run this test for two monthsbecause that's when you're going to have enough examples to saystatistically confident that this is better than this other one. And thenother times you'll run it through and you have so much volume and it'll besuch an improvement. You may only have to run it for a day or a week. Again,I'm using time as a, as a measure here. It can also just be volume. Right? Soif you're sending a number of emails that may not be time bound. But thingslike that can kind of give you a sense for when you've reached statisticalsignificance. Okay. Oh, that's so interesting. So there's no, as withmany things, there's no one size fits all with this type of testing and itreally is, the information at the end is going to be completely differenteach time you test. So how do you and what KPI s do you measure against whenit be testing? Yeah, I mean you're...

...primarily looking for lift andconversion. So if you send this email and you're looking for somebody toclick a button in the email is the incremental lift. 5%, 15%. that's ameasure you're looking for. And then you do want to look for statisticalsignificance because if you're sending millions of emails and you get 1/10 ofa percentage lift, that may be hugely valuable to your company. So the liftmay be small but if you can prove it is valid and it's over a large sample, um,that can be just as compelling. And then in the end, what information areyou looking for from the A. B test and I'm curious to hear some stories andexamples about what you've discovered after a B testing, especially to yourpoint earlier of saying like we found out we were wrong. Yeah, I feel thatany test is pretty much a success and that's going to sound weird because,well either it worked or it didn't, but things generally land into buckets.Either they were successful and you got incremental lift and you're going tomake a change to your approach or it didn't and you've learned somethingimportant in the process. Um, so I'll give some examples. I always feel likea simpler flow is better for a user. So if I can just fill out three fields,I'd rather do that than fill out nine fields or fill out multiple pages in aflow. I just genuinely believe that the shorter is better. A B testing willprove time and again, that that's not always the case and that teaches yousomething. So you may actually do things that are counterintuitive andyou'll learn about your users, you'll learn about how they move through theprocess. So when tests don't work, you learn something that helps you for thenext test or helps to continue to iterate and improve on your process. Isthere anything that tall you specifically that you started doing,you thought surely this will work or it won't work and it ended up being theopposite. Anything specific. There have been a lot. I think my, my, my favoriteexample of that is I used to think html emails were great, right? They lookbeautiful. They have colours and design and what we've done over time as we'vesaid, okay we're going to a B test and email again and again and again untilwe get the email to perform extremely well. And so what we've seen is thatoftentimes the most effective thing is extremely short emails, all plain textand ideally looking like they're written by a person, which is prettycrazy, right? Because in marketing we want to add design and we want to makethings look aesthetically pleasing. But oftentimes that's the antithesis ofwhat the user is going to interact with most effectively. And so yeah, reallybasic things actually tend to perform better than things that as a marketeryou may think are really appealing there often enough. Hey everybody Loganwith sweet fish here, if you're a regular listener of GDP growth, youknow that I'm one of the co host of the show, but you may not know that. I alsohead up the sales team here is sweet fish. So for those of you in sales orsales ops, I wanted to take a second to share something that's made us insanelymore efficient lately. Our team has been using lead I. Q. For the past fewmonths. And what used to take us four hours gathering contact data now takesus only one where 75% more efficient were able to move faster with outboundprospecting and organizing our campaigns is so much easier than before.I'd highly suggest you guys check out lead I. Q as well. You can check themout at lead I Q dot com. That's L E A D...

I Q dot com. All right. Let's get backto the show. That's so fascinating to, because you run the A B test, you findthis out now it impacts your tactics and how you communicate with thisparticular persona with this particular example. Did it save time and moneymoving to something very, very basic and simple versus this. Did it, did itgive up some free up some bandwidth for your team? Yeah. I mean for us, yes,it's I think it's eye opening in the future. Right? You, you know, once youhave an email template built, it's pretty easy just to hit the send buttonagain. But then in the next time you're going to run a different campaign oryou're going to do things differently, you realize, look, we don't need to godown this whole path of creating all this complexity. What's the message wewant to say? How do we say that very concisely and simply, And how do wejust push that message out there? So it's definitely allowed us to sort ofleapfrog right to the better answer in the future with additional campaignsand that saves us time of having to go through that whole path and and haveall these emails that are potentially less less successful. That's so cool.So is there ever a time when a B. Testing or maybe testing in general isa bad idea? Yeah, I would say there's a there's a few examples of when it isbad. So one is if if you can't get a valid test, don't do an A B test, right?So if you have traffic of I don't know 10 people A. b. 5-1 and five together,you will never hit statistical significance. So the test will never bevalid. Never a reason to do that. And then the other example is if you reallyonly have one shot at something right, if you're taking a half court shot at abasketball game and if you make it you win whatever prize like you can'tpossibly a B test that you only have one shot. So you know, don't bothertrying to use a tool that's sort of not uh these tools I'm curious is justsomebody who doesn't know a lot about A. B. Testing. Are these tools things thatyou rely on every time or is this is a B testing a skill that you can yourpredictions get better and better or have you found that you're stillutterly surprised sometimes? Yeah, I mean, so the tools themselves, a lotare built into marketing automation tools. So we use market. So for exampleyou can, within that tool you can run a test on an email to say this muchtraffic here, This much traffic here. This easy comparison. There are toolslike optimized lee that allow you to manipulate your website. So there's alot of like actual tools to do that. But the things you're testing will beall sorts of different things and I think you can get better over timebecause you've seen what's worked and what hasn't in the past. But if I wereto go to an entirely new company and try to do it again, I think you need toa B test a lot of these things because the dynamics are different. The usersare different. Um it needs to be uh fit to the scenario. We also work with aconsulting firm called Cro metrics and this is all they do. All they do is uma B testing and optimization and uh they're they're very heavy on the theoptimize the side and they work across clients and so they bring theirfeedback from other clients. But what works for one client doesn't work foranother. So there is some improvement, but you need to really think about itwithin your use case and everybody's use cases is unique. So for somebodywanting to do what you're suggesting to employ a B testing as a as a regularrhythm in their marketing department, what is step one? Yeah, that's a that'sa good question. I mean I would say test it. Right. So I know, I know we'vebeen talking, you know, pretty narrowly...

...to a B. Testing and I would put that ina broader umbrella of testing, but give it a try, go do a test, see if it worksfor you to see if see if you learn something, there's nothing reallystopping you. Um, there are a lot of ways you can kind of brute force it,you can um, you know, if if your marketing automation system doesn'tallow you to do that, we'll be just a quick thing. We'll try to send twodifferent emails that are different and measure them. It's there's plenty ofways to do that. So give it a try to see what you learn and uh, if it worksfor you, I think you'll be a believer like me and you'll keep on keep ondoing it. I'm curious in terms of things that youcan't necessarily measure. Do you think that A B testing has increased yourability to be creative because you know that you can try things and you don'thave to kind of like I said earlier, put all your eggs in one basket andwith this one shot idea, has it increased creativity? Yeah, I think, Ithink it creates openness. I think it allows you to listen to other peopleand it allows you to give them fair feedback. So a very common, I haven'texperienced as much, but what I've heard from a lot of others at a B testis somebody within the organization or a senior executive will say we shoulddo x. We need to change our website and you know, it shouldn't say contact us.I should say give us a call, I don't know. And by having the ability to a Btest, you don't need to debate whether that should be the case or not, yousimply run an A B test and then you come back and you say you were right.See look at the statistics were better. We should we should change the wordingaccordingly or you can say we tried it. Here's what we found the results aren'tthere And there's no debate. You know, you're able to communicate to somebodywhy something isn't done that way. So I think it allows you to open up two moreideas from more people and there isn't much cost to do it because you learn inthe process and you can always move things back. Do you think there, arethere any naysayers to a B testing? Like is there any reason why whysomebody would be against a B testing? Yeah, I mean it takes extra time. Right?If if you need to send an email and you just want to send it, it only takes oneiteration. If you are going to a B test, you need to do a little more work. Sothere is a opportunity cost in terms of your time, if you're at a very smallcompany and you're the only marketing person is the opportunity, cost oftesting what you're doing more valuable than doing something else, maybe, maybenot, you definitely find in, in larger organizations, they tend to spend moretime on this kind of thing because uh, they'll see more value and they've gotmore resources. Um, but yeah, again, it always depends on the context of theorganization with, with somebody, they've started testing, they're tryingout there taking step one. Is there any warning sign or anything that youexperienced as you became more and more proficient in A B testing? That was a,that was a flag or, or like a don't do what I did, that kind of thing. Yeah.So I don't, I don't know if I would define this the way you asked thequestion, but I would say this is something that everybody should beaware of when they A B testing, which is and and others have spoken to havesaid the exact same things. You go through peaks and valleys. So you'regoing to have a time where you won't have anything to test and because youdon't have any ideas and you'll need to go through this iteration and ideationprocess to come up with tests and then you have other times, well, you'll haveso many tests that you can't possibly run them all and you have to kind ofwait or kind of sequence them and prioritize them. And so when there's alot, everybody's on board with a B...

...testing, everybody thinks it makessense. But when you don't have many people think that, well we should justgo do something else or we should move on and that I would caution againstbecause you're going to come back to that. So be consistent. Get yourselfinto the valley, use that as a reason to come up with more ideas, seek morefeedback from other people. Get creative, but you're going to havethose dips and depending on the way you're resourcing the program, what isstructured internally? That may create pressure because people are saying,well, what are you testing right now? And if you say we're not testinganything that can be hard, but you will come out of that valley and you'llcontinuously go in and out of the cycles. And uh, you should, you shouldtry not to ride the wave, you should try to be steady right through thecenter and be consistent with your testing practice. Oh my gosh, what sageadvice. I'm curious. Do you have an example of that from your own career?Yeah, I mean, I would say uh, so again we engage third parties and we usedifferent tools for this. So when you don't have a lot of tests, people say,well, why are we, we should just get rid of that tool and we should nolonger work with the agency. I remember that happened, I think a year into ourcycle and I think we're five or six years now into having this program setup and so, you know, you see it regularly because you hit those valleys,but we wouldn't have had the success we've had today had we not beenconvinced that there's value and and kind of stuck it out through thatthrough those valleys. I love that. I feel like I've just gotten a massiveeducation and I'm going to go A B test. But if there was anything from thisepisode that you'd want people to take away from this, aside from go trythings and go test them, what would it be? The way I look at? A B testing isit's one element of overall testing and I'm a huge believer in testing, right?Uh I will rarely sign a contract for more than a year for example, because Iwant to test up, I want to see how it works and then I want to make adecision, right? I always believe in that iteration, cycle testing to me iskind of a sub component of decision making or judgment and so it is just atool to sort of help you make good judgments and good decisions. So yeah,I would say the takeaway for everyone here is, you know, kind of regardlessof who you are, what level you're at or what you're doing. Decision making isreally the important element of any business or role or function and A Btesting and testing in general, isn't it amazing mechanism to help thatdecision making process. It can't always be used but it's a it's a reallyimportant skill to home and leverage over time. I love that A B testing cansupport the muscle development of decision making as a whole. Is there?I'm just curious just to pick your brain for a second about decisionmaking. Was there any any resource or book or experience that really kind ofhelped you navigate the world of big business decisions? For sure. So Ithink we've been talking very deep on the data side and I as a person, I'vealways really enjoyed being data driven and seeing data and having that driveyour decision making. In fact, I think I over balanced that that was I was toomuch of that early in my career. The book blink by Malcolm Gladwell is anexcellent book which is sort of the Counter to the data driven view, whichis that your mind and your brain the way you think you're actually able tosee a lot of uh answers way earlier than the data will actually tell youthat it can create bias and it talks...

...about some of those challenges. But itwas a very pivotal thing for me to read that and say, you know, maybe I'm maybeI'm being so data driven that it's not allowing me to be as nimble and asagile and as quick in my decision making and there are times when youjust know and you shouldn't, you shouldn't wait. So it is all a balance.Um and I would definitely encourage people that are data driven to readthat book because it will, it will tell you that maybe it's not always theright decision to, to approach things that way. Oh my goodness! Well I like Isaid earlier, I've just been given a crash course on a B testing and a bookrecommendation to top it all off. This has been so wonderful bob. Thanks somuch for joining me on GDP growth. Absolutely. It was wonderful to be hereand uh looking forward to hopefully doing this sometime in the futurebefore I let you go because I can't let you go without this. How can peoplelearn more about you and Talia? Yeah, well if anybody wants to contact me,Lincoln is definitely best get past this fan filters and I'm pretty easy tofind. There's not many bob block lt's out there. My company is Talia, uh T AU L I A uh you can go to tally dot com to learn more about us. Well, thank youso much for joining me today. It was great being here. Thank you so much.Are you on Lincoln? That's a stupid question. Of course you're on linkedinhere at sweet fish. We've gone all in on the platform, multiple people fromour team are creating content there. Sometimes it's a funny gift for me.Other times it's a micro video or slide deck And sometimes it's just a regularold status update that shares their unique point of view on B2B marketingleadership or their job functions. We're posting this content throughtheir personal profile, not our company page and it would warm my heart andsoul if you connected with each of our evangelists, we'll be adding more downthe road. But for now you should connect with Bill Read, our Ceo KelcyMontgomery, our creative director Dan Sanchez, our director of audiencegrowth Logan, Lyles, our director of partnerships and me, James Carberry.We're having a whole lot of fun on linked in pretty much every single dayand we'd love for you to be a part of it.

In-Stream Audio Search

NEW

Search across all episodes within this podcast

Episodes (1619)