When Data Goes Wrong, with Aaron Wollner


In this episode, Benji talks to Aaron Wollner, the Chief Marketing Officer at Quontic.  

Discussed in this episode: 

  • How to fight against data manipulation
  • Pulling insights from the data
  • Working to build a marketing team that implements insights from the data regularly 


+ Marketing Brew does the hard work for you, dropping a quick-to-read, free newsletter in your inbox every weekday. Covering essential topics from influencers and advertising to social media & more, Marketing Brew never misses a beat. Check it out now, click here.  

+ If you’re hiring, you need Indeed.

Sign up and get a $75 credit to sponsor your first job for better visibility, more applications and quicker hiring times. Stay in control with payment billing options, no long term contracts, pay for only what you need and pause spending at any time.* 

Claim Your Credit   

*Sponsored Job credit offers available only for new U.S. accounts posting a job that expires one year after account creation. Upon expiration of credits, users are charged based on their Sponsored Job budget. Terms, conditions, and quality standards apply.

Conversations from the front lines and marketing. This is B two B growth. Today I'm excited to have Aaron Wahlner here with me. He's the CMO over at quantic and Aaron, welcome into B two, be growth. Thanks so much, glad to be here. Yes. So, I know in our pre call man we packed a lot of discussion in and when we were getting acquainted, just talking about man. What's the most helpful topic we could talk on? And you said something then that I think is just a really good starting point for this episode and for us today. You said when it comes to data, it's easy to go wrong and it goes back to inductive verse deductive. So if you're trying to just basically prove a point with the data set, you're going to be able to do it. And I was like Yep, like yeah, you hit the nail on the head right there. So talk about for you in your career Erin how you've seen that play out and why this matters a lot to you. Yeah, for sure. And and this is one of my favorite topics just in general. I'm a bit of a student as much as I am, you know, a leader to to my team and hopefully you know more or less within the industry as a marketing leader and a thought leader, but I truly am a student and one of the biggest learnings has come from my observation, especially over the past, I'd say, five years. It's really been pretty, pretty sharp and visible, which is, you know, everybody comes to the table with their data and so who's right and WHO's wrong? And it's even less important about right and wrong, but I think just openly, openly acknowledging that, you know, while numbers are black and white, right, three is a three, you know how you use that three and how it's presented. It is often, you know, misrepresented, and so so I think that's really where things get get interesting and you know, I think it just comes down to that, that awareness, and if you have that awareness then you can sort of engage in discussion. And okay, so when you when you see that, when you say that, what do you mean right when this goes up and that goes down and you're seeing those two things be related? Tell me more about that. Right. So I think it's more about that awareness that, you know, a data point is in the end of the discussion. It's the beginning of the discussion. Great way of saying it. I want to know before we go further down this rabbit hole, because as a CMO, obviously data matters, but there's also this aspect when someone thinks of a cmo or someone thinks even, let's just say, of someone in marketing, they're not going to naturally jump to two numbers. So do you find yourself as an outlier among CMOS? Do you find yourself as as having a unique voice and that's that's helped you when it comes to data or like? I don't know. I just wonder what that's been like in that development of really loving this as a topic for you and how that's influenced your your CMO role. Yeah, the two word answer is not anymore, which is great, right. I think that the well Cmo, the chief marketing officer a isn't that...

...old, right, and they're you know that the famous Harvard Business School paper that was written probably six seven years ago at this point, that the average lifespan of a chief marketing officer is less than three years, was like two point something right. So that tells you something. That tells you really what it tells me is is that the role is poorly defined and evolving quickly, and so I think that we're sort of settling into a spot where it's better understood what's expected from a chief marketing officer, and that really is growth, right, and that's more sort of connected to the business. And the tricky part is that there's different types of chief marketing officers and that that article in that Harvard put out a number of years ago went into this. But you know, the CMO at PepsiCo is a fundamentally different than Cmo than you know CMO at a fintech startup, and the CMO to fintech startup is going to be way more numbers oriented and probably came up in that environment where it was all about cost per type metrics and being laser focused on acquisition as opposed to sort of big brand place. So it's it's an interesting sort of evolving thing. It is very interesting and that idea of just having your eyes on revenue and on lead creation, I think, is a big deal and a big part of your role. So it does make sense. And it's interesting though, because different people, different CMOS, will come in with like a heavier, more design background, or some will come in with a heavy data background, but ultimately it's it seems like those that rise to the top did a lot of work in to understand the business more holistically, and that's obviously in any position in business. But the more you understand from a large scale it's it's going to be beneficial to your career. So that makes sense. Okay. So jumping back into just the data side, if the problem is data manipulation, and I would say it's rampant, I want to paint the clear picture of what the cost of that is. I mean, I guess play out the horror film that is data manipulation in your opinion, and maybe like a real world example for us, Aaron, sure, yeah, I give you one from yesterday. So we've got a pretty healthy a B testing program on on my team. That's well run in my opinion, and we take a scientific approach. We have hypotheses and you know, we pay close attention to statist still significance that we've got general sort of best practices, and so we have a good we have a good program right where a B T always be testing. It's one of the one of the acronyms we have on the group with the group, and yesterday we were looking at the latest results from the last few months of tests and the biggest increase came from a test where we removed F D I C right, like, F D I C insured in terms of hey, we're a bank, you know your your deposits are insured. We removed that from the top of the page and that told us that it increased conversion or APP starts. And the more we looked at it,...

...the more we realized that we didn't really keep to our best practice rules and we let that test run only seven days. And one of the one of the rules we have is, you know, no, no, two weeks are the same. So two weeks minimum for tests, even though the separation was strong. So this is where things get tricky, right like there's no one right way to do things. That that's the hard part. So we did the right thing in terms of we've got statistically significant separation between A and B, but we didn't really let it run long enough and even those rules are fuzzy. And so you would conclude that or you could conclude that we should take F D I C ensured, you know, away from this site entirely across all, you know, bank product pages, before you go and do that, think hard about that one test and maybe run one another test exactly the way you ran it the first time or run a similar test elsewhere. So I think the broad application of an exciting looking result is a really easy trap to fall into. So, with that being literally, like a very fresh example, that was was the conclusion. Basically the insight like we need to just run a longer test or we need to go back to our best practices. Exactly we're going to run the test. Yeah, I think it's interesting because part of the breakdown, and you you said this really well. It's actually had me thinking ever since our first conversation, but the breakdown between data and actual value or actual insight. People think like well, I have all these these numbers, I have all this information, but that doesn't naturally lead to like the right next step, the right insight, the way forward, and so this is something that you've been a student of but also like helped define. Talk to me about how would you move from data to actual insight? I would love to just break that down for our listeners. Sure. Yeah, so, my last cent in the agency World I ran the data analytics practice us and had a pretty talented group and we developed sort of our own methodology and it was it was a fourth step process. M that started with data and then stepped into information, then into knowledge and then then to value. And so that's sort of a four phase or four step approach where you have sort of you know, you can look at each step in isolation and understand how you went from a number on the left, right the data, all the way to value, right, and and so just to kind of go through each step. Data would obviously be a data point. Right, think of like a back end data base where you'd have a set of raw numbers. So you know, you know three percent and right, so you've got these two numbers that use rat and on. That's data. To make that information, right, you have you'd have to sort of carve that out more clearly and define that better in terms of the time period, in terms of relative movement over time, right, so three weekly...

...reporting or exactly like like web traffic over time. Right. So now you're starting to look at information. Right. So you went from data to information. To go from information to knowledge, it really is about more more context, right, so that that's where you sort of zero in on the thing that you're trying to figure out or the interesting thing that you're that you're looking at. And again that's just more context. You could be slicing it in terms of a type of traffic, rights organic traffic, and you could be looking at organic traffic going up over that time period and the amount of PR that you put out. Right. So you're trying to correlate to things. For example, what is the relationship between the stuff that we're putting out there and PR for the brand and organic paid search for your brand term? Right? So now, now you're in. Now you're in step three, you're you're in that knowledge space and then getting from there to value. That's where the magic happens and I think that's where insights live and you know, that's where you sort of create hypotheses and it's supposed to lead to tests, right, because I think all all great marketers, you know, tests before they just roll out for the most part. And Yeah, so we created sort of this this process where we can go from, you know, a number to delivering value in a little bit more of a systematic way. It's interesting because you could look at data and draw like like, you can walk through that whole process, but to actually create true change, like you'd also have to be able to identify issues in the data right, like wrong data. Then you walk it through that whole process. Information, knowledge, value could lead you one way, but there's you have to be able to actually look at the data and and see any sort of inconsistencies or or real issues there. What stands in the way of that? When we're looking at the data? Are there ways we see it wrong or ways we get it wrong? No, I think if you follow that a right, it's not going to lead to this beautiful, you know, nugget of wisdom every time, right, but it will help you sort of follow a path and understand where you are in that process. So I think that just being able to sort of you know, let's say you're in a meeting and you know your your peer brings some data in terms of the product. Right. Let's see, you have an application to sign up for a new bank account, like we do. Right. So if they bring in, you know, data around the application to submit rate, just sort of making up a metric and how that's been volatile in to two picking a time period, right, you can help. It helps you contextualize. Where does that live on sort of the spectrum? Right, is that data? Is that information, is that knowledge or is that value? And then you that can help you sort of situate like where, where do we start this conversation? Because again, when you when you look at a data point, you know with your peers, it really should be the beginning of the conversation at the end. What I like about...

...thinking about data in this way and these four steps is it actively exposes kind of how hard it is to go from data to valuable insight. And once you know the steps, obviously there's there's a part of it that's like it gives some ease, but in reality I think people, a lot of people, will jump from data thinking they have an insight and they miss those steps in between. I wonder if you zoom out and you look at how in like the B two B space or businesses at large think incorrectly. What what does it look like to maybe pivot towards more effective use or pulling of insights from data? Yeah, I think from especially from a B Two b standpoint, you know you're talking about a longer sales cycle. You're typically not talking about you know, let's just say something like hundreds of thousands of consumers that are purchasing clothing. Right. That's the typical sort of B two C model. So it's actually much harder. It's not necessarily good news for the audience, but you know, I'm not saying anything that that probably the folks in the B two, b were a don't already intuitively understand. But you know, if your sales cycle is a year, let's just say, right, and you're selling software to enterprises, you don't have tens or hundreds of thousands of data points to to look at and sort of you know, draw from. And then you've got all these wacky outliers, right, like a company, for example, that we spoke to in two thousand seventeen pop back up on our radar because of this really unique situation and the leader that had the software from another company joined the company we talked to in so there you go. You've got an example. You technically have a data point, but what does it mean? What do you do with that? Right? So, just to sort of finish that off right, that could spark an insight of okay, let's scrub our database and see if we can identify, you know, prior customers, are current customers who have moves companies in the past six months. Right, and now we're talking right, and now, you know, we're not debating the numbers, we're not talking about how hard it is to draw conclusions from so few examples, but you're sort of, you know, putting a good idea to work based on not an insight again, but something that's further upstream that sort of you know, is able to propel the discussion forward. B Two B growth will be right back. M M. It feels like the hardest leap would be from knowledge to value. If you were thinking about it almost as like a muscle, if you will, and like needing to strengthen that knowledge to value muscle, like, what would you you provide as like the way forward for that, Eran? What what can we do as B two B organizations to to strengthen that knowledge to value muscle? To me it's business understanding, right, and the B I person, the business insights person, is one of my favorite people. Wherever I am, you know, I make sure that that person is by my side, because having good context of the business, how the business runs, the mechanics...

...of it, will really help you crystallize and jump from knowledge to value. Right, because what matters you know how is revenue actually being generated? You know what's there between a smaller customer versus a bigger customer. So to me it's really strong under a really strong understanding of the business and, and this has just worked for me personally, common sense, like I do not want to under emphasize this, and I actually remind my team pretty regularly. If you've got a good, strong sense of you know what makes sense, that goes a long way and it actually helps you create really strong insights, believe or not. Can you give me an example like what you mean by that or like how you would emphasize that to your team the common sense side? Yeah, just to go back to the C D F I example, right, and I can do another example as well if you like. But you know, to not jump out of our chairs at lift and conversion based on removing se D F I, right, and to apply some common sense there of okay, so that that, you know, icon is meant to represent security. Right, your money is secure. So does it make sense to remove that across the board? Does it make sense to lower that down on the product page? Like so, thinking more practically about put yourself in the consumer shoes, you know, makes sense of the thing that you're testing and what you're trying to achieve. And always, you know, one of one of the things that we one of our core values at quantic is know the goal, right. So always know the goal and I would put that in the bucket of common sense. So, you know, I think that again, it's not you know, as marketers we see we see, you know, positive result and we just want to sort of, you know, passing around town. And before you do that, you know, I think you know, apply, apply a layer of hey, what do I think this actually means? What are we testing here and what are the implications of that? Yeah, if you're gonna give like the B two B growth audience and in our community, a homework assignment of sorts and invitation towards improving ourselves and the way we think of data, after this conversation, what would you advocate for? What what should we be thinking about maybe changing or starting Eron? Yeah, I think it's a little bit of statistics. I think that in the B two B world, whether you're in bad or marketing or somewhere in between, I think having at least an appreciation and understanding of the basics of statistics is so important, right, so that you can sort of be more confident in the numbers. Right, I think the biggest thing is, you know what is statistically meaningful, right. So, for example, we just put out survey to our broker audience for the B Two B side of our mortgage business and we sent it out to I don't know, let's say fifteen thousand broker partners again on the B Two B side of the mortgage business, and we have so far received a little bit under a hundred responses. Right. So you could rush that to press and of a whole plan around like...

...how we're going to use that in a Webinar and draw people in. Hey, you know, listen to what your peers are think about the mortgage market going into que for it's really it's going to be a great campaign that we're gonna put out there in a number of different ways. But, you know, and equals, right, like there there is a statistical requirement there that you need to, you know, check the box that you're not, you know, giving out results that are not statistically valid. So I think no matter who you are, what you do, having some statistical basics what will go along way in Mi co well, I really appreciate you taking time and being with us on B two B growth today. I know data is a recurring topic, obviously that we're thinking lots about and something that people are always trying to to refine that skill. I find the data, information, knowledge value thing to be something that's stuck in my mind, both in the context of B Two B marketing but even outside, just in like life, when people say a stat now er and I'm thinking about what you said. So thanks a lot for that. But tell us a little bit more about quantic and the work you guys are doing before we close out today. Absolutely. Yeah, so we're at an exciting phase as a company. So quantic is a digital bank that was a community bank. It's about ten years old, a little bit older, and we fully went through digital transformation right, which is just basically a buzzword at this point. But the cool part about it for us is that we can point to no branches. We can point to, you know, fully bringing in and servicing customers digitally online. So we did it. We actually kind of did a sink or swim approach, which was a bit scary, and closed down our last branch about two years ago. So an exciting time. We've we've grown our our customer base about four x in the past eighteen months. So I know the name of the game here is growth. So so that that's a nice data point to sort of know, showcase that. But a lot has gone into that. So I got a talented marketing team and and data analytics, of course, is it's a big part of that. And sort of the last thing that I'll highlight here is that we're innovators and we see that as a real differentiator for us and some of the stuff we've innovated and put out there recently is a wearable payment device in the form of a ring, so you can pay for stuff with a ring on your finger. Super Cool and it comes free with your checking account. We were the first digital bank in the metaverse, which was just sort of a really interesting exercise, and we explore the metaverse and invited our customers to explore the metaverse. And I believe it's your cover photo on Linkedin as well. Right. Yeah, it's so cool. It's so cool we have like there is no quantic branch, but now you've got this sort of you know, digital three D quantic branch in the metaverse. It's just it was super on point for us. It was a ton of fun to execute and I think our customers really enjoyed us sort of inviting them into the...

...metaverse of showing them, Hey, what is this thing all about? So we just we just put a bunch of innovative stuff out there, including a crypto rewards card as well, so earning one point five percent back in the form of Bitcoin. And Yeah, the Digital Bank side of the business is a ton of fun. And then we are we're mortgage lenders as well and we live in a pretty unique space in terms of lending out to those who are often overlooked, outside the box, borrowers, and we've got a whole sort of, you know, set of mortgage products tailor fit to that audience. So, yeah, it's been it's been a fun couple of years here at quantum, fascinating. I have never been in the banking World Erin but I love just the innovation and it's it's really compelling too, from the outside looking in to see what you guys are up to, and so it's been an honor to get to chat. People can go to what's the website for people to check out quantic quantic DOT COM, simple and easy to remember. Love it. Aaron Walner, thank you so much for being on B two B growth today. Man, it's been been a pleasure chatting with you. Absolutely appreciate it. Bendy. For everybody listening, if this is your first time checking out B two B growth and you have yet to follow the podcasts on whatever podcast player you're listening to this on, go ahead and do that. We're having conversations that will help continue to fuel your innovation and your continued learning, and if you want to reach out to me, you can do that over on Linkedin. Would love to chat with you about marketing, business and life. Will be back real soon with another episode. Keep doing work that matters. B Two B growth is brought to you by the team at sweet fish media. Here at sweet fish, we produce podcasts for some of the most innovative brands in the world and we help them turn those podcasts into micro videos, linked in content, blog posts and more. We're on a mission to produce every leader's favorite show. Want more information, visit sweet fish media DOT com. MHM.

In-Stream Audio Search


Search across all episodes within this podcast

Episodes (1805)