Carl: Hello and welcome to the SaaS growth podcast. This week we're here with Sahil Patel, the founder of Spiralyze, a company specializing in conversion rate optimization for B2B SaaS How are you today, Sahil?
Sahil: I'm doing really well, Carl. Thank you
Carl: Give me a quick rundown of Spiralyze and what you do and how you came to be.
Sahil: Yeah, I'm the CEO at Spiralyze. We are an A B testing company. There are 34, 000 websites that run A B tests somewhere on their website, and we crawl and scrape all of them. So we find the best A B tests that have worked for other similar companies, and we run those for our clients.
Carl: Oh, interesting. So how has that worked for you? And what results do you get for your clients doing it that way, rather than more traditional approach to A B testing?
Sahil: Yeah, I would describe the difference. We call it predictive CRO, predictive conversion rate optimization. And the difference between that and what I would call best practice CRO, what most folks are doing, traditionally doing, is that we run tests where there's aggregated show that it's already worked. Because what most people end up doing is they run some, what I would say, they pick the low hanging fruit, and then it gets a lot harder. So then they look around and they go, Oh, Hey, our competitor I just noticed all their call to action buttons are orange. Maybe we should try orange. I saw that our other competitor, their call to action is get your free trial and ours say start your free trial.
Let's try that. What you end up with is a lot of meek testing. And me testing doesn't move the needle and it doesn't get everyone excited and it doesn't generate a lot of return on investment. And by the way, if you're going to do conversion rate optimization and more than just dabble in it, there's a meaningful investment to do it.
We can talk about why that is, but it's, it is it's not a an easy undertaking. But if you start with instead tests where multiple companies. Especially ones that are similar to you, and there's a lot of ways to define similar to you. By the way, certainly it's someone operating in your industry vertical, but it's not just that.
It's more than that, and they converge on an answer. It's a much better starting point to run a test, number one. Number two, it allows you to take bigger swings. Not changing the button color, you're not changing a little bit of text. You can really rethink And reimagine the whole experience, whether it's on your homepage, your free trial, sign up your pricing page, take lots of big swings.
And when you take big swings, sure, some of them are going to swing and miss, and it's going to be a quote loser, but you're also going to find some big wins, the double digit gains that have a big impact on your business.
Carl: That's the nature of these sort of optimization tasks, right? Because sometimes the optimization is the wrong way. Sometimes we get it right. But until you prove that, you don't know. What
Sahil: Yeah, to get big wins, you got to take big swings.
Carl: sort of things have you found have worked with these big swings in the past for some of your clients?
Sahil: Yeah, I'll give you three great examples of things that we never would have found on our own, if me and our designers had sat in a room with a whiteboard or a zoom whiteboard and said, okay, what should we test next? I don't think we ever would have come up with these, but we saw other companies do them.
And we knew it works. We said, why don't we try them? And that's the nature of experimentation. Try new things. Okay, first one is called showing the product. Sounds like a simple idea, but it's amazing how many companies don't do it. And by the way, I think for today, everything we're talking about is really the starting point for the context is B2B SaaS.
So I'm going to borrow from some other verticals, but everything I know is how do we apply it. In the world of B2B SaaS. That's the world that I know. That's where I have something to share and some insights. Let's start with showing the product. It's a really simple idea. If someone is going to check out your, they're going to buy your CRM tool that you sell they're there to buy software to show them the software.
Let's borrow from, let's think about where we borrow this from. Think about, Carl, can you think about the last time you saw it? An ad for a car. Can you think of what ad it was?
Carl: I can't probably like a Nissan or something on YouTube.
Sahil: I'll make an intelligent guess of what was probably happening in the ad. It probably showed now you picture this in your mind and you tell me if I'm getting this right, probably showed the car driving on a either a nice road like a curvy road with lots of trees.
If it was like a sedan or sports car. And if it was more like a truck or an SUV, it probably showed it going up a hill people camping. Can you picture one of those? Can
Carl: Oh, definitely. I've seen hundreds of those ads. I've definitely seen the Ford ads where they have them driving over like basically hills and gullies and all sorts. Yeah. Yeah.
Sahil: where are these hills and who's driving on them? And of course, can you picture the little subtitle that it says? Do you remember what it usually says on these ads?
Carl: No I'm a
Sahil: I'm just adding a little bit of color.
Carl: pick me on this
Sahil: You need to say something like, closed roads, do not try this at home because it's not real driving conditions. 99 percent of people are not taking their truck over a stream, climbing rocks. They're going to, taking their kids to like football practice or gymnastics or ballet.
Taking their dogs to the vet, taking pictures. Now, I want you to picture this in your mind. Imagine the same exact ad. Whatever the people were doing it. I want you to subtract the automobile from the ad. What would you be left with? What would be in the ad?
Carl: In the case of the Ford, it'd be a bunch of hills. No one's, yeah. Or the windy road.
Sahil: It'd be really weird. And would it get people excited about checking out a Ford?
Carl: Not without the Ford, no.
Sahil: Yeah. Yeah. The Ford F one 50 without where the F one 50 is just it's a bunch of lifestyle images, but you could do it. You could show some people, they got their camping gear. They got their dog. They're going on a weekend trip in the wilderness. But if that's all you got, and you didn't actually show them putting the stuff in the back of the truck, driving up the hill, just the ad doesn't work. This is a great example of learning something from AB tests from other companies and applying it to B2B SaaS. Many verticals long ago came to the conclusion that you got to show the product somewhere in your ad. It sounds like a really simple idea. You're selling trucks. You got to show the truck.
You're selling musical instruments. You're selling a guitar, you gotta show someone rocking out on the guitar. Now you gotta, you can show some other stuff like the people, they got their camping gear, they got their dog with them, they're having a marshmallows at the fire. Sure, we call those lifestyle images, but you do the lifestyle without the truck.
It's not that.
Carl: Yeah, I've been doing some audits on people's landing pages and onboarding, and that's one of the things I always look for is before I even buy your product, do I know what I'm gonna get?
Sahil: Do I know what it is? Yeah.
Carl: yeah.
Sahil: Now our brain processes images much faster than words. You gotta read words, you gotta think about them, you gotta synthesize them, you gotta, you see an image, you see a truck and you go, this is a car ad. You see it software for something that looks like a sales pipeline. You go, okay, this looks like some kind of sales tech, CRM, sales enablement, something showing the product on a landing page.
Let's take a classic B2B SaaS. Start your free trial that one of those landing pages or a demo request. Almost all B2B SaaS. It's one of those two things we found when you show the product. On average, you get 20 percent more conversions and as an A B test one in two times 52 percent in fact, it beats the control.
Carl: It sounds easy. For something that's so easy to do, like you've already got your software, it's just a quick screen recording. That's all it needs to be. But yeah, you should definitely do it.
Sahil: it doesn't have to be perfect. There's lots of edge cases. Oh, we don't have our product is stale. We're just redoing the UX. Our product is complicated. Oh, we have seven products. Which one are we going to show? I can unpack those if we have time or it's important, but just as a principle, the data shows this is a test that has worked for many other companies.
It's also worked for many other industries. And as an A B test, it's something that everyone listening can at least try as an A B test. You don't have to have a heavy sophisticated, robust A B testing program to try something like this and then measure. Is it getting more like number one number two?
You said something really interesting, Carl, which was you have audited landing pages. And the first thing you ask yourself is what the heck is this product? Did I hear the kind of right observation there? Tell me
Carl: more or less it. That's more or less it, right? So I think that it's very important for I was looking for more of a free trial conversion side of things. So helping people with their customer onboarding. And one of the first things you need to do is set your expectations for customers.
And when they go in one, whenever you click your free trial button, you've actually had a handful of opportunities to already start onboarding them and get them like at least aware of what your product's going to look like. So that when they arrive, finally, you've got the opportunity for them to go through familiar flows that you've already shown them, things like that.
So yeah, it is that. Removing the surprise. Yeah.
Sahil: Moving surprise. I call this the one second test on a landing page. Your audience, let's put who the audience is audience for a landing page. Most of the time is a stranger. They don't know you. They don't know your business. They may not even have heard about your brand, especially in B2B sass. We're not talking about, direct to consumer brands that have spent hundreds, millions or hundreds of millions building a brand presence like Apple or Nike, where you just know it, you see the Nike swoosh, everyone has the brand.
Now, do they want to buy the product? Different story, but there's that high level of what marketing people call unaided recall. Most B2B sass don't have that. They don't have the luxury of that. So they're changing and they typically make up their mind in about one second. Are they even going to stay on your page now?
We're not him out. Are they going to buy? We're not him out. Do they want to sign up for a free trial? We're not him out. Are they evaluating? Is this the right product for my organization? Or does this satisfy the jobs to be done? Or my technical reports? Those things are happening in a second. Second, they're just like, am I even in the right place? It's that's the mindset to put yourself in and there's two really interesting tests. Two things I've learned from doing CRNL for thousands and thousands of websites. One is, and again, this is a really easy one. I love to do this with executive teams is take the landing page or take your homepage even better. click. You do this in a Google Chrome browser, right click and choose Google translate and put your homepage in a language that no one on your team speaks like Arabic or Urdu. Or Hindi. Okay, now I'm picking things that in the U. S. Most people don't speak and then show your home page to five people that you don't know, like just colleagues or friends or neighbors and just they want people who don't know that this is your company and just ask them.
What do you think this company does? Categorically? Okay, now you're not looking for they make sales enablement software for mid sized businesses, right? Based in Finland. Okay, that's specific. What you're looking for is something like they make accounting software or they make business software.
Even that would be pretty good. It's amazing how often they can't do this because the landing page, the homepage, I think, especially for the homepage, it's relying too much on the copywriting and not enough on simple images that instantly tell your audience's brain they're in the right place. Right now.
In real life, everyone's reading your website in their, whatever they're probably fluent languages, or at least what they're probably what their main language. So you're going to get the benefit of the copywriting. I'm going to talk about the copywriting a moment. You're getting the benefit of the copywriting and the imagery.
But when you remove the copywriting from the equation, you get much faster to some approximation of what your audience's brain is making a snap judgment on. I also like this because it's fun. It's a little bit of a parlor trick. And you can see as soon as you suggest it to someone, they start getting a little bit nervous because they know they can tell they're like I think we're going to have a problem.
Yep. Our homepage has people lifestyle images of someone wearing a business suit, looking happy, just like every other company. No one on earth is going to know that we're that we make a FinTech solution for. Private equity company, right? It's just fake, and it's fun. It's great with the C suite, too, because they have to confront most of the time.
Go ahead, Carl, you're
Carl: No, I'm sorry. It just sounds basically what the practice is making your homepage look like what you are, and you're saying that a lot of places fail on that. And the too generic and I, we've all seen this top photos of, someone smiling with a nice shirt and a, like a key chain, like a badge
Sahil: Yeah and I think the other mistake just, and not, I love what you said there. You evoked a particular image. The professional end user, happy person. I think you're right on and specifically the re there's a couple reasons people end up with it. And one of them is actually for a pretty good reason, which is the executive team somewhere along the way. Got it in their head that if they show. An image of their customer of their ideal customer marketing speak. We call that the ICP ideal customer profile or ideal customer persona that they will recognize themselves.
So if you're selling some kind of billing software to restaurants, you might show the the shift manager at a restaurant. They may even have some Real but misguided data that shows heat maps or mouse hovers or eye tracking that shows. Oh, when you put a person's face, everyone's everyone looks at it and they're looking at. So there's high engagement. It's working. And this is a great example of imposing what you want to be true.
Fitting the conclusion to the data instead of the other, or sorry, fitting the data to the conclusion instead of the other way around. As humans, of course, if you see another face, you are instinctively drawn to that face. But it has either no or little predictive power about the person's likelihood to convert because they see a human face. You can make lots of speculation about why that is, but the data is very clear, like the fact that someone's eye is drawn to someone's. It doesn't mean that they're now more likely to buy or that they understand what's going on. But that's where it's rooted. It's not, just putting aside all jesting.
It's, they probably had some data. They probably had a consultant tell them like show a picture of Your audience and they recognize that this is the right place for them. In B2B SaaS, you're so much better off. Show them something instantly tells you what you are. And if you're not sure if you're in the right place, do the one second test.
Do Google Translate. Show it to five people. Can they guess what you are? The answer is no. Just keep doing it till you find a better answer. Show a different image. See if you get there. Okay, so that's number two. Here's number three. You mentioned the copywriting and I'm going to go on there because the thing we just talked about, the one second test, it subtracts the copywriting from the equation.
So it's a fun way to isolate the power or lack of power on the imagery on your homepage. to bring it into the real world, of course you have no one really looks at websites that way. They're an English speaker. If you have broad aspirational. Headlines. The headline is the biggest font on your homepage or your landing page. No one can tell what you do. And my guess is, Carl, you've had some of this experience auditing landing pages. What's that been like? Tell me. I bet you can come up with some of these headlines off the top of your
Carl: The thing is, I almost just discard them once I've read it, but it is that sort of empowering synergy or something like that, or it just doesn't mean anything. It's great, but you haven't told me anything about who you are and what you do.
Sahil: You've got, I like to call it word soup. And cliche salad, you just string a bunch of marketing jargon together. Or you put together something immediately. It's like next generation contact management software. Number one, like number one, what, like, how is that even true? Like, how would you know that you're not, look, let me say, there are some if you get like the Gartner. Rating that legitimately puts you as number one in your your category. You should brag about that all day long. We know it's really powerful. Holds persuasion for most companies, some self declared self congratulatory description. Just it doesn't tell your audience anything. We can put aside my opinion about whether it's earned or deserved or undeserved factually.
It doesn't help your audience figure out. Are they in the right place? And should they stop scrolling? Or just say, this is yet another website, and I don't know what it is, so I'm going to go to the next one. Most B2B SaaS companies are in categories 20, 30, 40, 50 other companies that are legitimately, they share that category. And if you're lucky enough, they actually found you. You got to do something to set yourself apart. Here is the A B test. So I just, I was setting the stage for great A B tests that we've seen work, learned it from lots of other companies. Call it. Bold claim quantitative headline. So you want to scribe a benefit that your end user has, and you want to quantify it and you want it to be believable.
Carl: like
Sahil: And it's so simple.
Carl: percent or something like that.
Sahil: Sure. By the way, you could check out the Spiralyzed homepage. By the way we. We we drink our own champagne on this one. And by the way, I'll be the first to say we're always learning and we're always trying something new. And there's no perfect answer to any of this.
That's the beauty. That's the, why you should be AB testing. If you take something like that, let's say you, I was working with a company that makes financial software that improves the process of Closing your books at the end of the month and generating financial reports and they classic next generation software, blah, blah, blah, blah, blah, jargon, vague vague.
What the hell is it? Probably 50 companies in this category. And we workshopped some, just some really simple ideas. Cause I said, okay, why do people buy? And the person I was working with, she said you can make reports faster. And I said how much faster? And she said in our opinion, it's 90%.
And I was like, okay, No one believes anything is 90 percent better. Like it may be true, but you have to haircut it a little bit to make it credible. I said, okay, could we agree? Maybe like 30 percent would be more believable, even if 90 percent is true. She goes yeah that's fine. And I said, okay, faster than what I was like, cause I, I have been in tech reporting system.
I'll admit it's good. Not great. If you're really 90 percent I'm like what do you do? Are you automated everything? And she goes, Oh no, our audience are usually people that rely on spreadsheets and we're going to bring them into the modern age of automated cloud based reporting software.
So I said, great, could we do something like this? FPNA software 20 percent faster FPNA or FPNA software. It's 20 percent faster than spreadsheets. And she goes, yeah, boy, that's a mouthful. And I go, yeah, don't. Use that literally sit down with your copywriters and get it to where you get it to feel rooted.
But we just put it next to each other. And she didn't even need to AB test it to know that this was going to be something provided better clarity. Now the outcome got to AB test it to know if it works or not. But these are three great things show the product, make it instant past the one second test and use a bold claim quantitative. That The data shows an aggregate. It works over and over again, and there's three tests. I think many, maybe all folks who are listening today could try workshop. And if you've got the tools to A B test it run some A B test without a heavy investment to run those tests.
Carl: So the interesting thing about the quantitative thing, I've always called that establishing a dream. So like we use when you are selling to customers, it's like they need to know what they want out of your software. So it seems that's an excellent way of doing that. And if you can't establish that, it's really hard to tell your own customers here's why you should use us.
And the fact that it's okay, weird. Even faster than spreadsheets. If you're using spreadsheets, you're like shit that's me. And I know exactly what I'm going to try and do when I use the software and exactly what it's going to do for me.
Sahil: Exactly. Now, look, if you're already using Oracle, this is not for you.
Carl: Yeah. That's it. It's not for you.
Sahil: there may be a different value prop. If you're going to migrate from Oracle or NetSuite, I think that's who owns Oracle now, to PackageX. That's a different story.
Carl: So the one thing about AB testing, that it's definitely a problem, especially once you get to the sort of past clicking the free trial button or something is just really having the numbers that you need to get a good sort of data set. So how do you manage that?
Sahil: Yeah. Ko, I think what you're asking is broadly, you want to run tests so you
Carl: you don't have a lot of traffic.
Sahil: yeah, you don't have a lot of traffic. And just as a general, you don't need a an advanced degree in statistics to know you need to measure. The reliability of your test.
Carl: 10 people isn't a reliable
Sahil: 10 people is not going to do it.
And you have to balance that against Here's a challenge all of us that work in the B2B SaaS world have compared to say A direct to consumer website that gets thousands, perhaps millions of visits and clicks and purchases in a much shorter time, like in a day on over here. You have B to B sass. Let's say you're making you're selling we'll go back to example.
You're selling fintech software billing software for restaurants Sure. There's a lot of restaurants, but there's only so many of them that you check out your website. And over here, you have like booking. com, like the gold standard of AB testing. They've been doing it for 20 years. They claim they're running like 100 different tests every day or some ridiculous number.
And they're testing every single possible thing on the page. And most people think they are probably much more The fintech company than booking dot com. So there's a couple of things you can do. And by the way, here, I'm going to tell you where my answer is not going to go, which is in place for a lot of the I call them the.
The high risks of purity, and they're going to, I think, talk you out of a B testing because they're going to, they're going to introduce all kinds of very sophisticated and, by the way, very valuable methods of calculating the how much power the test needs. and the statistical significance and the confidence level and why those are three different things. And by the time you're done listening to them, you're going to be convinced that no one except booking.
com and Amazon should be running A B tests. And I just want to go on record by saying, I think this is terrible advice. And I'm not saying these are terrible people. They're really good people, they're smarter than I am, they know more about A B testing than I, they've forgotten more about A B testing than I'll ever know, okay?
If you're out there and you're listening, or you know someone that does this, I want you to, I have this self awareness of where I stand and where my expertise is versus these people. I think they're well intentioned, but just not practical. Here's what I would do instead. First. You should run big swing tests. This is a lay person's way of saying don't test little tiny things because the smaller the effect size, I'm going to use a little bit of statistical jargon here.
The longer it takes to see if it made a difference. If you change your button color from orange to green, first of all, I'd be shocked if that actually made a difference, but it might take you months and months to detect what is the
Carl: You have to get there, yeah.
Sahil: Okay. What does that mean? It means you usually have to combine multiple ideas into a single test. The purity priests, the high priests of purity are going to cringe and they're going to go, this guy doesn't know what he's talking about. If you change multiple things, how do you know what made the difference?
Carl: Yeah,
Sahil: You need to run a multivariate test. You change one thing, one test, you run them again, then you do an A and a B and a C and a D.
And I'm just going to call bullshit on that. Because if you do that, you're going to run like three tests a year. And who is going to do that?
Carl: I suppose the problem I'd call that is that those are called hill climbing optimizations, right? Where if you change the button color, all that does is it might improve on your current approach. But if you want a more dramatic change, then I think what you're saying is right, is you've got to try much bigger swings.
Sahil: Bigger swings.
Carl: yeah,
Sahil: Now, big swing almost always means changing three, four, five, six things on the page. And I want to directly answer the concern, but it's a valid concern. All my kind of joking aside, it's about a concern. You changed four things, how do you know which one made the difference? And my answer is you do not. You are 100 percent correct. You change four things. You don't know which of the four and can you attribute that we move the form to the left and we brought it above the fold.
And then because we brought it above the fold, we had room to put the social proof, the client logos higher on the page. Also, we changed a quantitative headline and we turn the wall of text from a wall text to set up bullets. Okay. Did they all work? You don't know. 100 percent valid.
Carl: My question is, does it matter?
Sahil: It matter?
Carl: totality of it works, if it all works, and you've got a 10, 20 percent increase, does it matter which little individual bit?
Sahil: Take the win! Take the freaking win! Go to your CFO and COO and say, dude, or dudette, we just got it! 15 percent lift. Let me tell you, your sales team is going to be through the roof because they're getting more leads. Now, can you get better leads, higher quality leads? Because more is not always better. Sure. I think that's outside the purview of today. There's lots of techniques you can do about how to improve the quality. You can do lead scoring. You can ask qualifying questions. You can, Be more targeted with the imagery and the copywriting to make sure it's really appealing to your ICP rather than just a bunch of random people.
Okay, really important. You don't stop. Take the win and stop and I'll go home. That's you have nailed it. You take the win. And let's make an analogy. You're an athlete. And you want to get better for the next race. You go to your coach and your coach says, Hey, Carl, let me ask you, Carl, would you play sport? Did you play sport growing up? What was your sport?
Carl: yeah, I played soccer.
Sahil: Soccer. Okay, awesome. I'm a soccer player too. You're getting ready for the season. You got diet, you got fitness, you got ball control, and if your coach says, Carl, I want you to, I want you to do more interval training to work on your high burst speed.
I want you to mix it in with some running, so your endurance is better during the course of the game. I also want you to work on your touch on any ball that's coming in the air, because you're great at trapping when the ball's passed to you. You're not so good when the ball's in the air, okay, and you're getting ready for the season.
Are you going to ignore those things and just do one thing? No!
Carl: Of course not, no.
Sahil: Of course not! Because, but Cole, how are you going to know, how would you know which one made a difference in your performance?
Carl: It doesn't matter.
Sahil: No! You have a chance to win, right? You're going to do all this. You're going to improve your diet, you're going to do some sprinting, you're going to do some long range running, you're going to get the, you're going to have someone kick the ball to you, you're going to practice your chest traps. All of those will make you a little bit better. Same thing is true in A. B. testing. You should try multiple things, take big swings, take the win.
Carl: I suppose it makes a lot of sense when you don't have as much traffic. Just trying to generate dramatic a change as possible. So cause if you would do those really minor optimizations to get, again, any sort of statistical significance, you'd need thousands and thousands of trials, which could, if you've only got a thousand people a month come to your site, that's suddenly five to 10 months to get to work out whether you should go blue or little green on your call to action.
But it's, and it's who has the time really?
Sahil: Who has the resource? There's an opportunity to cost all these things. And now let's pair that with the, we talked about this, because when you put it, by the way, in the terms of a soccer player getting ready for the season, everyone would be like, oh yeah, I get it. Yeah. Better diet, better interval training, better long distance, better ball control.
You do all those things to get better. The other thing you do. I'm going to borrow a phrase from outside the business world called kill your darlings. I have, are you familiar with that phrase? Have you heard it before? What's the context where you've heard it?
Carl: I've been a software developer for years and years. It's really easy to get too attached to some of your old ideas and it's just sometimes, things change and your dialing isn't as good for you as you thought it was at the time.
Sahil: Do you know the root or at least the attributed root of that particular phrase?
Carl: Not by name.
Sahil: There's a lot of variations on it
Carl: been around forever.
Sahil: Yeah, it's been around forever. It probably goes back thousands of years. Okay, so it's on your idea. But that particular phrase, which I love because it's so short and succinct, and it's evocative.
Like as soon as someone says it, you can picture it. It's attributed, or at least the way I've heard it attributed is to the beat generation, the beat poets. And their context was is when you write something, the creative processes, you have to be willing to kill it if it's not working. You can't get so attached to your poem or your phrase or your choice of words that you're not willing to say, Nope, don't worry, I'm gonna try something else. I'm going to now lift and shift, borrow, apply that to A, B testing and conversion rate optimization is that when you are running a test, and testing is a cruel game, always have tests that I love that don't win. And there are tests that I'm like, this is, why are we doing this? And it wins. In the context of you have a lower volume site, which most of us in B2B SaaS have is. You want to kill your losing test as early as possible. Now, again, the high priests will call it peaking and they'll say it's not valid. And I'll say, Hey, we got to be practical here. Now, don't kill a test after one day, but if you've done some calculations, but I also want to add, like, all of these are not to say you should not be rigorous in your statistical analysis. You should absolutely be rigorous in your statistical analysis. You should, if you're as far as where you can calculate what's called the minimal detectable effect, you should do it, sample size you need, the statistical significance that you're willing to accept, the confidence interval, all those things, you should do them. And then what you should do is you should look at them and balance them against the practical realities. And if you run a test, and it doesn't, it's going to take you six months to run this test, I think you need to. Face real life because it's just for most companies is not just not practical, but I'm going a little off targets there.
Let me come back to the main point. If you run a losing test, if you run a test and it's really underperforming in control, eight, 10%, I generally will recommend to my clients, we can build that test in less than a week. You want to run it long enough to make sure that it's not noise, or you launch the test on a Wednesday and maybe you tend to get traffic spread over the course of the week.
Want to make sure you get some representation. In my experience, you rarely get a test that starts out losing by a lot and then flips. At best, it'll maybe go from losing to neutral. So I kill those tests. I kill the darlings early. If a test is neutral, if it's an incremental test, I also want to be clear it's, I don't want to say that there's no room for small tests.
There's room for small tests, but you should make sure you have a lot of room in your roadmap for big swings. A big swing that is the most common outcome for a big swing is it loses by quite a bit because it's a, you take a, you roll the dice and you come out wrong. swings that are neutral almost always stay neutral. And again, I would kill those pretty quickly. Not as quickly as something that's losing big, but you don't need to run it to 95 percent stat sig to know that it was about as good as what you already had. When you have a test that's winning is when you should be skeptical. And those ones you should run to 95 percent stat sig.
But what you're doing is you're buying yourself traffic. The opportunity to come to a true and valid conclusion by killing your darlings. Kill the losers early, kill the neutrals. When you have enough data, you don't need StatSig to know something is neutral. What that does is it buys you time to then be skeptical, appropriately skeptical of the winners.
Carl: makes sense. You don't waste the traffic almost on these losing tests.
Sahil: Yes! You just put it better than I did. Don't waste your traffic. That's my advice.
Carl: sense when you're only working with a few maybe a thousand hits a month. You don't want to be spending a whole month on a failing test or a test that you know is only going to come out neutral.
Sahil: Great way
Carl: perfect sense. Yeah. Cool. So I'm just going to go to your interview page. Yeah. The only other topic we've really gone into in a big way is the A B testing done right. So you've obviously you've got a lot of experience in a B testing and we have covered a lot of that. So if you got any sort of tips and tricks for people who want to start a B testing and aren't right now, what they should start doing other than these is there any sort of tools they should use or is there a more simple approach you recommend
Sahil: So great question. What are some ways that you can get started faster or sooner than later? Let's start with the tool stack. The best tool stack is the one you already have.
Carl: always
Sahil: Here's what I'd say is don't spend six months on an RFP, put together the right tool stack. Yeah. Just. The opportunity cost of that is really high. If you're at a big, huge company, you may not have that ability, but for the medium companies and below, you probably don't need to run an RFP on something that's gonna cost 3, 000 a month. So that's number one. Number two, let's say you don't have a tech stack. Here's what I wouldn't do. I wouldn't try to manually do A B testing. That's a recipe for sadness, and it's a waste of time. There's a lot of really good tools out there. I will tell you, there's three that I really like. And I'm in, in no particular order among the three.
One is called VWO. Great company really good product. They have a free version, gets you started. Number two is a company called Mutiny, also a really good product. In the B2B world, I think they have some particular strengths on what's called personalization or segmentation, where you show certain personas or audiences different kinds of imagery or content.
The details are outside of the purview of today. Number three is a product, I think a little bit less well known on the market called React. It's owned by a company called Metadata. io.
Carl: yeah, that's one of the ones I've heard of just,
Sahil: Yep. Really good. I want to, in the spirit of disclosure, share that Spiralyze has partnership agreements with all three of these. So I'm not entirely I should say I'm not entirely unbiased. On the other hand, we've looked at a lot of products, and we also use products that I did not just name.
And there's some really good ones out there. By all means, I'll come back to if you have something that's other than the three I just listed, and you already have it, just keep using it. That's what I'd recommend. But if you don't have one. I would put those three on your short list. They're really good.
I know the CEOs of all three companies, they really believe in what they do and they have good feature sets in the specifics of do they work for you and your website and whether you're on WordPress or, Webflow or something else. You're the you're a better judge of that than than I am. That's the first thing I think we've talked before take some big swings and three If you have the in house resources to do it, but many people won't because they'll look at you Oh, I need a copywriter. I need a dev. I need a oh, I need a designer if it's gonna look right sure start out small, there's a whole cottage industry of part time fractional people who have experience doing CRO and there's communities of these people that you can find and they will be delighted to lend you their expertise for a fair price.
And I promise you, you'll get there like anything when you borrow or rent other people's expertise, you'll get there faster.
Carl: I just think freelancers and contractors,
Sahil: Yeah. Freelancers. Yeah. It'll get you off the ground. It'll get you off the ground much faster than anything else. And then I think once you get some wins and you know what you're doing, then you can evaluate how do you run a more sophisticated high velocity program.
Now, for some companies that will be you'll have the means to do it in house. And if you can, that's great. And for some companies, it'll be bringing in a partner like Spiralyze. Yeah. And obviously I'm self interested in talking to the audience that wants to do that. But I recognize that's a small sliver of the world.
It's a growing sliver of the world. It's a small sliver of the world. And I'd love the opportunity to work with those folks. But, to anyone who's listening, you're the best judge of where you are on your particular journey in the CR world. And I would just say, figure out where you are on that curve first.
And just match if you're just getting started. Maybe you want to start with a couple freelancers. And you can you want to learn from their expertise and then you get you're doing the crawl walk run. You walk and say, Hey, why don't we try this in house? See if we can do it. And for those that have the. You want to really take it to the next level. That's why companies like Spiralyze exist, high velocity, big swing testing, where we're working with clients who want to get double digit gains in website conversion rate.
Carl: Something I see a lot when talking about A B testing and even just website statistics and analytics is there's all sorts of metrics, right? So you have things like bounce rate, whether someone scrolled and all sorts of things that some big companies like booking. com would start trying to optimize for an A B test for did he use the scroll, click on this, go to this page.
What do you think are the important things people should be tracking and trying to optimize with their A B tests?
Sahil: Here's a great place to start. Wonderful question. What should you start? Should you focus on things like scroll depth or clicks, other forms of engagement, let's broadly call that engagement. I would tell you don't focus on those things. I would say for the first two years of any CRO program. You should focus on actions that produce revenue or sales pipeline.
Carl: That makes, yeah, perfect sense, right?
Sahil: That's it. Just keep it simple.
Carl: all the engagement metrics are really just all you're really trying to do is increase revenue, right? So it doesn't matter
Sahil: are secondary.
Carl: yeah, if everyone's staying on your webpage but not buying, it's that's actually a failed A B test really, cause you made that money off it, so what's the point?
It's a bad investment at that point.
Sahil: And here's why I would encourage everyone to focus on that. If you're building or you're in the early stages of your journey on CRO, at some point to get the resources to do CRO program, you need to go to the people that write checks in your company. And. Those people rarely part with money unless there is a demonstrable return on investment. And you're unlikely to sell that ROI story. If you're talking about we got people staying on the page for 20 percent longer. They're scrolling down, 30 percent more edges. You got to put it something in like our sales pipeline is now 15 percent bigger. We got 20 percent more free trial starts, something like that. Big, bold.
Carl: Have they actually moved through the sales funnel? Have they actually moved through your sales funnel? That's the, I suppose the question, because if they haven't, then it's not really a metric you should, yeah. Perfect. Thank you. This has been a very good conversation, Sahil. So thank you so much for being here.
Is there anything, anyone, where can people find you? So Twitter, Are you on Twitter? Are you on LinkedIn? Where do you like
Sahil: You can find me on LinkedIn. I post every Tuesday and Wednesday at 7. 30 a. m. Eastern.
Carl: Awesome. Remember that. Sorry. Spiralyze. com.
Sahil: Yeah, and the website. So you can find, on LinkedIn, I'm posting under my personal profile. So just look me up, Sahil Patel. And then If you want to, if by the way, those things, I'm not really posting about Spiralyze. I'm just more postment things I've learned. So if you found what we did today, interesting techniques, learnings, something surprising, then I think you'll like what I'm doing on LinkedIn.
And I'm lucky because the CRO world is very visual. Yeah. You'll it's not just a bunch of text and it's not just me talking. It's it's very visual. And then if you're in that place in the in your journey where maybe talking to spiral eyes make sense. Yeah. Check out spiral eyes dot com.
Carl: Thanks again for coming and it's been great chatting. See everyone else next week.