Mandy:
Welcome to another episode of the Duoplane Podcast, Navigating Ecommerce Skies, where we take a deep dive into the ever-evolving world of ecommerce. Today on our show, we have Nils Koppelmann. Nils is a founder of 3Tech, a company that helps ecommerce, SaaS, and marketplaces grow through experimentation and conversion optimization. He has a background in software engineering and design and studied computer science in Berlin. He focuses on helping mission-driven companies grow online and enables them to make better decisions. Hi Nils, thanks for being here today.

Nils:
Hey, it’s nice being here.

Mandy:
How are you?

Nils:
I’m good, it’s not so sunny day in Berlin, but I’m doing well, what about you?

Mandy:
Good, same here. I’m in Vancouver, Canada and it’s been a very, very hot summer, but the last few days it’s been quite cloudy and I’m not mad about it.

Nils:
Yeah, it’s fairly similar here in Berlin actually.

Mandy:
Awesome. So today we’re really going to focus on A-B testing. So I kind of just want to jump into that. For those that don’t know, can you give a little explanation on what that is?

Nils:
So, I’ll take just one step back and go more into the why before we go into any kind of technicalities. So, whenever you want to make any kind of decision on your site, say you’re an e-commerce shop, you have your Shopify, and you’re like, I kinda wanna get this little extra out of my conversion rate, whatever metric it is you’re looking to improve, and you then have an idea. And you can, of course, go ahead and implement that idea right away. And but you’ll never know really if that really was like the change that you’ll see afterwards, if that really had anything to do with the idea, with the change you made, or if it was just random because maybe you changed something in your ads as well. So that’s where A-B testing really comes in. A-B testing allows you to test this change across your audience or across your visitors on your site, but basically you’ll split the group of people you’ll expose this change to into two groups. One, you will just see the version that it was before, and one group, and it’s usually 50-50 split, that then will see the change. This allows you then to basically, statistically see… if the change actually had any impact or if not. And if not, then you can go iterate and yeah, keep continually optimizing the site.

Mandy:
Awesome, so I do know that you help companies, like e-commerce companies, SaaS, marketplaces, that sort of thing. Can you go a little bit deeper on an example, maybe, of one of your clients? What industry are they in? What type of clients do you help on a regular basis?

Nils:
So I’ll go in a more broad sense, but there is a couple of examples you could see on my LinkedIn, for example, or my LinkedIn specifically. But generally speaking, the kind of clients we work with is an e-commerce company that is not just starting out, because for A-B testing, you need a certain kind of traffic in order to actually measure a difference and see a statistically relevant. change. But the kind of clients we work with usually have, I mean, there is also some clients that are different, but usually they have a very strong brand, a good following. Oftentimes, they evolve around one specific product, at least for the direct to consumer e-com brands or one product category of products. Then there is also marketplaces we work with that, of course, have like a huge… range of products usually. And what’s interesting to see is for these kinds of clients, the way we approach CRO and A-B testing is very different. Not in the sense that the frameworks we use are different, but the way we approach testing is different. For example, with the D2C econ brand, the test ideas often revolve around how can we better present, for example, like USPs of the product. How can we better go around the story of the line of products they have? On a big marketplace, that’s not as easy. Think about Amazon, right? Amazon is on our client, they do that all internally, but like just to give context, on a marketplace, you don’t have as much control over… every single product and tests, if you were to test like something content specific, could get very costly, very quickly if you want to change like 100,000 products, right? So in that case, for those who would usually and typically test more things on how to make the experience smoother, right? How to make it easier for people to add more products to the cart, stuff like that. But Generally, that’s the kind of clients we’d work with and strategies can differ, but yeah.

Mandy:
So you mentioned how you generally don’t work with new brands. So it’s more like established brands, established companies. Do you think that newer e-commerce companies, for example, shouldn’t be A-B testing? Or is it something that they should still do? They’re just not typically your client.

Nils:
Yeah, so I would rephrase that a little bit. The point I was trying to make is that typically, new brands don’t have the traffic to support A-B testing. That said, there is brands, for example, if they venture back to that, can already buy traffic from day one and hence support A-B testing. Typically that’s not really the case, but there is like fairly new brands that scale up so quickly that after a couple of months, they already have traffic volume required for A-B testing. That said… A.B. testing is not limited to on-site, which is what we focus on. But you can very well A.B. test on your ads, right? That’s what a lot of companies do in their performance marketing. They test one kind of ad against the other. And even if you don’t do that yet, and if you don’t have the traffic to support changes on your site, you can still test various angles, for example, like message angles in your ads. very valid tool for companies with lower amounts of traffic. And that said also, there is a difference between A-B testing and CRO. Oftentimes they’re like put in the same bucket, but A-B testing is just one tool. A-B testing is one tool to help companies make better decisions. And, but CRO is more the bigger picture, right? And CRO can be so many things. So… Smaller companies can do zero just as big companies can do.

Mandy:
Can you explain what CRO is to listeners?

Nils:
So, sure. So CRO stands for conversion rate optimization. And a common misconception is that it’s always about improving conversion rates, right? So, let’s go ahead and start with the conversion rate. It’s not. It’s a bit misleading, but still the fact remains that you want to improve conversions on your site, but not necessarily just the conversion rates. So, and the idea behind it is that you want to best understand customer and consumer behavior to then change your site or optimize your site to best meet those needs. And that’s basically CRO. The methods that then follow are depend on what you really want to achieve, but usually it’s a mix of using qualitative and quantitative insights and methods to get those insights and then create a plan on implementing changes that solve real user problems.

Mandy:
And can you give an example of A-B testing? Maybe this could be a client that you’ve seen a lot of success through some sort of A-B test, or however you want to explain that. But I think an example would be awesome for listeners

Nils:
Sure,

Mandy:
here.

Nils:
sure. I’ll make a very vivid example, even though we have video and audio, but I’ll try to explain the idea behind it. So we have this, let me think of a good one. All right, so we have this client that we were doing A-B testing, it’s an e-comm store, and we were trying, so they have a lot of products, now more than when we started with them initially. But they have, basically they sell nutritional goods. And I think one initially was the chocolate bars. But they would sell them in packs of, I think, I think 12 and larger. And they would have on their site, they would have a variety of options. So you could buy one, two, three or whatever packs, but the price would change. And we of course wanted people to buy the higher ticket item, which would be then the option to go for three packs or I forget how many they were. So we were thinking, okay, how can we actually solve this gap between what people thought? a chocolate bar would cost, because if you go to the supermarket you see, okay, chocolate bar is like 152, three bucks, I don’t know how much it is in Canada, but here in Germany you typically pay around 150 to two euros for it, but online it’s a bit different, especially if you more or less buy bulk. So we just thought about, hey, why don’t we… Why don’t we just show the indication of price per item? And there is an effect behind that, or a very similar one, I think that was coined initially by the New York Times or something, the pennies per day effect, which is very similar to what we used here, where we would just show the price per item. And we saw, I don’t know the exact numbers anymore, but we saw about a six or five, six, seven percent increase in conversion rates and a similar one also in average revenue per users once we implemented this feature, which basically just displayed, hey, this is how much it costs per item. That’s a very simple A-B test, but yeah. Took a bit long to explain, but yeah.

Mandy:
So just to confirm, so you mean that instead of showing $10 for a bunch of chocolate bars, you would show the cost per item would be like

Nils:
Yeah,

Mandy:
$0.99 or something like that.

Nils:
yeah, so with this one specifically, we showed, we still had to, for legal reasons, we still had to show how much you’d pay in total, but we’d focus then on the price per bar, because that’s really what people cared about, and that they were not overpaying or something on that.

Mandy:
Right.

Nils:
And of course,

Mandy:
Thanks.

Nils:
the bar is very delicious, but yeah.

Mandy:
Awesome. And so what are some of the key steps involved in the process?

Nils:
Yeah, so, like, it’s very easy to have like 10, 20 ideas. So whenever I go on a site, I was like, oh, I have an idea here how to improve that and this and that, but. we try to follow a more concrete process there where we, of course, the expert review and a deconstruction is usually part of what we do to gather insights, but there is so much more than that. So for example, we look at session recordings. Sometimes we even do user testing where we try to understand what’s the real, like, where do people really struggle using the site or… following through with certain actions like putting something into the cart or finding a certain product. So there is this discovery research phase that we always do. And then we usually have like a list of 50 or something insights and then we condense them down into a couple of ideas, say 15 or 20 ideas. But now you have a backlog of 15, 20 ideas, and you’re like, shit, now I need to do something with that. And that’s where prioritization comes in. So we have a framework that we use where we basically give a score to each of these ideas based on a certain number of factors. And then we can decide, okay, so we have now three ideas that are top ranking for the homepage, three ideas for the product detail page, three for the cart, and… so forth. Thanks for listening. And then we can create what’s called a roadmap. So, and a roadmap is basically just for all the pages you wanna test on, putting a date, start and end date usually for every one of those ideas. And that’s when the ideas become experiments. And then we create a design. The most important step actually I forgot to mention, but we create a hypothesis. So, hypothesis for every one of these ideas, change, why we think that works, in the best way, in the best case, we back it up with a psychological pattern or something, or some research that we did, and then also, yeah, the kind of metric we want to improve with that. at least that we want to change. And yeah, and then we take all that, create an experiment from that. There’s also a good deal of coding involved usually. And yeah, we just press start, at least for the first experiment in the roadmap. And then we wait. And at the end, we look at which one of the variants won.

Mandy:
And are there like common mistakes that you know typically you want to avoid or that people come across when they are conducting AB tests?

Nils:
Yeah, there’s a good number of mistakes. I’ll give one that’s like very easy to make and it’s especially if you’re starting out with A-B testing, it’s what’s called peaking. So, thank you. We’ve just gone through all of that, a lot like this load of work and you’re like finally my A-B test is live and you feel like I kind of wanna look if it’s, I mean, I know my variation is better, right? So you go into the tool you use to evaluate your results and you like your look. And that’s a mistake, especially if you look with the intent to stop the test. So depending on the kind of statistical model you use to evaluate the success of your test, there is different criteria to stop a test. But there is also some, I call them hygiene criteria that are involved in making the decision to stop a test. And usually that is, I mean there is exceptions to that, but usually that’s like let it run for two weeks so you have two business cycles. Some people argue that’s not the way to approach it, but that’s how we do it for a for most clients. But, and then, for example, if you use Bayesian statistics, you’d want to wait until you have a certain level of confidence, is the wrong term in this case, but a certain level of confidence that, or probability that your test is leaning towards one variant. But… So there’s, yeah, this peaking, again, depending on what statistical model you use, the stopping criteria are a bit different, but yeah, peaking is one of the mistakes. Another one is… So, I mean, we are in agency context. We are in the luxury position that we don’t necessarily have to use the same person for evaluating the test and the same person for, like, creating the test initially because that helps, like… take bias away, the person who analyzes the test results doesn’t necessarily care cares about which variant won because they didn’t spend as much time creating it. In smaller teams, you usually don’t have that luxury. But that’s why we try to also like when we work with clients, just share results openly. But yeah, sometimes that could lead to confirmation bias in a way. You want your work to yield positive results. Yeah, it’s stuff like that. Then there’s a huge amount of statistical mistakes you can make, of course, in all of that, in analyzing, but maybe that gets too technical.

Mandy:
The confirmation bias actually makes a lot of sense to me. I never really thought of it that way, but I think you’re completely right. It’s almost like you get this attachment to it, right?

Nils:
Yeah, I mean, I see that in myself too. Like sometimes, actually there’s another one that’s very similar to that one is when you see a result of a test that feels too good to be true. it’s most likely not true. And it’s called the Twyman’s Law. Probably I’m mispronouncing it, Twyman’s Law, I think. And there is a book by Ronny Kohabi and a couple others that also covers that. And yeah, probably check your numbers because there is probably some kind of error along the road.

Mandy:
Interesting. So just going back to companies that do have limited website traffic, are there any strategies that they can do to employ to effectively carry out A-B tests?

Nils:
Yeah, so I mean, the one I already mentioned is like doing it on an ad level. That is if you’re running ads. Otherwise, so there is a statistical reason why that is. So you usually need a certain number of… people involved in the experiment, and if you just don’t have that kind of throughput of traffic, then you won’t be able to make a real decision. Then you could just as well just flip a coin and hope for it to be numbers up. But what you can do is still use qualitative data. That won’t really be A-B testing, but it will give you… to a certain level of certainty, insights into what your users really want. And that’s probably what I would definitely lean towards. And one oftentimes underutilized tool is what’s called review mining. And if you just think about how many reviews you’re getting on your typical store on a product, or if you’re also on Amazon, or even if your competitor has product on Amazon, you can leverage that because real people write something about your product or a very similar product about their concerns, even if they don’t like something about the competitor’s product. Wow, that’s amazing. Now you just learn something you can put maybe on your site and show how you’re different. And I think that’s a very interesting thing. Same thing with analyzing chat results, if you have a live chat or something on your site, or other customer support channels. Stuff like that is very, very helpful. And also looking at and interpreting, and that’s where the art is, but interpreting session recordings that you can use tools like Microsoft Clarity, Hotjar, or Content Square, probably if you’re a bit bigger, but especially like, probably won’t have that budget if you’re low traffic, but yeah.

Mandy:
So if the experiences that don’t have the knowledge to do A-B testing themselves, what are some of the key factors that they would want to consider when they’re selecting a company to help them do that?

Nils:
Yeah, so there is a couple options. I mean, of course, I would vouch for an agency because that’s what I’m running, but there’s also the option to work with independent freelancers, which sometimes can be a good choice, right? But especially if you’re looking to build up a testing program, I think… Couple of the criteria to evaluate an agency or a company you can help you on that is like. like check if they actually can do what they say. So like check their testimonials or their past work, if that’s not online. Like actually, I’m a very big fan of like discovery calls. Just hop on a call and see also if you’re getting a good feeling about it. I mean, that won’t be reason enough, or shouldn’t be reason enough to go with one company, but I feel like A-B testing or like in collaboration with our clients, it’s a lot of the time but also a very intimate relationship in that you spend a lot of time together, not necessarily all the time, but over, like it just accumulates. So you wanna make sure it’s also, you feel comfortable working with these people. Yeah, so for me, that’s a very important thing. Also from a vendor perspective, right? I wanna work with people that are fun to work with and are also open to new ideas. So yeah, that would probably be things I would focus on. And then also keep in mind that budget is important and be honest to yourself what you can really afford. And for example, the reason I’m saying it is if you really need to have these zero efforts to save your company, probably going with an agency is not the right step right now. Because what… especially as if you focus on experimentation and A-B testing and you want that company to help you with that, if you can’t live through even a streak of losing tests, this is gonna break the relationship because you’re coming with the wrong assumptions, with the wrong expectations, and you won’t have the mindset to support what’s really needed for A-B testing. And yeah, probably then I would just do different things because then you’d have a different problem. But if you’re really open to experimentation, if you’re open to learning, if you understand that at the end of the day, the learnings and even the tests that lose are so valuable, then I think it’s a good place to start from.

Mandy:
I think you made a really good point there where there’s companies that might think of A-B testing and just optimization in general as kind of like a way to save their company, but really it should be done when they just want to grow or push their company forward.

Nils:
Yeah, so definitely. And I was giving a talk in Amsterdam earlier this year about how I think experimentation is especially useful in these times of economy being a bit difficult. And that’s true, but as you said, it’s not the saving mechanism, right? But it’s a way to make sure, or at least to a certain level of… of certainty that you’re making the right decisions and that the decisions are supported by facts and not just pure opinion. So

Mandy:
Right.

Nils:
yeah.

Mandy:
Yeah, I think that’s a great point. And so, you know, if there’s a company that is at the point where they’re like, okay, A-B testing, it’s the right time, how should they go about starting an A-B test? Are there any best practices to follow?

Nils:
So, yeah, sure. I wouldn’t say necessarily that there is a best practice as to what kind of AP tests to run or what change to run, but there’s certainly best practices as to how to run an AP test or how to run a program for that matter. And one thing I already mentioned, like never skip the hypothesis stage. in if you can do as much research as you as you have as you can or at least sufficient research not as much as you can that might go overboard but yeah stuff like that I think I would I would suggest as yeah and also like invest not only in tools but in people. I feel in the A-B testing world there’s oftentimes a lot of tools pitching you that they will solve your problem when really it’s not the tool that will solve your problem. It’s just merely there to… I mean there is a lot of great tools. We work with a lot of amazing vendors but… at the end of the day, the ideas won’t change. So you still need people executing it. You need people coming up with these ideas, doing the research, telling the right stories to management is also like, depending on again, how big your company is, but telling the right stories of what these experiment results mean, this is super important and a skill at the end of the day. So yeah, process over tools and definitely, and people as well.

Mandy:
Definitely. And I know we talked about this a little earlier. You gave that example of the chocolate company, but are there any other successful case studies or examples of businesses that you can think of that have benefited from A-B testing?

Nils:
Sure, I mean, probably for the audience, a lot of the cases that I would talk about don’t mean much because they’re usually German, so, or at least the majority of the ones that I can talk about. But we’ve worked, for example, with a car platform here in Germany, or. probably the DACH region, where we’ve helped them basically sell a lot more cars by using A-B testing and sending people down the financing route, or the way there, actually. That would be an example. We’ve helped like Jim, Jim chain in the past. Not the typical e-comm case, but still very relevant. Yeah. And even for companies, like I want to mention that because you said it earlier, it just comes to mind. It’s okay if A-B testing doesn’t work for you right now. Don’t push it. If you don’t have the traffic to A-B test, rather don’t A-B test and come to the wrong conclusions that you can’t statistically prove. Sometimes doing, and this is what we offer also to smaller companies, sometimes doing a conversion audit or like a conversion deep dive basically. is very helpful already. It’s a good start. And that’s also what we, again, do for a lot of smaller companies, just to help them kickstart and get them into the right mindset. And oftentimes, the feedback for stuff like that is like, hey, whoa, I’ve never thought about that. And yeah.

Mandy:
In what ways can A-B testing contribute to the growth and success of an e-commerce company?

Nils:
I would say very vastly. If you, in my opinion, if you have enough traffic and if you don’t A-B test, you’re being very arrogant in the way that you assume that you have the best version of your website or of your shop already out there. And that tells me, I mean, this might be a hard statement, but that tells me that you don’t want to understand the users any better. And so in a way, A-B testing, and CRO in general, but especially A-B testing, is a tool that if applied correctly, is like really good customer service, because it helps you improve the experience where not only your like, like apparent conversion rate will benefit, because… That’s just an indicator of how happy our customers to shop with you. This is also part of the misconception about CRO. You will never optimize the conversion rate, but you will optimize the behavior of your customers or of your prospects to the degree that later the conversion rate will reflect that. All these metrics are just reflections of how good your shop is, of how good… or how easy your shop is making it for consumers to shop with you or do other certain connections. So yeah, it’s like in a way, it’s like as valuable as customer service or other functions. So, and this is generally something that I think about a lot is experimentation is not just a method. It’s in a way a business function. And if you have the means to do it and not do it, it’s being in a way cruel to your customers and to yourself.

Mandy:
Yeah, definitely. And I have more of like a personal question to end this

Nils:
Sure.

Mandy:
off here, but you’re obviously very passionate about A-B testing. It’s quite niche when I think of marketing agencies who generally do a whole bunch of different types of things. This seems to be your focus. How did you get into this? What makes you so passionate about A-B testing?

Nils:
So. I like this question. It’s fun. And yeah, you’re right. I care about a B testing and experimentation in the bigger picture a lot. And I think this comes from initially, or this stems from me being very curious as a person. I was always the kind of guy I remember once sitting in the car with my dad and he was like, Oh, you asked so many questions. But yeah, that’s so important. Because curiosity is at the core of being an experimenter of wanting to improve something. And I always wanted to, and I was always building stuff or improving stuff. So, I mean, for the longest time that wasn’t in the… in A-B tests or in CRO. But initially I was helping, even as a teenager, companies with their websites. That later became a job. And at some point I realized, oh, there is, because… as an agency, initially we started out as a web development company, but at some point briefings had a goal in there and people wanted to improve their conversion rates. I was like, oh, what’s that? How does that make sense? And I realized, hey, people don’t just want shiny websites, they actually want to fulfill a business goal or reach a business goal. And that’s where I was like, finally, I can apply this curiosity and this drive to optimize And yeah, that’s how this slowly evolved. And I discovered A-B testing, found a wonderful community of people who experiment and realized that there is a huge potential, not only just to do this as a service, but huge potential for companies to utilize experimentation for their business functions.

Mandy:
Very cool. I love hearing the reason why people do what they do. Thank you so much for all your insight and wisdom today. I think listeners

Nils:
Sure.

Mandy:
are gonna gain a lot, whether they’re brand new to A-B testing or they’re already in there a little bit, I think that they’re gonna gain a lot from this conversation. So thank you so much for sharing.

Nils:
Thanks for having me on and it was a pleasure talking with you.

Mandy:
For those listening, you can check out 3Tech at 3tech.de and Duoplane at duoplane.com. You can find and subscribe to our podcast wherever you listen to your podcasts and we will catch you next time on the Duoplane podcast.