Big Swing or Incremental, Which Ecommerce CRO Testing Framework Wins?

What happens if we remove the Gordon Ramsay approved section in the TV commercial

“Like what happens if we remove the the Gordon Ramsay approved section in the TV commercial? Like what what kind of lift does that does that bring to the page or drop to the page?”

““Then we did this big facelift that we also split tested, that included a lot of the things we we learned in our iterative testing. And now we're on and that like kind of brought us up here, and now we're going back into the iterative testing to see like which of these sections are more important. Like so, now we're kind of going back into the All right, we we have a new experience that's converting better, and we feel like it's just an overall more aesthetic on-brand experience.””

Oh, if this like Gordon Ramsay social proof section when we removed it, that drove the biggest drop

““So you can see like which which removal, which variant, which section removed drives the biggest drop in ARPU and that tells you which section is is driving the most performance on the page and you can almost like stack rank all the different sections based on like which everyone has the biggest drop in ARPU has the most importance in driving performance on that page and that can kind of like steer you a little bit in terms of where you go and optimize next. Like, oh, if if this like Gordon Ramsay social proof section when we we removed it, that drove the biggest drop, okay, well maybe we go and like focus there because assuming that the inverse is also true, if we go and improve that section, it'll also drive the biggest lift versus going and improving like a different section on that page.””

Where UGC in the US was in like 2019

“Ja- uh UGC in Japan is like where UGC in the US was in like 2019. Like you could launch a new UGC ad and it would have like a 6% outbound click-through rate and CPA will be cut in half.”

““But if you go look at what we were doing, it was just like offer ads offer after ads offer after offer ads. So the big thing this year was launching more evergreen creative and just like training the algorithm to go and reach more new people, which we've successfully done. So that was like phase one here this year is like we need more more upper funnel, more creator led ads. And it became clear in the first quarter that like the egg pan was our big unlock over there.””

We need more upper funnel, more creator led ads

“So that was like phase one here this year is like we need more more upper funnel, more creator led ads.”

““So the big thing this year was launching more evergreen creative and just like training the algorithm to go and reach more new people, which we've successfully done. Ja- uh UGC in Japan is like where UGC in the US was in like 2019. Like you could launch a new UGC ad and it would have like a 6% outbound click-through rate and CPA will be cut in half. And it became clear in the first quarter that like the egg pan was our big unlock over there. Like we launched a bunch of UGC and the egg pan UGC had like a third the CPA of any other product.””

We launched a bunch of UGC and the egg pan UGC had like a third the CPA of any other product

“Like we launched a bunch of UGC and the egg pan UGC had like a third the CPA of any other product. So, like great. We've just unlocked our hero product. Fantastic. That's a huge learning.”

““Ja- uh UGC in Japan is like where UGC in the US was in like 2019. Like you could launch a new UGC ad and it would have like a 6% outbound click-through rate and CPA will be cut in half. So that was like phase one here this year is like we need more more upper funnel, more creator led ads. And it became clear in the first quarter that like the egg pan was our big unlock over there. Struggle with some inventory. We stocked out a couple times. Now we're in a better place.””

The big win we've had lately is not necessarily peppering in another 15 egg pan UGC creators

“So, the big win we've had lately is not necessarily peppering in another you know, 15 egg pan UGC creators. It's been how do we drive up how do we drive up AOV on the egg pan and and also simultaneously get more sales on these other products.”

““So, I was like, all right, this doesn't That's fine, but we were over indexed on the egg pan. Um and AOV did come down. So, basically the the existing funnel was, you know, egg pan ad mainly UGC going to this egg pan landing page, going to the egg pan product page and that was that was a winning funnel. So, the new test was the hypothesis was, well, if we instead of directing people from the lander right to the product page, take them to a collection page and we create like, call it three new bundles that are just combining the egg pan with our other top selling SKUs, which is like a fry pan, a high-sided pan and the rice pot and then maybe one SKU that's like all four of them together and then we can like position a discount.””

The existing funnel was, egg pan ad mainly UGC going to this egg pan landing page

“So, basically the the existing funnel was, you know, egg pan ad mainly UGC going to this egg pan landing page, going to the egg pan product page and that was that was a winning funnel.”

““It's been how do we drive up how do we drive up AOV on the egg pan and and also simultaneously get more sales on these other products. So, the new test was the hypothesis was, well, if we instead of directing people from the lander right to the product page, take them to a collection page and we create like, call it three new bundles that are just combining the egg pan with our other top selling SKUs, which is like a fry pan, a high-sided pan and the rice pot and then maybe one SKU that's like all four of them together and then we can like position a discount.””

I just think it's a great example where people get

“So I just think it's a great example where people get so hyper-fixated on like more ads, more ads, more ads, and often it's like once you find an unlock with your ads, you can often like look down funnel and and continue to optimize that way.”

““So we're by just like we're we didn't have to introduce a bunch of new ads around these products, but we're less over-invested on the egg pan now, which is great for inventory levels and and a number of other reasons. And now Shopify's up like again, this is coming from like very small numbers, but it's up like 160%. And we've been pretty jazzed on this. And now we have like a very optimized destination, we can kind of go back to the ads and start developing more winning ads and like keep stacking these wins.””

Fixing the destination before buying more traffic is also where a Shopify development and CRO agency like ECORN earns its keep. Collection pages, bundles and product page layouts are exactly the kind of changes brands can test once their ads are already working.

We have like a hero card as the first SKU in the collection

“We have like a hero card as the first SKU in the collection, which also has a like a hype reel overlaid on it. So, it very much was not a single variant test.”

““We also like this collection page, it's kind of going back to what you talked about earlier. We actually added additional components other than just a pure collection page with additional SKUs. Like we have a hero section with a nice image that kind of explains the product. It was like, "How do we just make a much, much, much better experience, which includes multiple variables?" And then of course, now we can go and we're kind of back into some of these more iterative optimizations, which is It's interesting.””

I feel like I was at like Homestead in 2020 buying like launch a new UGC ads in an account that's never launched UGC

““Right like how we're setting up the test, like we've run similar AOV boosting tests in the US just around different products. Um and again, it's like it feels like buying media. I feel like I was at like Homestead in 2020 buying like launch a new UGC ads in an account that's never launched UGC and you're like like literally almost overnight the CPA on those new ads is half of what it has been for the last 30 days and you're like it's just like so obvious, you know, whereas at like our brand scale now, these these wins aren't as like big, right? You know, you you rarely launch a new ad that just like cuts the ad account CPA in half because you've been launching literally tens of thousands of ads for the last four years.””

Source: Marketing Operators, watch the interview