Building a Data-Driven Experimentation Program
Identifying and testing high-impact product improvements before launch
Product Manager - Digital Experience
2025 - 2026
Up to +57.% conversion lift
mobile ecommerce
TL;DR
THE PROBLEM
Product improvements were often launched without validation, making it difficult to predict which changes would meaningfully impact conversion and revenue.
THE SOLUTION
Established a structured experimentation program that identified opportunities, tested hypotheses, and only launched validated solutions.
MY ROLE
Partnered with a growth agency to identify opportunities, form hypotheses, and design experiments while leading product alignment and final launch decisions.
IMPACT
Three experiments completed — all statistically significant winners — with results ranging from engagement improvements to a +57.8% conversion rate lift.
THE CONTEXT
The case for testing before launch
As Product Manager of Digital Experience, I partnered with our growth agency to establish a formal experimentation program.
The goal was to move product decisions from opinion-driven to data-driven.
The agency brought expertise in experimentation and owned the testing infrastructure and execution, while I represented the product team and ensured experiments aligned with broader business goals and user experience.
Opportunity identification and hypothesis development were collaborative efforts between the agency and internal teams, combining product insight, analytics, and growth expertise.
Together we followed a simple process:
Identify opportunity → Form hypothesis → Test solution → Launch only if validated
THE TESTS
A hypothesis, a variant, and a data-backed decisions
TEST 01
A/B Test
Improving product discovery in the mobile navigation
Hypothesis: Adding product highlights directly into the mobile navigation would give users a faster path to top products and increase downstream engagement.
Variant 1 - Category highlight
Category image highlights at top of the menu
Variant 2 - product callouts ✦
Product highlights at bottom of the menu
✦ Winner
Result: The test produced measurable improvements across the purchase funnel.
+1.50%
PDP Views
Goal Metric
+25.4%
Checkout Starts
Secondary
+10.2%
Add to Bag
Secondary
+14.6%
Conversion Rate
Secondary
What we learned: Small improvements at the top of the funnel can drive measurable impact all the way through to purchase.
TEST 02
A/B Test
Increasing product visibility on listing pages
Hypothesis: Displaying products in a two-column mobile layout would allow users to scan more products per scroll and increase engagement with product pages.
Baseline - single column
One product per row
variant - two column ✦
Two products per row
✦ Winner
Result: The two-column layout produced modest engagement lifts but a meaningful improvement in revenue per user.
+1.8%
PDP Views
Goal Metric
+1.1%
Add to Bag
Goal Metric
+13%
ARPU
Secondary
-0.7%
Checkout Starts
Secondary
What we learned: Increasing product surface area allowed users to evaluate more products quickly, ultimately improving purchase value.
TEST 03
A/B Test
Reducing early friction on the mobile homepage
Hypothesis: Introducing a category carousel at the top of the homepage would reduce early navigation friction and accelerate product discovery.
VARIANT A - Text Only ✦
Text only pills
✦ Winner
VARIANT B - Thumbnail image
Two products per row
Result: Both tested variants outperformed the control. The text-only carousel performed best across every metric.
+10.9%
PDP Views
Goal Metric
+28.1%
Add to Bag
Goal Metric
+57.8%
Conversion Rate
Secondary
+21.7%
ARPU
Secondary
+14.3%
Checkout Starts
Secondary
What we learned: Clear, simple navigation pathways can outperform visually complex solutions when users are trying to quickly reach products.
IMPACT
Small experiments, compounding product impact
Across the three experiments:
engagement increased across multiple funnel stages
add-to-bag and checkout start rates improved
the strongest experiment delivered a +57.8% lift in conversion rate
More importantly, this program introduced a repeatable experimentation framework, allowing product teams to make confident decisions backed by data rather than opinion.
REFLECTION
Building a culture of experimentation
The most valuable outcome of this initiative wasn’t any single experiment — it was establishing a process for making product decisions backed by data.
Instead of debating ideas, teams could now:
form a clear hypothesis
validate it through experimentation
launch only when results demonstrated measurable value
This shift moved product conversations from opinion to evidence, enabling more confident decisions about where to invest design and engineering effort.
Product Management
A/B Testing
Growth Experimentation
Conversion Optimization
Data-Driven Product Decisions
Agency Collboration
Mobile