Walmart Runs A Potentially Confounded Button Color A/B Test

Leak #108 from   |   Jun 26, 2024 Global

Walmart just ran another iteration of a button color a/b test as part of their navigation tests (seen here and here). This time, it looks their team attempted to isolate the button color change (a yellow vs blue background on the search button). However, upon looking at this more closely we can see that there is a possible confounding situation. The contrast of the icon in variation's hover state has has dropped heavily (barely legible).

For all we know, such changes may already have tiny effects that are barely detectable (and therefore make no relevant difference at all). Nevertheless it might be good to check for such confouders. 

A - Jun 3, 2024 Screenshot
REJECTED Confirmed Jun 26, 2024
B - Jun 6, 2024 Screenshot

Highlighted UI Changes From This Leak

  1. Changed Button Color

    Instead of a dark blue background, the variation used a yellow one (possibly aiming for contrast from a somewhat complementary color choice).

  2. Decreased Icon Contrast

    The hover state however, has a strong decrease in contrast on the icon (dark on dark situation).