|
KENT: So tell me a little about how this YouTube project came to be. I understand that a lot of it came together due to Curtis Lee.
MICHAEL: Curtis, we used to work with Curtis when he was at Google, he was one of the guys who worked with us on GWO [Google Website Optimizer, used for A/B and Multivartiate Conversion testing] things at Google. So when he moved over to YouTube and wanted GWO things done, he came to us.
NEAL: Yeah. We'd worked with him back when he was at Google and he had thought that we were pretty good. They really wanted to foster a testing culture in the organization, and hadn't run a large scale GWO test on YouTube before. Meanwhile, YouTube wanted to increase account signups, so it all came together.
KENT: Tell me a little about developing the test and what kind of things you were thinking about when developing this recipe?
MICHAEL: It started with a heuristic review, analyzing potential weak spots, stumbling spots, areas for improvement, and then developing hypotheses about what could make pages better. It was, in essence, a pretty simple experiment but what was interesting was that it had very high traffic, how many combinations there were, and how successful it was... really, there are so few websites you could do something like this on.
NEAL: I think Michael originally wanted a test with like 6000 combinations or something insane. It was ridiculous. Wanted to test absolutely everything...
MICHAEL: *laughs* There was, at one point, 1800 combinations, not 6000, but we ended up with just over 1000.
KENT: That's still a huge amount.
MICHAEL: Gigantic. One of the largest GWO experiments ever performed - with respect to traffic combinations tested, and probably the largest with respect to volume of traffic...
KENT: Does anything change when you're dealing with a lot of combinations?
MICHAEL: Well the main reason why you don't normally do that many combinations is that you're spreading out the data between so many different versions that its going to take forever to get results. It's only on a page like YouTube that you can do it. So that was what was fun about this: you have freedom to test as many combinations as you want (within reason) [GWO allows you to test up to 10,000 combinations]. Normally, you have to really scale it back and restrict the number of things you test, but YouTube's traffic gave us much more freedom than you usually get.
KENT: What ideas did you have that you decided not to go with?
NEAL: There was an animation. At one point we had an animated button that... it wasn't animated but it did kind of rotate. It was like an animated gif and the message might change or the image might change. We didn't go with that though.
KENT: Is there anything you can say about language use in the context of this kind of test?
MICHAEL: We thought that the original messaging didn't really stand out, it was too subtle, so people would miss the sign up link. I think the original version was "sign up for a youtube account" which didn't sound like fun - it wasn't a benefit or anything - so we thought we'd play with different wording, like join the community and upload videos, and all of that. Then we went to [Youtube] with them and they came back to us and gave us confirmations and rejected a few. Like we couldn't say the word free.
KENT: Really?
MICHAEL: Yeah, apparently Google doesn't like that.
But yeah, the big questions was what benefit was going to be most compelling. What do people perceive as the greatest benefit? Youtube had some specific ideas there and we had some ideas and we did some in person informal lab user testing as to why people would want to join. Was it to upload videos? To rate and comment on videos? Youtube thought that playlists and annotating videos would be important, but I would have thought that for newbies that wouldn't be that exciting.
The crux of it was that there was only room [on the page] for one benefit statement, so we wanted to develop a whole bunch and test each to see which one would turn out best. Then, we wanted to see which combination [of benefit statement and call to action] would work the best. Unfortunately, they decided to test each piece individually.
KENT: What determined your color choices and design, and can you address the potential for contradiction between design aesthetics and testing?
NEAL: Color we took largely from the design, we inferred a color palette based on what was there. As for testing, the idea of the test is to present different variations and combinations to users, and you express this through design...
As for contradictions, well... I guess (and this is very reductionist) the point of aesthetic design is to make things blend and be harmonious but if something's very harmonious, then its not going to stand out and grab attention. The boss used to say that he wanted to be able to see the call to action from six feet away, and in order to see a call to action from 6 feet away it has to be pretty big and pretty ugly.
MICHAEL: Yeah and as we thought, the ugliest one won.
Also, there was the box [around the benefits section] itself, and how the box was laid out. The original was very subtle, grey on grey, and the one that won (which was the one we thought would win) was white with a black box around it. It stood out more than the subtle grey one.
[ed note: observant readers will notice that YouTube has recently decided to adapt their current grey box, but add a slightly smaller black border]
KENT: What concerns did Google/YouTube have?
NEAL: They were concerned about GWO breaking YouTube, because whenever anyone hits the page some GWO control script runs, it serves the variations from Google's servers to the user (and for the sake of this I'm going to differentiate between Google and Youtube) so they get the JavaScript and CSS from YouTube's servers, and then content from Google's servers separately, which adds latency into the whole thing. So, if you have all that going on, it might impact the user experience and their servers.
Then, of course, Google was worried that 6 gazillion users might flood Google's GWO servers... And yeah, we probably did have too many combinations and they were concerned about that.
MICHAEL: ...the head technical guy at Google for GWO suggested we have fewer combinations, but we said "no, we're not going to restrict our test, fix your tools!" *laughs*
KENT: Are there any concerns you have post-test?
NEAL: Well, the results were good. We are very happy with them. We got a victory, which was good. The problem is... I wasn't too thrilled with the aesthetics of some of the variations, though they were very effective - they succeeded in their goal of increasing signups.
Ultimately, the most compelling aspect of the test recipe was the big ugly button we stuck in the masthead, and, quite frankly, I didn't like that one, but from the start we knew it was going to win. It was big and red and ugly, and I liked the white one which was nice and harmonious, but ultimately didn't stand out. Aesthetics versus business goals.
The thing is, YouTube's homepage is very noisy, there's like fifteen video previews that are rotating, then ads for chrome, then there's some ads for like Doritos or something, and so for something like a little link that says "sign up" to stand out we need to add a big ugly red button.
So when people load the page for the first time [the big red button] is going to jump out at them, then hopefully [they will see] the area on the right hand side where we had the messages and icons that spell out the benefits. The idea is motivation at the side, then a call to action at the top, and if they didn't see one they might see the other, or they may work in synergy.
KENT: And you credit that to the majority of the success?
NEAL: Well obviously it was a number of factors and distinguishing which out of the entire combination accounted for so much is all but impossible, but I'm all but sure that the red button played a major role in the eventual success of the test, yes.
KENT: And what were the results?
MICHAEL: 15.6% increase in conversions, which is... well... I can't say that, but let's say that with the number of visitors that Youtube gets, that number is huge.
| 
