The Client
YouTube is the global leader in online video sharing. Alexa.com ranks YouTube as the fourth most trafficked site in the world. It gets over a billion views a day.
A significant proportion of YouTube’s users interact with the site without ever signing up for an account. Unhappy with this, YouTube decided to perform multivariate testing on the website's homepage, aiming to increasing the rate of account sign ups.
To design and develop this test YouTube approached VKI Studios.
YouTube's traffic provided VKI Studio's with opportunity to devise a massive multivariate test, one which contained over 1024 variations. It was the largest GWO test to date. Amongst other things, this experiment involved testing the graphical treatment of the signup call-to-actions (CTAs), along with different benefits which accompanied said CTA's.
The test produced an outcome very quickly, and managed to increase YouTube's sign up rate by 15.7% - an amazing increase given the volume of traffic hitting the homepage. Moreover, the test conclusively demonstrated that GWO was robust enough to run a large multivariate test on an enterprise level site.
The Challenge
Signups are a valuable business objective for YouTube. There are several reasons for this:
- YouTube relies on fresh video content being constantly uploaded by users. An account is required to upload a video.
- Once a user signs up for an account, YouTube is able to provide that user with added site functionality, such as video playlists, favourites etc. This additional functionality helps to increase both user engagement and retention.
- Crucially, YouTube is able to gather information about account holders during the sign up process. In addition to this, as members interact with the site over time, their behavior and interests provide additional information. This insight into their members is critical in enabling YouTube to fully leverage and target their onsite advertising.
- Finally, YouTube intends to increase the amount of premium content that can be purchased on the site. User accounts are required for premium content.
The business goals identified above provided a compelling case for increasing account signups. To do this, YouTube needed a systematic, data driven methodology. Early on, they realized that split testing their home page could achieve this, but as they had no previous experience running a GWO test on the site, they required the expertise of an Authorized Consultant.
VKI Studios became Website Optimizer Authorized Consultants in 2008, and we have since established ourselves at the forefront of optimization consultants worldwide. Our exceptional track record in producing winning results for first time testers like YouTube made us a perfect fit for their needs.
The Solution
Our plan for increasing the rate of account signups contained the following steps:
- Research: Review client website, service offerings and users.
- Hypotheses Development: The ideas we want to test.
- Test Plan: The detailed description of what the test will look like.
- Creation: Test Assets: the creation of the individual test elements.
- Test Setup: The technical implementation of the test.
- Review of Results: The post test review of the data and takeaways.
Research
Our process started with the identification of a test page and success event. With this pre-identified, we reviewed the GA data, the homepage, and researching the types of people using the page and what they were looking for. By exploring the purpose of the webpage and the intent of the users interacting with it - and specifically, understanding the gap that exists between user intent and page purpose - we are able to pose questions about how best we can tune the page’s performance. For example, if a user lands on the homepage and is looking for a video, what benefits could we offer them to change course and sign up for an account instead? If they land on the homepage and want to create an account, how could we ensure they are able to easily achieve this goal?
Hypotheses Development
The research and ideation outlined above culminated in the creation of test hypotheses. These hypotheses form the foundation upon which the test is built and answer the questions uncovered during the research phase.
For the homepage, we had three main hypotheses:
- We proposed that we could persuade users who were not intent upon signing up when they arrived on the site to do so by clearly spelling out the relevant account signup benefits.
- We hypothesized that the most compelling sign up benefits for YouTube’s users were related to participation in community, creation of playlists, rating, and uploading videos.
- Finally, for those users that were interested in signing up, we hypothesized that by increasing the prominence of the signup CTA we would enable more people to find and use the signup feature.
Test Plan
The test plan is our formal test recipe document that outlines the test structure, test goal, test hypotheses, and the test elements that we will use to compete against the original page and validate our test hypotheses.
On the YouTube homepage, there were three elements related to account signups in the header and the right side column. These are highlighted below:
Sections 1- 3: (NB: Section 2 was a new element for the test so it appears blank in the original)
Figure 1 - The Test Sections 1 - 3
The three areas highlighted above served as our experimental control. These were the elements of the page that we would test our new ideas against in order prove or disprove our hypotheses.
For the banner, we created two sections to test our hypothesis that by increasing the prominence of the Signup link we could ease the path of those users already motivated to sign up. We had two different techniques for increasing prominence that we wanted to test, and this required the creation of two test sections.
Banner - Section 1
Section 1 involved the Sign Up link itself.
Figure 2 – Section 1 Text Element
Sometimes, the smallest of changes can have really big impacts, so for our first test variation we wanted to test something both subtle and straight forward. Our first variation tested the effect of changing the "sign up" text to all caps.
- “Sign Up” (The control)
- “SIGN UP”
Figure 3 – Section 1 Test Variation
This was the simplest of our sections. It contained two variations, the original (the lower case “sign up”), plus our new upper case variation.
Banner - Section 2
This section involved adding a second multivariate section into the banner related to the addition of a new element to attract the user’s eye.
Figure 4 – Section 2 Test Element
We wanted to test the impact of adding a speech bubble pointing to the “Sign Up” link. We had a couple of ideas for the presentation of the speech bubble and the text contained within it. Our copy points were:
- “Join YouTube Now!”
- “Join the YouTube Community”
- “Create your YouTube Account”
- “Sign up for a YouTube Account”
- “Join and Participate”
While the presentation of bubbles varied as follows:
Figure 5 – Sample of Section 2 Test Variations
In total, including the control, there were 16 variations for this section. (See the appendix for a full list of Section 2 test elements).
Here is a sample of what Section 2 looked like with a test variation inserted.
Figure 6 – Section 2 with Test Variation Inserted
Right Column - Section 3
We wanted to use the Section 3 right column element to test the hypothesis that, if we presented a sufficiently compelling sign up benefit, we could persuade those users who were not intent on signing up to act otherwise.
Figure 7 - Section 3 Test Element
To do this, we developed a number of copy points that we felt would motivate YouTube users to sign up. They went as follows:
- “Want to customize this homepage?” (The control)
- “Don’t just watch, participate”
- “Rate and comment on videos”
- “Create playlists”
- “Subscribe to your favorite uploaders”
- “Upload your videos”
- “Be seen around the world”
- Rotating benefits (This version that rotated through all of the above)
In addition to the different copy points, we also created a number of different presentation styles to aid the prominence of the copy points. For example:
Figure 8 – Section 2 with Test Variation Inserted
In total, including the control, there were 32 variations for this section. (See the appendix for a full list of Section 3 test elements).
Test Results
The launched with 1024 test combinations.
- Section 1: 2 variations
- Section 2: 16 variations
- Section 3: 32 variations
- In total: 2 x 16 x 32 = 1024 combinations
This was a very large test, and not something that we would typically recommend for most sites, but given the massive traffic levels YouTube generates, and thus the massive sample size we could achieve, we knew that we would attain statistically significant results relatively quickly. As things turned out, we were correct, this was the largest GWO test run to date (in terms of traffic volume), and statistical significance was attained with two days.
The test results went as follows:
Figure 9 – YouTube Homepage GWO Test Results
As indicated by the results above, Combination 28 came out on top with a 15.7% increase over the control.
The control:
Figure 10 – Original YouTube Homepage (The Control)
The winning combination:
Figure 11 – Combination 28, the Winning Combination
The Results
The top performing combination had a 15.7% lift in conversions, which meant a lot more YouTube sign ups. This was a very significant increase and YouTube was extremely pleased with the results.
Of the 1024 test combinations used, a dominant percentage of them outperformed the control. This strongly suggests that our initial test hypotheses were sound.
By increasing the prominence of the signup link (highlighting it with the red banner bubble) we produced a significant increase in signups. In addition, we also found that the effectiveness of Section 3 could be improved by increasing its prominence (different box style), along with providing a more compelling benefit.
Reviewing it section by section, we find that not all our test assumptions came out on top. For Section 1, the use of all caps text for “Sign Up”, proved less effective than the control. It finished a fraction of a percentage behind the original.
Section 2, proved much stronger. All of our test variations performed well. Interestingly, the design of the bubble seemed marginally more important than the copy points, with the red bubble variations outperforming all other styles. We deemed the “Sign up for a YouTube Account” copy as the strongest copy point, however, as it was performed well with each of the different bubble styles.
For Section 3, the pattern was less clear but the results indicated that the copy was more important than the presentation. It should be noted, however, that the top performing variation for Section 3, did not actually appear within the winning combination. This suggests that while combination 28 may not have had all of the strongest elements, there was a synergy between the elements it contained, that made the combination of elements stronger than the sum of their individual parts. This is the type of outcome that is hard to uncover outside of a multivariate experiment, and once again, proves the value of testing.
