Yammer Tutorial: Registration Conversion
When we interviewed candidates for product managers, we asked how they would change our registration process and why. The most common response was to remove steps that looked unnecessary. That sounds good, let's try.
Want to test your product manager skills? Take the test "Reduce the registration process" (about 30 minutes) before reading the post.
Christina Lucy proposed to remove two steps from the registration process and replace them with the addition of information in the product itself. She suggested that this move would simplify the task for new users and increase their flow.
')
The idea was pretty simple. When a person goes through the registration process, he has not really seen the product itself. You can imagine what the user might think if you ask to add your own photo or join some groups. If you ask to do this after the user has become acquainted with your product, he will most likely do it. In addition, we have slightly simplified the registration process, so that more users will pass it without interrupting.
With these thoughts, Christina structured her test:
Remove the step "Add your photo" from the registration process and add a banner to the main page that asks the user to add a photo.
Remove the "Join Groups" step from the registration process and add the "Suggested Groups" section to the navigation links on the left.
Move both of these steps out of the registration process.
The original registration process is as follows: (1) the user fills in the information in the profile, (2) the user subscribes or invites colleagues, (3) the user joins existing groups, (4) the user uploads a photo.
Everything went wrong, as intended ...
For new users in test groups, the frequency of posts has decreased and fewer users were attracted every day.
Based on the test results, Christine added a photo upload banner to the product. She did not add the "Suggested Groups" module and did not cut a single step out of the registration process.
The reason for removing steps from the registration process was to attract more users to the product. Despite the fact that the two options achieved this (deleting joining groups and removing both steps), Christine noted that it was much harder to keep new users. The number of attracted users per week improved slightly, which, of course, is very important, but it turned out badly for user activity - the frequency of posts decreased by 4%, and the photo load by 34%.
Explanation: After users completed the simplified registration process, testers began familiarizing themselves with the product without photos and groups. They were less involved in Yammer than the group that went through the original registration process, understood less if Yammer was valuable for them, and wanted to post something and use the product less
Good question. Christina and the analyst analyzed the data to see if invitations to new users compensate for the fall in retention. They found that this was not the case, and decided not to shorten the registration process.
The number of existing users with photos uploaded was about 9%, which was understandable. Since they had already completed the registration process, we didn’t have to worry about their retention, which was complicated by the cuts. As a small bonus, we noticed an increase in the number of people invited by existing users.
True, there was a small problem - the number of groups created decreased by 6%. Christine accepted this because group creation was not so important for the Yammer network, and our intuition suggested that profile photos are important for retaining the user in the product.
Explanation: The banner for adding a photo is large and takes all the attention from the entire left column to itself, so users do not notice some menu items, such as creating a group.
We either did not consider them important, or their P-value was too large to be statistically significant. As I mentioned in the test, we usually ignore results with a P-value greater than 0.3, and we are skeptical about indicators whose P-value is greater than 0.2. Scientists usually ignore everything with a P-value greater than 0.05, so we are still quite humane.
This function performed the exact opposite task. Users began to join groups at 18.5% less. Despite this, a strange increase in the number of posts appeared, but it was not large enough to compensate for this.
How can removing steps from the registration process be a bad idea? From this experiment, we realized that the registration process is a great place to engage the user in the product, even if they haven’t really seen it yet. This is counterintuitive, but user behavior is often just that.
Also this post shows that A / B testing not only helps to choose from 41 shades of blue . A / B testing can help check for risky changes to a product.
If you liked this post, read the Drew Dillon post about why statistics are not the enemy , and why you shouldn't be afraid of them.
Take the test
Want to test your product manager skills? Take the test "Reduce the registration process" (about 30 minutes) before reading the post.
Christine's hypothesis
Christina Lucy proposed to remove two steps from the registration process and replace them with the addition of information in the product itself. She suggested that this move would simplify the task for new users and increase their flow.
')
Original registration process
The idea was pretty simple. When a person goes through the registration process, he has not really seen the product itself. You can imagine what the user might think if you ask to add your own photo or join some groups. If you ask to do this after the user has become acquainted with your product, he will most likely do it. In addition, we have slightly simplified the registration process, so that more users will pass it without interrupting.
With these thoughts, Christina structured her test:
Option 1
Remove the step "Add your photo" from the registration process and add a banner to the main page that asks the user to add a photo.
Request to add a photo on the product page
Option 2
Remove the "Join Groups" step from the registration process and add the "Suggested Groups" section to the navigation links on the left.
Suggested groups in the navigation links section
Option 3
Move both of these steps out of the registration process.
The original registration process is as follows: (1) the user fills in the information in the profile, (2) the user subscribes or invites colleagues, (3) the user joins existing groups, (4) the user uploads a photo.
Test results
See full results here.
Everything went wrong, as intended ...
For new users in test groups, the frequency of posts has decreased and fewer users were attracted every day.
What did Christina decide?
Based on the test results, Christine added a photo upload banner to the product. She did not add the "Suggested Groups" module and did not cut a single step out of the registration process.
Why did she do that?
The reason for removing steps from the registration process was to attract more users to the product. Despite the fact that the two options achieved this (deleting joining groups and removing both steps), Christine noted that it was much harder to keep new users. The number of attracted users per week improved slightly, which, of course, is very important, but it turned out badly for user activity - the frequency of posts decreased by 4%, and the photo load by 34%.
Explanation: After users completed the simplified registration process, testers began familiarizing themselves with the product without photos and groups. They were less involved in Yammer than the group that went through the original registration process, understood less if Yammer was valuable for them, and wanted to post something and use the product less
But what about the increase in sending invitations?
Good question. Christina and the analyst analyzed the data to see if invitations to new users compensate for the fall in retention. They found that this was not the case, and decided not to shorten the registration process.
Why did she add a banner for the photo?
The number of existing users with photos uploaded was about 9%, which was understandable. Since they had already completed the registration process, we didn’t have to worry about their retention, which was complicated by the cuts. As a small bonus, we noticed an increase in the number of people invited by existing users.
True, there was a small problem - the number of groups created decreased by 6%. Christine accepted this because group creation was not so important for the Yammer network, and our intuition suggested that profile photos are important for retaining the user in the product.
Explanation: The banner for adding a photo is large and takes all the attention from the entire left column to itself, so users do not notice some menu items, such as creating a group.
Questions:
What about metrics / results you didn't mention?
We either did not consider them important, or their P-value was too large to be statistically significant. As I mentioned in the test, we usually ignore results with a P-value greater than 0.3, and we are skeptical about indicators whose P-value is greater than 0.2. Scientists usually ignore everything with a P-value greater than 0.05, so we are still quite humane.
Why didn't Christina introduce the "Suggested Groups" module?
This function performed the exact opposite task. Users began to join groups at 18.5% less. Despite this, a strange increase in the number of posts appeared, but it was not large enough to compensate for this.
Results
How can removing steps from the registration process be a bad idea? From this experiment, we realized that the registration process is a great place to engage the user in the product, even if they haven’t really seen it yet. This is counterintuitive, but user behavior is often just that.
Also this post shows that A / B testing not only helps to choose from 41 shades of blue . A / B testing can help check for risky changes to a product.
If you liked this post, read the Drew Dillon post about why statistics are not the enemy , and why you shouldn't be afraid of them.
Source: https://habr.com/ru/post/192008/
All Articles