Do not index
Do not index
User engagement is often taken to be the actionable link to revenue that teams should focus on. Create more engagement, create more revenue, in a nutshell. This is why growth teams design for multiple logins, Weekly Active Users, or Day Seven Retention. In this episode, your hosts discuss why the logic of “more engagement = more revenue” is flawed, and present four arguments for why you’re missing out on big opportunities by manufacturing engagement.
Sections
(03:23) Argument #1: The Lowest Common Denominator
Yohann: I had a swimming coach when I was a kid whose teaching technique was to just throw you into the pool. And I understand that everybody who ends up becoming a competent swimmer has been in the water, has jumped into the pool…but getting into the pool doesn't exactly mean that you know how to swim. Anyway, what ended up happening was he would throw me into the pool, I would flop around close to some place that I could hold onto like the railing. And the minute his attention was elsewhere, I would get out. And he would see me again, and he would throw me in again, it was just a never-ending loop of misery. And ultimately I did learn how to swim, but it was no thanks to him.
When you're designing specifically for engagement, you are catering to the lowest common denominator. You're saying, "what do all of the people who convert have in common? Engagement. Let's design for that." Yes, they do have that in common, but they also have a bunch of other things in common that might be more meaningful that you're not paying more attention to. It's literally the fact that all swimmers have to jump into the pool that you're designing for. You're designing for people getting into the pool, not teaching them how to swim.
(08:22) Argument #2: The Opportunity Cost
Yohann: You've only got a limited number of touchpoints that you can use to interact with the user. Most conversion flows we’ve observed are twenty or thirty touchpoints at maximum. Most onboarding email campaigns are seven to ten emails. So with the limited number of touchpoints that you're working with, focusing on engagement comes with an opportunity cost — you are missing out on the opportunity to interact with users with touchpoints that will meaningfully move them toward conversion.
Samuel: If you're trying to give a dog a pill, you can either hold its mouth open like a crocodile wrestler and try to throw it in the back of its mouth or you can wrap it in a meatball and have the dog eat it up. We recommend the latter when managing relationships with users at scale. Meet them where they are, help them get to where they want to be, and strategically pop up on their radar at times where it's relevant for them, rather than just saying “come engage with us.”
(14:52) Argument #3: Meaningful Engagement Patterns
Samuel: Trying to force more people into your product doesn't mean that those people are gonna convert into customers at nearly the same rate. So if you do have a robust pool of users who are free but are providing a more valuable experience and environment for the people who aren't free, that makes sense to me in terms of wanting to prioritize engagement. But if you are an invoicing app and you have a two-week free trial, there are a lot of levers that I would try to pull before I tried to just say, "Screw it, can we just get them to log in a bunch of times?"
19:43 Argument #4: Strong vs. Weak Correlation
I think we can make a distinction between weakly correlated and strongly correlated with revenue. Yes, engagement correlates with revenue, but I would put it more in the weekly correlated side of things because just because someone logs into the app seven times doesn't mean they then go on to engage meaningfully with the value that the product offers. Just because they log in doesn't mean they're making meaningful progress towards their outcome.
An outcome that both the user and the business care about is strongly correlated with revenue because it defines what’s bringing the the two parties together at all. No one uses software to use software.
Samuel: Designing for engagement doesn't seem inherently antagonistic to users but it’s definitely not mutualistic, because it’s not aligned with what the user is trying to do.Their goal is not to log in and spend a bunch of time in your app, but to go through your app to the outcome that they're pursuing in their life.
Transcript
Want to read along?
Toggle Transcript
Yohann: Hi, I'm Yohann.
Samuel: And I'm Samuel, and you are listening to the Self-Serve SaaS 'Cast, formerly known as the Value Paths Podcast. And today we have a spicy hot take episode for you, provocatively titled, "Is Engagement a Vanity Metric?"
Yohann, do you want to kick this one off?
Yohann: Yes, absolutely. I feel strongly about this one as a user who's been forced to engage with software, forced being the key word here, and as a person in software, because I feel like we can do better than just make people engage with our stuff. So I'm particularly excited for this episode and I've got a few arguments in the bag. But before we dive into those, let's lay out the Engagement Argument. Let's lay out, charitably, what the opposing side says.
Samuel: I'm not saying I'm on any side at the moment for the record.
Yohann: Okay. You can be the voice of reason here, and I'll be the crazy dude on the hill yelling at everybody in the town.
Samuel: All right. I would love to be playing that role for once, so that sounds great.
Why Companies Design for Engagement
Yohann: Okay, so the Engagement-based Argument is essentially that if you create engagement, it will result in good things down the line. Everyone who converts engages with the software and therefore, if you can create more engagement, you will ultimately be creating more conversions. And this is believed to the extent that most growth teams are given engagement mandates. For example, the goal that the growth team is working towards is to have people log into the app seven times. Or to make sure they come back for a second session or to make sure that they engage with the onboarding experience. These are common engagement metrics that growth teams typically work towards. And in this episode, we're excited to really put that belief under the microscope and see if it holds any water.
Samuel: Fair enough. And then on the other side of the equation, when we ask, "Is it a vanity metric?" What do we mean by that exactly? I guess that's a throwback term to the whole Lean Startup thing, shout out Eric Ries et al. But I guess my interpretation of a vanity metric is: a metric that you theoretically can engineer and optimize, but that doing so does not really strongly correlate with your business succeeding better.
Yohann: Absolutely. That's my interpretation as well. It makes you feel good, but it doesn't actually result in revenue.
Samuel: All right, so when we put the microscope lens on, we're saying, "under which circumstances does driving engagement, especially just engagement for engagement's sake, really produce positive outcomes for the business?" And if not, it's hard to conclude that it is not a vanity metric in some sense.
Argument Against Engagement #1: The Lowest Common Denominator
Yohann: Okay. So let's kick things off with my first argument, and I call this the argument from the Lowest Common Denominator. I had a swimming coach when I was a kid whose teaching technique was to just throw you into the pool, and I understand that everybody who ends up becoming a competent swimmer has been in the water, has jumped into the pool, but getting into the pool doesn't exactly mean that you know how to swim. And anyway, what ended up happening was he would throw me into the pool, I would flop around close to some place that I could hold onto like the railing of the pool. And the minute his attention was elsewhere, I would get out. And he would see me again, and he would throw me in again, it was just a never-ending loop of misery. And ultimately I did learn how to swim, but it was no thanks to him.
Samuel: Okay. Gotcha.
Yohann: And I'm using that as an analogy to say that I understand the logic of where the Engagement Argument. Is coming from in the sense that everyone who converts has engaged with the product, but not everyone who engages converts. Make sense so far?
Samuel: It does make sense to me. I mean, I guess we're saying correlation, causation here in some sense, but with the, uh, swimming teacher metaphor, I was also getting a kind of a read on, uh, the difference between logistical compliance versus purposeful action is that sort of in the neighborhood you were thinking as well?
Yohann: Yes, yes, absolutely. Correlation and causation aside, and meaningful engagement aside, what you're doing when you're designing specifically for engagement is that you're catering to the lowest common denominator. You're saying, "what do all of the people who convert have in common? Engagement. Let's design for that." Yes, they do have that in common, but they also have a bunch of other things in common that might be more meaningful that you're not paying more attention to. It's literally the fact that all swimmers have to jump into the pool that you're designing for, you're designing for people getting into the pool, not teaching them how to swim.
Samuel: Okay, in a software analogy that would be like focusing on activation at the expense of retention in some sense.
Yohann: I wouldn't even see it as an activation/retention distinction because you could create engagement that doesn't even lead to activation. Logging in seven times, for example. If that is your activation metric then, we went over this in the Aha moments episode, you're going to have a tough time because... just because people have logged in seven times doesn't mean you've made it any easier for them to go through the process of getting to conversion. Activation is so far removed from conversion that you're not paying attention to a huge part of the process i.e., everything that comes after logging in seven times.
Samuel: Okay. There's part of it to me where, of course your best customers are going to be demonstrating above average engagement, but if you try to just engineer people to engage more, it doesn't necessarily mean that they're going to convert into customers at the same rate or at the same level of quality.
Yohann: Right. This is another thing we went over in the Aha Moments episode, which is not all your users are pursuing the same outcome, so manufacturing engagement without paying attention to what kinds of outcomes people are pursuing means that you're not really supporting anybody.
Samuel: That speaks to me for sure in the sense that, um, if when you look at trying to engineer engagement, if you're looking at Day Seven Retention, Monthly Active Users, whatever your metric is, and you wanna have more people logging in, basically, it doesn't seem like that's an inherently antagonistic perspective to take, but it's definitely not aligned with what the user is trying to do.
Their goal is not to log in and spend a bunch of time in your app. Their goal is to go through your app to the outcome that they're pursuing in their life. And so, I really hesitate to say, like, that it's adversarial or an antagonistic or anything like that, but it's not, mutualistic, I guess you could say.
Argument Against Engagement #2: The Opportunity Cost
Yohann: Absolutely. And that's a perfect segue, actually, into argument number two, which is that you've only got a limited number of touchpoints with which you can interact with the user. We've gone through a number of conversion flows and like the longest ones are, users look at 20, 30 screens before they upgrade, counting all of the branching pathways and onboarding emails, like, how many times can you reach out to someone in two weeks... Seven? Ten onboarding emails at the most? So with that limited number of touchpoints that you're working with, if you are going to focus them all around engagement, that's a huge opportunity cost that you are paying. You are missing out on the opportunity to reach out to them and use those touchpoints to speak to them better when you're just trying to get them in the pool.
Samuel: Yeah, I agree with you on that one. I mean, we talk about the.... I don't know if I should do the whole metaphor again. Oh, I'll do the really fast version. If you're trying to give dog a medication, you gotta hold its mouth open like a crocodile wrestler, and try to throw it in the back of its mouth. And usually it doesn't work out. It's a big problem. Or you can just put the pill in a meatball and the dog eats it up. And when we talk about lifecycle messaging or just managing relationships with users at scale, what we're really talking about is meeting them where they are, helping them get to where they want to be, and strategically popping up on their radar at times where it's relevant for them, rather than just blasting out to people and saying, come engage with us.
Yohann: The way I see it, each touchpoint is kind of like a poker chip, and you've only got 10 poker chips to play. So you want to use those poker chips as effectively as you can. And building off of what you said, Samuel, if you wrap the pill in a meatball, you might not have to use all 10 poker chips. Maybe if you speak to what, if you speak to the outcomes that users really desire, you might need 3 poker chips instead of 10, which is just a way more efficient way to do things.
Samuel: Yeah, and ironically, I think that the user is a lot more open to receiving your poker chips, to really push the metaphor past its limits. But, like, if I sign up for a service and they start blasting me with really tone-deaf emails that are one-sided and, you know, don't have that mutualistic aspect that we've been talking about and are just saying, "we miss you log in again." Or, "Hey, did you know that we're really easy to use? Why don't you log in and see?" Or things like that. I'm gonna be a lot less receptive to what they're sending and my threshold of tolerance of the volume of those kind of messages is gonna be way different than if I sign up for a service, I don't know, let's just say podcast hosting, and I start receiving emails that are actually relevant to helping me get my podcast up and running, and maybe also start getting emails that are like, "You just crossed 50 listens. You just crossed a hundred listens, you're on fire, you just crossed 250, 500. You just got your thousandth listener," whatever the, like, if it's emails like that about how much ass I'm kicking, great. I'd be happy to get 20 emails in, in the span of a week or two under those circumstances.
So, it's kind of goes both ways, like. Your the poke, the limitation on the poker chips is how much time and attention can you put toward different aspects of your user experience. And I think your dollar goes further if you're doing it in a mutualistic way and. I don't know what the, this is really, the currency exchange works better too? Something to that extent? But like, you know, I truly believe anecdotally as a user, and just having studied this, that people are significantly more receptive to emails when they are about things that they care about rather than when they are not.
Yohann: Right. I really like I really like the example you provided. You know, like, this is your 500th email.
Samuel: No, I meant like 500 podcast listens or-
Yohann: Right, right, right, that being a big milestone that you celebrate. When I was researching onboarding emails way back when, most onboarding emails that I came across had a big button at the bottom of the email that said, "go back to the app," or "back to app," or some version of that.
And it's like the app was where all of the good stuff was happening and that irrespective of how you were fitting this product into your life or what was happening with, since we're using the podcast example, what was happening in your podcasting life, or what was happening with your podcasting audience, it was almost like that mattered less than what you were doing in the podcasting app to this podcasting company. And yeah, when you focus on engagement, you just ignore all of the other things that have to come together for a user to find success.
Samuel: Yeah, it's just as a litmus test, there are emails that go out, and it's not just emails I mean it's any kind of lifecycle messaging or whatever, but there are emails that go out that are maybe reprimanding users for not being a good cog in the machine of the company. Whereas, as a user and as a designer, I like to think of it more as like the company, the service that they're offering, is just a cog in the machine of the person and the life that they're pursuing. And like you want to be taking on a cog mentality, not telling people that they're not fitting into your system as well.
Yohann: Yeah. Gosh, that's so good. Yeah, it's just the mismatch of perspective becomes really clear when you see it like that. Like, I think you are an NPC in my story, and you think I'm an NPC in yours.
Samuel: Totally. You gotta be willing to be the NPC and it's amazing how much more engagement, speaking of engagement, how much more engaged people are based off of a service oriented mindset.
Argument Against Engagement #3: Meaningful Engagement Patterns
Yohann: Yeah. Okay. Argument number three is, and you brought this up as well, meaningful engagement. What kind of engagement are you creating? Because engagement for engagement's sake helps no one, it doesn't help the user and it doesn't help you.
Samuel: I think you could, I apologize for interrupting. I've gotta jump in here. I think that you could make an argument that driving engagement for engagement's sake is an unhealthy and unethical kind of thing. A lot of times in our industry there's a regrettable amount of information out there about how to build habits or, you know, change people's behavior and, sort of like co-opting people's behavior or overriding their own pursuit of their best interests to comply more with the numbers that you're trying to create.
And so when it comes to designing for addiction, habits, behavior loops, et cetera, I personally have become comfortable getting more and more distant from that in my growth and design career.
Yohann: It's also a tactic, engagement for engagement's sake, that is left over from the social media model of growing a software company. And it might not really work in a B2B setting. For example, your invoicing app doesn't need users to log in seven times. It just needs users to log in whenever they create an invoice. So what does meaningful engagement look like and why does it follow the pattern that it does?
Samuel: Ultimately, this goes back to the main premise of Vanity Metric or not, where if it doesn't have a strong connection to improving your business's likelihood to succeed, especially in terms of revenue, it might be a vanity metric. And there are a lot of business models where getting people to engage more, even if they don't convert to customers, is still a completely viable strategy.
So as you mentioned, anything you know, social, of course, the more people that you have involved, the richer the network, the more content that's being created by users, et cetera. It's just going to improve the value of the offering itself. Games. So if you're Mario Kart and people wanna play Mario Kart against each other live or go to a Fortnite and play against random people, et cetera, having a whole active community can be a benefit to whoever the people who do buy the, the battle pass or whatever it might be.
So, shades of freemium in a sense, where there are times where getting people in and getting them to routinely engage and never become customers is still worth something within a business context.
But if the business model of the software company is business oriented in a B2B sense, then it's be, it's a lot harder to make the claim that, "Oh, if we can just get people to, to log in, you know, seven times then there's, they're much more likely to convert to customers." Where, like, of course that's the case, because the people who are more likely to convert to customers are gonna be making more use of your product.
And just trying to force more people into your product doesn't mean that those people are gonna convert into customers at nearly the same rate. So I guess long story short, one major consideration here is, if you do have a robust pool of users who are free but are providing a more valuable experience and environment for the people who aren't free, that makes sense to me in terms of wanting to prioritize engagement. But if you are an invoicing app and you have a two-week free trial, there are a lot of levers that I would try to pull before I tried to just say, "Screw it, can we just get them to log in a bunch of times?"
Yohann: Absolutely. And it's just so wasteful to create those kind of social patterns in an app that doesn't have that social element, right? You are managing the resources at your disposal much more efficiently if you get the person to make progress along the timeline of getting their first invoice paid, for example, instead of just logging into the app and exploring it.
Samuel: I think we agree on that. And I have to admit, I I am pretty interested in finding out what your fourth argument is. Would you care to share that with me and the listeners overall?
Argument Against Engagement #4: Strong vs. Weak Correlation
Yohann: The final argument is, again, something we talked about before but didn't really dive into, which is correlation, not causation. I think when people design for engagement, the belief is that the rising tide will lift all boats, you know? That kind of mindset where if we can just create engagement, then it's better than doing nothing. And if everybody who converts engages, then it will make the people who were going to succeed anyway... will nudge them to be successful faster and-
Samuel: -and it could help encourage the people who might not have converted, to cross the threshold?
Yohann: Right. However, I think we can make a distinction between weakly correlated with revenue and strongly correlated with revenue. Yes, engagement correlates with revenue, but I would put it more in the weekly correlated side of things because just because someone logs into the app seven times doesn't mean they then go on to engage meaningfully with the value that the product offers. Just because they log in doesn't mean they're making meaningful progress towards their outcome, and the outcome, on the other hand, is more strongly correlated with revenue.
Samuel: Because it is more strongly correlated with the purpose that's driving the person to use the app to begin with.
Yohann: One of the reasons we love outcomes so much is that it defines what's driving the relationship between user and business. It's the outcome that's meaningful to both parties.
Samuel: If you're gonna try to engineer behavior, would you rather engineer people developing a habit of logging into your app or would you rather engineer some really cool successful milestone that people are coming to you to try to reach.
Yohann: That's a much better way of putting it. What kind of user behavior are you trying to create? If you look at user behavior as the rising tide that lifts all boats, then you want to veer towards the strongly correlated side rather than the other.
Samuel: I completely agree. And then, I mean, I guess for the listeners are maybe wondering how do you find the strongly correlated ones? Because based off of a lot of anecdotal experience that I've had working with different companies, despite people's best intentions and despite having never-before-seen levels of technology to be able to slice and dice the data and perform business analysis, a lot of the growth goals or product goals that are handed down from the executive level are, really are, things like log in seven times or create three projects or things along those lines, which... if that's the best that people are coming up with through sifting through the data, do we have a different way of sifting through the data that we recommend? Or what's so special about our approach that lets people tap into things that they couldn't see before?
Yohann: I think that's a very valid question. If you are not designing for engagement, then what should you be designing for?
Samuel: What are the things that have the stronger correlation? How do you find those?
Yohann: Ultimately, what's different about our approach is that it's very focused on these meaningful outcomes that we talked about, the outcomes that are valuable to both the business and the user. It is very likely that these outcomes take place closer to the conversion event. So getting an invoice paid, for example, is much closer to conversion than creating your account or-
- chronologically, yes, user timeline-wise, I mean, it's much closer to conversion than creating the invoice, for example. So once we identify these meaningful outcomes, we create paths to them and optimize those paths by looking at the drop-off from step to step to that particular outcome.
Another thing that's distinctive about our approach is that we don't just look at the mass users as a whole. We look at cohorts, we divvy users up by week or month. And the way we optimize these paths is to have the trend going up of the number of users who over time are able to make it to those meaningful outcomes rather than just a user behavior very early on in the user timeline that's weakly correlated with revenue.
Samuel: So instead of focusing on how do you get your top-of-funnel, new users to stick around a little bit more on average, you're saying how do you find the people who are sticking around and do have those positive user/customer traits, and then reverse engineer what's so meaningful to them and try to get more people who have that desire to the same successful point.
Yohann: Yeah. To go back to the analogy from the beginning of the episode, the people who did learn how to swim, we're asking them how their experience went and what happened and what their, what challenges they had to overcome and how they really did it, so that we can become better teachers and get more people, not just to get into the pool, but once they get into the pool to do things that are meaningful to them so that they become good swimmers.
Samuel: And then to even think of it one level further and say, what is the machine that learning to swim is a cog in? And like, what does learning to swim unlock for you? If I were a swim instructor, I would be starting with that, you know, first and foremost. Try to understand how learning to swim fits into their life, and then tailor their swimming learning journey around getting them to a point where they feel satisfied about how it serves the bigger, even more compelling context in their life.
Yohann: Right, right. So just to summarize, the Argument From Engagement is that if you can create more engagement, that's a good thing because everybody who converts has to engage and more engagement will result in more revenue down the line. However, we have four objections to raise.
First, it's true that everyone who converts has engaged, but not everyone who engages converts. So if you design for engagement, you are designing for the lowest common denominator and not anything more meaningful than that. Second, speaking of meaningful engagement, what kind of engagement patterns are meaningful to your users? Not everyone who becomes a good customer needs to log into your app seven times. So what kind of engagement patterns matter more than others? You don't wanna just create engagement for engagement's sake.
Third, if you design for engagement, that comes with a huge opportunity cost because you only get a limited number of touchpoints to communicate with users. You're better off using those touchpoints to communicate about things that they care about. That allows you to use fewer touchpoints and make those touchpoints more effective. And finally, engagement might be correlated with revenue, but at best, it's weekly correlated and not strongly correlated. An outcome on the other hand, particularly an outcome that is meaningful to both users and the business. That would be strongly correlated with revenue. And you're much better off designing for one of those.
Samuel: If you can find them.
Yohann: If you can find them, we have a whole process for how to do that.
Samuel: If you're willing to do it.
Yohann: Oh my gosh. Yeah. That's... don't even get me started.
Samuel: Let me not end this on a sour note. I think that was a delightful recap to a delightful conversation. And I would love to hear if anybody else out there has some thoughts that are in line with our thoughts or contrary, critique questions, concerns, whatever you might have, if you want to email us at podcast@selfservesaas.com, we promise to read and respond to everything that gets sent unless it's hate mail.
Yohann: We probably gonna respond to the hate mail too, I mean, knowing us.
Samuel: I would respond by marking it as spam. But anyway, we are very tolerant people. So, if you're worried about offending us, very unlikely, would love to hear from you.
Yohann: Thanks so much for tuning in, and I hope you're enjoying all of these episodes. We've officially crossed the mark of season two being more of more substantial over season than season one.
Samuel: let us see how far we can push it from here, shall we?
Yohann: Yeah, absolutely. Thanks for tuning in and we'll see you next time.
Written by
Written by
.png&optimizer=image&quality=80&width=280)