Lots of folks/companies are going to lose long-term as a result of bad A/B testing driven decision making. I suspect the decline of the Facebook feed might be attributed to this as well (maybe even the recent polls of Trump's twitter practices[1]). Generally these sorts of things don't really consider long term fatigue and other important factors, and take short term micro-benchmark wins as success. On the flip side, some attribute not A/B testing to the failure of the SnapChat redesign[2].
Is there anyway to do this properly? I can imagine the problem will get worse and worse as we reduce human evaluation of A/B test results and automatically make real-time decisions through software and ML.
Any metric or KPI is going to get gamed in harmful ways if it's prioritized too highly. So the answer is to use A/B testing to make sure you're not regressing on the goals of a design change, and judiciously, to validate whether your change is moving things in the right direction.
But beyond that you have to do the hard slog of mastering the discipline of UX, real, human-centric design, and accept that not everything important is measurable.
We have plenty of examples of how doing real UX instead of playing a numbers game can differentiate your business, Apple has applied this philosophy consistently over many years.
The root of the problem is a "fuck you, market share at all costs" culture that has come to permeate Silicon Valley. And you can argue (somewhat cynically) that this philosophy makes sense in a blue ocean where you have no competition and just need to gobble up people and turn them into cash before someone else does. But I think going forward this mentality may actually become a liability as more humane alternatives to heavily despised products emerge. Many of the current crop of giants seem to have forgotten that a company's most valuable asset is always its brand.
I agree with most of what your wrote, but "hire an expert" or "you just have to learn" are non-answers to the question that I am seeing a lot in these comments including your:
> mastering the discipline of UX...
There is no reason to be ambiguous about the practice of good UX design.
The main mistake made by most A/B testing is to think that just because you do some testing, it will magically make your UI somehow better. You can make any tests you want, but how do you know they the right tests? And test results do not interpret themselves. There are no obvious conclusions to be drawn from the results of a test.
The correct way to do user testing is to start with clear, articulated goals, then select tests to see if the goals are failing or succeeding. Anything else is just back-dating hypotheses - a known anti-science practice.
The other misinterpretation of A/B testing is that it will make user interfaces "better" for the end user. Testing can only support or disprove the stated goal and it works just as well for goals that are foolish, short-sighted, self-destructive, and malevolent as it does for those that are ethical and beneficial.
The way to do it properly is to hire experienced, competent UX designers and human computer interaction specialists, who understand the problem space and how to keep people engaged without addictive malpractice / dark patterns. Metrics are useful but they are what they claim to be - measurements - and nothing more.
You wouldn't choose which house to buy based on how large it was measured to be, but you would expect a surveyor to make those measurements.
Speaking as a web and video game designer, I can't wait until I can hire someone to think about these things for me.
Once this becomes the goal, the game is already up. It doesn't matter whether you use A/B testing, focus groups, or brilliant designers who operate on intuition. By targeting engagement, you're treating your users as a means to an end and falling afoul of the categorical imperative [1]. You're focused not on meeting their needs as users but your own needs as an engagement-driven platform.
The answer to this puzzle is to take engagement right off the table. An ethical business model is one in which you treat your customers as fully autonomous agents. As soon as you look past people's autonomy and see them as creatures of habit with psychological traits that can be gamed, you stray from the ethical path and find yourself searching for other rationalizations.
Apologies, I was using engagement in a different sense. As a game developer I want to keep people engaged with the game - not just arbitrarily "hooked" but actually immersed and invested in the game I have made.
I would say: talk to your customers, find out what their needs are and try to make your product better at fulfilling them. Focusing on metrics is going back in the direction of treating your customers as means to an end: like grain to be harvested or sheep to be shorn.
Instead try to think of your customers the way an old barber would think of his clients: as friends with individual needs who trust your experience in a particular problem domain.
Of course, this approach is extremely unlikely to lead to a billion+$ exit. But if your goal is to become a billionaire and you want to be as stridently ethical as possible, I don't know how to help you (and would invite you to further examine your priorities).
It’s unclear to me that highly trained professionals are any better at this than a/b tests. Usually it’s the professionals themselves telling us that they’re better.
I guess it depends on where any individual places the task on the skill/luck or simple/complex scale. If the task (in this case, a design) is very simple — how many users who want to complete their voter registration form are able to — then experts may have an advantage.
OTOH, if it’s a complex space like facebooks news feed, I don’t think experts are the way to go. They simply think that they are.
FWIW, I’ve run hundreds of multivariate tests in my career. Every place I go, the design team is upset by the process, for the same reasons argued here. Every time, I challenge them to come up with 1 winning variation and 3 losers. If the designer-identified “winner” performs best, i pay them $10. If any of the losers wins, I get $10.
I have yet to meet a designer that can, a priori, pick the winning variation better than random chance would suggest.
Expertise is not valuable in an unpredictable context. Complex user facing systems like the news feed are impossible to predict.
I’m channeling Kahneman and Mauboussin here, but their theory explains my experience very well.
Trained experts in short term user tricking into more engagement would be marketers and advertisers. Not usability experts. Usability is not easily measurable with A/B tests, as those are usually not long term enough and not done well enough to make any conclusions.
Note that talking to customers is not easy too. To the point that you need to psychoanalyze all the feedback to understand why they make those comments. Useful feedback is very rare. Customers are just not usability experts, they don't really know what they are talking about.
Probably lots of things that require direct user feedback, like output from focus groups such as customer attitudes towards your product or customer impressions of utility and value.
This makes sense. It’s like how clickbait headlines would, if they previously didn’t exist, improve click rates. But long term if your platform was all click bait it would fatigue the users. Or at least, one might think so...
Clickbait + low quality content is what fatigues users.
Personally I don't mind clickbait if after doing whatever it takes to grab my attention, the creator delivers on an actually solid high quality educational/entertaining content.
Personally I don't mind clickbait if after doing whatever it takes to grab my attention, the creator delivers on an actually solid high quality
Captain Pedantic, reporting for duty, and that’s not actually clickbait. If the masterful baiter of clicks actually delivers, then it wasn’t bait. I mean, what’s to complain about? Turns out there is a way to target stubborn belly fat, five minutes a day, who knew?
>>People are hired for roles when company cannot afford not to.
Only if people have a stake in the decline of a company, and have something to lose. Other wise in these days of executive networks, and people getting in their golf and dinner buddies- Execs mostly care about nothing at all.
The only people to lose anything are a few low level employees and may be shareholders. Shareholders mostly don't care either. People who hold large block of shares, are generally diversified. And since nothing fails at all at once- They are covered too.
That’s a bit reductive. You may be right, but it would be less facile to say that poor designs can be made by anybody, but that a mismatch between design and business objectives led to bad results. Not all business people are morons, and not all designers are good at design.
If the business objectives lead to a design so bad that it hurts the business, surely the people who came up with these objectives didn't do a very good job?
I mean one way to do it properly is to actually listen to your users. So many corporations claim to have that as a priority and when you look at real, actual feedback, they really don't. How hard is it to look at the masses of complaints about autoplay or notification spam or... any numbers of dark patterns you could name and decide that you shouldn't do that?
Yes I remember this, every time, as Facebook, Twitter, and Instagram made it more and more difficult to keep your feed in chronological order.
It is nakedly a psychological trick to give you a mixture of things you've seen before and new content to make you feel like you should try again more often - the slot machine strategy. No one asked for it.
Of course it helps that ads are easier to integrate into a feed when you don't even know what to expect in your feed in the first place.
So now, not only do these platforms not give a feed of pages and people I subscribe to, they litter my feed with ads and 'popular content' that I never subscribed to! otherwise how could they charge money to increase your organic reach?
what a crock! It was a marketers' dream to have their latest news and coupons delivered to your inbox and have you actually look at them, but if you subscribe to businesses you want to follow, how can the ad network make money? So they turned everyone into a customer of their ad platform, and held your content hostage until you paid up. This has effected numerous friends that are small content producers that maxxed out facebook's 5,000 friend limit and all they wanted was to invite all of their fans to their next event.
Other POV: I only check FB a few minutes everyday or so and that algo, maybe ironically, is what allows me to do so.
You have 1000 friends each interacting once per day with FB in a news feed worthy way (posting, joining an event, commenting etc). Are you supposed to look at 500 daily news feed posts to get a 50% chance of knowing what that friend you really like is up to? (pushing the trait but you get the idea)
Instead having to seep through endless activities I don't care about I can scan 10-15 posts per day and they are the relevant ones from the groups and friends I want to interact with.
--
I'm fully with you on the point you make about business/content creator you volontarily subscribed to having to pay extra for you to see their stuff.
It would also be less volume of stuff if Facebook only POSTED THE THINGS WE WANT PEOPLE TO SEE. No "likes" or such. Give people a clear idea what they're actually sharing/posting.
FYI-- I have many of the same complaints and found that the browser extension https://www.fbpurity.com/ is able to clean up a lot of the cruft as well as put the timeline back in chronological order.
Its a phrase that is often used but it is usually not as simple as that. Imagine that your existing users have a very different set of needs and priorities as new users coming onto your platform, or not yet users you want to attract. This means making hard decisions that will not please everyone.
Or, as you mention, your shareholders may require you to generate revenue faster and then you are left with a lot of bad decisions to make and your only choice is to pick the least damaging one.
> your existing users have a very different set of needs and priorities as new users coming onto your platform, or not yet users you want to attract
In this case you should side with your existing users who 1) you already convinced to sign up and engage with your product 2) helped you build your product in the first place.
I’d take an established power-user any day over a potential new user that has yet to prove themselves (they might quit the platform as quickly as they joined).
Of course, some idiots don’t understand this, and “Twitter” is the result.
> In this case you should side with your existing users who 1) you already convinced to sign up and engage with your product 2) helped you build your product in the first place.
sometimes you don't have that luxury, because "you need to grow".
Apple doesn’t listen to users either. Think headphone jack. I believe it comes down to a core set of values that guide you daily through tough decisions. Like: we believe we should not trade privacy of our customers for profits (just made this up on the fly). This will rule out a lot of stuff.
Corporations have no priority except to make money. If making money is at odds with positive user experience (which is almost always is), making money wins.
>> I mean one way to do it properly is to actually listen to your users. So many corporations claim to have that as a priority and when you look at real, actual feedback, they really don't. How hard is it to look at the masses of complaints about autoplay or notification spam or... any numbers of dark patterns you could name and decide that you shouldn't do that?
> Corporations have no priority except to make money. If making money is at odds with positive user experience (which is almost always is), making money wins.
That attitude has a similar problem: socialism has been gaining in popularity, and I'd say that's a direct result of corporations over-prioritizing their money-making over both customer priorities and other social priorities (like paying reasonable amount of tax).
I had been a customer of theirs for years, starting with the DVD rental. After a while they started adding the streaming content which was cool, but then they eventually separated the two services and switched my plan from DVD rentals to exclusively streaming content. All without any action on my part or any consent from me.
Eventually, I quit Netflix because between Prime Video and Hulu, I was covered for my streaming needs and with the exception of a few shows, was unimpressed with Netflix's original content so it was a waste of money for me.
Fast forward to a week ago, I decided to give Netflix another shot. What amazed me is the dark patterns you speak of. Like downloading content for offline viewing.
I was without access to my normal ISP and had to use cellular data so I opted to download a show. 1 episode was 100 MB and I'm on Google Fi so it was basically $1 of data usage which I understood. But then when the episode was over, I stepped away from my phone to go to the bathroom and when I came back, I found that the episode had been deleted from my phone, and that the next episode had automatically downloaded without any action on my part incurring another $0.90 of data (not to mention deleting the episode I had intentionally downloaded making the first $1 a waste)
I saw at the top of the download section that there was something called "Smart Download" enabled that claims to do exactly that - delete the last episode and download the next, but it's only supposed to happen on wi-fi which wasn't the case for me. I turned off the feature as it was obviously broken and tried contacting customer service.
The CSR I talked to then told me that the app cannot delete or download material without me clicking a button - which contradicted their description of "Smart Download" that I had just read, and he explained there was no way for them to process a refund or credit to my account because they don't have any way to do that.
This seems like a pretty major problem.
Even when confronted with a direct customer complaint, their response was just to deny, deny, deny rather than taking even minor steps to rectify the issue. This seems mind boggling from a customer service perspective when companies like Amazon will bend over backwards to keep their customers happy. Even Hulu will credit you a free month of service when they goof - they even do it if you try to cancel the service because you can't find anything good to watch. Netflix's service isn't good enough to justify this type of behavior and general apathy. Customer-last seems like a crummy business model.
I can feel your frustration with the service but I'm unconvinced that "download the next ep" is a dark pattern and more that you're a niche that their UX currently isn't catering towards.
I do a lot of cellular/tethering and know some of the pain points but can recognize that the needs are quite different to the regular user who may consume stuff at home then want to take stuff with them on the go.
Normally I see that UX often caters to the lowest common denominator which leaves power users and niche users sidelined.
My personal bug bear on this subject is that YouTube no longer buffers entire videos and holds them there - presumably this saves them lots in traffic costs and additionally allows them to up sell YouTube premium
Sorry I didn't see this immediately, I don't think that downloading the next episode is a dark pattern in and of itself. But I do think downloading the next app over mobile when the description explicitly says it wouldn't is questionable. And I think deleting downloads automatically without any good reason is so far removed from assumed functionality of a download to be a borderline dark pattern.
Without the "Smart Download" enabled the episodes persist until I delete them or cancel my service, that's the assumption with a download. Smart download immediately deletes whatever you downloaded as soon as it sees fit - which in my case was erroneously assuming I was on wi-fi. And this was my first download, it's not like it had some storage space restriction to deal with. I had ~16 GB free and it was 100 MB.
The next episode feature aside, it seems weird to say, "Okay, download this one" and then it saying, "Okay, so you meant download it and then delete it".
And to your Youtube point, I think it depends on the use-case. I fully agree with holding the buffer there, but if I'm on mobile data, I might not necessary want it to just load as much as it can as soon as it can because I pay per MB.
I actually love the smart download option, it allows me to always have the next episode downloaded when I leave the house.
This seems more like a CS fault than anything. Netflix is a huge company and CS agents are outsourced/so far from the actual product team that they operate on guidelines/info that might actually not be true/outdated. I'd say it's just a problem with a huge organization.
I'd assume #num_customer_complaints is not part of the A/B metrics. If people complain but don't follow through with actions A/B probably should ignore it.
I'm not an MBA, but I imagine there's something where if you wait until you know for a fact that customers are leaving for specific and known reasons, you're waiting too long to fix it.
What if they follow through with it 6 months or a year later? Your a/b metrics won't catch long term effects. Maybe a change gets people more engaged short term, but drives them away long term.
Others say "hire competent UX people", but I don't think it's necessary, and I don't think doing that will help.
Even the best UX person won't help you if management - or processes they set up - push for microoptimizing profitability at all costs. If the competent UX person doesn't have morals and ability to push back based on that, they'll either leave, or - worse - implement the bad things that much more efficiently.
The way I see it, companies must realize where their profit/growth optimization borders on user abuse, and then not cross that line (or be made to not cross that line).
All the morals in the world are meaningless when there is no agency.
I’ve yet to find a UX team I’ve worked on with enough political capital to win any conversation which results in a net-decrease in profits, regardless of the ultimate benefit.
Too many quarter to quarter direction to overcome the profits-today-at-all-costs momentum which grips management.
I have my differences of opinion with most UI/UX teams I worked with (in particular, I don't buy the trend of making software as easy as possible at all costs; it leads to dumbing down and reduced utility). But I'm not blaming UX people for this. They have as much to say in this topic as developers about third-party JS bloat - i.e. we can all complain all we want, out of practicality or morality, and ultimately the decisionmakers don't care.
At Google (while Amit was there) trained manual raters were very important part of determining whether a change should be launched or not (especially for relevance). I'm not sure how the process changed since he left, as I see much more wrong auto-corrections by Google.
It is as you say. The key is that you always get what you optimize for. You can optimize for short term addiction or for long term sustainability. The issue is that large factors of these goals are incompatible.
This reminds me of a few years ago when they did "cash for clunkers" to stimulate the economy. Tons of people took advantage of it and car dealerships has two amazing months........ followed by four months of almost no sales. All that happened was that people who would have bought one in the next six months took advantage of the incentive now. It didn't create any new demand for cars or stimulate anything. If you only measured success during the first two months you might have been convinced that it was a roaring success.
>This reminds me of a few years ago when they did "cash for clunkers" to stimulate the economy.
Psst.. Are you ready to feel old today? The Cash for clunkers program was from 2009. Yeah, really.
(The federal program in the US at least, I don't know if states ran their own version since)
One of the best examples of this problem is something entirely outside of software. Coke and Pepsi. It's well known that people prefer Pepsi to Coke in blind taste tests by a very wide margin. Yet in the market Coke outsells Pepsi by a very wide margin. This anecdote was used for quite some time as an example of the power of advertising. But it turns out the truth is something much more simple. While people do prefer Pepsi in blind taste tests, that's because in blind taste tests you're only getting an isolated sample of the product. To cut out the software metaphor, people prefer Coke when it comes down to drinking an entire bottle of the product. It has nothing at all to do with advertising or false preference - Pepsi was just testing the wrong thing.
The point here is that just because people think they want or like something doesn't necessarily really mean they do, especially when what they're 'consuming' is in some way different than the regular long term consumption of the product (as you hit on). I don't think it's much of a coincidence that some of the most famous marketers, most relevantly here probably being Steve Jobs, weren't particularly interested in what the consumer thought they wanted.
From a software engineering point if view, the problem may be not A/B testing correctly.
From a sociological and personal point, problem is the companies which effectively make money off of stealing people's time and data for profit by making them agree to short term gains through addictive content.
What we as responsible adults need to do is to find out what addictive psychology is and put an end to these opportunistic constructs through law making.
In my opinion, time is better spent on cultural skills and pursuits, not flocking to "the next big thing" you are spoonfed.
This is perfectly useless advice. You've hidden all the magic in vagueness. Everybody thinks they have common sense, so of course they already think somebody with common sense looked at the design.
I'd suggest either just giving an honest rant or giving actual advice. And for that you have to really unpack what you mean by "common sense" until somebody who doesn't have "common sense" can have a chance of understanding and fixing the problem.
I would ageee with you for cases where both sides had a strong argument, but in Snapchat’s case everyone was complaining about the new UI, so I’m not sure how to describe that sense - people who complained are from all different ages & backgrounds so it’s not like they had UI design experience or anything - “common sense” is probably the only thing they all had in common and that thing alone was enough for them to determine the UI sucked.
In this case, simply doing some basic user-testing would’ve been considered common sense and would’ve brought up the problem immediately.
Basic user testing would’ve been recommended by anyone with even a tiny bit of common sense.
Either the company truly didn’t have anyone do user testing, or they did and were ignored (which doesn’t surprise me either considering the child they appointed as “CEO”).
I'm not sure why the quality I'm talking about is so "mystical", when hundreds of thousands of people on social media seem to have it just fine (given their reaction to the redesign), as well as pretty much anyone who talks about the Snapchat UI.
Maybe the problem isn't with my rant, but that somehow the people running that company do indeed not know what they're doing?
I'm afraid you keep missing my point. If you have some specific question about it, feel free to ask it. But I don't feel like me giving another explanation will move things along, so if you're content not to understand it, that's fine by me.
I think there needs to be some sort of human ethics oversight for running psychologically harmful research experiments under the guise of A/B testing. When did the practice of 'informed consent' on human test subjects become optional?
I find it interesting that everyone’s alternatives all involve much more subjective decision making progress than A/B testing. Not that it necessarily makes them worse but it partly explains why it’s so popular to A/B test.
It's popular because math and stats are always right. Even if you've hacked your A/B test to the point of where you should have zero confidence in the results, you can still point to them and say "math says this is the best decision, so we should take it".
The problems associated with NHST are well documented (three that are especially awful: multiple comparisons lead to uncontrolled rate of false positives; large sample sizes leading to statistically significant but substantively irrelevant effects being "accepted"; hard thresholds of statistical significance are arbitrary and the difference between a significant effect and insignificant effect is rarely significant). In addition, there are problems that emerge from using NHST as part of an A/B test: namely, using live, rolling samples to identify significant results rather than prespecifying sample sizes; and making atomic changes as though they are additive without re-testing joint hypotheses of multiple changes.
But when you come down to it, the question you want to ask is "Is B better than A?" and the question you do ask is "If A were truly no better than B, how often would a sample of size n drawn using the same sampling procedure I think I'm using produce the impression than B is as much better than A as it is apparently in my observed data?", and two problems are that these aren't the same question and almost no one knows they're doing the latter.
To be totally fair, one of the most common problems with NHST (the null hypothesis is patently absurd) isn't necessarily a problem in the A/B UX case.
Not sure which of these in specific the grandparent is referring to, but I suspect they and I are on the same page in general.
There is always a difference between condition A and condition B. It will be detected with careful enough measurement and/or large enough sample size (spend enough money).
Unless you can predict what this difference is expected to be with some precision beforehand it is not possible to legitimately attribute it to your favorite reason.
It always sounds like the people working on these redesigns are not heavy users of the product. And by that I mean legit heavy users, not dogfed users.
Personally I've said half-jokingly that the redesigns are for job security because admitting that the design is perfectly fine and changes have no guarantee of improvement greater than the adjustment cost would result in a pink slip.
Although that does point to a common problem of 'design for purchasing manager' where software is designed for the sake of pleasing people who don't actually use it and merely adds a bunch of useless cruft that would rapidly become an irritating obstacle to users but looks impressive on first blush.
Cycle metrics so that you don't get stuck on local optimization minima.
It's also helpful to have long term and short term optimizations to avoid burning fats, like measure both conversion and yearly conversion because it's more than likely that a discount for a week will not increase yearly conversion but will reduce yearly earning.
This was learned hard by steam, they made the sale event so big the whole game industry changed their pricing practices dropping in time much much more slowly than what it used to be.
I keep being more and more impressed with Apple to be honest. Whether it is desktop UX, phone touch UI, laptop trackpads, or even how seamless Apple Pay/FaceId works, most of what they push stands up to the test of time relative to the seemingly more data-driven experiences.
This is exactly the way capitalism is supposed to work. Companies correctly prioritizing their long-term wellbeing over short term gains. Netflix is watching what's happening to Facebook, and deciding that that's not the path they want to take, even though it would likely be good for their profits in the near term.
I wish we could see more logical long term ideas. So much of it is short sighted I feel. I often describe stuff such as Facebook as immature technology that doesn’t understand what it is capable of, and doesn’t seem to care about the negative impact.
Ya, it is unfortunate. But I think it is the process working the way it's supposed to. It sucks that we have to touch the stove sometimes to learn that it's hot, but that seems to be how it works. Facebook touched the stove and their stock price got burnt. Now everyone else is learning from that so it doesn't happen to them.
I think it's the system working as it should, and I just hope people don't over-react and try to throw the system out when it seems to me to be actually doing a pretty good job at healing itself here.
Nothing drove my adoption of prime video faster then the auto play on Netflix. To their credit I contacted them asking if there was a way to turn it off and at least spoke with a friendly person about how awful it made the experience.
I find it annoying on youtube though, for some reason.
Edit: Oh, wait, are you talking about autoplaying preview trailers or autoplaying the next episode? I like the next episode feature, autoplaying trailers are annoying.
Binge watching is not that different from binge drinking - it's a path to oblivion. You might still have enough health to deal with it, but the health itself dwindles in the process.
You got it. I want to be able to open the app and flip through my options in peace. Having the preview play automatically makes the experience into a sick game of hot potato. Go too slow and you are blasted with random preview music with every click...
Kids shows are worse though. Netflix will sometimes helpfully restart the entire series when you finish it. Super Halloween Monsters all day long with no intervention needed woo.
On the other hand my Amazon Video app (Roku in my tv) is slow as hell and I'm lucky if it doesn't crash (1) on launch, (2) between the homescreen and the video I selected, (3) between the episode selection and the video actually playing, (4) sometime in the first 10 minutes, (5) randomly thereafter.
Prime Video on Fire TV boxes now autoplays some selections in the menu (maybe just Amazon original content?). Now I have to mute my TV with Prime Video just like is required for Netflix.
Prime doesn't have the bizarre, useless, space-wasting auto-expanding selections like Netflix though.
Netflix stream selection UX is so infuriatingly atrocious.
The worst aspect is how it shrinks the credits and cuts them off prematurely, so you have to go and click on them, in case you want to watch them. Many shows have extra scenes at the end of the credits, for example. Or sometimes you just want to listen to the music, like at the end of Silicon Valley which has some great tracks.
I really dislike the modern Netflix UI. I miss the old Netflix, where you could rate your shows with stars, it did a good job of actually suggesting things, and didn't force you to make choices you didn't want to make.
I second this. The auto play is super irritating, dosen't let me browse in peace and has led me to exit Netflix and try something else more than a few times.
The convenience of the app based remote has allowed us to function relatively well without a TV remote at all. One drawback is the lack of remote volume control - so I'm interested in a peaceful experience while browsing.
The Netflix web interface hs a mute button that only applies to trailers, which the poster I was replying to was referring to.
And yes, since you ask, my TV’s mute button doesn’t do anything when speakers are connected via the optical cable (I suppose it sends some event and the optical -> analogue converter I use to hook the TV into my ancient sound system just doesn’t know how to handle it). But that isn’t Netflix’s fault.
Several Netflix UI changes have been pretty anti user. I'm more upset at the way they mask an incomplete catalog by pretending most Hitchcock films or French new wave don't even exist. I'm more upset about auto play. I'm more upset at the retooling of the recommendation engine away from finding gems you didn't know you would like to just finding ways to convince you to watch Netflix owned content.
Adding a badge system? Not wonderful, but basically just brings them up to par with almost anything else on the internet. If we want to oppose badge systems, and maybe we should, we should start with the entire gaming industry and social media.
Netflix has so many other sins that they have raised the bar on my outrage. That in itself is sad...
Personally I didn't like their decision to "randomly" switch the movie's image thumbnails. This way they keep catching my attention on titles which I already decided I was not interested in. Quite annoying.
This is my most disliked feature as well and it causes the opposite, inconvenient effect for me: I add to my playlist a movie or show that I want to watch in the future, days later, I go back to my list and it takes me an unnecessary extra amount of cognitive effort to find the same item because it had a different thumbnail back when I first added it. So irritating.
I decided to use W10 application solely to avoid autoplaying in browser version but recently they introduced that amazing "feature" even there. But what's more annoying is that my currently watched series tends to pop between various genre sections on the titles grid, not mention that sections themselves are also being swapped every now and then just to lure me into watching something. No idea if that happens also in the browser version but I assume it does.
And I also noticed that my profile has different images than one used by my partner - but that's not a surprise from what I recently read; netflix algorithm uses some heavy profiling - more black characters on thumbnails for those who seems to be black person etc.,; and the algorithm thinks I am black person because I'm watching Star Trek DS9...
I should have been clear, I meant the trailers. I'm browsing for specific content but getting blasted by ads for Netflix originals.
> did you ever think maybe you should just not subscribe to it?
I often don't. They've pushed me from a habitual subscriber to an occasional subscriber. Perversely that means I'm opting out of providing them ux data.
But in general people can have complaints about something even if it's not a dealbreaker. For another example, the ux on aging isn't great, but it beats the alternative.
It’s really painful to browse Netflix, I want to read the descriptions, but I get a loud trailer with every selection. There’s no way to turn it off afaik, so I mute my tv whenever the Netflix app is open.
Netflix's interface was best when they still shipped DVDs in boxes. Ever since they stopped that it has been a steady descent into the dark patterns of social / mobile / eyeball counting user interface fail.
Really? I've been a Netflix user since 2013 (so after the mail service) and I think it was the best around 2014/2015. When they had great content, just gotten success with their own content, and things seemed to be booming.
Now the content is subpar; every movie I've wanted to watch in the past three months is not on the platform. Add to that, the UI is incredible tedious to navigate. Everything _has_ to be "Because you watched XYZ" - and never the kind of content I'd be willing to settle for.
Around 2008, Netflix signed a deal with Starz that brought their enormous catalog to Netflix's streaming service. It seemed there were almost no well-known movies in English that you couldn't find streaming on Netflix. That was, until the deal expired in 2012. Those were the glory days of Netflix streaming, in my opinion.
Yes! At one point there was a way to get all movies by score paginated in a simple HTML table without even pictures. Just nice, dense, sortable and filterable information.
The only better interface for me personally was when I used their API via Ruby and was able to sort exactly how I wanted.
That might have been helpful for you, the user, but it was likely a problem for Netflix the corporation. I remember people scraping those tables and creating websites like "Check which shows are available in your regional netflix".
Because every international Netflix variant had a much poorer selection of English-language titles compared to Netflix US, it must have hurt international adoption.
Unsurprisingly, Netflix changed their site layout so that such tables were no longer "exposed" to the end user.
Envelopes, whatever. They cared more about interface and recommendations when they were an e-commerce company and not focused on pushing original content.
It was always envelopes. I actually remember when their streaming was first coming out and you were limited by how many hours/shows you could watch in a given week.
I agree with the above poster that 2013 - 15 Netflix was the best. Transparent ratings systems and (to me) more relevant content with higher quality originals on average.
But a lot of the most interesting titles are stuck on “very long wait” which seems to mean they’re not planning on buying another copy to replenish their library.
It's always two steps forward, one back. I've used Netflix on a half-dozen devices and the latest version on PS-4 is the worst. Some improvements, but then adds things like five-second autoplay at episode end and autoplaying ads for shows when you view the menu.
I no longer feel confident if I add something to my list that I'll be able to find it later, mainly because the interface has changed so many times that I have to remember what to do.
This was a silly product feature, it would have killed the quality of their viewership data.
As soon as you reward someone for watching, they'll figure out how to script it. They'll load it up, set it going turn off the speaker and the monitor and go to bed.
It wouldn't have increased my engagement, but it would remove me as a data point!
They can probably flag autoplays in the statistics? And, to be fair, Netflix is not making more money for more streaming, or have I missed something. Autoplay should actually cost them money for bandwidth wasted in the case the customer don't want the next episode.
I would not consider autoplay the next episode a dark pattern.
Are we talking about 2 different types of autoplay? Yes, the autoplay of the next episode feature can be a waste of bandwidth, but that's why they put up the "are you still watching" after X number of episodes/TRT/whatever. Besides, it's a nice feature when you are trying to marathon through stuff, especially when they allow you skip openings/recaps of something you just saw. I pushed for my company to allow the Play All on DVDs to do this very thing.
It's the autoplay of trailers/previews while browsing their library that is the most annoying. This is definitely a waste of bandwidth for me. And while maybe not a dark pattern, it's annoying as much as popup UI stuff or in social feeds, ads in news, etc.
Hindsight is 20/20 but this idea is definitely one of the worst ideas I've heard when it comes to applied gameification. Perhaps unlocking some sort of extra content would've been better, but handing out badges is such a deliberate way to signal to parents that "we are trying to hook your kids".
I find it very obnoxious how the Netflix app for Apple TV autoplays previews of titles when you’re browsing and doesn’t let you disable that behaviour. As a result I rarely open Netflix except when I already know what I want to watch.
Autoplay is a cancer that spread across the popular media platforms. Facebook, YouTube, and Netflix seem to have all abandoned the idea that the app shouldn't play distracting video and sound without prompting. (I haven't checked Hulu in years, I don't know what they're up to.)
Remember when it was annoying pop-ups that auto-played video and sound? Now it's a feature to extract as many watch-time minutes from users as possible. I guarantee it's all A/B testing looking for a few points of uptick in watch time.
Just yesterday, I found that YouTube on my TV reenabled auto-play on it's own even though I disabled it long ago. Even though I doubt it was intentional, I found this incredibly frustrating.
How could it be unintentional? At best it was a deliberate ignorance of the consequences of a settings schema upgrade. Just like how bank errors always are in favor of the bank.
Realistically... Very easily? The default is to have auto-play on, I can imagine many scenarios where settings get lost. I've not heard general outcry about this so my best guess is that it's an edge case.
I know a few other cases regarding auto-play and youtube too.
I've had it re-enable itself a few times. I'm sure it's due to the extra add-ons and such I use that's messing with it. A few of my friends don't use the same add-ons I use, or they don't use any and have had it happen enough times for them to complain to me about it.
My friend first found out his auto-play was re-enabled after youtube ate up most of his monthly bandwidth cap in one night.
So we're definitely in the minority, but it does seem to happen to different people with very different configurations.
Twitch auto played when it was embedded on a website and used 6GB before I realized causing my internet to be shaped. Fortunately it was only a couple of days before the quota refreshed. I now block the ttvnw.net domain using a browser addon.
There is none of this nonsense on Hulu (from my phone to Chromecast). I recently dropped Netflix for Hulu simply because of the more interesting portfolio.
Same here. I literally found myself scrolling Netflix for many, many minutes on end. It essentially became Facebook. I don’t know if the issue is their catalog has expanded with so many originals that it’s hard to find something to watch and or that not having ratings (to tell if something sucks) me to not watch.
Sure they have some stellar content but they also have a lot of crap.
The fact that Netflix no longer has a rating system is quite sad. That was one of the high points of Netflix back in the day - the ability to read reviews of shows/movies before committing to them. Now everything has to be “recommended” to me.
IMHO most of Netflix's catalogue is crap... Or their recommender system just doesn't get me. It seems to have overfit on anime, even though I just watched a couple a long time ago.
Plus if I want to watch current TV or more recent shows alongside originals it’s available on Hulu. Netflix has lost a lot of the more bingeable shows and it really doesn’t have any current content.
I get that Netflix is a different beast than Hulu in many ways. But for cord cutters Hulu feels more like the total package.
Hulu does have autoplay, at least on the Android TV app, which is why we use it much more than we do Netflix. If there weren’t a few Netflix original shows which my wife and I really enjoy, we’d cancel our subscription.
Kind of playing devil’s advocate here since there’s so many negative comments on this feature, but is it possible that this was a well-intentioned design decision on the part of Netflix?
I personally like having the ability to preview content rapidly—I can glean a lot about a show/movie in just a few seconds of video (title/thumbnail alone is low-info). The old pattern was ~5 clicks + buffer time to watch a preview (if one was even available). Also, opening Netflix signals an intention to receive audio+video—this isn’t nearly so bad as e.g. ESPN’s site randomly blasting you with a video as I’m trying to read an article.
Anyways, I agree that software should be as user-configurable as is reasonable, and it seems that these days apps like Netflix are completely bereft of meaningful options. Also much of the content that Netflix pumps out feels like the output of a generative algorithm designed to cover market segments vs. anything guided by a sort of original artistic vision.
(PS—can we collectively pump the brakes on the “X minor annoyance is cancer” lingo?)
For me there is definitely a bit of a "hype gap" for most shows on Netflix and those previews make it a bit smaller. A title and some random faces just don't get me interested in anything. I remember actually wishing for trailers on more than one occasion.
But it's easy for me to not get upset about auto-playing trailers because my primary client never got updated to support them.
One thing I'm always reminded of when I hear about an HN reader bemoaning a feature/policy: Behind every feature change (like auto plays), there's a product manager who's tested changes to show that it maximizes some short-term metric.
Users get the products they "want", not what they need — or deserve:
Make Something People Need
> We are surrounded by well engineered — and increasingly addictive — technology that shapes what we do and how we think. Especially when you work at a large tech company, how do you trade giving users what they want and your own business’s profitability, with the negative effects that this leads to?
>I find it very obnoxious how the Netflix app for Apple TV autoplays previews of titles when you’re browsing and doesn’t let you disable that behaviour. As a result I rarely open Netflix except when I already know what I want to watch.
I suspect it's a result of poorly done UX research.
If you frame the question as "How can we increase engagement with the homescreen", adding noise does increase engagement.
Basically the % of people interacting with the home screen increases, but the number of people viewing the home screen decreases.
Ironically this means less time spent exploring...
Netflix often generates some interesting micro-genres and I used to scroll through a bit looking for things to add to my list - now if I don't have a specific title in mind I don't open Netflix at all.
Though I'm not sure I should complain - less mindless watching has lead to me reading more research papers and news. Being more productive.
The whole point is for users to spend less time exploring so they don't realize the selection of decent content is actually quite limited. Netflix doesn't care if people don't like it as long as they keep paying for a subscription. People will complain about this but not cancel the subscription. People will cancel the subscription if there's no content.
>The whole point is for users to spend less time exploring so they don't realize the selection of decent content is actually quite limited.
I actually disagree. I think Netflix has several massive troves. For example I recently worked my way through Frasier, then Cheers. There's a lot of hit shows like Star Trek, Friends, Buffy etc.
I agree the selection is not perfect, but for me, there's a lot of stuff that I wanted to check out in the 90s but was too lazy to monitor a VCR for. There's also a decent enough amount of original content - Black Miror, Bojack for example, that make me feel I get my money's worth.
Maybe it's a generational thing... Netflix has so much more than any Blockbuster.
If not Netflix, what service? Who has better selection? Who is their rival.
IMHO Netflix is getting better at creating great content - Bojack Horseman for example is superb. They have the name brand recognition. And at the end of the day, the price is right - people can justify 12 bucks a month.
IMHO Netflix's rival is not Hulu or Prime Video - it's the local library. Many library systems nowadays offer DVDs and BluRays. Why pay a monthly fee when you can get the same titles from the library as fast as you could. Basically, people may be willing to trade cost for speed. (Heck sometimes I can get things faster from the library than I can from my DVD plan)
My wife hates this behaviour because you never know when it's going to show previews for intense thriller or even horror content. It's bizarre that they would show this to an account which has literally only ever been used to watch Friends, How I Met Your Mother, Gilmore Girls, and similar shows.
I think the issue is that the autoplaying background promotions have nothing to do with the recommendations— it's mostly just Netflix pushing their exclusive content.
Same with Netflix app on Roku. I keep keep using up/down/left/right to keep stuff from playing, which is pretty silly that I have to do that to keep stuff from playing. I'm trying to use my own attention to read title and try to make a decision and it's trying to blast unwanted information at me.
Yes, the autoplay preview is very annoying, but I also find annoying the little information provided in the thumbnail gallery. To see more info about a title, you have to select the thumb to go to a full screen view of the title. Except now the actual video starts to play even though there is clearly a play button that was not clicked. At this point, all the play button does is hide the overlay. The buttons have no real meaning/purpose.
And I don't understand the economics of this at all. Don't they pay for bandwidth? Shoving out crap that no one asked for costs Netflix money, doesn't it?
So why do they do it? What's the business case? I know they aren't idiots over there. What do they know that the rest of us don't?
Are the content producers actually paying them to force-feed those previews?
This is the heart of the net neutrality debate, is it not? ISPs claim that Netflix doesn’t pay for all the infrastructure required to carry their bandwidth.
They don't pay for preferential treatment at the expense of other users -- and they shouldn't -- but they pay for the bandwidth they consume just like you and I have to. Otherwise we'd all have OC3840 connections running to our basements.
I doubt they're right, and/or I doubt they told you the truth. This feature actively impairs discoverability by putting the user in a Skinner box where they have to push a button, repeatedly, to make the negative conditioning stop. In my household, it mainly serves to make us "discover" what's on Amazon Prime Video instead.
Of course, many of Netflix's UI features have historically been engineered to make it hard to tell just how limited their selection really is, so autoplay does fit that pattern, at least.
I have given up and just not used Netflix before because of that awful obnoxious feature. There's nowhere you can navigate to in that app that doesn't play video, with sound - except presumably settings, but that doesn't help me choose a show. Awful, anti-user design. I hope the trend of people consciously fighting back against deliberately addictive apps continues. Personally I'm putting my smartphone in a safe next week and getting a crappy Nokia and seek g how long I can go.
Adding to wishlist: rotten tomatoes and IMDb score of each movie clearly visible when browsing in addition to Netflix rating. Maybe with a link back to IMDb/rt/... It’s infuriating that I need to have laptop or phone with me when browsing movies to see how they are rated.
The IMDB scores are one of the biggest benefits of the Amazon UI. The ratings are actually meaningful unlike the totally useless Netflix ratings. Netflix ratings typically range from baffling to insulting.
We have decided to cancel a few streaming services and Netflix is the first to go. It literally gives me anxiety to open it as I feel like I have to flip through it so fast to avoid the previews.
My “smart” TV is really a dumb, laggy POS and that DUN DUN is completely unpredictable. Sometimes it happens, sometimes it is lagged, sometimes it doesn’t, if we have the sonos bar turned up, wooo buddy it will make your heart skip a beat.
> As a result I rarely open Netflix except when I already know what I want to watch
You'd be better off getting rid of Netflix altogether. The acceleration of addictiveness[1] caused by tech companies making their platforms ever more sticky is anti-human.
If there is a film/series you really need to watch, there's always Bit Torrent.
This behavior is not confined to Apple TV. Isn't it the same on every client? Roku and the web interface are both guilty as well.
It's so obnoxious that I would consider paying extra to 1) shut this off, and 2) have a sane implementation of some sort of helpful/powerful search filter.
i end up waving my mouse around like a spaz to find a place where something isn't auto-previewing. I like the idea of a mouse over preview, but it needs work
That decision alone makes me want to cancel Netflix. I’m close. I hate that feature. Zero reason they can’t let you toggle that off, they just have no respect for users.
At the very least, it shouldn’t auto-play the audio, only the video.
Netflix app on the Roku does the same. Quite annoying when you’re paused on the interface for a little bit and then all of a sudden something plays/roars at high volume.
People who don’t use it are probably more likely to churn relative to customers who watch content, and would be less likely to share Netflix with friends. And that expected loss due to churn is probably more expensive than the resource costs of a customer who watches. I don’t have the actual answer but they undoubtedly perform this kind of calculation.
On a Bay Area programmer salary Netflix is dirt cheap, but for many people 120$/year is enough money to consider canceling. Making Netflix a nessesity is a great way to make sure it's one of the last things cut from any budget.
Hulu is also cheap but you probably don't pay for it. And you are indeed the perfect customer as opposed to the typical one. For many people, 10$/month is substantial enough that they would cancel the subscription or go to a competitor if they felt like they weren't using it enough.
The article wasn't really worth the time. It tried to make a mountain out of a molehill: Netflix gave badges for watching children's programs as a test of a feature to make Netflix more appealing to watch. The author of the article Yohana Desta just loosely connected it to some narrative where "Netflix is evil and out to zombify your children" for the purpose of, ironically, getting people to read the article because that narrative is more appealing. Just another piece of trash online article designed to catch your eye. No substance, no value. Just throw it in the trash bin and move on.
> Adults sure, but what decent human sets out to get kids addicted???
Quite a few. Have a look at the free-to-play games in the app store of your phone/tablet; it's both edifying and horrifying. Psychological addiction has been refined into a high art and, unfortunately, has been quite successful at separating children and young adults from their money.
It is not at all surprising that interest in these techniques is spreading beyond the games industry.
Yeah, the biggest issue here is that no tech giant has burst because of these unsavory practices yet. It’s tough to say, don’t do this or that, when your competition will and the consumer won’t care.
> the company blatantly tells its investors that its competitors aren’t just other streaming platforms, but rather literally anything viewers do in their leisure time that is not watching Netflix—quite literally including “going out to dinner with friends or enjoying a glass of wine with their partner, just to name a few.”
Why optimize for screen time? They get payed through monthly subscriptions not per minute. Shouldn’t they optimize for user satisfaction, or lock in then?
Viewing things like going to dinner “the competition” is, in my opinion, immoral. This is Netflix serving their shareholders instead of their customers.
Article has a brain damaged writing style and takes two pages to tell us what the feature is. Apparently, you get a badge for watching a show’s entire series of episodes.
It's calling the writing style brain-damaged (making it a metaphor). Similarly, saying "your writing style is beautiful" and "you are beautiful" express very different meanings. Just as an ugly person can have a beautiful writing style, a healthy person can write something in a brain-damaged writing style.
"Everything" isn't offensive, no. But some things are, to some people, and sometimes for good reasons. We should try to be a bit more careful in our talking and thinking, I think.
I think, too, sometimes, but not too much, or else I get no sleep, but please don't hate my use of commas and run on sentences, I swear I'm making a point.
This sort of rhetorical question will not produce HN worthy content either. There's a down vote button if you disapprove a comment or at least state why you found the comment inappropriate. I'm not a native speaker and just thought it was a figure of speech.
Kudos to Netflix. I pay Netflix money every month so I can watch a wide selection of movies and tv episodes. Apart from the script writing itself, I don’t want any games played with my attention or my kids attention. This is the big thing that sets Netflix apart for me from other internet companies: I know that they are selling me a service and not selling my attention to others. As long as they keep doing that and have a good selection of movies and shows they will have my business.
>This is the big thing that sets Netflix apart for me from other internet companies
Oh please. By some miracle every Netflix-produced thing has a >90%-match with my preferences, yet somehow when I type in movies that I want to watch I find Netflix had them in their catalogue but never bothered to suggest them to me.
They want you to see self-produced stuff and then a show and not a movie (so you stay on longer) and not to find the best matches for your taste.
Also, giving them Kudos for not doing something that should have been a no-starter in the first place sets a pretty low bar for giving someone Kudos.
A) A company that makes money from you spending as much time on their website as possible
B) A website that makes a set amount of money per user because you voluntarily pay them every month
Company A is not inclined to optimize for user happiness, whereas company B absolutely is. If these two companies do not take different approaches, then one or both of them is doing something wrong.
Edit: And for the record, I do think that on the whole, Netflix is behaving far too much like Company A despite having Company B's business model.
Yes, I think there should be a difference between companies A and B and obviously Netflix has its upsides like no Netflix-unrelated) ads. However... your edit sums it up pretty nicely ;)
Is there anyway to do this properly? I can imagine the problem will get worse and worse as we reduce human evaluation of A/B test results and automatically make real-time decisions through software and ML.
[1] https://www.businessinsider.com/poll-shows-trump-appears-to-...
[2] https://www.engadget.com/2018/10/05/snap-evan-spiegel-app-re...