Opinion What happens when your data sample isn’t representative of the public? We already know, for 2015 is turning out to be the year of the dodgy opinion poll. And it isn’t just politicians who are being stung.
In Windows 10, Microsoft says it will hide running apps from the task bar, breaking a twenty-year-old convention which started with Windows 95 – potentially causing confusion for hundreds of millions of regular users.
It’s doing so because fanbois who signed up for Microsoft's public Windows preview programme say they marginally preferred the change.
Let’s leave aside for whether the change will cause confusion or not, and examine how and why the decision was made. Last week, Microsoft’s Gabe Aul wrote that the Windows 10 designers had listened to the self-selecting “Insiders” in making the choice, and treated the feedback from the sample as if it was scientifically valid. One option had a satisfaction rating of 4.2, the other 3.8.
"We also observed that users are 34 per cent more likely to be strongly satisfied with the filtered Taskbar and three times less likely to be strongly dissatisfied compared to the global task bar."
And that’s the problem right there. The self-selecting sample might not actually be representative of the views of the regular Joe or Joleen.
This year, two national elections – the UK and Poland – have sprung major “surprises” that defied all the opinion pollsters. In both cases, it turned out the people the pollsters sampled weren’t giving them the full picture.
In the UK, it seems, people withheld their true intentions from the pollsters, although other factors were in play too. Every expert predicted a hung Parliament and yet the Conservatives won a small but definitive majority.
In Poland, only two out of 22 of the most recent polls gave the eventual winner (Duda) a lead. The defeated president had led the polls since September. The pollsters oversampled metropolitan areas, understating the eventual winner's support among rural voters, and turnout was high. His victory “surprised the party itself” – but then they’d been listening to pollsters more than their own supporters.
In both countries, the poll sample did not reflect the views of the population. Too much weight was given to data that didn’t support the conclusions that were then drawn from it. Microsoft has exposed itself to the same peril with its Insider programme.
Opening the Windows Insider programme to anyone who wanted to join had merit as a bug-squashing exercise. But it was also more than that: it demonstrated Microsoft was in touch with its user base.
After the years of the Sinofsky rampage, which via Windows 8 forced drastic changes on to users, Microsoft felt it needed to be more of a listening company. Windows Insider was a “listening exercise”. It showed Microsoft now cared much more about the user experience; design could be democratic.
The specific problem here is quite straightforward. Normal people don’t sign up in large numbers to try out very rough alpha software, or at least not knowingly. This means only the most devoted fanbois and developers have been using early builds of Windows 10, which has been very much an Alpha quality release until last week, when a much more mature build arrived.
Clicktivism and Win 95
Windows 95 introduced a taskbar which showed all the running programs, and all were visible by default. If you couldn’t see an app (as opposed to a background service or daemon), then it wasn’t running. But when lots of apps were running, this resulted in very compressed and unreadable taskbar buttons.
Microsoft tried to clean this up in Vista and Windows 7 with button grouping and scrolling. Buttons for older running apps may require an extra click to be made visible – but they were still somewhere on the taskbar, just out of view.
However, Windows 10 adds a new dimension with the introduction of virtual workspaces or multiple desktops. Apps might be running but not visible - which is something entirely new in Windows (the Mac has allowed you to hide applications, but Windows has minimised them instead, leaving some kind of visible stub). So the designers have to solve the question of whether an app running on a different workspace should be displayed or not. Did the absence of a stub mean the program was closed, or just not visible on this portion of the work space? Over the beta program, it’s tried both. Aul says the Insiders have voted.
It sounds trivial, and I think in this case, most Windows 10 users will be actually able to cope just fine. Despite the break in convention, Microsoft will still show a popup list of running apps. And Microsoft has much bigger problems with Windows 10 desktop – it’s still a long way from prime time, yet the company is publicly committed to a summer release. This means OEMs hoping to sell those back-to-school laptops will be disappointed if it isn’t ready.
The problem isn’t the decision per se, but that it was taken on shaky evidence. Microsoft stressed to me recently that Insider Preview feedback is just one part of the input into a decision. It still runs qualitative UX feedback sessions, too. But Gabe Aul has just demonstrated what really counts. Unfortunately.
Microsoft is fun to write about again these days, because so many cast-iron assumptions that ruled its business have been thrown away. Microsoft used to agonise over the loss of licence fee revenue all over its business.
But with Ballmer gone, it doesn’t care any more. At the Build conference, Joe Belfiore plugged a Windows 10 phone (licence fee: $0.00) into a display with a keyboard, and carried on, the phone substituting for a PC. The Office software (license fee: $0.00) dynamically adapted to the new display. That would have been unthinkable under Gates or Ballmer. Under the old regime, whoever suggested it would have either been sacked or sent to Siberia.
Listening to a tiny self-selecting sample has its perils too. Design democracy is a great buzzword – IKEA uses it – but like real democracy, it only works if every person has one vote. Skewed samples aren’t democracy. Back when the Insider Preview for Windows 10 was announced, I joked with Microsoft that the release would include manga wallpaper. After all, that reflects the Insiders' tastes pretty well.
Don’t bet against it. ®