Friday, October 16, 2015

The Long Burning Hype of Hype Indicators

(motivated by From Turing to Watson: The Long-Burning Hype of Machine Learning)

Hype is bullshit. It is not true, but it is also not necessarily false. In fact it has only a tenuous connection to the true/false dichotomy/continuum. It is only exclamation.

It is the hot-or-not score. It is the Time magazine weekly up or down cultural indicator. It is based on empty anecdotal perception, vaguely perceived frequency of mention or frequency of thought or coolness or I don't know what.

It is barely a measure of anything other than the .

The Gartner Hype Curve is also hype. It attempts to inform about the hype stage of many closely related items at once. But it turns out that is a piece of hype itself. The hype cycle is a well-hyped pseudo-scientific (non-evidenced based) proof-by-look-there's-a-picture.

Here is the general pattern:




It is very compelling. I have to be honest and say that that's exactly the timeline of how I think of things. At first I've just never heard of the thing. Then one mention.Then three in one day, then I hear and think of it all the time, then I get just sick of it, nauseated at the thought. Then it comes back as an accept everyday thing. Here's an example of a set of items from the 

But... really? That's just a made up story. It seems to match what I think of as a story of popularity. It seems to match a good Hollywood drama: hero has early success and downfall and then third act of redemption.

And it is just the vaguist notion of mood swings. And also what is the point besides entertainment, or schadenfreude or rooting for a comeback? 

Look at the following. So sciency. Look at all the data points that you can follow year after year (the data viz is a bit hard to read the course of any particular item, but that's a minor quibble in comparison to the the central problems).

The hype curve might be a useful thing, if only it measured something that is 1) coherent and 2) based on evidence. Introspection is a great inspiration but it is not measurable. 

What is the meaning behind the graph? Also whatever the meaning what is the data underlying the graph?

The easy answer is the source of the data. It is simply the 'educated' guess of Gartner analysts. Not an actual number. 

Take any particular item. Does it follow the curve? Does it match the icon? Each label is a hype term, but has its own definitional problems. Each term can be vague, have multiple meanings, and have multiple incommensurate sources.

Note also that the shape and timing of the graph is the same for all items. The different point icons are the only appeal to different scales for each item.

So time, the x-axis, is incoherent (unmentioned context based for every item). Different thing might move along the supposed curve at different rates.

But what about the y-axis? Is it popularity, that is, how often an item is mentioned (mentioned in tweets or on google)? or is it how successful an 'item' is (let's say quantitatively, money, earnings per year?) Even these ostensibly measurable concepts are problematic because of definition of terms (is one label the same or different than another).

And once you nail down what the measurement should be, Gartner isn't doing any kind of such measurement, and there's no guarantee that the hype curve shape is a common pattern. It may be that an item's curve is up then down (then dead). Or it may have a steady rise. Or it may have multiple hype peaks at different scales. Or frankly it may have a curve like any stock, up down, steady, with most any pattern imaginable.

If you want to make the hype curve useful. Pick a meaning (or meanings, heck go wild and have many types of hype) and then actually measure it. And only then will people... wel they won't accept that unconditionally, they'll also complain about the coherence of the concept measured and measuring difficulties. But at least it will be scientific evidenced based hype rather than just empty celebrity bullshit hype.

No comments: