Can Deep Learning be applied to Automated Deduction?

Deep learning is just a deeply layered neural network, and by deep they mean more than one internal layer. It has recently gained much attention because of its successes with images and go and speech and all sorts of things plain old NNs weren't doing so well at.

But what about automated deduction? That's not rhetorical, this is entirely speculative and answer free. I am wondering if DL (or any ML technique) could be thrown at AD (or ATP (automated theorem Proving) whatever you'd like to call it).

First the optimism: wow wouldn't that be cool, a method that would prove really hard mathematical theorems (the ATP community has to work hard and do a lot by hand to do things like flyspeck) or even . But desiring the outcome doesn't say anything about the implementation.

Next the pessimism. By analogy, images and speech/text are iffy optimization problems. Lots of perturbations get you the same thing. But logic (and similar combinatorial problems) are all or nothing. It is either true/proven or false/wrong. how could an approximation optimization method translate to logical combinatorial problems? Where do you get the scads of supervised instances required by DL? Millions of tagged images from MNIST but from TPTP literally only thousands.

So, I don't know. But just because I don't know how to do it doesn't mean it can't be done.

A small nitpick about a small p-value problem trope

Lately the p-value has been getting a lot of press, almost entirely bad (tip of the iceberg). Whatever it is and means has been up for discussion as it hasn't had since NPHT was created (the Bayesians have been fighting against it, or rather fighting for alternatives, since the 60's).

Hidden in the middle of this storm, or a small tornado on the side, is the issue of data fishing or p-hacking. Since only a p-value of less than .05 is considered 'statistically significant, only such values are considered publishable leading to the problems of: selective publishing (ignoring significant non-results), and p-hacking (if one p-value isn't good enough, change you hypothesis and testing little by little until you get a p-value below the threshold). The problematic trope is stated in roughly the following manner:

At p-level threshold set at 5% (or .05 or 1/20), all you need is 20 studied hypotheses to get one hypothesis that is significant by chance.

It's so obvious!  With 1/20th probability, you need 20 tries to guarantee a hit! The intention is that you shouldn't make many hypothesis tests at a time, otherwise you'll get some false hypothesis stated as true.

But you may notice with that wording that it is a classic gambler's misinterpretation, 'the run has to end!'. Each hypothesis test is independent, and so the probability of the next test will not change if all the previous tests are all hits or all not or whatever.

Whatever you think of p-values, and whatever you think they mean, they are probabilities. Probabilities of what is complicated and nuanced and misleadingly stated and problematic and the firm basis for statistical inference for the past hundred years. But still, they are probabilities of something and a strict threshold of 5% of accepting a hypothesis over rej... forget that verbiage. it's a 1/20 probability event.


So now we're in the realm of basic probability (and its own difficulties) but they should be shared by Bayesians, Frequentists, Kolmogorov..ans, Keynesians (he had his own!). So any hypothesis has a probability of .05 of being positive, a hit. What's the probability of a hit in 1 trial? 5%. What's the probability of a hit (at least one hit) in 2 trials? 3? n trials? Those are harder but only a tiny bit, basic probability/combinatorics. What's the probability of at least one hit? One minus the probability of no hits at all. What's the probability of no hits in n trials? (probability of no hit in one trial)^n. They are independent events so you multiply. Final answer:


\[
P({\rm hit\ in\ n\ trials}) = 1-P({\rm no\ hit})^n = 1- 0.95^n
\]

Well, not final exactly, it's just a formula. We don't yet have a good picture of what it means in realation to our intuition about 'it'll take 20 to make sure we have a hit'.

So then a picture:



It starts at 0, rises in exponential decay asymptotically to 1, but is a little slower than you'd expect for an exponential because the base is so close to 1.


The usual way to present such probabilities is, like the birthday paradox, to say how many trials it takes to get %50 chance. Multiplying .95 a few times we see that it takes 14 trials for there to be more than 50/50 chance that at least one item is a hit. To get to 95% chance, it takes 59 trials. There's no guarantee that there'll be a hit, just the probability gets smaller and smaller.


I make this point to... well, to pick a nit. You do more experiments, the more likely there will be one that is 'statistically significant' totally by chance. Intuition and logic lead to that immediately. But the logic is never done and one step of it doesn't lead to correct two steps. I do realize it is a bit of a mouthful and difficult to digest to say 'after 14 trials there will be a fifty-fifty for a false positive'? 50/50? It's not obvious how that relates to 5%, but the erroneous 20*.05 = 100% does obviously relate.


In the end, to say '5% means 20 experiments', which seems so directly and intuitively obvious, is wrong. In the right direction, but wrong.

A serious proposal about Daylight Saving Time

People everywhere hate the spring change in clocks for Daylight Saving Time.

First, it's annoyingly tiring having to get up one hour earlier. And feeling like you can stay up a whole hour later. And this leading to wanting to stay in bed that much longer even more. Annoying.

Second, the average effect of this annoyance on the population. This leads to, statistically, 'studies have been done', of reduced physical performance leading to accidents (lower light levels from sun than in the previous week), and an increase in heart attacks and strokes (not exactly excuses to lie in bed a bit longer).

DST changes things twice a year. In a preindustrial society, without fast travel and communication, time zones are not useful. DST is useful for... for what I'm not sure. More daylight in the evening? That's not for work, that's for ... I don't know, kids to stay outside and play longer?

But if that is desired, but also we'd like the feature of no changing of wakeup time, I have a radical new idea, which though it sounds strange, is not entirely a crackpot idea. My proposal:

Have clock time pegged to sunrise.

And call that time 6am. No matter where you are on Earth.

First there is the practical consideration, then there is the implementation. Practically, it fulfills the feature of never changing. You always get to wakeup at the same time. Nowadays we all choose to go to bed pretty much irrelevant to sunset anyways. And the farmers don't have their delivery schedules screwed up when the milking cows don't care about these crazy human practices.

Implementation may be the most difficult, and here is where some of the subtleties come out. However, this is the 21st century. The point is that sunrise is equivalent to longitude, and this is determinable now by any variety of methods but also by pure calculation (whose coefficients were determined by those methods or in the end skywatching).

We all have at our disposal computation devices that are minute in bulk (phones, watches, RFID chips) with GPS positioning (via satellites), that can determine longitude within a few meters (2?). So sunrise will be a few seconds off from someone 10 miles west (15 degrees = 1 hour sunrise difference, 1 degree ~= 60 miles ... 10 miles ~= 24 seconds). This continuous difference sounds like it is insurmountable... except by computation, which is currently trivially executable. You're having a phone meeting with someone at 9:30am, you're in downtown Chicago and they're in Kansas City, MO (~415 miles west). That's around 7 degrees or 28 minutes. If the appointment is at 9:30AM Chicago Loop time, in Kansas City it is ~9AM. The appointment making software would calculate the time for the recipient based on their address. For coordination there can still be a UTC which is the time at one particular longitude.

There are some very minor difficulties that are barely in the realm of practical considerations. Depending on your latitude, noon (or six hours after sunrise) may not correspond to the highest point of the sun in the sky, but that's the case now anyway (just not as much). Also for higher latitudes, sunset will change drastically in the spring in fall (sunset coming later one day to the next by up to 10 minutes, making the feel of the evening change within the week.

There's cultural precedence for this. Many ancient calendars count the change of day (but not when to rise or go to bed) to sunset, so why not sunrise. Our mental perception of the 'next day starting', despite the European style (derived from Roman) changeover at the sun point opposite to sun's zenith (yes, I'm avoiding 'midday' and 'noon' for the moment),  is truly when we wake up around dawn or more naturally at dawn.

So that's it. Free idea. Go forth and implement. Just give me credit. If there are problems, I'd like to blame the future implementation.

Of course, this doesn't fix the problems of Jews and Muslims north of the Arctic Circle with practices on cooking or fasting concerning sundown.

Review: Ready Player One

"Ready Player One" by Ernest Cline, is a sci-fi novel about a kid, living in an impoverished near future, escapes like most of the population into a super-powered virtual world invented by a Steve Jobs style nerd. The plot of the novel is following this kid trying to find an 'easter egg' planted by this eccentric inventor bajillionaire, all by playing different video games or by using pop cultural references from nerd culture () most of these from the 70's and 80's (the dawn of blockbuster movies, and PCs and video games (arcade games, PC games, and pre-Mario Bros game controllers).

I don't know. The nerd-pop-cultural references were all there. Star Wars, Star Trek, Blade Runner, Monty Python, War Games. Joust, Defender, Dig Dug, D&D, all well referenced, lines of dialog. All well suited to my personal memory of teen years. Lots of excitement and virtual explosions and real explosions and plot twists and I'm sure hidden easter eggs in the novel itself.

But I'm an old dude. I don't see how kids these days (2011) would appreciate the references.

The tone is very much 'young adult novel'. It should be advertised as such, because it's not Philip K Dick if you're expecting that. The concerns are all of a cliché 18 year old (does that girl like me? I will obviously win this game having played it literally hundreds of times or literally never! I hate my welfare aunt who I live with since my parents died.)

The writing is like Asimov. I don't mean that as a compliment. Despite its obvious attempts at being modern and PC, they all come across as clunky white-male-teenager-privileged like a nice mormon dad coming to terms with a child caught drinking coffee. The main character's best friend on-line, a dude like himself but cooler, turns out in real life to, wait for it, it's shocking, are you sure you won't be shocked, a shy black overweight lesbian. Oh and the girl I like in the virtual world whose avatar is totally hot, in real life, she's also hot except she has a port wine stain on her face. The horror. The shame. The forced lesson of understanding of others without being other.

And the plot should be considered simply a string of the most unmotivated deus ex machina's and Mary Sue. And this is two levels deep, both in the engineered MMORG world and in the simulated games within that world. Oh did I mention this one hidden rule that you get 40 lives if you die the right may in this video game? Convenient that the best friend of the inventor comes to help out the gang at the last minute. Oh no, the opponents destroyed absolutely everything (virtual) with a  pixel bomb (sorry, that's not in the book but it's the shortest way to give the same idea), but the one tiny thing needed to complete this one magic task left happens to have survived? I'll win a bajillion dollars, I think I'll share it with my friends even though I won the game at the end, I'm a great guy!

The book is a lot of fun (moreso if you recognize the references). But it's mostly processed sugar and wish-fulfillment. Should be a fun movie, which should be able to avoid most of these clunkers.

An annoying argument/information trope

You know what really bugs me? When you're reading an article and it says:

"Last year, 350 bajillion Americans stabbed themselves in the face with a fork"

Yes, that's tragic. But the additional thing that annoys me is that nowhere in the article does it say how many people are using forks when they don't stab themselves in the face or at all, how often did people do it last year or in the world, or what the data source is, or if this was a meal accident vs kitchen clean up accidents, or what about people stabbing others like that, or if this is counting repeat offenders or stabby motions or what.

But the biggest annoyance of all is the bigness of the number. How big is that number really? Why that number (which usually sounds like it is quite a bit more than the number of forks or people with the capacity to do such a thing). It's like magical thinking, saying a big exact number invokes some mystical 'wow'. It's a very specific thing for a very fuzzy idea. Self stabbing bad. A lot bad. Wow, lots of bad things is bad. Wow.

Oh, so you want me to be specific? OK, specifically any article anywhere that mentions a number. Mostly science popularization articles.

Sure, the main problem is the base rate fallacy, but it's so much further down the scale than that. It's the 'Wow, a number' fallacy.

Chomsky vs Google, Structuralism vs Behaviorism

It's 2015. Google has been in business almost 20 years (since 1998). Their primary tech model for web page search is (still) to split all findable web pages by words, make a word/concept vector for each page using the entire corpus of words and rank a word search using links to other web pages (the PageRank algorithm). It is primarily a data driven process rather than an expert driven process. Of course there are lots of add ons, hand-tweaking, special cases, but at the center is an automated process of using what words people have used and what they've linked to.

It's 2015. Almost 60 years ago, Chomsky revolutionized linguistics (assuredly pursuing already existing trends in structuralism). Though the revolution started in substance with 'Syntactic Structures', he also put in a a mortal wound to (already dying) behaviorism with his critique of Skinner's Verbal Behavior. Skinner's thesis was that people learned language by example, by mimicking the thousands of utterances, assimilating the patterns heard. Chomsky's critique was that, in so many ways, this was wrong. The variety of known languages had a narrow set of commonalities not explainable by the broad possibilities allowed by reaction conditioning (fixed action patterns, operant conditioning, stimulus response). People seemed to learn from negative information. Chomsky's criticisms were so incisive and convincing

So behaviorism went out of favor, and Chomsky's linguistics is still like Newtonian mechanics; even if there's a linguistic Einstein, it will only be a slight correction to the very accurate approximation that is Chomskian language theory.

But Google. It is so obviously successful (sure as a company, but their search capability). More relevantly, Google Translate is essentially an ngram analysis. It is really good! Surely it makes many groaning mistakes on uncommon languages, but as time goes by it is more and more successful. It is a poster child for behaviorism. It is behaviorism embodied in a machine. And it works so well.

The point is that Chomsky won 60 years ago with, not exactly rationalism but more like introspection (which had lost earlier). And now Google is winning with pure un-introspected descriptivism.

But why? Why is statistical NLP so (currently) successful (when based on single words or short sequences of words, not constituent phrases) and syntactical NLP (POS, parse trees) is, well, not exactly wrong, just not terribly useful?

Maybe it's machine performance and having lots of data.
Maybe the words themselves have a lot of meaning and just being in the same sentence is enough.
Maybe understanding parse trees is very informative but only under toy conditions; the bag of words has the bulk of the information.

Google is killing Chomksy. It's like saying that Popper killed Logical Positivism. He did, sort of, but along with others, and anyway it's not really dead, just a basis for what comes afterwards.

Metaphor is a Metaphor. A Leaky Abstraction is a Leaky Abstraction

Bear with me.

Metaphor is a metaphor. A metaphor is a transfer is a carrying over (meta- (GR) = trans- (L) = over, across, -phor (G) = -fer (L) = carry, bear). A metaphor is a transfer of meaning from one literal sense to a figurative one where, hopefully, the transfer to the new domain keeps things one-to-one.

Imperfect metaphors are called leaky abstractions (at least in software). And 'leaky abstraction' is a leaky abstraction. They leak because the literal source of the abstraction leaks through to the abstraction layer. An abstraction or a metaphor is metaphorically used as a container of fluid some of which leaks out (but onto what? which is the source and which is the target (in another metaphor of metaphor)). You can't take the abstraction layer literally, you have to know something about the underlying source.

Also, 'figurative' is figurative or metaphorical; I often take metaphor as a synecdoche (or metonymy; synecdoche is a metonymy of metonymy) of figure of speech, in that it, figure of speech, is not literal. A figure is a picture, which is a good metaphor for metaphors... or figures of speech. 'Literal', on the other hand, happens to be somewhat non-literal because it is about writing, which is a metaphor for verbatim, that is the primary surface definition. 'Literally' has been used non-literally (that is as a general intensifier) literally for ages, but universally (that's hyperbole which is a figure of speech which is just a lie that we all agree to and not metonymy) recognized as wrong.

Borges said (in This Craft of Verse) that Lugones said (in Lunario sentimental) that "all words are dead metaphors", which is a dead metaphor because nothing has really died. Or rather the original meaning died or faded away very slowly, but you could resurrect it a little if you tried. So it's a little leaky.

We're swimming in metaphors!

Also 'Metaphors We Live By', by Lakoff and Johnson

Technical Debt is a Leaky Abstraction, but so what

Technical debt is a recent term in software engineering used to describe potential later problems that may be caused by known decisions now. For example, suppose a customer asks for a feature that can be implemented cheaply and quickly, but it will introduce security holes or be very difficult to generalize if done quickly, or will prevent another rare feature from working without lots of rework. You're making things work now for some pain later. You would have much less total pain overall if you 'do it right' today, but cheaply and quickly are more important.

(from collab.net)

Saying 'we have a lot of technical debt in our software' is very loose talk for 'we have either a lot of bugs that no one is complaining about' or 'we don't do code review so I bet there's a lot of crap that gets deployed'.

A leaky abstraction is another metaphor in software engineering. When one creates a new layer intended to hide all the details of a lower layer, For example, floating point numbers are leaky because they try to hide all the gross details of finite bits per number approximating perfect precision of the desired  numbers, but sometimes you end up having to be aware of the underlying implementation because they leak through the abstraction (eg a+b usually equals b+a except when b is much smaller than a, and to deal with that you need to know some details of how the floating point ops really work under the hood).

Technical debt is a leaky abstraction in that the metaphor cannot be taken too literally; if you try to follow implications of the words, it breaks. Debt is a balance in a ledger, you can owe or be owed a value. Technical debt, being very qualitative, is hard to put into numbers, you just have a vague sense of 'this is bad' vs 'this is really bad' vs 'this is tolerable'. The debt isn't really about the features themselves but about the time and mental effort needed to implement things.

The first picture is a terrible explanation, the following is much better:

(from commadot.com)

All I'm saying is that technical debt is a leaky abstraction, a faulty metaphor. There's no paying back, it's just cleanup.

Using the term is great because it is more politic than saying 'The code is a mess and needs some cleanup. Features are really hard to add'.

Names make theories make names

Richard Feynman, the Nobel Prize winning physicist and Challenger disaster O-ring explainer, has a couple of anecdotes about naming. 

One is about the 'map of the cat', where Feynman had to give a talk to his graduate zoology class. In preparation, he went to the library and asked for a map of the cat, to which the librarian responded "You mean a zoological chart!"  (I'm guessing 'what a funny thing to say') Then later when he presents this, his fellow bio students say "We know all that!". Feynman says that they "had wasted all their time memorizing stuff like that, when it could be looked up in fifteen minutes."

The other story is much earlier in his life. All the (nerdy) kids on the playground try to one up each other on what their dad's taught them. "What do you call that bird? What about that bird?". But Feynman's dad said  something like it's called X in language Y, Z in language W.

You can know the name of that bird in all the languages of the world, but when you’re finished, you’ll know absolutely nothing whatever about the bird. You’ll only know about humans in different places, and what they call the bird. So let’s look at the bird and see what it’s doing—that’s what counts.” (I learned very early the difference between knowing the name of something and knowing something.)

What is the point about these parables? He goes on to explain that names don't explain anything. He explains that these technical terms for anatomy or for bird varieties or for whatever science are just terms, they're not explanation themselves. There is a tendency even for these technical terms to be magical invocations, the totems of a closed guild, supplications to the gods, when these terms are barely paratactic gestures of pointing, a superficial 'behold', an inarticulate label with no explanation of the depth of the experience. "What kind of bird is that?" "It's a sparrow" "Why does it do that?" "It's a sparrow." as though emphasis is explanation enough.

Except... what do you expect? Is naming so terrible? Do you want to do away with naming and simple move on to the much more interesting explanation? Is naming so simpleminded?

Where did the names come from? Without this being an explanation of historical linguistics, presuming terms like gastrocnemus, sparrow, and inertia are somewhat random and distinct, these labels are hooks for the concepts. A lot of thought, using other labels or conceptual manipulation, led one to label this object or concept one thing, that another. If our thoughts are not necessarily tied to language, communicating them certainly is (though a gesture or picture can go a long way, like a shrinking ring in ice water). 

Names and words are little theories in themselves. We learn most of them superficially, but eventually we acquire their nuances. A new word, like inertia, is opaque to the newcomer, but at some point in time, the scientist or experimenter was playing with a number of concepts, and eventually some concepts coalesced out of that thinking and one of them was given the label 'inertia'.

Yes, knowing a name doesn't explain anything. Or rather, it explains very little. Knowing how to use a name is nontrivial, but has little explanation to it. Answering why requires being able to manipulate a number of names, but having those names is necessary. 'Black hole' is the culmination of a lot of thought. That it is a thing is the consequence of lots of thinking. And the start of a lot of thinking. Once you get to that concept, a lot of thought has gone on and not having that term would be a great loss, leaving us to swim around a number of concepts of relativity but don't quite say what they really mean. Feynman is saying that it is dumb to stop at the names of things. Sure, don't stop. But don't let that stop you from naming things because it'll be that much harder to continue without the name.

A name is itself the end product of a theory, and it makes further theories possible. 

Prescriptivism vs Descriptivism: which is worse?

Prescriptivism vs Descriptivism: which is worse?

These two words are used to describe one's attitude towards language usage; at its very simplest do you prescribe or describe how you speak, what are people supposed to do vs what people actually do, 'should' vs 'is'. When these terms are thrown around (and I do mean thrown, like mud pies) it's almost always meant to sting.

Objectively, prescriptivism is usually understood to mean keeping strictly to the formal rules of a language and descriptivism is more about discovering and recording the rules of language however people say things (whether they match the formal version or not). Newspaper editors and school teachers are often the supposed standards of prescriptivism and linguists as that of descriptivism. Your secondary teachers are teaching you the rules of good grammar, and the linguists are being scientific about what the rules actually are.

M-W's third edition dictionary (1961) is often cited as a classic of the descriptivist abyss, putting in words of dubious provenance, all the good profanity (which at least got some kids to crack it open at least once). The dictionary was decried by many as the nadir of pandering to idiocy, the last gasps in the decline of western civilization.

Informally, from the prescriptive point of view, descriptivists are 'anything goes'; whatever people say is what is allowed, nothing and no one is wrong, there are no mistakes, 'ain't' and 'between you and I' are now OK and that is just wrong and descriptivists are avatars of the decline nay the destruction of western civilization, regression towards the mean, the twilight of the idols, the idiotocracy, the worst are full of passionate intensity.

From the descriptive point of view, prescriptivists are stuck up old school marms, who make up arbitrary style rules, say 'you can't do this-you can't do that', split infinitives, prepositions at the end of a word, singular they, when people have been saying it that way forever and you just made that rule up because you are warped, frustrated old man. P's try to enforce their made up rules, when it's just their own repetition of some one else's peevish peeving on personal style. Descriptivists say that prescriptivists single way of speaking is reprehensible elitism and that they think they're are morally superior to others, and any other patterns are slack jawed, uneducated, lower class.

But that's just the tendentious version.

From the prescriptive point of view, there really are mistakes that people make, infer for imply is just wrong, 'literally' for not literal things is just wrong and native speakers just do not say it that way. A common error is not necessarily a common alternative.

From the descriptive point of view, there are many patterns out there for the same thing. Different contexts have different rules. People will say different things in different situations, one way speaking at the press conference and another in the bar, and neither is wrong (or the differences show up in different contexts).

People may very well avoid a split infinitive and prepositions at the end of a sentence in writing for stylistic or esthetic reasons, but in speech there's hardly getting around it what with all the phrasal verbs in English. The double negative ain't no (= "isn't a") problem as a perfectly everyday way of speaking for some varieties of English (that are, as the linguists say, not highly socially respectable (= redneck or AAVE)) or grammatically and logically appropriate either, two negatives making a positive or a form of understatement (that was not an uncomplicated sentence).

And this is where it comes down to the real difference.

Descriptivists are really prescriptivists at heart. Descriptivists just recognize many more varieties than prescriptivists, and those varieties tend to be much more informal or used by socially non-pinnacle subpopulations. People who are called by others prescriptivists do seem a little judgy, and people who are called descriptivists do seem a little too accepting of things that are  (i.e. errors). But if you just label large groups of patterns as varieties, Prescriptivists are just talking about (mostly) a single variety, the newspaper/college paper variety, and descriptivists allow for a wider range of varieties, informal or regional or inarticulate (well, maybe not the last one). There are still mistakes. It just depends on the context or variety you're in.

Bayes Theorem != Bayesianism

Words are slippery. They have many meanings. Change an ending and you change everything. Not exactly everything, but enough to confuse everybody.

Bayes theorem is not the same as Bayesianism.

Bayes theorem is an elementary mathematical truth of elementary probability.

Bayesianism is a big trend in statistics for creating and interpreting new statistical tests.

So it is fairly obvious now that they are different. They are certainly related, but they are still different things altogether.

Some details are in order.

Thomas Bayes was a man who the famous Bayes Theorem (of elementary probability) was named after.

Also, the direction of Bayesian statistics was similarly named after him. Because of how the theorem thinks of probabilities.

Sure, all three, the man, the theorem, and the statistics philosophy, are related. But not as closely as you would imagine.

First, Bayes was an eighteenth Presbyterian minister in Kent, England. I'm sure he was quite important to the people around him, but the theorem named after him, was never published by Bayes himself. The theorem was stated de facto by someone else in passing, and also in passing mentioned that Bayes had discovered it. This someone else was Richard Price who was also probably important to the people around him but never had anything like a theorem named after him.

Now to the theorem.

Symbolically, Bayes theorem is, at its simplest,

$$
P(A|B) = \frac{P(B|A) P(A) }{P(B)}
$$

Simple enough, if you know elementary probability. Translated to English this means that the probability of an event $A$ having occurred, given that you know $B$ has occurred, can be computed from the probability of $A$ divided by the probability of $B$ times the probability of $B$ given $A$.

Big deal right? So what, right? The clever twist to this is that it is allows you to reverse the direction of causality, to use the past history of A's and B's together to determine the probability of something you don't know about $A$ from something you do know (the other three things on the right hand side). The calculation is elementary, you just count all your events (where $A$, $B$, both $A$ and $B$, and neither have occurred). $P(x)$ is the fraction that is the number of events where $x$ happened divided by the total number of events. $P(x|y)$ is the (note all fractions are between 0 and 1).

The proof is also elementary. From the explanation, we can see that $P(x|y)$ is the fraction the number of events where both $x$ and $y$ occur divided by all the $y$ events (whether $x$ occurred or not). That is

$$
P(x|y) = \frac{P(x) {\rm \ and\ } P(y)}{P(y)}
$$

Since $x and y$ is no different than $y and x$ this means that

$$
P(A|B) P(B) = P(A) {\rm \ and\ } P(B) = P(B|A)P(A)
$$

Divide the two ends by $P(B)$ and you're done.

This is a very short set of inferences, almost purely arithmetic. Thinking of the Venn diagram, a division is really the proportion of the subset in another larger set. A little more complicated, but giving even more insight, is to note that if you look at a 2x2 contingency table, A given B is the proportion in a row, and B given A is in a column, the theorem allows you to move between row and column.

To use it, tabulate a number of events where both $A$ and $B$ may have occurred. This allows you to calculate

It might seem at this point that the theorem is a lot of extra thought work when, if you have the contingency table already, just compute anything you want right there $A$ given $B$, $B$ given $A$, whatever. The idea of using the theorem is that often you are not presented with the contingency table, but have some good idea of the different values. The theorem allows you to update $A$ given $B$, if $B$ given $A$ is somehow magically known to you already. Actually often you don't know $B$ either.

To make this more concrete (a common example), let $A$ be 'patient has toenail cancer', $B$ 'new test on toenail is positive for cancer'. And as you may suspect, tests are not perfect. Sometimes they raise a false alarm, they are positive when there really is no cancer, and sometimes they miss the cancer, it is negative when there really is cancer.  And usually there is a huge cost in discovering if the patient really truly has cancer (like the patient dies and you do an autopsy, or you do a biopsy and discover the cancer too late to do anything). So you totally know $P(B)$, when your tests are positive, at least for your lab because you just count. You may have a good idea of $P(A)$ because of national statistics, but your much smaller set of patients may be unusually prone to the disease or unusually healthy. And you may have a good idea of $P(B|A)$ because you know of your patients already who has cancer and who of them had a positive test. And what you want to know is the probability that this one new patient really has cancer, given that the new test turned out positive.

So this says for a given positive test, you can calculate the probability of cancer (there are lots of class examples showing how this is not trivial, that lots of tests are made, to ensure that you don't miss a cancer, the test is very sensitive and may include a lot of false positives. Then if your test is positive, there is still a low chance of having cancer just more than if the test was negative.

But the point is you don't always have the full contingency table at hand. That's what makes Bayes theorem so useful.

To give some perspective, Bayes theorem is very useful theorem but it is very simple. It is one of the simplest theorems ever. It's mostly just a convenience of calculation. It is simple on the order of the theorem that multiplication distributes over addition. It is almost trivial in proof, and its application is mostly just a simplification of calculation. Calling it a theorem is on the order of calling Monaco a country: it is certainly a country but in practice is  more like a small but popular and overpopulated district in an overpopulated area of a much larger popular country.

Now to Bayesian Statistics. Statistics in general as a discipline is only a couple centuries old (and that is stretching it). Its mathematical foundations with axiomatic probability and statistical distributions based on the central limit theorem and the normal curve and special functions and practice centered on p-values really started taking off in the early 1900's with Pearson, Fisher and others. This manner of doing statistics was eventually named 'frequentist' statistics in distinction to the newer trend called Bayesian statistics. The trendy new field was called that, not because frequentists did not use or believe in Bayes theorem, but because in its manipulation of distributions relied on computing new distributions based on prior ones, analogous to how Bayes theorem computes new probabilities.

The point is that Bayesian statistics is not some super exploded hypergeneralization of Bayes theorem but rather a large set of mathematical machinery that allows one to compute in many different ways how well different hypotheses are supported by data. Instead of the frequentist procedures t-test and ANOVA, that have very strict assumptions on the distributions of the data, these new procedures allow a parameter of the hypothesis of the existing distribution which then gets updated by the procedure (or if you don't have an idea of the distribution, you can always assume the uniform prior).

You'll note I can say quite a bit more about Bayesianism, but I have spent more about Bayes theorem. Volumes could be written (and have) about Bayesianism and the religious wars between Bayesianism and Frequentism (one could have a meta-religious war whether that war is religious or substantive). There are extremely specific things about Bayes theorem I can write, but about Bayesianism it is easy to stop early without going into lots of complicated math.

So Bayes theorem does not mean the same thing as Bayesianism. Bayes theorem is a tiny almost trivial calculation in elementary probability, with a lot of uses. There is no controversy about the theorem. Bayesianism (or Bayesian statistics) is a forceful trend in statistical practice with a large set of alternative theoretical and practical procedures. It is a controversial trend among statisticians, or rather it was controversial in the 70's snd 80's and is mostly mainstream among them now right along side traditional frequentism.

So don't let the similarity of the names lead you to think they are the same. They are similar and related, but not that much.

• If you are thinking of a calculation of a probability of $A$ given $B$ using the probability of $B$ given $A$ (a very very specific usage), then you're using Bayes theorem.
• If you're thinking of a trend in statistics that avoids distribution assumptions, or rather allows you to specify your arbitrary distribution assumption, then you're talking about Bayesianism.

In short, Bayes was a guy who is famous for a theorem he may not even have stated, much less proven, Bayes Theorem is an elementary probability calculation, and Bayesianism is a broad trend in statistical practice.

In defense of astrology

Astrology is the study of personality and personal events with respect to celestial objects and ones birthdate and location. It is often used in prediction of events of a person's life and their personality. In the European tradition, astrology is mostly based on the position of the sun in the sky, which constellation of the zodiac it is in, on a person's birthdate.

Most people are aware of astrology through the idea of a daily horoscope (often printed in newspapers on the comics page next to the word jumble or chess column) (these days with the increasing marginalization of newspapers, I don't know where people experience it).

Many people view it, and dismiss it, as entertainment, on par with palm reading or Tarot cards, not something to be taken literally. Those with a scientific or practical bent might dismiss it outright as an empty myth or BS because it is, to use the jargon, not falsifiable. There are a some who still believe that it may work.

But the general consensus is that it is BS. All I want to do is pedantically (but not to contradict this consensus which I share) show that Astrology is indeed scientific in the technical sense, just one that is not supported by the data. A science that has been falsified is still scientific, just disproven.

There are two common applications of astrology. One, based on your birth sign one out of twelve, to predict, is very vaguely the events of the day. For example (taken from a random horoscope site; it is very exemplary of the kinds of writing):

Today you urgently need to understand and master the art of balancing the physical reality with your vision. While your plans are ambitious, you have to understand the actual obstacles top these plans. Otherwise, you are headed for a collision course in spite of all your good intentions. You also need to understand that your plans may be conflicting with those of someone else who is as determined and ambitious as you.

(source (no link to the specific text for a day, just generic for the sign))

These tend to be the greatest source of disbelief: how can everybody having the same birth sign have all those things happen? These read like fortune cookies that could be true of anyone whatever their birth sign. Accidentally read the fortune of another sign and that is just as likely to happen or not. Astrological predictions like the above are a great example of confirmation bias in action; all you need to happen is for one thing to match what happened that day (if you read the horoscope after the day), or you just may make one of things happen, and you'll come to 'believe' that horoscopes work.

The other common application is a prediction of your distinct personality. A natal chart, the position of major celestial bodies with respect to the constellations and each other at the the time and place of birth, is used to predict personality. The position of the sun is the most important, telling you your 'sign', but the moon and planets and their relations are important. This facts lead to statements of the form like 'The moon is on the cusp of Libra and Virgo, and Mars is in opposition to Venus but square with Saturn meaning that...' followed by some explanation involving the personalities of those celestial bodies, inspired by their mythological stories.

(image from wikipedia)

My defense is that all these things make astrology not simply a myth but an actual scientific theory, scientific in the sense that it could be the case, and that we can test whether predictions by it or its parts hold or not and there is a postulated mechanism for these effects that holds up. Of course, none of it holds at all. My defense is simply that astrology is legitimately falsifiable.

Counter to most opinions, I consider astrology to be falsifiable because one can make a study of daily horoscopes or a study of star charts and individual's life events or personalities. There is very scientific measurement of celestial body positions. There can be predictive personality tests (but many currently personality tests are currently not considered particularly good at prediction, for example Meyers-Briggs personality test (MBTI), the somewhat similar Big Five test is supposedly somewhat more predictive).

St. Augustine's dismissal of astrology because twins don't have identical outcomes is motivating to me but scientifically could be problematic. Twins do share quite bit of personality. This thought experiment this could be modified only the slightest to be more supportable; compare the lives of two children born the same day in the same hospital. The general predictions of 'astrological theory' can barely be supported. Some lifestyle and class similarities will predict most of the similarity between the two children.

The horoscopes in newspapers or your astrological sign ("You must be a Libra!") are, I agree, not scientific because they are so vague as to be barely even functional much less predictive (meaning that the pronouncements they make can barely be mapped coherently to life events or personality.

But an astrology based on actual measurement could be falsified and therefore is a scientific theory. Its effects just have never been established.

By the way, the Middle-eastern-based astronomical astrology (the greco-roman constellations-sun-moon-planets and mythically associated personalities) may have exact predictive power but there's one huge glaring plain old mistake in execution. The constellation the sun is in, one's sun-sign, as usually given today, is mismatched with the prescribed dates.  The twelve ranges of dates are set on the sun based along the ecliptic and the positions of the constellations. The constellation labels were set 2000 years ago. Because of the (astronomically measured) precession of the equinoxes, the position of the sun during the year has shifted a little more than the space of one constellation.

(source: wikipedia)
That is, on March 21st (the spring equinox, that start of the zodiacal year), when tradition states that the sign of Aries begins, in the year 2000 the sun is actually as measured in the sky against the backdrop of the constellations, already part way through the next constellation of Taurus.

So this is not an error of interpretation or logic, it is an even more elementary error of data recording/reporting.

'Thing Explainer' review

'Thing Explainer" is a picture book by the engineer-cartooner Randall Monroe (who does xkcd). This has so much stuffed into it, not just information but also manner. It is two major things that are at odds, so I don't which one to present first. So I'll just do it: it's a book that 1) uses only the first 1000 most frequent words in English (so it has to use some cleverness for technical concepts) and 2) presents technical concepts.

- How it's made and How it works. If you like 'Cathedrals', Lego, science, whatever, this is 'How it's made' but for more sciency things.
- At the same time, it is Oulipo-style constrained writing. The author took an arbitrary restriction (not exactly arbitrary) and, for all the labels of the hugely entertaining drawings of scientific concepts and engineering artifacts, made them entirely using the 1000 most common words in English.

The note at the end of the book states what his Oulipo rule really is (and gives the words he allows himself:

Note: in this set I count different word forms as one word
I could have said ship but I stuck to boat because space boat makes me laugh.

And he doesn't feel like he has to use every word in the list (this isn't serial music); he doesn't use a couple extremely common but profane words. The funny thing is is that he doesn't force himself to absolutely strictly adhere to the rules.

But the point he intended comes across. He wanted to explain abstruse scientific comments in a manner understandable by those early in the language (and thought) learning process. Language is not thought, so those who understand the concepts already can find the technical terms that match up what the author translated into slightly more verbose 'simple' sentences and similarly, the accessible vocabulary is used to get across the same concepts seemingly necessarily captured in abstruse technical vocabulary.

There's another side to using 'basic' English. The bad side of this is that it is deliberately dumbing down science. Simplify the language removes true things. It is intentionally anti-intellectual.

Or rather, it is nominally anti-intellectual, it could be anti-intellectual, but it is not. It is not the diametrical opposite of anti-intellectual, which I take to be obscurantism, because the whole point of the exercise is to be intellectual, present knowledge, and it is intentionally not obscurantist, because it is intentionally trying to present knowledge in a digestible form.

One exercise is to attempt to translate back to technical language. What is the scientific terminology that was the source of the circumlocution. What exactly is a 'fear water'? Oh, it's the adrenal gland, or at least adrenaline. When you translate technical terms, which are mostly stipulated and have their single technical definition, into multiple non-technical terms, there is the likelihood that the new term will be using high frequency words commonly used for lots of things already. So the new term requires some more frequent use to make sure it applies to that original singular technical concept. For example, 'power box' is used for electric battery but surely could be used for other boxes associated with power. But it is easy to see that with repetition, we could get used to just using 'power box' for just those items we currently call electric batteries.

Highlights:
- Includes descriptions of: a nuclear bomb, the ISS, the periodic table, a smartphone, the LHC, a jet engine, cell anatomy. Look, I could just list every page. It's both science and technology.
- 'thousand' isn't in the top 1000, so discussing itself is already constrained (it's 'ten hundred').
- 'Nine' isn't in the top 1000, so it is given as 'one more than eight', 'almost ten'.
- 'Gold' is the only element name that remains on the periodic table.
- the tree of life. Most animal names are fairly uncommon nowadays, now that few of us live on farms or the wilderness. So lots of liberties taken here. But a lot of the descriptives work. For example, 'gray tree jumper' for squirrel, 'sea dog' for seal, 'pocket babies' for marsupials, 'sweet thing' for fruit are all excellent. But there are some clunkers like 'food often in cans' (I have no idea), 'small dog' (is that a fox?). Here, with so many examples, it's nice to see how one can be paratactic and perceptual and that's often what one finds in other languages (because it is so hard to see the many times it already is the case in your own). Nice inclusion of 'water bear' for water bear/tardigrade/moss piglet/slowstepper.

I want to see the translation to Chinese. Well, the translation back to English. So I wonder if there'd be a difference if the Chinese were created from scratch (thousand most common characters or thousand most common terms) (presumably translating the given text directly would, or should, be dumb.

Also, I'd like to see Euclid's Elements done the same way.

One negative. If you're like me, you're getting older. I mean that with respect to the book. Meaning with respect to how it is printed. Meaning... ugh. My eyesight isn't terrible, but the font size is so small that it is a chore to read. I'm sure it wouldn't be so bad if my eyes were younger.

Sorry, two negatives. I am disappointed it is so short, only 60 pages but some fold outs.

Holy crap, a skyscraper!

What's with dictionary definitions for metaphorical usage?

Like a Jeopardy contestant giving an anecdote about their life changing experience when a pet dog tore up a favorite slipper, I have something I am terribly upset about.

I have noticed some online dictionaries giving metaphorical definitions. By this I mean that for a word, giving a meaning entry that is metaphorical, not its literal meaning.

For example, 'to devour'. Without checking, this means to eat ravenously. But it's easy to see that, say, a paper shredder could be said to devour some documents,

You may well note that many words, rather most words, really almost all words have multiple meanings (except for highly stipulated technical terms, and even then things can get loose). Our perception is usually that a word has one meaning and that's that. But then we notice that, well, that same spelling can be used for more than one distinct concept, usually nearby.

You may then well note that for many words, there really is a primary meaning: its meaning out of context that everyone thinks of first, and then secondary meanings, ones that appear in different contexts, that are slight extensions of the primary meaning, or used in analogous situations, not literally.

Here is the example for 'devour' from google:

de·vour

• eat (food or prey) hungrily or quickly.

• (of fire, disease, or other forces) consume (someone or something) destructively.

• read (something) quickly and eagerly.

The first is the primary definition, the second a metaphorical one, the third... huh? That is definitely not what 'devour' means. Sure, one can easily use it in 'I devoured the sequel' meaning that I read the sequel quickly and eagerly. But that's not the meaning of 'devour'. That's not what 'mean' means. It's too specific. Does the omission mean you can't watch a movie voraciously? How come 'reading' is more devour-like than other metaphorical uses? This isn't right! If you include read, you should include every other possible metaphorical usage. But of course that is too laborious to imagine.

The difficulty I'm having is the demarcation line. When does a reasonable metaphorical usage of a word become dictionary-entry-worthy?

Taking the title word 'incensed', which was not deliberate, its primary and only definition is around 'angry', and no mention of the ostensible literal meaning which might have been 'burned like incense'. It already is a metaphor. The only definition is non-literal. So putting in metaphorical usages is necessary. At what point of semantic drift, at what point of leaving the original does a dying metaphor become dead, and at what point does the altered meaning move from quantitative difference to qualitatively requiring a new entry?

A close analogy is with suffixes. You can take any word in the dictionary and find some suffix that applies that will create a perfectly good word. 'Neologistically' is my favorite. 'Neologism' to 'neologistical' to 'neologistically'. Probably not in any dictionary, but perfectly understandable, sounds like a word, and is (arguably) undeniable as a word. Does it need to be in a dictionary? At what point do lexicographers decide not to include a possible variant?

There are a number of possibilities. Checking multiple dictionaries, most don't have the strange 'read' entry, only Google and Macmillan. What I suspect is that there is a tendency to require definitive alternate usage for an additional entry to be made when the entries are edited by humans. And that Google and/or Macmillan introduce metaphorical entries mechanically and its easier to be lenient. The latter two dictionaries certainly need human oversight; that is, the 'read' entry isn't a mistake but a lower threshold.

This will require looking into the editing policies of the various dictionaries.

Fix it now! Necessary minimal examples for change in English

The problems (two of them) are pronouncing can vs can't and fifteen vs fifty (or really more generally -n vs -n't and -ty vs -teen).

Well, it's not global warming, but there are a couple problems with English, specifically pronunciation, that are problems in information exchange. It's not some style thing, like the inarticulation of textspeak or emojis, or plain semantic errors like 'literally' for when it's not literal.

It's about pronunciation where the actual information being transferred is at it's most distinct. And the distinctions are being softened to almost disappearing but the information is as substantive as you can get. This screams out for language change, but...I just don't see any that people are unconsciously doing. The details are:

- can vs can't - In American English, standing alone, these two are identical except for the addition of the 't', which stands out in only the most labored of articulate speech. Word final 't' is unreleased in AmE, meaning that though 't' is usually aspirated elsewhere, but at the end of a word it is not.  This results in a difficulty for foreign language learners for hearing the distinction and for producing the distinction, what little there is.

But what's hard for foreigners can be hard for natives and even an area for language change (see the cot-caught merger).

• I can do that
• I can't do that

The intonation/timing for these is different (easy to see for native speakers but difficult for non-natives). But in high noise areas, these will be impossible to differentiate. "Can you move this 400 lb sofa off my chest?". "I can do that" vs "I can't do that". The answer is kind of important.

The same situation holds for a few other contractions involving 'not': could/couldn't.

My proposed solution: I can't think of a good natural one other than encouraging people to not contract when there is a 'not'. But for the positive version, the most natural way would by to shorten it considerably as '/kn/' or for both emphasize the sentence into nation "I cn do that" vs "I can't do that" (where the intonation would be terribly weird for the version of opposite polarity).

- -ty vs -teen:  The teens and the tens, thirteen/thirty and above, are similarly rooted but dictionary-pronounced distinctly. thir-TEEN vs THIR-tee. Different stress, different last syllable.

Except in regular speech these are about as close as you can get in pronunciation, and differently from most other close pronunciations (or eevn homophones), context rarely distinguishes them. "How far is it from the last to the next gas station?" "Oh about  thirt.. miles." "What? Is that thirty or thirteen miles?" "Thir-Teen" "What? THIR-ti or thir-TEEEE-nuh?" "The first one." "Oh. Then we're going to run out of gas."

The context doesn't say which one is more likely, and pronunciation has to be very articulate to tell the difference. The stress isn't pointed enough to tell. This is a perfect time for native speakers to unconsciously do something (make a slight change) to encourage better understanding (to avoid the waste of energy).

My suggestion? It's gonna sound bad but... for 'thirteen' say 'thir-TANE' or 'thir-TEEN-uh, and for 'thirty' say 'THIR-tuh'. To me these are the least unnatural and closest to natural pronunciation. Spelled out they look awful. But this is my prediction/suggestion.

I know. It sounds terrible.  But I don't want to run out of gas.


NB This is entirely about General American English... I can't expect to speak for other varieties of English. Also, I speak authoritatively about my own variety...but I may be biased and there may be free variation and some people may speak with more articulation than I can. So in one sense I am saying 'I am right' (which may be annoyingly authoritarian) but in another I am saying 'here is what I think I see, some others with great authority see it too, you can check now too now that you're aware of the possibility'.

Docs don't want to see your FitBit records, but ...

Docs don't want to see your FitBit records, but they do want your Holter monitor.

Docs do want you to exercise, or rather keep an active life and intense exercise is a good part of that but also make sure you don't sit down all the time, just walking and standing up and moving around calmly is probably a better thing to shoot for.

Having a FitBit or other exercise tracker, they consider that great, of course, because it means you are thinking about your fitness and trying to keep active.

But your PCP (your family doctor) does not want to see your FitBit records. They don't care how many steps you make a day. They don't care that you missed a few days last week (OK all last week. but the week before I made it every day!). They don't care about the fine detail of every second of every hour. They just want to know that you're using it. Or that you're making progress. Or that you're unable to hit the targets (knee surgery?).

Yes, the FitBit data is 'big data', and big data is useful...in general, but your PCP can't do anything with all those numbers. Your PCP isn't a big data consumer. Your PCP just wants to know (for the most part) Yes or No (or better than last year). Yes, totally get a FitBit, and totally because you think it'll please your PCP, and totally tell your PCP that you have one, and they'll totally be genuine proud of you. But don't think they can do anything with your data.

Maybe, just maybe, your cardiologist wants to know some of these details. Wait, no, your cardiologist doesn't want to know your FitBit data. What FitBit is tracking is not useful for them. Steps per day? Heart rate pattern over a time period? Oh...maybe a cardiologist does want to see a continuous EKG, like what you get from a Holter monitor. That is a specialist who can understand that data under a very prescribed situation. And with intensive study of that data as a specialist.

If say these fitness trackers somehow morph into a full body tricorder, measuring all possible metrics continuously, then maybe just maybe, it will become useful as a big continuous stream.

I'm sure the FitBit company data scientists could tell something interesting about behavior and fitness by working with sports physicians, epidemiologists, internal medicine in general, to discover interesting patterns of use of the product and specific health metrics.

But for you? Right now? Are you using a fitness tracker? Your doc does want to know that. Are you meeting your fitness goals everyday? Yes, your doc want to know that. Does your doc want to see your heart rate for the past month continuously by minute? No, your doc does not want to see that.

"Big ideas in mathematics"

My answers to the survey What do you consider to be the main "Big Ideas" in mathematics?

It's all about proofs:
- truth != proof (Goedel's Incompleteness theorems and what follow)
- logic and math still support each other (Reverse Mathematics)
- proofs are not just algebraic manipulation. They give meaning (Combinatorial proofs)
- automation in proof assistants will accelerate mathematical progress by humans (automated deduction)
- the most abstract of abstract nonsense (Category theory, no this not particularly about proofs)

These are all the biggest of big ideas in mathematics. All the others that are specific to an area are what I would consider too... contingent. Topology, algebraic geometry, algebraic K-theory, as important as they are, don't have any far reaching ideas. OK, number three above, combinatorial proofs, I just like a lot and is somewhat like those three I just disparaged.

Abused dataviz: periodic table and subway map

In addition to wordles, here are two more often abused data visualizations, the periodic table and the subway map.

Both diagram methods are intended to show that among a set of entities, there are many subsets, for the most part mutually exclusive but they have informative intersections. Think of the Venn diagram as the canonical diagram of subset relations. A subway map should have very few intersections (only a handful of entities are in more than one subset, the interchanges or transfer stations). The periodic table has a lot more structure, in fact, as a special case two-dimensional table, the full set can be split into mutually exclusive subsets in two distinct ways.

Take for example the original periodic table.

(from Science Notes)
What a great invention. Mendeleev compiled a bunch of disparate facts, similarities of elements, into a single visualization. The dataviz wasn't perfect, because there were gaps. But the picture was almost a theory, an extrapolation from data, that by 'testing' (further exploration) was confirmed by elements that fit nicely in those gaps. There have been attempts at organizing that chemical information in different ways but Mendeleev's holds primacy.

Nowadays, a periodic table is used for organizing a large set of items that have some similarities. Except the similarities have only tenuous systematic patterns. The point to the chemical table is that they fit nicely into rows and columns according to number of shells and number of electrons in outer shells (which predicts chemical properties nicely). The modern use of these periodic tables seem not to care what patterns in reality there are, just that pretty colors and list. Often the items in a column are not really related, and often they don't go from simple to weighty.

(from Expand via pinterest http://www.xpand.com.au/ )
In this example, the table is simply chart junk. The colors specify the mutually exclusive subsets but the rows and columns say absolutely nothing about the entities.

The periodic table of dataviz has some attempt at using the structure appropriately in the far left and far right columns, but in between its a mess. The site is great for examples, I'm only criticizing the use of the periodic table as the viz method. Note that almost all periodic table viz's use lockstep the funny unbalanced form of the table rather than fit it to the data (the properties of the entities). Instead the entities are shoehorned usually without any reason at all.

The point to a periodic table is that everything in a row should somehow be similar, and everything in a column should also somehow be similar. Also there should be some kind of progression from simple to complex down a column.

When is it appropriate to have a periodic table? When your set of items has two clear dimensions. There can be lots of gaps, or more in one position than another. But the two dimensions need to be clear. Also use those labels! Make sure everything in a column needs to be related. The rows don't necessarily have to be exactly related but at least of roughly the same complexity.
---

Subway maps are diagrams of connectivity of train systems. As a dataviz, they show that certain sets have a handful of points of intersection. Within a subset (shown by a line or track in the system) all the items are related. So when two lines intersect, that item must be a member of both subsets. Unfortunately, many 'subway' maps don't even bother with convention. They'll group items on a line that are only tenuously related, and then a 'transfer point' (an entity on two or more lines) ends up having little to do with either.

What makes a 'subway' map good is when the subsets have very few common entities. That will translate to only a few interchanges, making the diagram easier to create and less busy. It's a plus if you can order the entities along a line in a meaningful fashion (there is some inherent ordering).

Most uses of the 'subway map' dataviz, just like with the periodic table viz, either take a literal subway map (London's usually for obvious for obvious dataviz design homage) and shoehorn entities in, or make up their own but don't bother to make the lines and interchanges act like sets and intersections.

(from Becoming a data scientist) Using some domain knowledge, the items on each colored line aren't very coherent subsets, and their interchanges aren't really common between the two intersecting lines. There is quite a bit of overlap among these entities, lots of subset relations and intersections, but they are unfortunately not even bothered with.

A subway map viz is appropriate for a set of entities if those entities separate nicely into mutually exclusive subsets, with a handful of single entity intersections. If there are many intersections, then there are many constraints on how the lines meet each other.

Consider each entity as as having a list of features. If all the entities have a single feature that partitions the set (these are the subway lines) with very few entities with more than one line (the transfers) then the subway map is appropriate. If all the entities have two features, each partitioning the set in two distinct ways, then a periodic table is appropriate.

These dataviz strategies may well be meaningless chart junk simply to display a list with some structure. They are certainly esthetically pleasing (just like wordles!), but for the most part used irrelevantly. Most lists of entities are easily separated into sublists, with little extra structure, r quite a lot of complicated structure. The subway viz is good is there is a very little bit of common properties. The periodic table is good if there are two mostly coherent discrete dimensions, they don't have to be numbers.

The primary complaint is that the template is ostensibly knowledge based (scientific looking, 'sciency') but that the data poured into them just doesn't have that structure; the structure is a red herring. The dataviz should add something, should give you knowledge about the entities. If the items are on the same subway line, they should have some commonality. An entity in a periodic table should be similar somehow to the other entities in the same column and also the same row.

The alternative, when there is not enough structure, is a simple set of lists. If there is too much structure (lots of common features with little discernible pattern) is to use a venn diagram which captures all the possible intersections.

Or maybe I'm just complaining about incoherent sets and it's not even at the level of the top level structure being a red herring. It's a red herring that it's a red herring.

In defense of publication bias and p-hacking

Publication bias and p-hacking have recently come under a few attacks even though it's not a new thing... not new ... at all.

Publication bias is the tendency to publish study results that have p-value (the statistical measure of 'significance') if the p-value is <= .05 (5%), the magical oversimplifying cutoff stated originally by Fisher as a.. well... magical oversimplification because he found that most people didn't really understand 'really' what p-values mean so he gave this as  a quick heuristic for significance.

'P-hacking' is the tendency for researchers to massage the data, the experimental design, the statistical method to improve the p-value output just over the threshold to 'significance' mostly to get around the publication bias.

These are problematic for different reasons. Publication bias leads the literature to ignore the non-information or weak information or non-results, things like 'X doesn't predict Y very well', or 'Treatment Z really doesn't really do much'.  These are things that would be nice to know, so a researcher in a field can then either avoid studying such unimportant things or discover refinements of the data that do show something important.

'P-hacking' is a bit more pernicious because it can lead in the direction of outright fabrication. Fixing poorly recorded data removing an outlier (somewhat reasonable, but it is controversial), but this is almost in the direction of fabricating data itself.

Given these well attested problems with publication bias and p-hacking, something about them is actually not so bad, in fact, they are an outcome of very reasonable and desirable scientific behaviors. This is not a justification for 'a road to hell is paved with good intentions', rather that some part of both is actually a good thing.

First, what is bad about publication bias (tending to publish only positive results)? From a literal, rational viewpoint, it is obviously denying half the story, creating all the false negatives. You should report all your results positive and negative to get a good picture. But that is a false equivalence. Positive results are not positive instances of a coin flip. Positive results are the interesting results. Interesting is something new and compelling. The null hypothesis is dull and lifeless. We already knew the null hypothesis. The null hypothesis is the air we walk through constantly. Reporting positive results is like pointing out a new pathway in the forest. Experts in the field, especially editors of academic journals, se many many results on slightly different phenomena. They have a good sense of what is new and important, and what has been done over and over again (and is maybe replication), but they also have a sense of what isn't important or isn't positive in the field. I'm not saying that negative results should not be published, but I do say that they don't need the boosting that positive ones do. A negative result is usually not that interesting. (In the natural sciences, that is; in more mathematical sciences, they are a different sort of thing, and often earth shattering)

And for p-hacking, sure, it is gaming the system. Hacking and gaming are things you do to improve something that are, let's say, hors du combat, outside the system Once you take a measurement, it is something to be gamed. Two runners competing for the fastest time? Train harder, eat better, lean at the tape, starting blocks, better shoes, bend the rules, shave your hair, make up new rules, take meds, get surgery, make up rules about these rules. The difficulty with experimentation and science is knowing what is cheating and what is allowable. For p-hacking, there are rules, things to do, things that are encouraged, things to avoid, and things you just can't do.

P-values are a third order measurement. First, data is the most primary measurement, a stopwatch, a rule, whatever. A statistic (like an average) is a measurement on data, you take a bunch of data and measure that set of data. For the mean, it gives you an idea of the center of the data. Then you can measure the p-value which is a measurement of how reliable the statistic is. At each stage of measurement, gaming can take place. You can manipulate data (remove outliers, 'fix' values) or manipulate the statistic (choose another, sample differently), manipulate the p-value (pick the best one, correct for multiple comparisons) and at each stage of gaming you can do it legitimately or not (no bias or much bias; yes, what the bias is is well underspecified).

The p-value itself is a time honored quantity associated with statistics. It is notoriously subtle and notoriously difficult to teach those subtleties. But it is a very useful measure of quality. It shouldn't be thrown away, just used carefully. When you mix dangerous chemicals, you do it under a fume hood, wear goggles, and have first aid nearby. When you calculate p-values, you make sure you don't calculate many on the same data and pick the best one.

Sure, sure, sure, publication bias and p-hacking are, as stated in that manner with their expected tendentious meanings, to be avoided as such. But the scientific process that results in those things, are not entirely evil. They are natural drives for knowledge and expression of knowledge and convincing people of knowledge. By saying 'natural' I'm not being lenient. Those drives have a correct part and an incorrect part. The part that we label bias and hacking are essentially bad, but the other part is not and is good. Not everything is bad about them.

Progress in making Star Trek tech real

There's been a lot of cool sci-fi technology over the years in Star Trek: transporters, phasers, faster-than-light travel. And by sci-fi I mean 'convenient but impossible stuff that helped get the plot move a long'. Star Trek TOS introduced a number of things, NG a few more, the movies I can't think of anything more than "an excess of chronotron particles has created an anomalous rift in the space-time continuum".


But it's kind of funny - since the original series, engineers have almost taken these sci-fi things as a challenge. "Star Trek can do it in the 23rd century. I will make it happen now!". Some things have actually happened; some have been found to be physically impossible. And all sorts between.
Here is an inventory of these plot devices/technologies and 'our' progress (star date spring 2015, 50 years after TOS). 

• transporter - (from The Guardian). It (presumably) records all the positions and velocities of all particles in an abject, and recreates them ... elsewhere. This is theoretically possible and experimentally shown to work on individual electrons. But it needs a lot of work to scale this up to safely transfer humans. It just seems computationally infeasible and would take inordinate amounts of energy to make sure that all the atomic particles get transferred to just the right place. Some scenarios of this require that you have to kill your own doppelganger (it's really just a duplication device). Also, there are myriad niggling details like accounting for the relative speeds of the source and target locations (planets and satellites spinning around at huge velocities). However, one of the side benefits of transporter technology is that it allows you to remove infectious diseases and compute your genome, which, if the quantum relocation problems are solved, would surely be a piece of cake to do. But might allow mishaps aplenty as in many Startrek subplots, and other movies like The Fly. Not impossible, but very difficult with today's technology and economics. There are claims that transporting of objects has been done (see The Guardian article), but I consider that cheating of a form that just isn't cricket.That's a form of manufacturing (which may suffice for the entry on replicators).

• automatic door opening - https://www.youtube.com/watch?v=6CSmkym-Stw  This is old news. Every grocery store has had these for years. Either through motion sensors, RFID chips, or weight pads

• artificial gravity - this is a difficult one. What does it mean to have this technology? Without a large mass, can you simulate attraction to a flat surface? Or does a spinning cylinder which mimics gravity count (you'll be able to walk mostly normally, but water will still form messy blobs instead of pouring straight down)? The latter gets a lot, but the former...I don't think there is any physics that says this is at all possible. There's no graviton generator.
  
  However, one can cheat reasonably here. You can 'simulate' gravity on a surface as long as you make the surface accelerate towards the thing you want to exhibit gravity against. Make you spaceship accelerate towards a destination at 32 ft/sec/sec and It feels just like Earth! (turn around halfway and slow down at the same rate, because deceleration is just acceleration in the other direction. Note that this requires lots of fuel, fuel the whole way, rather than just coasting.
  
  Or the old fashioned spinning cylinder would work (with a large enough cylinder), going around a circle is acceleration, too.
  
  Neither of these is cheating, those are real equivalents of gravity. But if what you want is localized gravity, say over a square yard, as different from the adjacent square yard, then no, that's not going to happen.
  
  Progress score: by thinking of gravity as acceleration then yes in some contexts. But, arbitrarily, no, physically impossible.

• invisible force fields or shields (for protection in battle) - same thing. This is actual magic. In that science is not involved. OK, this might be cheatable with lots of magnets. Progress score: never

• cloaking device - The ability to make yourself invisible. Or maybe they've rethought what it means into a solvable formation. Yes, there is technological progress on this, wrapping light to go around an object, or camera on one side projecting to the other. Progress score: sometimes totally yes, other times in progress.

• tractor beam - If there were arbitrary artificial gravity, it could be slightly modified to produce a tractor beam. Except I've heard of laser/magnet/ultrasound pincers for very small scale manipulation in air of particles (like dust). Progress score: not at all. Cheating: maybe

• phasor (weapon for stun or kill) - sorta, not really. Like solar energy, there is the basic technology there, but it just hasn't progressed well. There are 'energy beams' but they take a lot of energy to produce, which is probably too difficult to engineer down to a handheld. But what about those green pointer lasers? I guess making you close your eyes is something. Progress score: sort of, very slow

• photon torpedo - I don't know what's inside that thing. But we have had bad enough destructive bombs since before Star Trek came out. Progress score: already done, but maybe it doesn't glow or have flashing lights on it as it approaches its target, probably for good reason.

• faster than light travel - This is obviously theoretically impossible, even for information, forget objects, All that wormhole stuff is nonsense (at least wormholes that allow an entire ship to travel through unscathed). Dude, if you go into a black hole, you'll be ripped apart before you even get to the event horizon. Of course there may be some cheating thing (transporters?) Progress score: never

• computer disks -
  
  
  
  ( from filmjunk) quickly surpassed and obsoleted. I love this one as an example because it was one of the mini minor details in the TV shows that really made it great. Instead of big file folders of paper. "Just a small piece of plastic? Wow, the future is crazee!." That was the 60's. Then with PCs in the late 70's/early eighties, the floppy disk could be used as a replacement for paper. and it slowly got smaller until the early nineties the 3.5 in disk was widely used and pretty much identical in function to the ST disk. By the mid 2000's, we have thumb drives, barely noticeable on your keychain. And now people keep everything in the cloud. We don't even need a physical device to keep our files, it's just there in the ether. ST had the idea, people created it, people surpassed it, no one uses physical devices anymore.

• communicator - Cellphones were invented in the '80s and have improved ever since, even better with apps (smart phones). I guess there are some limitations like we can't talk to the ISS from the ground with our phones (maybe someone at NASA can?) Wait.. we can tweet to the ISS on our phones. Progress score: Done.
• talking computer - yes, to great extent. It's not perfect.It doesn't do all languages automatically. But for European languages, it gets syntax  and word choice mostly right (except for a few dings). Slow but steady progress by scientists over the years have created this (despite the bravado of the earliest years of AI claiming it could be done in a couple of years)
• tricorder (handheld noninvasive medical analysis) - not yet, but getting there. There are all sorts of tools now for blood sugar, blood oxigen, temp, etc. Also, radiology is slowly miniaturizing. Next hurdle, hand held DNA sequencer?
• universal translator - crazy impossible AI in the 60's. Hardwon but infinitesimally incremental progress over the years has recently resulted in a passable intermediate stage by Google. Kind of the same explanation as Siri. It's not the best but it's workable if you don't talk too fast.

• DNA analysis - I mentioned the genome when I discussed the transporter and tricorder. Currently, if you have the locations of all the atoms in your body) presumably one can then go through some process on that data to get your DNA sequenced. But without (which the tech to produce the transporter would allow) genome can be done by a big machine for <$1K, but analysis and miniaturization not yet.
• time travel - OK everybody, just shut up. Time travel is not possible at all. At least not in any way like in popular fiction. Wait, I'm sorry, Planet of the Apes got it perfectly right. You can travel into the future (we're already doing that right now, right?), but faster than normal. That is, if you travel at a non-trivial percentage of the speed of light, your time will be slower than others, and when you come back, everybody will have aged more than you. Other people will think you popped in from the past. That's about the extent that physics allows time travel. That's it. No going back in time (we'd have noticed people doing it already), and if you could you couldn't do anything that hasn't happened already. It's all just a plot device that appeals to successful primate brains. When the rodents take over (look at their hands!) and start telling stories, they'll come up with it too.
• replicator - not really, but minimal progress. I consider 3D printing to be one part of this for which there has been great progress (and still more to go). The 3DP will create the physical substrate for an object (the parenchyma if you will). The missing part is the chemical or substance part. If you want to replicate a ham sandwich, you can (could, with some work) 3DP the shape and consistency of the bread, tomato, lettuce, ham, and mayo. But for each of those you still need the taste. And that will need some chemical engineering that I am not aware of. Subtleties in taste, not everything in the grocery store can be imitated with artificial flavoring. Progress score: early stages of prototyping
• holodek - we're almost there. In the 90's there were rooms called 'The Cave' where your environment was displayed on the walls, and you held onto a control device that noted your orientation and movement. It was neat. On the way but not the best. The gaming community has created very life like images and environments. If you call it virtual reality there are VR helmets now. Not totally VR, not total immersion, but even closer. Basically what is needed is better plots and virtual acting.

Did I forget some glaring ones? Sure, the communicator. We've done it. Next.

And surely there are a lot of technologies mentioned before and after, not in ST, that are worthy of discussion. But this is about ST.

For many of these, where I say no or impossible, I'm pretty much challenging you to go around it, by rethinking the rules or outright cheating. That's how engineering works. To get from A to B better than a horse or bird is not to add more legs or implement wings. So maybe you can make a levitating car (but why?) as long as you have a track or ferromagnetic surface.