Tuesday, September 27, 2016

"Probabilistic programming languages" aren't

I'm looking for truth in advertising: "Probabilistic Programming Languages" or PPL is a recent term to describe a very useful new paradigm in statistical modeling computing... but it's not a set of new languages. It is much more restricted to very particular kind of statistical modeling as opposed to the broad sense of probability.

As someone who likes a little consistency in language use, for words to have meanings you can mostly rely on, I am bothered by this usage (just as I'm bothered by the similarly mystically enticing marketing term Deep Learning). Here is a  very representative description of PPLs:

Probabilistic Programming (PP)
There’s a revolution in Computer Science called Probabilistic programming (PP) where programming languages are now built to compute with uncertainity in addition to computing with logic. This means that existing programming languages can now support random variables, constraints on variables and inference packages. Using a PP language, you can now describe a model of your problem in a compact form with a few lines of code. Then an inference engine is called to automatically generate inference routines (and even source code) to solve that problem. Some notable examples of PP languages include Infer.NetStan, BUGS, church, Figarro and PyMC. In this blog post, we will access Stan algorithms through the R interface.
from a blog article on PPL (which also tries to introduce new but uninformative terminology, MPML).

I expect words to mean things, and despite liking metaphorical usage in literature and expository writing, not calling a technical thing what it is sounds too much like slimy obscurantist marketing practice. If it is misleading in any way, it is suspect. Suspect maybe not in venal terms, but more likely suspect in intellectual depth.

For the record, the difficulties in the passage above are:

  • There's no revolution, not in computer science, not in programming languages, not in AI. Maybe there's some recognition that there is some progress in usage, but it is incremental.
  • No new programming languages are being built. No programming languages are being modified to accommodate new probabilistic data types. This is the biggest clunker. There's no new programming language thing at all. What is new is packages or libraries or functions, in the existing programming languages. PyMC is a library written in Python, and used in Python as native Python. Stan is written in C++ but it is not a syntax/semantics, just a library that is accessible from existing languages (R, Python, Matlab, Julia, etc).
  • The idea of operating on distributions as a type is not actually new. Mathematica and Maple have had object oriented implementations of distributions, allowing operating on those distributions functionally. What these PPL packages add is approximation algorithms to compute values for Bayesian inference using Markov-Chain Monte-Carlo (MCMC) , which is fancy talk for calculating a number approximately. Pretty much very analogous to computing a p-value.
  • All these PPLs are just library add-ons to existing languages. So in that sense don't worry that you have to learn a new syntax. You surely will have to learn how to use the library.
  • It's not about probability in the large. Most all languages have probabilities already (restrict floats to the range 0:1). Some people are creating packages that make it more easy to use probability distributions (which some languages already had libraries for), and to manipulate those distributions (and make statistical inferences from them. But, no, it's not a revolutionary new alternative to languages with logic using probability. It might be a revolutionary library of functions that will make manipulating and computing with distributions and models easier, nut it's not a new language.

A programming language is a syntax and semantics and a compiler. A library is a set of functions written in a programming language usable by programs written in that programming language. One language can use functions written in another programming language as long as there is an interface.

To call a programming language probabilistic (or extension to an existing language that would render in my eyes the extension to be probabilistic), there would have to be a basic data type, like integer or boolean, that corresponds to a probability distribution (for the purposes of efficiency in compilation).

This is certainly a diatribe about naming. I have no qualms about the use of these PPL libraries. It's just the name. Distributional Modeling Libraries may be more accurate and doesn't have the same punch but is not actually incorrect like PPL. But if you're advertising faster-than-light speed, those words come with a lot of meaning, and should actually provide that without a lot of qualifications.

No comments: