What is artificial intelligence?

Most people think of artificial intelligence as a walking talking robot that looks and acts almost human, and that this is the goal of AI, to make a simulacrum of all that a human does. But there are a lot of nuances to this, well, frankly a lot of major differences with what artificial intelligence really is.

I am writing this to clear up a lot of misconceptions that I read about AI, not to give _a_ definition, but rather to give _some_ definition to it.

So first, a first attempt. Artificial Intelligence is a computer method (rarely mechanical) that seems to do something only a human could do. A talking responding machine is AI, but a walking machine, with articulated legs, is not. Unless... well there it is, sometimes it is the application that is AI and sometimes it is the methods. And a chess playing machine....well is that AI or just really good engineering, exploring all possible game possibilities quickly?

We already have two nuances: how an AI is engineered, and what does the label apply to. The methods of AI engineering fall into two broad areas: human simulation, attempting to mimic the internal biological/psychological processes, and direct engineering, doing what it takes no matter what to make the engineered device act externally intelligent. A language parser uses scientific theories of linguistics to mimic how the brain is supposed to manipulate the text of a sentence to determine understanding. A chess playing machine uses alpha-beta pruning of deep game trees, contrary to the usual method people which is to look at only two or three moves deep and vaguely judge 'strength' of a position based on experience.

A lot of what was once considered AI is now considered plain old engineering. Once you see under the covers that the method used to look outwardly human is only a boring step by step recipe or lookup table, it loses it's 'wow' AI appeal and just seems like regular non-AI computing. in the 1600's, calculating machines were magical and might have been considered AI (if that were a thing then) because most people thought that people themselves doing it was magical much less a machine (OK this is a bit hyperbolic. what is 'most'? who are these people?). But you get the idea. A machine doing chess is magical (or has a small hidden turkish chess master manipulating the controls within). But Deep Blue in 1995, which beat Kasparov the chess grand-master, was pretty simple just using search trees, no ethereal silicon-embodied intuition ('just' = very rocket-sciency search trees and some hand-curated openings, gambits and end games).

An AI, with the indefinite article, is by popular usage, the thinking talking machine. Because of the potential for confusion the term for that nowadays is usual referred to as an AGI, Artificial General Intelligence.

But now to more substance. All along we were going along in vague style, not really knowing what AI really refers to concretely, relying on a presumed common idea of AI without really knowing. So here are some concrete examples of what AI is.

First, applications, which is what externally looks like everybody's general idea of AI

• vision - converting digital images into words 
• language - speech recognition, natural language processing and understanding, chatbots
• games - chess, go, crosswords
• problem solving/reasoning - word problems and puzzles, expert systems
• planning - taking a set of goals and initial conditions
• robotics - not the mechanical part of for example an articulated hand grasping an object but coordinating a path over a complicated landscape or separating parts on an assembly line
• learning and memory - most of computer science has been intent on solving the memory problem and has done it well. Financial statements, medical records, airline reservations, online libraries.

And then there's tools and methods, how it's actually done, the man behind the curtain.

• logic and other classical mathematical methods - for reasoning
• probability/statistics - most machine learning methods are within the realm of stats
• combinatorial algorithms - when a perfect algorithm for one domain works most of the time for another, then it's called a heuristic
• decision trees - chess? one big decision tree? Yes.
• optimization - specifically numerical algorithms for optimization like simplex or A*
• cognitive science - cognitive psychology, neurology (brain physiology, neurons), linguistics, so you know how it works in actual biology, so you can either be inspired by it (neural networks) or simulate it (parsing a natural language according to linguist's rules)

Apps and tools are orthogonal; an app can use different tools. So NLP is one application area of AI, and many tools are used by it: machine learning using large language corpora, and independently linguistic knowledge.

The man behind the curtain is an apt metaphor for AI because like in Oz, the apps look magical, but it turns out it is, well, not exactly charlatany; the methods are so...mechanical and spartan and inhuman and small tricks. For example, crosswords seems like you need to have not just broad and deep knowledge, but also clever ability with associative memory and puns to read behind the clues. The recent very successful AI methods for crosswords is just to use a dictionary search and almost ignore the clues themselves, by any means necessary no matter how cleverless.

With all that said, the motivation for this was what I find to be a lot of misuse of all these labels, one term being used for another. So here's a bullet point summary:

• AI is what you call the big show, all the different things together
• AGI (Artifical General Intelligence) is a very limited example of an AI app but the most salient, the 2001/HAL intelligent like spoken interface. This seems to be like the end goal of AI but currently unrealistic, or rather very realistic with very limited expectations.
• ML (Machine Learning), which is mostly just statistics, is a tool to quickly create AI apps from lots of data without having to rely on a domain expert
• ANN (Artificial Neural Nets) and DL (Deep Learning, which is just a big NN) is just (of many) ML techniques

A meta-note about NP-completeness and its current state of (lack of) knowledge of

Recently, Scott Aaronson wrote a short-book-length summary of the current state of knowledge about NP-completeness. This is an honorable tradition, starting with the classic and definitive book by Garey and Johnson, then Sipser, Cook, Wigderson, and Allender. Aaronson gives his spin on things (a little more on the quantum computing side) but hits well (like the others) all the great points: Turing Machines, reductions, completeness, the hierarchy of harder problems, what is conjectured to be the case (P != NP), etc.

What I want to do is point out the unsaid things, those things that are obvious to insiders and don't need to be said, but are inscrutable to outsiders. The outsider may realize it or be told it explicitly but not really get it understand it because, you know, math.

The most important thing is that, of all the things stated, all the theorems, all the papers and books, the great number of words and symbols written about the subject, all these things are written about what we don't know about NP-completeness. We don't know whether P = NP. We don't know whether NP = coNP. We don't even know whether Graph Isomorphism is in P (early 2017 Babai retracted 2015 quasi-polytime algorithm, just sub-exponential). We don't know. We don't know so much. Every day there are books and papers published that are essentially just outlining the boundaries of all the so many things we don't know about P vs NP.

Most of the theory surrounding P vs NP was developed in the 1970's. Lots of small results (with great effort!) have been established over the years. We do know now that NP = PCP(log n,1) (Arora et al. 1998), NEXP is not in ACC^0 (Williams 2011). No, not small results but proofs of things that we really expected to be easy. Some specific problems have been shown to be in a particular class unknown before: Primality is in P (AKS 2004),  LinProg is in P (Karmarkar 1986)  UGraph Reachability is in L (Reingold 2004), but one usually expects it (relatively) easy to find an optimal algorithm for a problem...it just  turns out that all these were very difficult to discover.

But all these are minor results. One-offs. They may spark a cottage industry in their subcommunity and are undeniably important. But they are... they are only making infinitesimal progress towards P vs NP. In fact, because of this meta-intractability (get it? the problem of intractability is itself intractable), no one actually is currently attacking the direct problem of P vs NP. Most of the (great amount of) work goes into ancillary situations, extending minor results, changing the problem, all in the hopes that there might uncover one little tiny bit that makes the difference. It feels like panning for gold in that you dig up tons and toms of material, wash it through multiple levels of gratings, wash it some more, refine the dirt, wash the dirt some more, and if you get a couple of tiny specks of non-dirt that just turns out to be copper, you call that a good day.

As far as history of science goes, this is a rare situation. Usually, a bunch of data is collected, some patterns are noticed, you collect a lot more data (and all this new data is useful and interesting), these patterns look like they repeat in different weird ways, then a consolidating theory (a scientific experiential theory that is) is formulated that generalizes and explains the prior data, and then a search is made for even more data which then either confirms the theory or encourages refinement or maybe even over-throw of the first theory. Similarly in math (which the P vs NP problem is a part of), a bunch of small theorems are proven, then some (usually a group but usually a single person gets good press) see a pattern, add a few definitions and prove a big theorem that combines them all, and then further consequences of that big theory are applied to more instances (note that yes, in math it is very rare (never?) for a big theory to be overthrown because a proof is a proof (right?) of course it is, but the refinement comes either in how many new more complicated problems to explore are created that go beyond the first theory.

Anyway, the rare situation with P vs NP is that we've essentially gone almost 40 years now with the same state of affairs of lack of knowledge. The progress that there is is not steady, and any gains seem underwhelming at least from the outside (the first outsider thought is "You're so smart, shouldn't you have figured that out a long time ago?"). Even the most minor seeming result is astounding (NEXP is a very intractable class and ACC^0 a very tractable one - how come it was so hard to show NEXP not in ACC^0?).

Also, back to more basic things. All this literature, all this theorem proving hasn't really established, hasn't really given a definitive proof of the pinnacle question P vs NP...but it's almost a scientific proof, a proof by having lots of examples in mostly one direction. All the reductions, all the intermediate problems, al the proofs of no proof by method X, are really supporting evidence (not mathematical definitude) just supporting the particular direction that P is a strict, proper, much smaller subset of NP, that NP problems are actually harder and we really can't find exact and fast algorithms for NP-complete problems. That is, even though we have 40 years of not proving one way or the other, it (almost) all goes in the direction that they're probably not the same.

Practicality vs Esthetics in DataViz

We want both. We want for something to be practical and nice looking as possible. We want the shiny doorknob to look elegant, but we also want it to work smoothly without having to jiggle it, and not to need cleaning all the time. Sometimes one has to be given up for the other, an industrial assembly line may be a little grimy all the time but it gets the job done; of course, cleanliness may be a desired property of the object and practicality and esthetics share a common cause.

Sometimes we want something that is esthetically pleasing and superficially practical but not necessarily perfectly practical, like a watch with no face numbers.  The esthetics is the desired intention.

That's all philosophy diatribe to justify something that bugs the crap out of me.

Basically wordles are the worst. And periodic tables (except for The Periodic Table which is the best). And usually subway diagrams (except for actual city subways). Here's one for semantic web technologies:

This! This is the worstest.

If you know what each of the entities are, and you make all sorts of qualifications, maybe this makes sense a little. It makes only the slightest bit more of sense if 'A above B' means A is 'built on B'. But then there are all sorts of 'If you did that, then why did you do that?' questions (why is encryption and signature off to the side only for some, why is logic and proof separate, is Unicode really such a huge important base technology, etc etc). Wait, isn't a namespace a particular kind of URI? There are many variations on the 'Semantic Web Stack', but each in its own way has all these "I don't get why they did that?" problems. This is all about esthetics (Nice color combo!) and little to do with imparting coherent information. No, you will not learn anything from this. Wait...what the hell is 'signature'?

Trust vs Depend

"I trust that guy as far as I can throw him"

"I can't trust him to complete the project on time"

'Trust' is used in two different ways. One is the usual opposite of falsehood. If you can't trust them, they are a liar. This presumes intent and is almost demonizing.

The other way is dependability. Trust of the outcome. If you can't trust someone this way, it's not a reflection of their evil intent but about ability to execute. This is very different from falsehood. You can actively do something about this.

The other way you can only seek other sources of information.

So instead of 'trust' 'use 'depend on'. 'Trust' makes it sound like you think they're lying. 'Depend' just means there is doubt without judging.

Testing Software and Machine Learning

Testing is a major part of any non-trivial software development project. All parts of a system require testing to verify that the engineered artifact does what it claims to do. A function has inputs and expected outputs. A user interface has expectations of both operation and ease of interaction. A large system has expectations of inter module operability.  A network has expectations of latency and availability.

Machine learning produces modules that are functional, but with the twist that the system is statistical. ML models are functional in that there are one-to-one correspondence between inputs and outputs, but with the expectation that not every such output is ... desired.

Let's start with though with a plain old functional model. To test that it is correct, you can test the output against all possible inputs to check if. That's usually not feasible, and anyway is in the more esoteric domain of proving program correctness. What is preferred is instance checking, checking that a particular finite set of inputs gives exactly the corresponding correct outputs. This is formally called 'unit testing'.  It usually involves simply a list of inputs and the corresponding expected outputs. The quality of such a set of unit tests relies on coverage of the 'space'. Edge cases (extreme values), corner cases (more than one variable in an extreme value), important instances, generic instances, random instances, etc. Also, any time a bug is found and corrected, the faulty instance can be added to the list to ensure it doesn't happen again.

An ML function is, well, but its name, also functional, but the expectations are slightly different. An ML model (the more technical term for an ML function) can be wrong sometimes. You create a model using functional directly corresponding inputs and outputs, but then when you test on some new items, most items will have correct outputs, but some may be incorrect. A good ML model will have very few incorrect, but there is no expectation that it will be perfect. So when testing, a model's quality isn't yes or no, that absolutely every unit test has passed, but rather that a number of unit tests beyond a threshold have passed. So in QA, if one instance doesn't pass, here it's OK. It's not great, but it is not a deal breaker. If all tests pass, that certainly is great (or it might be too good to be true!). But if most tests pass, for some of 'most', then the method is usable.

There are three major parts to creating a machine learning model that need to be tested, (where something can go wrong and where things can be changed): the method or model itself, the individual features supplied to the model, and the selection of data. The method or model itself is the domain of the ML engineer, analgous to regular coding.

I can almost go so far as to say that testing is almost integrated to a large part already within ML methods. Testing use systematic data to test the accuracy of code; ML methods use systematic data in the creation of a model (which is executable code). And so if an independent team, QA or testing, is to be involved, they need to be aware of the statistical methods used, how they work, and all the test-like parts to the model.

Let's taking logistic regression as an example. The method itself fits a threshold function (the logistic function) to a set of points (many binary features, one continuous output feature between 0 and 1). Immediately from the regression fitting procedure you get correlation coefficients, closeness of fit, AUC and other measures of goodness. There are some ways to improve the regression results without changing the input data, namely regularization (constraints on the model), and cross validation. For the features, mostly independent of the method), first there are the number of features, how correlated they are, how individually each feature is predictive; each feature could be analyzed for quality itself. And last for the selection of data, also independent of the method, there's selection bias, there's separation into training, validation, and test sets.


Where QA can be involved directly or indirectly:

- ensuring metric thresholds are met - analogous to overseeing unit-test coverage of code)
- questioning stats methods (like being involved in architecture design
- cross-validation - both makes the model better (less overfitting) and returns quality metric
- calibration of stats methods - quality of prediction
- test and training data selection - to help mitigate selection bias
- testing instances - for ensuring output for specific instances (must-have unit tests)
- feedback of error instances - helps improve model
- quality of test/training data - ensuring few missing values/typos/inappropriate outliers
- UX interaction of humans with inexact system - final say - does the model work in the real world via the application, has interaction with people shown any hidden variables, any unmeasured items, any immeasurables, any gaming of the system by users.

The latter seems to be the most attackable by a dedicated QA group that is functionally separate from an ML group, and all the previous ones seem to be quite on the other side, only the domain of ML to the exclusion of a QA group. But hopefully the discussion above shows that they're all the domain of both. There should be a lot of overlap in what the ML implementers are expected to do and what QA is expected to do. Sure, you don't want to relieve engineers of their moral duty to uphold quality. The fact that QA may be looking over quality issues doesn't mean the data scientist shouldn't care. Just as software engineers doing regular code should be including unit tests as part of the compilation step, the data scientist should be checking metrics as a matter of course.

Notes from China-Japan trip

The following are my unedited notes taken on my iPhone on a two week trip through China and Japan: Beijing, Chengdu, Yangshuo, Honk Kong, and Tokyo. Reading through them I notice a few autosuggest errors, but also some that look like autosuggest errors but are really intended, just me being weird. I also notice that there is quite a bit that is not here, either because we were too busy doing things for me to type up, or because I didn't think of it as a thing to write, despite my memory now realizing it was a thing.

The actual itinerary was:
Day 0,1 Fly from Boston to Narita, Narita to Beijing, taxi to hotel
Day 2 Beijing morning - Tiananmen square, early aft - lunch, late afternoon Acrobats/magic show
Day 3 Beijing free day, walking around SanLiTun, local hu tong
Day 4 Beijing morning Mu Tian Great Wall aft/eve flight to Chengdu
Day 5 Chengdu morning Jin Li St, lunch Szechuan cooking, aft People's Park, evening Chinese opera
Day 6 Chengdu morning Panda Zoo, afternoon Chengdu to Guilin flight
Day 7 Guilin to Yangshuo, morning drive from Guilin hostel to Yangshuo, aft Li river cruise
Day 8 Yangshuo morning - bike ride along rivers, climb Moon Hill arch
Day 9 Yangshuo to Hong Kong travel
Day 10 Hong Kong morning - Victoria Peak, aft - Peninsula Hotel high tea
Day 11 Hong Kong, Lantau Island morning Buddha, aft - Tai O fishing village.
Day 12 Hong Kong to Tokyo flight
Day 13 Tokyo - WTC and bus/boat tour
Day 14 Tokyo - Joypolis amusement center
Day 15 Tokya to Boston flight

 ---

China Japan

Day 0 (Jun 25-26)
Flight
A little more room than domestic
Ian port asks to trust. Cables everywhere.
Two people in back cabin w face masks. 3, no 4
1/2 of area of plane is 1st and business class
Already annoyed that I can't look up what actor that is in imdb

Day 1 crosses the date line in flight. Lots more flying.
Narita airport
By the time at Beijing airport I think I've seen every possible JAL entertainment option. Every movie even Japanese and Korean ones.
Fried rice at place downstairs from hostel.

Day 2 Beijing
Tian an men square huge but still people everywhere.
Women wear more dresses. Men spit. Some People sit on their ankles. Iranian and Nigerian tour group.
Guys rolling up their shirts to cool off. Ugly
Among the throngs of tourists. After TAM lots of hawkers but also lots of people asking for money. Guy with one arm
Restaurant not organized looking.
Little girls all dressed up
Big areas, but then smaller garden areas very nice.
Young Woman w cat ear head band
So many Chinese tourists. I thought I was special.
Breakfast bao zi, jiao zi, banana
Lunch kung pao chicken, chao Cai, vegetables, frog legs.

Bathroom at hotel has an toilet w water saving two flushes. Shower to open floor.

Dinner Peking duck, cabbage. Cauliflower, salt duck bones

Day 3
Breakfast at hotel. Meh.
Complicated subway trip to sanlitun mall area which ended up everything we wanted to see being closed. Complicated trip back. Raining hard by the time we got out.
Lunch at place across st duck and fried noodles. Really good.
Ping pong in public
Lao Beijing, I heart BJ, lots of public spitting by old men, clear g of throats
Small groups wearing monks or karate clothes
Electric mopeds that are entirely soundless

Day 4
Breakfast at place next door.
Road out of Beijing lots of nice new and old skyscrapers buildings. Plants decorating highway.
Cars look identical
English/pinyin everywhere almost always.
Everybody has an iPhone (everybody, or Samsung)
Lexus dealer
Road out of city is all tree and shrubs growing for sale.
Closer to mountains, very touristy but not many tourists.
Lots of isolated shrines (serious well decorated) just sitting in a field or on top of a small ridge

Great Wall - amazing. Someone forgot a level when building

Lunch - subway. Tastes the same everywhere.
I saw grape leaves growing. I smell... Cows? Pigs?
Lots of brightly colored exercise equipment next to playgrounds
Two camels sitting in bright green river basin. No water.
Our bus driver really likes passing everybody no matter what. He might slow down or speed up if a car comes the other way.
Roof top water heaters.
Everybody growing a very small crop of corn outside their walls
Gas stations seem empty even though highways full of cars
More people smoke but not everywhere. Female truck drivers

Dinner on airplane - measly Lang pao chicken. Seaweed salad muffin yogurt-thing sweet rolls (called bao zi)

Day 5 Chengdu
Breakfast baozi rice porridge bean sprouts savory flan
Jin li st cicadas yak meat rabbit heads, crossbow

People's Park-100 things

Cars go one way on one way streets but everybody else does what they want. Nobody is driving crazy but lanes seem optional. Safe but worrisome. Not speeding at all just questionable choices and behavior by cars, bikes, peds all.

Stores - 100 things to say

Chinese opera -
Taxi cheap
Cheng du fancy, so many new buildings, luxury stores
Post-modern story within a story
Mask changing
Grating female opera voice,
Very traditional music and dress mixed with very modern music and effects.
Chinese english Korean Japanese subtitles
San guo, 108 stories

TV - stupid, just like US
Motorcycle repair on the sidewalk.
People tend to try to speak minimal English if they can.
Skyscrapers go on and on.

Day 6 Thu - cheng du Pandas
Tibetan restaurants/stores
Lots of subway stops
Business skyscrapers give way to apartment skyscrapers. I see a spot where they missed.lota of construction. Lots of mopeds
Cicadas at panda zoo
You want to hug a panda

Lots of kids with shaved heads but a heart or circle.

Huge outdoor mall area as first couple floors of skyscrapers

Taxi driver left us off a block away and then continued on past front of hotel ahead.

You can see the sky in Chengdu (not in Beijing)
People walk across the st without a care with cars coming on either side.
Roads have lanes for bikes/scooters.
Free condoms in lobby of hotel

Day 7 Guilin to Yangshuo
Raining
Potholes
Beautiful streets in Guilin (riverside hostel really nice along river)
Guilin dirt is red
Everybody has a smart phone
Banks everywhere
Motorcycles and scooters in rain doesn't stop them
Rice
Constant change in crops. Beautiful mountains
Patchwork crops

B landscape totally flat but scattered with limestone hills

Nice cars from everywhere. Volvo Skoda Mitsubishi BMW



Etc toll ezpass

Yang shuo new built buildings with no siding look abandoned in the middle

Li river cruise amazing. Pictures are a weak copy of reality.
Trash in water at harbor
Selling dried fish onboard
Taking pictures of the 20 yuan with li river landscape on bill and in background.
tourists are almost entirely chinese (again I thought I was special)
Talked with older couple about kids. A younger man tried to talk (in sorry but I'm shy. The language thing or the social situation)
Water buffalo, bamboo, edging on river, cormorants, houses along river,

Day 8 Yangshuo
Breakfast - noodles

Raining off and on humid

Bike ride/moon hill - too much to see
Rice paddies, continuous cultivation of everything, small villages w new concrete roads, poster with alternative medicine flames, dams, pet dogs not saying hi, fruit trees, melons, grapes, lotus, wildflowers, Palm trees bamboo exercise parks other tourists, mao shrine in house. Talking with tang up mountain steps


Moon hill natural arch - easy short climb through bamboo etc forest so muggy can't tell if rain or sweat.

Lunch - Ian said best yet, taro chips, peppers not as spicy as claimed, egg and tomatoes, potatoes, beef w onions and veggies

Driving - people drive very slowly. But don't seem to follow right of way, or rather they do but not lines on street. Horn honking is loud but meaning is polite, to tell you that I'm about to pass

Got a lift with a motorcycle 3 on. Totally safe except very questionable
Truck filed with dogs of same type
Shopping in Yangshuo - 'local market' of touristy things snack aisle had all sorts of vacuum packed fish or meat things, chicken feet.

Dinner - Nolan fried rice, Shohreh fast food buffet. Ian spicy noodles. Faux tiramisu, Black Forest cake, bowls in plastic vs buffet confusing.


Day 9 travel Yangshuo to hk
Yu long river
Building everywhere
Potholes!!!
Cell usage everywhere
Shohreh says How dare they have tolls on roads in China
G65 off 543
Less pinyin in guilin, lots in Yangshuo
How did hills in guangxi develop?
Xanadu Putao 535
Gas station teardrops on highway Sinopec
Hills go on forever in every direction
Little terracing of rice paddies I guess because everything is so flat between the hills
Scooters and motorcycles with umbrellas either attached or holding
Highway 2 yrs old
Bamboo scaffolding looks flimsy
Highway sparse
Red dirt
Er 508 exit near G72
Tree planting in grids like France
Rumble strips 3 3 2 2 1 1
Now lots of hills far off in the distance
Cloudy consistently so no sun, have to use compass
Hard to balance typing and looking
Hills shorter and less steep, connect together
Individuals working in fields not so quaint. Hard work
Cars mostly new. Mini work trucks look much older
Gravestones at base of limestone hill
Exit s22 01
Store fronts entirely open everywhere we've been, even large hardware groceries toys motorcycle repair
Shrubs along highways
Wide bike scooter paths along main roads

On train to Shenzhen
Little villages nestled within tall hills. Lots of new bldgs they don't want to put siding on many, some dirt roads
I see a non-chinese person and want to stare.
Big empty train platforms (Guilin was full)
Guangdong flat broader bigger fields more taller bldgs
Large ponds with pumps in the middle
We had the kindergarten car. So many excited singing kids

Shenzhen metro
Little kid outed me as meiguoren
Littler kid has uber alles t-shirt

HK
2 customs
Ian says why do they need the same info that's in the passport copied onto a form?
Great air conditioning
So many mountains on the mainland side of HK

Jordan - like NYC smashed into 10squarr blocks. A seller in the night market swore at a customer for walking away. Great Japanese candy store.
Pakistan halal, Thai Vietnamese, doner kebab.
Multicultural: Malaysian, African Indian, southeast Asian.

Day 10 HK
Breakfast dim sum fulun restaurant
Cruise ship, modern mall
People are mostly very curt ad humorless
Skyline of HK island
Lunch at peak restaurant (noodles, turkey club, fried rice)
Peninsula hotel tea
No dinner (Ian had fried chicken, cucumbers, noodle soup) I had milk tea.

Day 11 HK
Bfast - McD's
Metro real nice try
Lantau islandvoiin
Cable car to Buddha - enlightening
Bus to Tai O fishing village - bust, false advertising. So many squid related food products.
Ian wonders that people still love like that.
Drivers are 'normal' here, drive faster, follow rules
Scenery great
Infrastructure for rural env really modern. Feels European.
Squat toilets ok w shorts.

Lunch
Subway (I had mango lassi and samosas from Ebeneezer's)

Dinner
Fulum fisher mans wharf restaurant
Yao had us wash our glasses and bowls and cs
Waitstaff surly and miffed by other waitstaff, walk away in the middle of the order
All mostly retired peopl

Misheard: ff7 sadder than titanic

Had laundry done at a chinese laundry

Day 12 HK to Tokyo
Bf cheap buns
Lunch on Cathay pacific al, sns pork
JR Narita line
Japanese rice fields much neater larger and fuller than Chinese
People dressed less casually (not formally at all) wait... It's rush hour, everyone is coming back from work.
A handful of masks


YMCA br room very 69's looking
Guy with masked sneezed terribly in elevator

Searching for sushi conveyor place we ended up at a ?hibachi? place

Dinner Sukiyaki - the travel shows never showed that at some places, pay at machine

7-11 had mostly people looking for convenience dinner

Day 13 Tokyo
Breakfast YMCA turnip/fish broth soup rice kimchi mini mushrooms
Always pay somewhere else first

Lunch at Hamamatsucho bus station cabbage salad

Everybody dresses the same. Everybody. Black pants, white collared shirt black or blue business suit. All work people.

2 Indian restaurants

Bus tour - 12m Tokyo, 30 area, 127m country
WTC
7-11 covers Tokyo too
Convenience stores, small restaurants everywhere
Clean clean clean
ATMs have english, but lots of places want cash or don't take
cc
Few bikes
Very modern concrete and steel bldgs (not much mirror siding)
Tour fact: most people follow both Shinto and Buddhist practices/beliefs

It raining constantly but people (and tourists) are still out.

Trains on time to the minute, and scheduled often

Chinese restaurants

Dinner at sushi manemura conveyor belt super excellent and cheap kind of loud 'not crappy'
People talk in restaurants, not on pub trans
Mistook tea for wasabi powder. Mixed with soy, really not the same

Review of Hanemaru ( in the Kitte building across from Tokyo station):
Conveyor belt sushi is a crass gimmick, like sprinkles on donuts or strobe lights at fireworks show.
But there's also the accompanying atmosphere and quality of the sushi. We were foreigners here but the conveyor belt was very practical for us, just take what looks good. Everything looked good.

And the war zone that was our table at the end with random stacks of miniplates and dribbles of soy sauce, all of us stuffed beyond imagination, only $15 a person. Luxury eating at regular prices


Day 14 Tokyo
Breakfast seaweed soup at y
Joypolis
Huge landfill developed area full of buildings k.

Urinals have video games, measures volume

Every so often a woman in a kimono or a young couple (just married)

Kabuki theater is such kabuki theater

Every joke has been made before

Dept store basements have the food shops

English (not just roman) on all transit.
Joypolis had written english instructions. Taxi drivers and info people and waiters very polite about no japanese speaking

Dinner selection of foods from food shops.

Day 15 Tokyo to Boston
TV shows have a small box in lower left or right of someone watching the same show and giving their reactions, just facial expressions

Cars just have a slightly different body shape.

Commercial Parking lot with space for two cars. That's it.

nobody has driers, everyone hangs laundry to dry

'Blonded' hair

House rooves some regular some 'chinese'. All small and shallow.

Taking pictures on a train is hard. By the time you realize there is something interesting to take, it's already gone.

Street and rail crosswalks everywhere and all used

I am what I eat or don't eat on purpose

I grew up in Richmond, VA, a southern town but not deep south. As I grew up (1970's suburban white), I never felt southern (or northern, or anything), and people outside of Virginia asked my why I didn't have an accent. I don't (sez I!) but every so often I'll let through a "y'all" or "hunnert" (for 'hundred') and think it is the height of articulate speech to say 'Et-lanna' for Atlanta (I just say 'lanna'). And as to food, I grew up on McD's hamburgers and Taco Bell and HoHo's and DingDong's and bagel and baloney sandwiches and frozen mini pizzas like everybody else in every suburb in the US. Sure, there were a noticeably greater amount of pine trees, below the top two inches of topsoil the dirt was orange, and I got chiggers (once, and only when I came back to visit as an adult).

But I now realize there were a number of things I ate (or decidedly didn't) that say 'Southern':

• barbeque - I never liked this as a kid but love it as an adult. The only experience of this I had as a kid was a drive up that my mom liked. I got the burger. But now I really appreciate it.
• cornbread - The kind I had as a kid was dry and dense, almost peppery, coming in small fingers made from a metal indented pan to make them look like corn cobs. What I consider edible cornbread now is almost cake, sweet and thick.
• greens - collard greens, mustard greens, turnip greens, creamed kale, I loved them all, cooked with with salt pork.
• fried chicken - my mom was into health and stuff, so she didn't make it with batter or skin. Popeye's is the best ever, but you can't have it too often without having a piece of your heart die. Literally.
• okra - never had it as a kid and rarely as an adult. Some people hate it and some love it. I don't care.
• brunswick stew - the best way to burn your tongue. Great from a can or made from scratch.
• mac and cheese - never had it as a kid (did I mention my mom was into health and stuff). I still don't understand what's so great about it. But this is southern?
• watermelon - it was around -all- the time as a kid. It was OK, but made a sticky mess, sort of like cotton candy but wet. also seeds are a pain. I'm still ambivalent about it
• Krispy Kreme donuts - like Kleenex, these are the canonical donuts. But I didn't care for them. Again a little messy. DD donuts are real donuts. Sorry, South.
• grits - hell yeah. As a kid I had the instant kind where you add boiled water, which if you add the a little grated cheese and hot sauce are great (ooh and a boiled egg).  Waffle House and diner grits have so much extra ... something... that they feel like something else. Not bad but just ... too much.
• roasted tomato halves - best thing ever at fancy buffet events (after all the others). Fried green tomatoes is just weird like some foreigner thought it would be a good idea for the title of a quirky movie, and then people thought it was an actual thing. It's not a thing. Roasted (red) tomato halves are a thing, a real thing that is actually good.
• sweet tea - ugh...who would put that much sugar in their tea? I mean, unsugared is a little too ascetic but sweet tea? ugh.
• crab cakes ... ugh.
• pecan pie - that's southern? I thought it was just pie. Anyway, banana pudding kicks its ass. Not that pecan pie is bad. Just if you had to choose.
• fried pickles - ? hunh? That's a thing? And a Southern thing? I've never heard of that. Ever. Why try to gild the lily?
• hush puppies - mmm. Only ever had them on vacation, Virginia Beach or Nagshead at a seafood restaurant
• sweet potatoes and yams  (there's a difference?) - ugh. These things look funny! And taste funny! No way! Potato or sweet potato? Take the potato.
• tofu - a versatile basis for any... haha, just kidding. That stuff is so bland, you add it to dishes to take away taste. Also, not Southern.
• pig maw - eww. pass.
• black-eyed peas, lima beans, just beans - I avoided these every New Year's. Ugh. It's not that they taste bad, it's just that they don't taste good.
• biscuits - these are southern? They're not universal? I have nothing against them, they're OK (when not greasy)
• chitlins - that's just a funny word that people use on old west shows. Nobody actually eats that stuff. 
• smithfield ham - as a kid, tasted funny, why would you ruin a biscuit sandwich with weird meat. Now I appreciate it a lot.
• apple butter - if this is at all a southern thing, it's great. It's great anyway.
• deviled eggs - also, this is southern? They're great, but don't have too many, especially if they've been lying in the sun

Some of these items were off the top of my head. Some I was reminded of by the wiki list. Those on the list that are not here I just don't recognize, didn't think of as southern, or just don't care about. Also, I find it funny to go to a 'soul food' restaurant and wonder what's going on because it's just food.

Effect size versus statistical significance

One of the major tropes in the p-value wars is the difference between statistical significance and effect size. The usual (important) observation is that you can have a calculation on data that results in very small p-value, meaning very high statistical significance, but very small effect. And often this can be effected by increasing the number of instances: the more instances the smaller the p-value can be guaranteed, that the phenomenon is really not due to chance, no matter how small the scale phenomenon actually is. This is not to say that the phenomenon is not real, just that the phenomenon doesn't change that much in one direction.

This difference is presented often laconically ("Using Effect Size—or Why the P Value Is Not Enough") as:

Statistical significance is the least interesting thing about the results. You should describe the results in terms of measures of magnitude –not just, does a treatment affect people, but how much does it affect them.(Kline RB)

This makes it sound like you have two things that can be presented, and one is much more important than the other. But it's a false dichotomy. You want both. The magnitude is descriptive stats - how big it is. In an experiment on n individuals, fish oil tablets increased memory performance by 10%. If you don't know the effect size, what exactly beyond 'better' do you know about the phenomenon? Statistical significance is trust - how (mathematically) representative the sample is of the population. You can claim something is better but can you really trust the claim?

It's very easy to see how to manufacture a high statistical significance but low effect size - increase the number of instances. In fact, as you increase n, almost all statistical tests asymptotically approach statistical significance (for real world phenomena). Chi-squared is the worst!

A consistent high effect size (over samples) leads obviously to high statistical significance.

But it is possible to have high effect size and low significance.

So in the end, it is not one or the other. Both should be presented. The effect size tells you how different the sample shows phenomenon is, and the p-value tells you how much you can trust the sample that showed the phenomenon.

Annoying Sciency Tropes: Big effing number

Here's a really annoying pattern that comes up in science journalism (OK really any news story that involves a number) and it involves so many fallacies and misdirections and insults to intelligence that I can't over underestimate it. It is the presence of a Big Number.

The yearly output of carbon dioxide gas into the atmosphere is 50 bajillion tons. Wow, that must be bad because a bajillion is a lot. (Also 'tons'. You can have a ton of air? (of course you can that's physics, but it is counterintuitive enough to simply leave the reader with the simple incoherent feeling of 'wow').

The number of deaths due to the Iraq War of 2003 was approximated at 600,000. Of course that is terrible (any such death is terrible). But is it reliable? Is the scale right? How was the number arrived at? What groups are in that number? Is it overcounted? Undercounted? Adding a zero hardly changes the impact of the story but is still wildly inaccurate.

Million, billion, trillion are hard to distinguish. They're mostly 'really a lot', 'really really a lot', 'that sounds like a lot'.

I realize I'm giving these without context, but the point is that often news stories lack all context too.

There's a little bit of technical obscurantism going on (is a nanometer bigger or smaller than a picometer?) which expects education; that is, it is questionable whose fault this is, the one using the technical term or the one reading it. If the reader were educated, this is the best most accurate communication, what technical language nuances are created for. If the reader is not educated in these nuances (which are not nuances to the initiated), then what?

Part of the annoyance is that this is usually combined with a Base Rate Fallacy; usually no comparison data is given - no comparison with the total or comparable items, no context. For example, the debt of the US government is given (latest number) in news stories as $14 trillion. Obviously this is a big unfathomable number, but also there is nothing to compare it with, either the historical debt (what the trend has been over the past few years, what the debt in other countries is like).

What's the solution? For the reader, look outside the article for the base rate or trend. For the writer, supply that! Give something to compare with.

Bullshit

Bullshit. There's no better word for things that sound true or plausible but have no connection to or no support in reality or any attempt by the speaker to make that connection. The metaphor is weak but it captures the feeling of the realization about what someone else has said.

Usually a statement is called bullshit if it stretches the bounds of plausibility. If it turns out to be a falsehood, it is considered a lie. Harry Frankfurt wrote an entire book on the subject, trying to solidify it (ugh, metaphors) as a statement by a person who is not intentionally trying to lie but rather has no concern at all for its truth value. That is, a bullshitter doesn't care whether a statement is true or false. Frankfurt is a philosopher and is trying to shoehorn a word into his own internal concept or into one that is more logically amenable, which is to say I think that usage is charitable. A bullshitter is trying to lie; if the statement turns out true, they would be surprised.

But the label is problematic. Bullshit is a little taboo (or a lot given the context). What are the alternatives? There are many but they have their problems, too.

All the following words fit the syntactic pattern, in response to something said, "That's X".

The first category is the anachronistic nonsense words:

• balderdash
• bosh
• bunk
• bunkum
• claptrap
• codswallop
• fiddlesticks
• flapdoodle
• folderol
• guff
• hokum
• hooey
• malarkey
• piffle
• poppycock
• tosh
• tommyrot

What do any of these words mean? They're entirely opaque nonsense words, made up by someone long ago out of random sounds to sound like what they're describing. Another thing they all have in common is that currently (and for the past fifty years at least) no one in their right mind would utter these in sincerity, given that they sound like an old man smoking a cigar mincing an oath. All of these words are idiomatic, inexplicable. This description is almost the meaning, which may be the originally intended psychological effect.

Moving closer to reality, the next is the most populated category (and likely most thought of if not used), the actual shit category. Shit itself really isn't (metaphorical) bullshit - metaphorical shit is just worthless, but metaphorical bullshit is a damned lie. Many are minced (crap for shit), but at least they are analyzable.

• bullshit
• crap
• crock (of shit)
• horseshit
• horsetwaddle
• horsefeathers - minced horseshit, so to speak
• twaddle

Now there are real, relatable but still not literal, for mixed up messes, trash or sausage metaphors:

• garbage
• hogwash - sounds like nonsense to me but actually is pig slop.
• rubbish
• baloney

Actual literal non-onomatopoeic words that are close in meaning but just not quite bullshit and still somewhat old fashioned:

• blather
• bombast
• drivel
• trumpery

And finally the list of actual, literal words:

• nonsense

Of all the lists of related words I've seen for bullshit, 'nonsense' is the only direct literal word for it.

And that is the real problem here. Bullshit is not nonsense. Nonsense is words that make no sense. Bullshit makes sense, may be true or not, but may be misleading. Word salad (a salad made of words), gibbering (of an idiot), unconnected train of thought (of someone distracted) are all nonsense. Bullshit makes sense entirely. Just the intention (or reception) is different from usual truth valued statements.

Therefore there is no good alternative to bullshit.

Flounder vs Founder

In the series 'words almost spelled the same and almost mean the same thing, but are not'

Both mean many things but they come closest as things that happen to you metaphorically relating to the sea.

Flounder is the flat fish, and to flounder is to be like a flounder on the deck of a ship and flail about.

Founder, on the other hand is one who starts something (very different!) because they are at the base of things (cognate with the foundation). 

But to founder is not to found something but to begin the process of sinking, to founder upon the shoals. 

Certainly a flounder could founder on the shoals if it put itself into such a bad position, but that is less likely than that a founder of an enterprise would flounder before pivoting to a greenfield market (count the mixed metaphors!).

The etymology of founder is incontrovertibly via French fond from Latin fundus, the bottom. Flounder, the fish, supposedly is cognate with flat (obvious) and plaice (obvious biologically but not immediately obvious phonetically). Despite my metaphor about a fish flailing about, to flounder however has a controvertible provenance, probably mixed up with other similar sounding words, like flop and flail and fluke (is that another fish) in an example of phonosemantics.

So flounder, fish, flop. Founder, sink to the profound bottom.

If you know a word, you can't unhear it

Horse.

Big animal with a mane and long tail, big distinctive head large body with long thin legs. That word can't be anything else.

For a native speaker, any word in your language is like an automatic switch. It says what it is, there is no other thing it could be but what it says. It stuffs itself into your mind and there's no unstuffing it.

When learning a language, a word has this ethereal feel. It could mean something else, it could mean, well I'm not sure, is it some sort of ... no, is it ... bread? I'm sure it was an animal...or  maybe not.

This goes for syntax and phonology too. Lots of leniency as a learner, there is no wiggle room at all for a native speaker. For a native speaker the slightest deviance sticks out as something entirely different. Suppose you mean 'I bet a dollar', but you said 'I bit a dollar'. Why would you bite a dollar, that is crazy! Even if you're at a gambling table, people would get all bent out of shape. "Did you mean 'bet' or 'bit'. No one bites dollars here they only bet. Did you really bite a dollar?".

Insidious vs invidious

In the series 'words almost spelled the same and almost mean the same thing, but are not'

Insidious and invidious.

They both sound bad. One sounds like...well the other does, too. But they are distinct.

That snake slithering up towards you unannounced? Insidious. Shaking a snake in your face? Invidious. Both are pretty mean. Insidious is stealthy or under the radar. Invidious is plain ill will.

Insidious describes something that lies in wait to get you, and invidious is something offensive or defamatory. Cancer can be insidious, lurking in your body without your knowing it. Invidious doesn't hide; it's hateful right away.

Insidious didn't fall too far from the tree – it comes directly from the Latin word insidious meaning "deceitful, cunning, artful," from insidiae "plot, snare, ambush." That's pretty unannounced. Something insidious can even be attractive while doing harm, like an insidious plot to befriend your crush's girlfriend, so you can break them up. But often it's not attractive, just sneaky.

Invidious on the other hand is from the same place as envy. But it has slid over to mean just plain ill-will, where envy might come in but is not necessary.

You can be invidious in an insidious manner (being sneaky about your distaste for the other). But I think insidious carries enough negative feeling in there that it already includes the attributes of invidiousness.

More Star Trek Tech Progress

In a previous post, I listed a number of science fiction technologies used in the original Star Trek series and the progress our current society has made towards making them real. I also commented on how quickly we might achieve them.

I was just watching some reruns the other day...sorry, let me rephrase that in contemporary language in ephemeral tech language... I was watching a few of them them on Netflix (how long will the tech assumptions behind that phrasing last) and I realized that I had left out a number of things, some very obvious, some that seem like throwaways now. They are hidden, outside of the script but somehow mostly necessary assumptions for the plot.

- needle-less injections - Someone needs to be sedated? They're writhing in distress? With the 'hypospray' (which I suppose is hyperdermic), no need to be restrained when you don't have to find a vein. Just inject them without a needle via some transcutaneous mesh.

In some sense, the current transdermal patch effectively delivers chemicals to the blood stream but has its own limitations. Both aren't particularly effective for medications that need to be administered directly to the bloodstream for quick systemic dispersal (putting a bolus in your butt is slower release to the blood stream).

- earth diseases entirely solved, alien ones solvable in days with at-hand tech - every other episode someone contracts an alien disease. The other episodes some one is phasered in the face or breaks a leg, but they walk out of sick bay as chipper as ever. Medical science slowly does this, almost like this ST tech list, some things get solved perfectly (antibiotics?), some there's progress (chemotherapy), and some you solve well by cheating smart (colonoscopy screening). But in the show, any new unsolvable medical problem takes a long 48 suspenseful plot hours to solve.

- anti-gravity - elevators, walking on flat floors. That's just crazy. The only 'anti-gravity' is gravity from the other direction. But really, they need seat-belts even with anti-grav floors. For something like this which is technically impossible, a lot of tech cheating can go a long way

- deflector shields, force fields - a prison cell is fronted by it seems a painful invisible electric field. And the outside of the ship is protected from alien explosions (and the often forgotten cosmic particles and dust hit at interstellar speed). I presume if you can do this for a door way, you could do it for the outside of a ship so that the same tech would work for detention as well as protection.

- automatic pocket doors - doors are more privacy devices than security. They all seem to know when you're going to walk into any particular one otherwise like kids playing at the entrance to a grocery store, every single door would whoosh open as you walk along the hallway.

- bloop-bleep sounds - special futury sounds that alert you to danger or notification. These sounds were invented for the show (the communicator opening sound, the door opening whoosh).

Door opening sound
Red Alert
Tricorder scanning

OK, these are just fun. They're pretty much 1960's '101 Fun Experiments with Transistors!!'.

- the real time vital stats monitor - this is entirely doable, or even done, today. Maybe just badly. We can totally do this now. Get those ST set designers to work on the UX for the ICU/OR patient monitors!

Maybe the fancy scifi bit is in not having loads of tubes all trailed all over your body. That's practically possible now with bluethooth wireless.

- 'subspace' communication - this is very inarticulately presented in the shows. Sometimes they can communicate instantaneously over long distances, sometimes it is just somewhat faster than the ship. The physics of this is unclear to me. Somehow I expect that for the ability to transfer knowledge, there is some physics theorem that says that it is restricted by the same principles as the speed of light (i.e. if you could transfer knowledge that fast, you could have consequences (I don't know!) that would be equivalent to faster-than-light travel.

(oh almost forgot, bluetooth earpiece, pfft done that already)

Frankly there is so much in the original series that I'm sure I've missed even some big ones.
---

Much of these imaginations were on purpose, but then some were just natural tweaks (needleless injection) or even unconscious assumptions (antigravity).

As an aside, social progress is interesting to watch, both in the invented extended world timeline of ST (TOS, TNG, DS9, Voy, Ent) and the real one. It's almost grating how tin-eared white-male-centered TOS appears now in comparison to the less presumptuous later series. Ent itself isn't as embarrassing as TOS even though Archer is (again the captain) an alien-femalizer and his two male subordinates are always whooping it up with the brosplaining... ok maybe it is intentionally embarrassing (but Ent is only 150 years in the future, 150 years behind TOS you'd think there'd be a little more progress)?

I'm so sorry, Clippy

Clippy was Microsoft's 'intelligent' assistant that they added way back to MS Office 97.

Holy crap is that goofy looking! And those animated eyebrows were even goofier! And then he'd ask you really annoying questions like "Do you need help with that?" to things you don't need! Gah!

That was the general reaction. Clippy was universally mocked. Reviled. Other bad things.

But really, was it so bad?

Lately (20 years later) it seems that everyday a new 'assistant' is coming out. Siri, Cortana, Alexa, chatbots up the wazoo. Clippy asked you more questions, based on what you were doing and had answered already, just like a good chatbot. You didn't write to it natural language, but the intention was the same, help out the user by asking questions back and forth.

It was context dependent (Siri is not). It followed a decision tree to maintain context (most current chatbots have trouble with context). It doesn't use speech (like S/C/A) or even typed natural language (chatbots), but it was still an assistant to do things you weren't sure about doing.

Clippy didn't try to do NLP, but frankly the non-speech part of S/C/A and chatbots, just the text, is mostly picking out keywords using hardcoded scripts, not terribly different from hardcoded trees.

So what was so bad about Clippy?

The idea was right, lead people through tasks they are unsure how to do. The implementation as decision trees was both easy and well integrated. What the decision trees helped with...well maybe that wasn't so great. I know I turned off Clippy immediately because I knew what I wanted to do (haha no comment people from twenty years ago!).

But was that the main problem?

I believe the main problem was... the first thing I said, the superficially distasteful graphic and its smarmy animation. It wasn't with the suggestions (though those may have been too simple) it wasn't the use case (helping to write a letter) which was surely too simplistic, but it was just never pursued (unlike the more popular and well-deserved complaints about MS PowerPoint's Auto Content Wizard, a not too dissimilar concept to the assistant).

The problem was the front face of the feature. A weird laughable graphic. I'm sure the idea team thought it was cute and would draw people in with its informality. Fashion is hard to gauge. But the universal response of cringing was no fashion statement. It's hard for those people inside the design team/the makers to see what outsiders see. And sometimes organizations can be too...polite (I know I know, crazy. Sometimes people just don't tell you what they really think).

The major lesson of Clippy is that when designing an assistant/chatbot,  first avoid smarmy (and get honest opinions). The second, which is not a lesson that people normally take from Clippy because they already turned it off because of the first point, is to make the assistant relevant (I know that's a bit broad). That is, make it help people with things they really want. SOmetimes wording makes a big difference "Do you want help writing a letter?" Of course not , I know exactly what I'm going to say to that lawyer! "Do you want help formatting a proper legal letter?" Oh. Right. Yes. Where does the return address and salutation and letterhead go? Yes, that is what people want to be asked for and helped with. So even the subtlest wording can make a difference. The third lesson? Remember the context of prior questions. It just makes things easier for a human.

But what really killed Clippy? Just the graphic. Nothing else, whether substantive or not, really came into play.

Meetings are the worst

If I had to identify, in one word, the reason why the human race has not achieved, and never will achieve, its full potential, that word would be 'meetings'.- Dave Barry

There are different kinds of meetings. Some are to make other people aware of what has happened, some are for making a decision together, some are for getting things done, some are for brainstorming, some are highly structured, some are for meeting people you haven't met before.

Sometimes you can just show up at a meeting and wing it if called on. Sometimes you have to be prepared for a meeting. Either way, for some time before the meeting, whatever you're doing, you're thinking in the back of your head 'argh, a meeting, what will I say about progress on widgets?'.

Who likes meetings? The ones who call them. Organizing and leading a meeting makes you feel important. It's like a club, an exclusive club that other people aren't in. "Why wasn't I invited to that meeting? That affects me! I would have liked to say something!"

I find that the proliferation of meetings themselves comes out of a more modern desire to be considerate, transparent, and to be inclusive. You don't want to annoy non-key players. 

But it seems like there are always too many meetings. "My schedule is full, never time to actually do anything."

Some people are primarily managers and some primarily doers. A manager's job _is_ communication so their job _should_ be meeting with people. A doer is doing something (engineering the deployment, tweaking the spreadsheet, reminding clients about payments, making the training video). See this blog post:http://www.paulgraham.com/makersschedule.html. And they should be spending most of their time doing. Communication is necessary, but it cuts into time doing things.

Also, we're all a mix of these two. You may be a CEO and need to talk to the other execs, and clients, and investors, and give presentations. But you also need to prepare that ppt and practice your speech. And even if you're the assembly-line widget-53-orientation-setter-class-3, you still have to spend time discussing issues with your neighbors off-line.

Consider these tactics to make meetings shorter/unnecessary:

- have a clearly stated agenda (even as a document) with clear goals or if it is intended to be a discussion have boundaries and stick to them. An amorphous or non-existent agenda just means people will talk for an hour with little outcome.

- don't be shackled by hours and weeks. modify the schedule to once every two weeks, or just 1/2 hour (or both, or less).

- always try to end a meeting early (don't feel like the time has to be filled out). No one ever leaves a meeting early thinking they wished it could have gone on longer. If the end of the agenda has been finished, don't stretch it out, it's done. If there are any further questions. 

Some people spend all their energy trying to figure out when is the last moment to jump in with an idea or question and it may come at the end.

- assign a non-key participant to be a facilitator (this is difficult to get, to perform, and loses that person's time)

- stick mostly to guidelines but be lenient. If a talk veers off the agenda (a problem is being solved but the meeting is about discussion), then schedule a meeting for the problem solving -or- be open about modifying the agenda (just stating an agenda item is often enough and details haven't changed).

- someone, not a key-player, should take notes (could be facilitator)

- record action items discovered.

These last few tend to increase the energy spent on a meeting. They're intended to increase the worth of the meeting. If a meeting's importance doesn't call for this prep, then maybe it doesn't have to be that long or have that many people.

"Probabilistic programming languages" aren't

I'm looking for truth in advertising: "Probabilistic Programming Languages" or PPL is a recent term to describe a very useful new paradigm in statistical modeling computing... but it's not a set of new languages. It is much more restricted to very particular kind of statistical modeling as opposed to the broad sense of probability.

As someone who likes a little consistency in language use, for words to have meanings you can mostly rely on, I am bothered by this usage (just as I'm bothered by the similarly mystically enticing marketing term Deep Learning). Here is a  very representative description of PPLs:

Probabilistic Programming (PP)
There’s a revolution in Computer Science called Probabilistic programming (PP) where programming languages are now built to compute with uncertainity in addition to computing with logic. This means that existing programming languages can now support random variables, constraints on variables and inference packages. Using a PP language, you can now describe a model of your problem in a compact form with a few lines of code. Then an inference engine is called to automatically generate inference routines (and even source code) to solve that problem. Some notable examples of PP languages include Infer.Net, Stan, BUGS, church, Figarro and PyMC. In this blog post, we will access Stan algorithms through the R interface.

from a blog article on PPL (which also tries to introduce new but uninformative terminology, MPML).

I expect words to mean things, and despite liking metaphorical usage in literature and expository writing, not calling a technical thing what it is sounds too much like slimy obscurantist marketing practice. If it is misleading in any way, it is suspect. Suspect maybe not in venal terms, but more likely suspect in intellectual depth.

For the record, the difficulties in the passage above are:

• There's no revolution, not in computer science, not in programming languages, not in AI. Maybe there's some recognition that there is some progress in usage, but it is incremental.
• No new programming languages are being built. No programming languages are being modified to accommodate new probabilistic data types. This is the biggest clunker. There's no new programming language thing at all. What is new is packages or libraries or functions, in the existing programming languages. PyMC is a library written in Python, and used in Python as native Python. Stan is written in C++ but it is not a syntax/semantics, just a library that is accessible from existing languages (R, Python, Matlab, Julia, etc).
• The idea of operating on distributions as a type is not actually new. Mathematica and Maple have had object oriented implementations of distributions, allowing operating on those distributions functionally. What these PPL packages add is approximation algorithms to compute values for Bayesian inference using Markov-Chain Monte-Carlo (MCMC) , which is fancy talk for calculating a number approximately. Pretty much very analogous to computing a p-value.
• All these PPLs are just library add-ons to existing languages. So in that sense don't worry that you have to learn a new syntax. You surely will have to learn how to use the library.
• It's not about probability in the large. Most all languages have probabilities already (restrict floats to the range 0:1). Some people are creating packages that make it more easy to use probability distributions (which some languages already had libraries for), and to manipulate those distributions (and make statistical inferences from them. But, no, it's not a revolutionary new alternative to languages with logic using probability. It might be a revolutionary library of functions that will make manipulating and computing with distributions and models easier, nut it's not a new language.

A programming language is a syntax and semantics and a compiler. A library is a set of functions written in a programming language usable by programs written in that programming language. One language can use functions written in another programming language as long as there is an interface.

To call a programming language probabilistic (or extension to an existing language that would render in my eyes the extension to be probabilistic), there would have to be a basic data type, like integer or boolean, that corresponds to a probability distribution (for the purposes of efficiency in compilation).

This is certainly a diatribe about naming. I have no qualms about the use of these PPL libraries. It's just the name. Distributional Modeling Libraries may be more accurate and doesn't have the same punch but is not actually incorrect like PPL. But if you're advertising faster-than-light speed, those words come with a lot of meaning, and should actually provide that without a lot of qualifications.

Words that are almost identical in spelling and meaning but are not

Some words have a huge semantic overlap that there's hardly any distinguishing them. Bucket or pail? Is there any instance where one of these is right and the other obviously wrong? (probably, but rare).

Some some pairs of words are really hard to differentiate but they have distinct meanings, if only you study the dictionary. They have considerable vague overlap, but a verifiable situations where one or the other just don't apply technically.

First in a series:

tragedy and travesty:

That poor dog dying. Was it a tragedy or a travesty? It could easily be both. A tragedy is a particularly bad outcome. Most everybody dead at the end of Hamlet (sorry, spoiler!); that's a tragedy. A travesty is a horribly distorted copy of something. The elementary school play of the Cherry Orchard had no sense of anomie; that's a travesty but not really tragic. A travesty is often tragic; a travesty of justice ended up sending the falsely accused to jail resulting in the tragic demise of his entire family. There are tragedies that aren't a travesty. Most people dying are tragic, but really that's sort of the order of things eventually. There are travesties that aren't tragic; that elementary school play. Tragedy: Hamlet. Travesty: School play

Irish Words in English

Irish is obviously overshadowed by English culturally. So there are lots of vocabulary items in Irish that came from English. But it goes the other way too. There are a handful of terms that were borrowed into English from Irish: the Saxon overlords ('Sasanna' = England in Irish) picked up a few things from the Celtic underlings (pronounced 'keltic' with the Irish 'C').

Yes, in the following even though there is a link, some liberties are taken. So maybe it doesn't come directly from Irish but via another language that eventually borrowed from Irish. Or maybe. Or maybe it was just sort of maybe made up to sound Irishish. But most of this holds up.

banshee  From Irish for 'screaming spirit'. Haha, no that's what it is in English. In Irish, it is 'bean sidhe', meaning fairy woman.

britches  From Irish for pants. Well, not exactly, but it is the Irish word for pants so that's good enough for me for it to be Irish origin. 

brogue  An Irish accent, usually in the phrase 'an Irish brogue'. 'brog' means shoe in Irish, so they must be especially noticeable kinds of shoes to have turned into a symbol of Irishness.

clan  From Irish 'chlann' children, clan

clock  From Irish 'clog' clock. A tortuus path through other languages with Irish involved.

galore  From Irish 'go leor' enough, a lot. Often paired with whiskey. But works for anything in English.

hooligan  yob, rowdy young man, From ... well it's probably made up from Englisher's mispronouncing the surname 'Houlihan'. Like all Irish have the last name Houlihan. 

phony From Irish 'fainne' for 'ring (for your finger)'. Presumably costume jewelery.

pet  From Scots Gaelic 'peata' for tamed animal.

shanty Official sources say it's from French 'chantier' for cabin, but there are claims for Irish 'sean tig' for old house (which is not good Irish grammar). Fine, but it has that Irish sounding '-y'.

shenanigans  From ... probably also made up to sound Irish. But I count it!

slew From Irish 'sluagh' a big number of people

slogan From Irish 'sluagh-ghairm' for war cry (by a big number of people) 

smithereens  From Irish 'smidirin' small fragments    

spree - From Ir 'spraoi' for spree.

trousers From Scots Gaelic treu for pants. Scots counts.

whiskey  From Ir 'uisce' water. In Irish, whiskey is 'uisce beann' or water of life.

You'd think there'd be more!

What's most important for success in X?

On a very particular Q&A site, I saw the question "To what do you attribute your success?". A very broad question, intended for a very narrow audience, but any answer should apply generally. Many answers say something like 'tenacity' or 'embrace change' or 'have clear goals'. Which are all good. But here is my answer:

- good idea - addresses a real existing need in a way no one else ever thought of in just that way

- hard work - you can't just sit back and let people recognize your brilliant idea. Hard work can be either doing more (of the same stuff) or doing different (thinking harder)

- strategic details - getting the nuances just right can make the difference, that one tiny little thing, the subtle finishing touch whose lack

- good team - well prepared, good skill coverage, good connections

- luck - there are some many things that you don't know and can't know, historical accident, the whims of people, unexpectedly meeting an old friend

Any one of these can ruin things, but also any one can put you over the top into success. Sometimes you -can- sit back and let it run if you get real lucky (this is -very- rare, you're not -that- lucky). Often trying harder can compensate for bad luck.

The usual question is which single one is the most important, a good idea or a good team, or personality vs substance. The answer is always yes, but one can often compensate for the other.

Which one is the most important? The point of question like this is to determine which single one can you put your efforts into. Obviously you shouldn't neglect any but some are difficult to change like luck, but that's why you need to be well-prepared for when the opportunity arrives (or the disaster occurs).

EBM, precision medicine: literal but more

Often the metaphorical meaning of a word is used more often than the literal one.

Medicine is a technical domain, but lots of terms in it are metaphorical.

EBM, or evidenced based medicine, is a label for a suggested desire to literally base medical practice on evidence rather than 'what you've always done'. Does extra sugar intake cause hyperactivity in children? It's pretty obvious it does. Except when studies are done, there is no appreciable difference in activity afterwards between children ingesting more sugar and those who don't.

We have preconceived plausible notions, but it's always good to check more scientifically. Isn't medicine always working on evidence, and equally dismissive of unscientific, non-evidenced based things like homeopathy, or ingrained myths like feed a cold starve a fever? Of course, but still there are reasonable plausible things that just may not have an actual effect. EBM came to refer to a trend in RCT, randomized controlled trials, which means a trend in a particular kind of government funding. Also it became associated with expensive measures to confirm really idiotically obvious things and parodied by the idea of an RCT for the efficacy of parachutes. So sometimes EBM sounds like a good thing, and sometimes it sounds like a dumb thing. But it mostly means 'be skeptical, do an RCT', whatever the nuances of funding and sample size are.

Personalized medicine, as a term, is also problematic. Literally it's saying medicine should be directed towards the individual differences. But that's so obvious, you're not going to treat someone for a broken left arm when it's their right that's broken. How people use the term 'personalized medicine' nowadays is for when gene variations are known about the patient. That's it. PM, the non-literal version, is for the handful of medical situations where the different gene variation (on a small set of genes) suggests a slightly different therapy. The expectation is that the science will expand to include lots of gene variations and problems. Yes, doctors have been using medicine personalized to a patient's family history, environment, social situation, problem itself (duh!), etc. forever. PM currently refers to doing that same thing but with some gene knowledge.

These two terms, EBM and personalized medicine, are not incorrect, but they have a much more specific meaning than you think if you've never heard them before. If you use them all the time, then you (implicitly or not) know their narrow usage. Luckily people whether they know it or not, don't use these in the broader situations.

SQL JOIN Venn diagrams are only sort of Venn diagrams

SQL is a standard for querying databases. Despite questionable pronouncements that SQL is Turing complete, I hesitate to call it a language because its power is in using boolean logic in dealing with tables of data whose columns point to each other.

And often Venn diagrams, the go-to visualization for set operations, are used to help explain the process of table JOINs.

The interesting things is that set operations and table joins are not really the same thing. They're related but just not the same. Set operations, which are pretty much the same as boolean/logical operations, are simple to visualize. The picture is the universe of elements, a circle surrounds a group (a set) of elements with a property, and a set operation does something to one or more sets to make a new set.

(from Modern Dilettante)

SQL also has set operations that combine tables as though they were sets: UNION, INTERSECTION, DIFFERENCE. They simply do the same as the set operations; two tables with identical column labels have their rows combined into a single new table (UNION means all rows in both, INTERSECTION where the column/row entries match in value, etc).

But this is not how Venn diagrams are usually presented to explain SQL. UNION, INTERSECTION, etc, are not the most useful of operations (the WHERE clause of a SELECT is where the booleans are most commonly used). Venn diagrams are most often used to explain JOINs. A SQL JOIN first matches on a field from one table and a field from another (presumably a field of the same type or kind).

(source Codeproject)

These Venn diagrams explain the difference between inner, outer, left and right joins perfectly...except they are just from a different world than the traditional set operations.  A JOIN is intended to merge the information appropriately in the n by m relation (where the size of A is n and size of B is m). The universe isn't the set of rows of both A and B together. The universe is the product of rows in both. And the difference between inner, outer, etc, is purely with how the JOIN deals with NULL/missing elements in A or B.

An INNER JOIN keeps rows of AxB only where both A and B rows exist. A LEFT JOIN is only when the A part exists (B may or may not), similarly for RIGHT JOIN. An OUTER JOIN doesn't care if either a corresponding A or B exists. So the boolean idea does apply but in a strange way, only with respect to the NULL condition of the matching field. If the value of the field from A has no matching value in the field for B, then B is NULL or missing then (and vice versa).

So the Venn diagrams for SQL operations, I can't really say they are true Venn diagrams; they don't show the state of a consistent property over all elements of the universe. Or rather the universe is a bit more complicated (depends on A and B, their cross product) and the property being booleanized is whether element of one table is NULL. You can't just take an arbitrary universe of elements (with properties. With JOINs, you have to create the universe, the product, first before examining the elements (and whether the A part or B part of the new row is null or not.