# Grouping Rows of a Matrix

I spent a large chunk of yesterday afternoon doing something I thought would be simple (relatively speaking) in LaTeX. I wanted to group rows of a matrix (actually, in my case, a vector) with right braces, and label the groups. An example of what I wanted is in the image below. This seems to me …

More

# Big M and Integrality Tolerance

A change I made to an answer I posted on OR-Exchange, based on a comment from a well-informed user of OR-X, might be worth repeating here on the blog. It has to do with issues that can occur when using “big M” type integer programming models, a topic I’ve covered here before. As I mentioned …

More

# A Brief History of the (Near) Future: How AI and Genomics Will Change What It Means To Be Human

I’ll be giving the talk below to an audience of oligarchs in Los Angeles next week. This is a video version I made for fun. It cuts off at 17min even though the whole talk is ~25min, because my team noticed that I gave away some sensitive information :-( The slides are here. A Brief …

More

# Piecewise Linear Approximations in MIP Models

In the past, I’ve written about piecewise linear approximations of functions of a single variable. (There are too many posts to list here. Just type “piecewise linear” in the search box of my blog if you want to find them.) Handling piecewise linear approximations of multivariable functions is a bit more intimidating. I’ll illustrate one …

More

# How NSA Tracks You (Bill Binney)

Anyone who is paying attention knows that the Obama FBI/DOJ used massive government surveillance powers against the Trump team during and after the election. A FISA warrant on Carter Page (and Manafort and others?) was likely used to mine stored communications of other Trump team members. Hundreds of “mysterious” unmasking requests by Susan Rice, Samantha …

More

# US Needs a National AI Strategy: A Sputnik Moment?

The US needs a national AI strategy. Many academic researchers that could contribute to AI research — including to fundamental new ideas and algorithms, mathematical frameworks for better understanding why some algorithms and architectures work better than others, etc. — are not able to get involved at the real frontier because they lack the kind …

More

# More on “Core Points”

A few additions to yesterday’s post occurred to me belatedly. First, it may be a good idea to check whether your alleged core point $$y^0$$ is actually in the relative interior of the integer hull $$\mathrm{conv}(Y)$$. A sufficient condition is that, when you substitute $$y^0$$ into the constraints, all inequality constraints including variable bounds have …

More

# Finding a “Core Point”

In a famous (or at least relatively famous) paper [1], Magnanti and Wong suggest a method to accelerate the progress of Benders decomposition for certain mixed-integer programs by sharpening “optimality” cuts. Their approach requires the determination of what they call a core point. I’ll try to follow their notation as much as possible. Let $$Y$$ …

More

# Gork revisited, 2018

It’s been almost 10 years since I made the post Are you Gork? Over the last decade, both scientists and non-scientists have become more confident that we will someday create: A. AGI (= sentient AI, named “Gork” :-)  See Rise of the Machines: Survey of AI Researchers. B. Quantum Computers. See Quantum Computing at a Tipping …

More

# Ordering Index Vector with Java Streams

I bumped up against the following problem while doing some coding in Java 8 (and using streams where possible). Given a vector of objects $$x_1, \dots, x_N$$ that come from some domain having an ordering $$\le$$, find the vector of indices $$i_1, \dots, i_N$$ that sorts the original values into ascending order, i.e., such that …

More

# Quantum Computing near a Tipping Point?

I received an email from a physicist colleague suggesting that we might be near a “tipping point” in quantum computation. I’ve sort of followed quantum computation and quantum information as an outsider for about 20 years now, but haven’t been paying close attention recently because it seems that practical general purpose quantum computers are still …

More

# Recursive Cortical Networks: data efficient computer vision

Will knowledge from neuroscience inform the design of better AIs (neural nets)? These results from startup Vicarious AI suggest that the answer is yes! (See also this company blog post describing the research.) It has often been remarked that evolved biological systems (e.g., a baby) can learn much faster and using much less data than existing artificial neural nets. …

More

# Creating A New MIME Type

I struggled a bit this afternoon creating a new MIME type and associating it with a particular application, so I’m going to archive the solution here for future reference. This was on a Linux Mint system, but I found the key information in a GNOME documentation page, so I suspect it works for Ubuntu and …

More

# Robot Overlords and the Academy

In a previous post Half of all jobs (> \$60k/y) coding related? I wrote In the future there will be two kinds of jobs. Workers will either Tell computers what to do or Be told by computers what to do I’ve been pushing Michigan State University to offer a coding bootcamp experience to all undergraduates who want …

More

# Benders Decomposition with Generic Callbacks

Brace yourself. This post is a bit long-winded (and arguably geekier than usual, which is saying something). Also, it involves CPLEX 12.8, which will not ship until some time next month. I have an updated version of an old example, solving a fixed charge transportation problem using Benders decomposition. The example (using Java, naturally) is …

More

As I noted in yesterday’s post, one of the major changes associated with the new “generic” callback structure in CPLEX is that users now bear the responsibility of making their callbacks thread-safe. As I also noted yesterday, this is pretty new stuff for me. So I’m going to try to share what I know about thread …

More

# CPLEX 12.8: Generic Callbacks

IBM is getting ready to release CPLEX 12.8, and I had the opportunity to attend a presentation about by Xavier Nodet at the 2017 INFORMS annual meeting. Here are links to two presentations by Xavier: CPLEX Optimization Studio 12.8 – What’s New and CPLEX 12.8 – the Generic Callback. As with any new release, there …

More

# What “R” qualitative research methods?

I recently stumbled upon a post on R-bloggers entitled “Qualitative Research in R.” This title got me pretty excited, since I’m generally excited about most things R and since I recently helped teach a qualitative methods course, which has had me thinking about adding more ethnographic and other qualitative elements to my work. I’d also heard of …

More

# AlphaGo Zero: algorithms over data and compute

AlphaGo Zero was trained entirely through self-play — no data from human play was used. The resulting program is the strongest Go player ever by a large margin, and is extremely efficient in its use of compute (running on only 4 TPUs). Previous versions of AlphaGo initially trained on thousands of human amateur and professional games …

More

# Information Theory of Deep Neural Nets: “Information Bottleneck”

This talk discusses, in terms of information theory, how the hidden layers of a deep neural net (thought of as a Markov chain) create a compressed (coarse grained) representation of the input information. To date the success of neural networks has been a mainly empirical phenomenon, lacking a theoretical framework that explains how and why …

More

# Where are participants in American and Canadian teacher hashtags?

My dissertation research is focused on Regional Educational Twitter Hashtags (RETHs), which are teacher-focused hashtags that are associated with particular geographic regions, such as American states or Canadian provinces or territories. This isn’t the first time that I’ve done a project on this phenomenon, and it’s rewarding to come back to RETHs to answer some questions that …

More

# A Gentle Introduction to Neural Networks

“A gentle introduction to the principles behind neural networks, including backpropagation. Rated G for general audiences.” This very well done. If you have a quantitative background you can watch it at 1.5x or 2x speed, I think :-) A bit more on the history of backpropagation and convexity: why is the error function convex, or nearly …

More

# Minimizing a Median

$$\def\xorder#1{x_{\left(#1\right)}} \def\xset{\mathbb{X}} \def\xvec{\mathbf{x}}$$A somewhat odd (to me) question was asked on a forum recently. Assume that you have continuous variables $$x_{1},\dots,x_{N}$$ that are subject to some constraints. For simplicity, I’ll just write $$\xvec=(x_{1},\dots,x_{N})\in\xset$$. I’m going to assume that $$\xset$$ is compact, and so in particular the $$x_{i}$$ are bounded. The questioner wanted to …

More

# DeepMind and StarCraft II Learning Environment

This Learning Environment will enable researchers to attack the problem of building an AI that plays StarCraft II at a high level. As observed in the video, this infrastructure development required significant investment of resources by DeepMind / Alphabet. Now, researchers in academia and elsewhere have a platform from which to explore an important class …

More

# Updated Stepwise Regression Function

Back in 2011, when I was still teaching, I cobbled together some R code to demonstrate stepwise regression using F-tests for variable significance. It was a bit unrefined, not intended for production work, and a few recent comments on that post raised some issues with it. So I’ve worked up a new and (slightly) improved …

More

# Rolling Horizons

I keep seeing questions posted by people looking for help as they struggle to optimize linear programs (or, worse, integer linear programs) with tens of millions of variables. In my conscious mind, I know that commercial optimizers such as CPLEX allow models that large (at least if you have enough memory) and can often solve …

More

# Helpful resources for principal components analysis in R

I’m currently working on my dissertation proposal, which has meant exploring principal components analysis. I’ve worked with PCA before, but it’s been a couple of years, so I’m trying to refresh my memory, improve my understanding, and get this proposal moving! Along the way, I’ve found (and been recommended) some helpful resources that I thought …

More

# Robots taking our jobs

The figures below are from the recent paper Robots and Jobs: Evidence from US Labor Markets, by Acemoglu and Restrepo. VoxEU discussion: … Estimates suggest that an extra robot per 1000 workers reduces the employment to population ratio by 0.18-0.34 percentage points and wages by 0.25-0.5%. This effect is distinct from the impacts of imports, …

More

# Don’t Touch the Computer

Under what circumstances should humans override algorithms? From what I have read I doubt that a hybrid team of human + AlphGo would perform much better than AlphaGo itself. Perhaps worse, depending on the epistemic sophistication and self-awareness of the human. In hybrid chess it seems that the ELO score of the human partner is …

More

# Memory Minimization

As I grow older, I’m starting to forget things (such as all the math I ever learned) … but that’s not the reason for the title of this post. A somewhat interesting question popped up on Mathematics StackExchange. It combines a basic sequencing problem (ordering the processing of computational tasks) with a single resource constraint …

More

# Super-human Relational Reasoning (DeepMind)

These neural nets reached super-human (better than an average human) performance on tasks requiring relational reasoning. See the short video for examples. A simple neural network module for relational reasoning https://arxiv.org/abs/1706.01427 Adam Santoro, David Raposo, David G.T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, Timothy Lillicrap (Submitted on 5 Jun 2017) Relational reasoning is a …

More

# Probing deep networks: inside the black box

See also AI knows best: AlphaGo “like a God”: Humans are going to have to learn to “trust the AI” without understanding why it is right. I often make an analogous point to my kids — “At your age, if you and Dad disagree, chances are that Dad is right” :-) Of course, I always …

More

# Face Recognition applied at scale in China

The Chinese government is not the only entity that has access to millions of faces + identifying information. So do Google, Facebook, Instagram, and anyone who has scraped information from similar social networks (e.g., US security services, hackers, etc.). In light of such ML capabilities it seems clear that anti-ship ballistic missiles can easily target …

More

# Premature Obituaries

[T]he report of my death was an exaggeration. (Mark Twain, 1897) In a recent blog post, “Data Science Is Not Dead“, Jean-Francois Puget discussed and dissented with a post by Jeroen ter Heerdt titled “Data Science is dead.” Barring the possibility that Schroedinger shoved data science into a box and sealed it, both assertions cannot …

More

# Rise of the Machines: Survey of AI Researchers

These predictions are from a recent survey of AI/ML researchers. See SSC and also here for more discussion of the results. When Will AI Exceed Human Performance? Evidence from AI Experts Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, Owain Evans Advances in artificial intelligence (AI) will transform modern life by reshaping transportation, health, science, …

More

# Epistemic Caution and Climate Change

I have not, until recently, invested significant time in trying to understand climate modeling. These notes are primarily for my own use, however I welcome comments from readers who have studied this issue in more depth. I take a dim view of people who express strong opinions about complex phenomena without having understood the underlying …

More

# If This and That Then Whatever

I was asked a question that reduced to the following: if $$x$$, $$y$$ and $$z$$ are all binary variables, how do we handle (in an integer programming model) the requirement “if $$x=1$$ and $$y=1$$ then $$z=1$$”? In the absence of any constraints on $$z$$ when the antecedent is not true, this is very easy: add …

More

# Update Error: Wrong Architecture

Yesterday I ran into one of those mystifying glitches that, will infrequent, serve as a reminder that Linux is not for the faint of heart. When I booted my desktop system (Linux Mint 18.1 Serena), the system tray icon for the software updater was displaying a red “X” that indicates it tried and failed to …

More

# AI knows best: AlphaGo “like a God”

Humans are going to have to learn to “trust the AI” without understanding why it is right. I often make an analogous point to my kids — “At your age, if you and Dad disagree, chances are that Dad is right” :-)  Of course, I always try to explain the logic behind my thinking, but …

More

# Sharing “Implicit” Test Problems

The topic of reproducible research is garnering a lot of attention these days. I’m not sure there is a 100% agreed upon, specific, detailed definition of it, and I do think it’s likely to be somewhat dependent on the type of research, but for purposes of this post the Wikipedia definition (previous link) works for …

More

# Yann LeCun on Unsupervised Learning

This is a recent Yann LeCun talk at CMU. Toward the end he discusses recent breakthroughs using GANs (Generative Adversarial Networks, see also Ian Goodfellow here and here). LeCun tells an anecdote about the discovery of backpropagation. The first implementation of the algorithm didn’t work, probably because of a bug in the program. But they convinced …

More

Since I use the Linux Mint operating system, the obvious (if not only) choice for a LaTeX distribution is TeX Live. (If you are not familiar with, or are not interested in, the LaTeX typesetting system, you have already read too far in this post.) On Mint, Ubuntu and other Debian-type operating systems, you typically …

More

# Ginormous Neural Nets and Networks of Networks

Now that we have neural nets that are good at certain narrow tasks, such as image or speech recognition, playing specific games, translating language, … the next stage of development will involve 1. linking these specialized nets together in a more general architecture (“Mixtures of Experts”), and 2. generalizing what is learned in one class …

More

# Another Absolute Value Question

Someone asked whether it is possible to eliminate the absolute value from the constraint $$Lx\le |y| \le Ux$$ (where $$L\ge 0$$ and $$U>0$$ are constants, $$x$$ is a binary variable, and $$y$$ is a continuous variable). The answer is yes, but what it takes depends on whether $$L=0$$ or $$L>0$$. The easy case is when …

More

# The Future of Thought, via Thought Vectors

In my opinion this is one of the most promising directions in AI. I expect significant progress in the next 5-10 years. Note the whole problem of parsing languages like English has been subsumed in the training of neural Encoders/Decoders used, e.g., in the translation problem (i.e., training on pairs of translated sentences, with an …

More

# Fischetti on Benders Decomposition

I just came across slides for a presentation that Matteo Fischetti (University of Padova) gave at the Lunteren Conference on the Mathematics of Operations Research a few days ago. Matteo is both expert at and dare I say an advocate of Benders decomposition. I use Benders decomposition (or variants of it) rather extensively in my …

More

# Dangerous Knowledge and Existential Risk (Dominic Cummings)

Dominic Cummings begins a new series of blog posts. Highly recommended! It’s worth noting a few “factor of a million” advances that have happened recently, largely due to physical science, applied mathematics, and engineering: 1. Destructive power of an H-bomb is a million times greater than that of conventional explosives. This advance took ~20 years. …

More

# Pro Bono Analytics Is Growing Social

Pro Bono Analytics is a program by INFORMS (the Institute for Operations Research and the Management Sciences, for the acronym-averse), “the largest society in the world for professionals in the field of operations research (O.R.), management science, and analytics”. PBA “connects our members and other analytics professionals with nonprofit organizations working in underserved and developing communities”. …

More

# AlphaGo (BetaGo?) Returns

Rumors over the summer suggested that AlphaGo had some serious problems that needed to be fixed — i.e., whole lines of play that it pursued poorly, despite its thrashing of one of the world’s top players in a highly publicized match. But tuning a neural net is trickier than tuning, for example, an expert system …

More

# Mapping Trackball Buttons

For years, I’ve used a Logitech M570 wireless trackball with my Linux Mint PC. I generally prefer trackballs to mice — no need to lift and reposition after a bunch of movement — and I find that using my thumb, rather than my index finger (or, if I’m in a bad mood, my middle finger) …

More