Anyone can make an election model

Building the Economist’s election forecast model in pymc3

I’ve never been very interested in politics. But like many others, I got caught up in the hubbub of this past election, especially given the high stakes involved. So I started paying more attention to forecasts from FiveThirtyEight and TheEconomist, followed Nate Silver on Twitter, and even got distracted by the popcorn drama flame war between the two election forecasters.

As well as reading posts by Andrew Gelman – for anyone curious, he has some fantastic posts digging into the components of the Economist model.

All this also got me curious as to how hard it would be to build one of their forecast models.

As it turns out, it’s not too difficult. I can’t speak about FiveThirtyEight’s model, but the folks at the Economist were kind of enough to put their model and datasets on Github, so anyone can go through their code and reverse-engineer it. Their model is built in R and Stan – I have no familiarity in Stan, but I thought it would be a fun project to try to rebuild their model in Python with pymc3.

How the model works

If you just take a look at their repo, they state the inspiration for the model

“Improving on Pierre Kremp’s implementation of Drew Linzer’s dynamic linear model for election forecasting”

And if you follow the links, you get directed to this paper, which seems to be the source model. (By the way, I encourage anyone interested to read the actual paper, the math in it is pretty accessible and it’s a nice application of Bayesian modeling.)

This is the model we’re going to build in pymc3. It’s not exactly the Economist’s model, but it’s close enough – their model is a bit more complicated by trying to account include between state covariances and polling biases. I’m not a real election forecaster, so we’re going to keep it simple.

At it’s core, the model has two main components:

1. A “structural” forecast, based on economic and political factors
2. A series of random walks describing the trends of voter preferences throughout time.

We’ll break these down one by one:

1. Structural Forecast

The structural model turns out based on an earlier model known as Time for a Change, described in this paper. This model predicts the current presidential incumbent’s vote percentage from a regression on a few features, including the incumbent’s June approval rating and the latest Q2 GDP growth numbers. I don’t have much background on this, but supposedly this model worked well in the past.

There’s nothing Bayesian about this model – instead we’ll be using it’s prediction as a prior for the larger model

2. Voter Preference Trends

For this part, we assume that trends in state-level preferences are governed by two random walks: a national component \(\alpha\), and a state-specific component \(\beta_{s}\). Combining those, we end up with a stochastic variable representing state level preferences at time t.

(Logit is used to restrict the variable between 0 and 1)

The trend for \(\pi_{s,\ t}\) determines results in the polling data, given by the relationship

The \(\beta\) and \(\alpha\) parameters are then modeled as a reverse random walk (I didn’t know you can do this!), given by:

The structural forecast is used to set the time T (election date) prior on the state-level preferences \(\beta_{s,\ t}\) If you think about it, this makes sense, since the structural forecast is trying to predict what the preferences are going to be on the election date, not 90 or so days before.

\(\alpha\) is given a prior at time T of 0 – the reason being that on election day, we expect votes to be determined purely by state-specific effects. If you look at the paper, they describe it as “a national-level effect detects systematic departures from Beta”.

Even though this is now a reverse random walk, I think intuitively we can still think of this as a normal random walk. The key thing is that the prior is just on the last time step instead of the first. You can think this as saying, absent any data, we expect voter preferences on election day (day T) to be what we forecasted from the fundamentals model.

Show me the code

Let’s apply this model on data from the 2012 election. Here we’re just fitting a simple linear regression model Incumbent vote ~ June Approval Rates + Q2 GDP Growth

import pandas as pd
from sklearn.linear_model import LinearRegression

def fit_fundamentals_model():
    df = pd.read_csv('data/abramowitz_data.csv')
    train_df = df[df['year'] < 2012]
    model = LinearRegression(fit_intercept=True)
    X, y = train_df[['juneapp', 'q2gdp']], df['incvote']
    model.fit(X, y)
    print('R-squared: %f' % model.score(X, y))
    inc_prediction = model.predict(df[df['year']==2008][['juneapp', 'q2gdp']])[0]
    print('incumbent prediction: %f' % inc_prediction)

    prior = (100 - inc_prediction) / 100.  # Taking inverse and normalizing since we're predicting for Obama
    print('national prior %f' % prior)
    return prior

prior = fit_fundamentals_model() # 0.558804

And we get our prior of 55.8%.

To get state-specific priors, we can incorporate state level differences between % votes for Obama in the previous 2008 election, and apply those differences on priors

votes = pd.read_csv('data/2008.csv')  # 2008 results data
votes['vote_share'] = votes['total_count'] / votes['total_count'].sum()
votes['obama_diff'] = votes['obama'] - (votes['obama_count'].sum() / votes['total_count'].sum())

# states is an array of state names
state_weights = np.asarray([votes[votes['state']==s]['vote_share'].iloc[0] for s in states])
state_prior = np.asarray([votes[votes['state']==s]['obama_diff'].iloc[0] + prior for s in states])
# array([0.41770557, 0.41770557, 0.47770557, 0.63770557, 0.56770557,
#        0.63770557, 0.53770557, 0.49770557, 0.74770557, 0.56770557,
#        0.38770557, 0.64770557, 0.52770557, 0.43770557, 0.42770557,
#        0.64770557, 0.64770557, 0.60770557, 0.59770557, 0.56770557,
#        0.51770557, 0.49770557, 0.52770557, 0.47770557, 0.44770557,
#        0.56770557, 0.59770557, 0.59770557, 0.57770557, 0.65770557,
#        0.54770557, 0.36770557, 0.59770557, 0.56770557, 0.65770557,
#        0.47770557, 0.44770557, 0.46770557, 0.36770557, 0.55770557,
#        0.69770557, 0.60770557, 0.58770557, 0.45770557])

state_weights will come into play later.

def get_polls():
    polls = pd.read_csv('data/all_polls_2012.csv')
    print('polls.shape', polls.shape)
    polls['end.date'] = pd.to_datetime(polls['end.date'])
    polls = polls[polls['end.date'] >= '2012-08-01']
    election_date = pd.Timestamp('2012-11-06')

    # Breaking out data into 5 day intervals
    polls['days_to_election'] = polls['end.date'].apply(lambda x: (election_date - x).days)
    polls['day_idx'] = polls['days_to_election'].apply(lambda x: x//5)

    polls = polls[polls['number.of.observations'].notnull()]
    polls['number.of.observations'] = polls['number.of.observations'].astype(int)
    polls['obama'] = polls['obama'].astype(int)
    polls['obama_share'] = ((polls['obama'] / 100.) * polls['number.of.observations']).astype(int)

    state_polls = polls[polls['state'] != '--']
    national_polls = polls[polls['state'] == '--']
    return state_polls, national_polls

state_polls, national_polls = get_polls()

There are both national and state polls in this dataset. National polls will be handled differently.

Then we just need to fit our Bayesian model to polling data.

with pm.Model() as model:
    s_idx = pm.Data('state_idx', state_polls['state_idx'])
    s_day = pm.Data('state_day', state_polls['day_idx'])
    n_day = pm.Data('national_day', national_polls['day_idx'])

    # Defining state priors
    s_prior = pm.Normal('state_prior', 0, 1, shape=num_states) + state_prior
    s_sigma = pm.HalfCauchy('state_sigma', 1, shape=num_states)
    s_walk = GaussianRandomWalk('state_walk', sigma=1, shape=(num_days, num_states))

    # Defining national effect priors
    n_walk = GaussianRandomWalk('nation_walk', sigma=1, shape=num_days)
    n_sigma = pm.HalfCauchy('nation_sigma', 1)

    # Combining state and national effects
    state_effect = pm.Deterministic('state_effect',
                                    pm.math.invlogit(
                                        ((s_walk * s_sigma) + s_prior) + \
                                        tt.stack(n_sigma * n_walk).T
                                    ))

    # State polls
    state_poll_results = pm.Binomial('state_poll_results',
                                     n=state_polls['two_party_votes'],
                                     p=state_effect[s_day, s_idx],
                                     observed=state_polls['obama_votes'])

    # National polls
    weighted_national_effects = pm.Deterministic('weighted_national_effects', (state_effect * state_weights).sum(axis=1) / state_weights.sum())
    national_poll_results = pm.Binomial('national_poll_results',
                                        n=national_polls['two_party_votes'],
                                        p=weighted_national_effects[n_day],
                                        observed=national_polls['obama_votes'])

Note the part weighted_national_effects = pm.Deterministic(... (state_effect * state_weights)...), this is how the model is fit on national polling data.

Once the model is fit in pymc3, we can generate some charts from the resulting trace, such as the one below that shows the evolution state-level voter preference for Obama over time. The blue line is what the model learned, and the red dots are the results from the state polls.

Interestingly, we see a drop in Obama favorability around the beginning of October, right around the time of the Oct 3 debate, in which Romney supposedly performed well. Also, all the state-level preferences seem to have the same trend – if you look closely, you can see they all peak in fall on the same dates, which speaks to how powerful the national level effect (\(\alpha_t\)) is in this model. (You can see the same pattern on the Economist’s github)

From the trace, we can also get samples of the final vote probability for each state, and use that to generate the final election win probabilities.

total_effect = trace['state_effect']
total_votes_arr = []
for i in range(total_effect.shape[0]):
    total_votes = 0
    for j, p in enumerate(total_effect[i, -1]):
        state = states[j]
        if p > .5:
            total_votes += electoral_votes[state]
    total_votes_arr.append(total_votes)
total_votes_arr = np.asarray(total_votes_arr)

plt.figure(figsize=[10, 6])
ax = plt.subplot(111)
plt.hist(total_votes_arr, bins=50, alpha=.8)
win_pct = (total_votes_arr >= 270).sum() / float(len(total_votes_arr))
ax.axvline(270, ls='--', c='r')
ax.set_title('Obama Win Probability:  %d%%' % (win_pct * 100), fontsize=16)

And that’s it! Overall it wasn’t too complicated, the most annoying part was waiting for the MCMC sampler to converge. I don’t get use pymc3 much in my day job, so it was nice to play around with it some more.

Addendum

• Again, huge props goes to the Economist for releasing and open sourcing their model. Without their documentation and code to guide me, I wouldn’t be able to do this post at all.
• I should emphasize, getting the MCMC sampler to run a few thousand samples is actually pretty annoying. I don’t remember exactly, but I think getting 10K samples took over 40 minutes. The dataset really isn’t that large, so I’m guessing this is just from the complexity of fitting a 51 random walk variables.

The curious case of hazard rates in the UFC

I’ve recently scraped some UFC fights data, with the intention of applying some modeling on it in order to quantify fighter effectiveness. I’ve done similar work before applying a Survival Analysis model on hockey data to quantify player effectiveness, so I was curious to see what insights would come from applying the same type of approach on MMA fighters.

I was playing around with the Kaplan Meier estimate of the survival curve, using KO as the failure event, when I noticed something interesting.

There’s a large spike in the hazard rate around 300 seconds, which is right around when the 1st round ends.

What does this all mean? Well, for anyone still reading past this point, let’s provide some background first.

Hold on, what’s Survival Analysis

Survival analysis is a field of statistics used to model the expected lifetime of an agent, before a “failure event” occurs. For example, a typical question in survival analysis would be “What’s the expected lifespan of prostate cancer patients?”, or “What’s the probability cancer patients survive up to 5 years?” etc.

The core of survival analysis lies within modelling the survival function S(t), which defines the probability that the object of interest will survive up to time t.

The complement to the survival function is the hazard rate h(t), which is defined as the instantaneous probability that the event occurs at time t.

The cool thing about survival analysis is that we can define the “failure event” as any arbitrary event. For example, financial institutions use survival analysis to model defaults on loans, where the default is the failure event. E-commerce companies use it to model customer churn, where canceling a membership is the failure event.

In this case, since we’re dealing with fight data, I’m using knockouts (KO/TKO) as the failure event.

The data I’ve scraped comprises of the UFC fights (up until last month), including the duration of the fight, who the fighters were, and the outcome (KO/TKO, submission, referee decision, etc.).

To model KO survival as function of time, we simply need the total duration of each fight, and label for each fight indicating whether or not the fight ended in a KO/TKO or not.

Application

The lifelines library in python provides a nifty API for modeling survival functions. All you need to model the survival curve is a Pandas dataframe of fight duration and whether or not a KO/TKO occurred or not.

With the lifelines API, we can get a non-parametric estimate of the survival function through their KaplanMeierFitter.

from lifelines import KaplanMeierFitter
kmf = KaplanMeierFitter()

kmf.fit(surv_df_3['fight_time'], event_observed=surv_df_3['KO'])
kmf.plot(title = "Probability of Surviving (a KO)", figsize = [15,7])

And we get something that looks like this:

Interpretation of the curve is pretty straightforward. Each point represents the probabilty of surviving up to that point.

For example, there’s a ~82% chance of not getting KO’d within the first round, and ~70% chance of not getting KO’d within the first 2 rounds. Ultimately, it looks like there’s at least an 80% chance of not getting KO’d at all.

Notice the weird kink around 300 seconds? We’ll get back to that later.

Unfortunately, lifelines doesn’t provide a way to see the see the hazard function — however, we can approximate it by taking the shifted difference in ratio of the survival function.

survival_func = kmf.survival_function_
shifted_survival_diff = (1 - survival_func/survival_func. shift(1))
hazard_empirical = shifted_survival_diff.div(survival_func.index.to_series(), axis=0)

Essentially, what we are doing is this:

Where \(\widehat{S}(t)\) is the empirical survival function from the Kaplan Meier estimate.

Calculating and plotting the hazard rate now give us this:

Some things pop out:

• the hazard rate of KO is highest in the beginning of the fight
• the hazard rate declines over time, but spikes up again around 300 seconds (remember the kink?)
• after that, it seems to stabilize and stay relatively constant

It makes sense that the hazard rate is highest in the beginning, as that’s when fighters are still figuring each other out. In those early minutes, it’s also possible for one fighter to get lucky with a swing and knock the other guy/gal out.

Over time, fighters get into their rhythym, and are able to react better to their opponenets, which explains the stabilization in hazard rates.

However, the increase at 300 seconds is pretty odd. Recall that the hazard rate is defined as the instantaneous rate of failure, or the probability of failing at time t (given that the agent has survived up until then). This means that right around 300 seconds, the probability of a KO jumps by more than 3x.

And if we look at the distribution of fight times, there’s also a spike around 300 seconds.

So for some reason, the probability of a KO ~ 300 seconds is much higher than other times.

What’s going on here?

To be honest, I’m not sure.

It’s possible that fighters more likely to let their guard right before the 1st round ends. Or alternatively, fighters become more aggressive in the end of the first round, hoping to score a KO.

Or potentially there’s reason.

KO’s and TKO’s are judged by the sole discretion of the referee. The ref may call a KO/TKO if the fighter had been knocked unconscious, OR if the fighter had stopped defending himself/ herself. The second case is more ambiguous, and there have been complaints about early stoppage before. I wonder if a majority of these are early stoppages, and there’s some bias on the refs to call fights before the 1st round ends.

Conspiracy theories aside, what we can conclude is that fighters definitely need to keep their guard up in the seconds before the first round ends.

Recently, a large portion of my job involves building D3.js-based dashboards to help track different metrics over time. Since I work in advertising, a large majority of those metrics comprise of click-thru rates (# ads clicked / # ads shown) or conversion rates (# ads that led to a conversion event / # ads shown) of different advertising strategies.

The issue with clicks and conversion is that they are pretty rare events, so on a daily basis their estimates can fluctuate quite a bit. So if we put something like click-thru rates by data in a visualization, it ends up looking like this:

This sucks. It’s incredibly hard to detect if there’s any clear trend in the data. What’s the “average” click-thru? And what’s the deal with the spikes? Are those just normal fluctuations, or is it because of low impression volume? Hard to say.

So how do we do this? We could just calculate the moving average in the data before calling the visualization, but that puts extra overhead on the data pre-processing. And where’s the fun in that?

To address this, let’s create module that will display both the underlying data, and a smoothed moving average.

Coding time

Let’s start with a standard time series chart:

  var regularTimeSeries = function(){

    width = 450
    height = 150
    margin = {'top': 10, 'right':10, 'bottom':2, 'left':40}

    var yScale = d3.scale.linear()
              .range([height, 0])
    var xScale = d3.time.scale()
              .range([0, width])

    var line = d3.svg.line()
                .x(function(d){ return xScale(d.date) })
                .y(function(d){ return yScale(d.CTR) })


    setupScales = function(data){
      xScale.domain([d3.min(data, function(d){return d.date}), d3.max(data, function(d){return d.date}) ])
      yScale.domain([0, d3.max(data, function(d){return d.CTR}) ])
    }


    chart = function(selection){
      selection.each(function(data){

        setupScales(data)

        svg = selection.append("svg")
          .attr("width", width + margin.left + margin.right)
          .attr("height", height + margin.top + margin.bottom)

        svg.append("path")
            .attr('class','line')
            .attr("d", function(d){ return line(data) })
            .style("stroke","steelblue")
            .style("stroke-width", 3)
      })
    }
    return chart
  }


  var chart = regularTimeSeries()
  d3.select("#chart").data([data]).call(chart)

This is the exact code used to generate the visualization above. It’s assuming that the input data looks like this:

with CTR already calculated at each point in time. CTR is plotted on the y-axis, and date is on the x-axis (Note: I’m ignoring axis labels for this exercise.). The charting code is encapsulated in the regularTimeSeries function, and the last two lines generate the chart object and apply it on some data.

I’m skipping over most of the details of the code – I based the structure on on Mike Bostock’s reusuable charts framework, and also the multitude of on line charts in D3, so if you’re new to D3, I suggest you check that out before continuing. I’ve personally found d3noob and Scott Murray’s tutorials to be especially helpful.

Let’s modify this so it also plots a 7-day moving average, which should help smooth out the fluctuations.

We’ll start by adding some getter-setter methods that allow us to specify our x and y variables, and specify how to calculate the metric. This will help us recalculate the moving average CTR on each date.

chart.xVar = function(value){
  if (!arguments.length) return x
  xVar = value
  return chart
}

chart.yVar = function(value){
  if (!arguments.length) return y
  yVar = value
  return chart
}

chart.yCalc = function(value){
  if (!arguments.length) return yCalc
  yCalc = value
  return chart
}
chart.windowSize = function(value){
  if (!arguments.length) return windowSize
  windowSize = value
  return chart
}

I’ve also added a windowSize method that allows us to specify how many days to use for our moving average.

Now, our chart can be configured with those values specified, like so:

var chart = regularTimeSeries()
              .xVar("date")
              .yVar("CTR")
              .yCalc(function(d){return d.imps > 0 ? d.clicks/d.imps : 0})
              .windowSize(7)
d3.select("#chart").data([data]).call(chart)

With these specified, let’s add a function that will calculate a rolling sum of the data, based on the windowSize.

function rollingSum(data){      
  
  data = data.sort(function(a,b){return d3.ascending(a[xVar],b[xVar])})

  var summed = data.map(function(d, i){

    var start = Math.max(0,i-windowSize)
    var end = i
    var sum = {}

    for (var key in d) {
      if (d.hasOwnProperty(key)) {
        sum[key] = key != xVar ? d3.sum(data.slice(start, end), function(x) {return x[key]}) : d[key]
      } 
    }

    return sum
  })
  return summed
}

Taking that data as input, this will return an array that looks exactly the same, except that value at each data is the rolling sum of the past 7 days.

Now, we can easily apply the click-thru calculation on top of it, which we’ll also specify in a function:

function calculateYValue(data){

  data = data.map(function(d){
    d[yVar] = yCalc(d)
    return d
  })
  return data
}

Putting the above two functions together, we can come up with a new array of data that gives us a 7-day rolling sum of clicks and impressions, and a 7-day moving average of CTR based on these sums.

smoothedData = smooth(data)
smoothedData = calculateYValue(smoothedData)

Math Interlude:

We could also calculate the moving average of CTR as (CTR_1 + CTR_2 + ….+ CTR_n) / n; however, this puts equal weight on each date. This is undesirable in this case, because the number ads shown differ greatly on a day-to-day basis – days when 10 ads are shown shouldn’t get the same weight as days with 10,000 ads shown. Calculating the average as the rolling sum of clicks / rolling sum of ads allows us to overcome this bias.

The rest is pretty straightfoward – in chart(), we just need to add another path object that uses the smoothedData as its input source.

chart = function(selection){
  selection.each(function(data){

    
    smoothedData = smooth(data)
    data = calculateYValue(data)
    smoothedData = calculateYValue(smoothedData)

    setupScales(data)

    svg = selection.append("svg")
      .attr("width", width + margin.left + margin.right)
      .attr("height", height + margin.top + margin.bottom)

    svg.append("path")
        .attr('class','line')
        .attr("d", function(d){ return line(data) })
        .style("stroke","steelblue")
        .style("stroke-width", 1)
        .style("opacity", .75)

    svg.append("path")
        .attr('class','line')
        .attr("d", function(d){ return line(smoothedData) })
        .style("stroke","steelblue")
        .style("stroke-width", 5)
  })
}

We end up with a visualization that looks like this:

I’ve added some styling to emphasize the moving average line more, but this is much better! It’s much more clear where the trends are, and keeps us from being reactive to random spikes or drops.

Conclusion

This might be obvious to many, but smoothing your visualization matters helps a lot, especially when dealing with a lot of noise in the data.

Of course, you might need to be concerned about forcing trends that aren’t there, but that’s as discussion for another day ;).

Also, D3 is awesome!