Code

import string
from collections import defaultdict


class Node:
    def __init__(self, word="", parent=None):
        self.word = word
        self.parent = parent
        self.children = defaultdict()
        self.dists = None
        self.is_word = False


class Trie:
    def __init__(self):
        self.root = Node()

    def preprocess(self, doc):
        return (
            doc.lower()
            .translate(str.maketrans("", "", string.punctuation))
            .encode("ascii", "replace")
            .decode()
        )

    def insert(self, word):
        word = self.preprocess(word)
        node = self.root
        for i, char in enumerate(word):
            node = node.children.setdefault(char, Node(word[: i + 1], node))
        node.is_word = True

    def fuzzy_search(self, query, n):
        query = self.preprocess(query)
        matching_set, visited, stack = set(), set(), [self.root]
        while stack:
            node = stack.pop()
            if node not in visited:
                node.dists = self.get_levenshtein_dists(node, query, n)
                if node.is_word and node.dists[-1] <= n:
                    matching_set.add(node.word)
                if min(node.dists) <= n:
                    stack.extend(node.children.values())
        return matching_set

    def get_levenshtein_dists(self, node, query, n):
        if node.parent is None:
            return range(len(query) + 1)
        dists = [n + 1] * (len(query) + 1)
        dists[0] = len(node.word)
        prev_dists = node.parent.dists
        for i in range(max(0, dists[0] - n - 1), min(len(query), dists[0] + n + 1)):
            dists[i + 1] = (
                prev_dists[i]
                if query[i] == node.word[-1]
                else 1 + min(prev_dists[i], prev_dists[i + 1], dists[i])
            )

        return dists


class Index:
    def __init__(self, documents):
        self.trie = Trie()
        self.inverted_index = defaultdict(lambda: set())
        for doc in documents:
            for word in self.trie.preprocess(doc).split():
                self.trie.insert(word)
                self.inverted_index[word].add(doc)

    def fuzzy_search(self, query, n):
        return {
            doc
            for word in self.trie.fuzzy_search(query, n)
            for doc in self.inverted_index[word]
        }

Example

from simplesearch import Index

docs = [
    "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects.",
    "The Hrabri class consisted of two submarines built for the Kingdom of Serbs, Croats and Slovenes. The first submarines to serve in the Royal Yugoslav Navy (KM), they arrived in Yugoslavia on 5 April 1928, and participated in cruises to Mediterranean ports prior to World War II."
    "Did you know that Jean-Emmanuel Depraz (pictured) won a Magic: The Gathering world championship using three cards depicting the player who beat him in 2021?"
]

index = Index(docs)
index.fuzzy_search("Willipedia", 2)
## {'Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects.'}

Notes

• This demo is only optimized for time complexity, not memory.
• Obviously this not a refined search engine. It’s just efficient single-term fuzzy search and corresponding document lookup.

References

• http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automata
• https://julesjacobs.com/2015/06/17/disqus-levenshtein-simple-and-fast.html
• https://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html

The Four Critical Questions

All great solutions begin by asking the right questions. They seem like simple questions - that’s exactly the point. They are indeed simple, but not simplistic. The four following carefully crafted questions work wonders in virtually any situation. The first three are usually glossed over in the rush to answer the fourth.

1. What are we trying to accomplish and why?

   The question of what the project should accomplish - and more importantly - why it needs to be done, deserves fine-tuned attention because those answers drive everything else. In the rush to decide on the how, who, and when of a project, people often gloss over the why.

2. How will we measure success?

   This question is significant because Measures flesh out and anchor what the Objectives really mean. Until you define how success will be measured, even the most sincere visions are no more than highfalutin’ fluff.

3. What other conditions must exist?

   This third question puts your project, issue, or initiative into a larger strategic context. Asking this expands the analysis to include some of the outside factors which may disrupt your carefully crafted plans.

4. How do we get there?

   The majority of project teams I have worked with tend to delve deep into the details much too soon, or get sidelined by premature technical arguments. They gloss over the first three questions in a rush to get moving. The value of the fourth question comes from consciously placing it in its only, truly functional place in the planning sequence: Last.

LogFrames

While the LogFrame matrix may initially seem intimidating, the ideas it captures are basic. The four strategic questions offer a user friendly way to learn and apply this tool. These questions are inherently embedded in the matrix and answering them helps you design your project in a way that connect all the dots.

1. What Are We Trying To Accomplish And Why? (Objectives)
   The first column describes Objectives and the If-Then logic linking them together. The LogFrame makes important distinctions among various “levels” of Objectives: Strategic intention (Goal), project impact (Purpose), project deliverables (Outcomes), and the key action steps (Inputs).

2. How Will We Measure Success? (Measures and Verifications)

   • The second column identifies the Measures of sucess for Objectives at each level. here wew select appropriate Measures and choose quantity, quality, and time indicators to clarify what each Objective means.

   • The third column summarizes how we will verify the status of the Measures at eaech level. Think of the Verification column as the project’s management information and feedback system.

3. What Other Conditions Must Exist? (Assumptions)
   The fourth column captures Assumptions; those ever-present, but often neglected risk factors outside of the project, on which project success depends. Defining and testing Assumptions lets you spot potential problems and deal with them in advance.

4. How do we get There? (Inputs)
   The bottom row captures the project action plan: Who does what, when, and with what resources. Conventional project management like Work Breakdown Structures (WBS) and Gantt chart schedules fit here.

Aligning Projects With Strategic Intent

The LogFrame can be the cornerstone of any unit-level management system. However, this presumes that there is a sound, overarching strategy to begin with.

Strategy is the particular means chosen to get from where you are to where you want to go, selected from multiple possibilities and reflecting your vision, mission, and values. An overall Strategy (big “S”) usually consists of multiple strategic initiatives (small “s”), which are executed through programs, projects, and tasks.

Strategic planning steps:

1. Clarify the Planning Context and Issues - Be clear about your expected planning Outcomes and identify issues to include.

2. Involve Key Players - Decide who to involve in your process to build buy-in and stay-ini.

3. Scan Your Environment - Identify what’s changing in your environment; and analyze divvision and department plans to extract Goals your group shares or owns.

4. Revisit Your Vision/Mission/Values - Turn these “fluff“ statements into high-performance tools that energize staff and build shared commitment.

5. Sharpen Your Goals and Measures - Develop a meaningful performance scorecard that identifies how you deliver customer value.

6. Develop Core Strategies - Turn Goals into strategies, and test those strategies for impact against Measures to ensure smart choices.

7. Turn Strategies into Executable Plans - Using the Logical Framework. Let the responsible players flesh out implementation plans.

8. Follow Up and Continue the Process - Build momentum by revieweing and updating the plans while strenghtening the planning process itself.

The Strategic Action Cycle

1. The cycle begins with “Think,” the big picture strategic/program focus which follows the process from Chapter 4, or an equivalent strategic planning process.

2. Results of strategic thinking identify projects to be managed with the Plan-Act-Assess cycle.

3. Project plans created with LogFrames provide a solid foundation for action (execution/implementation) and Assessment.

4. The Assess block can complete the loop in three ways. If assessment shows that success has been achieved - as defined by project Purpose - the project can be considered complete.

   1. Project Monitoring is an ongoing process of tracking budget and schedule against deliverables and making tactical adjustments. It presumes the Logical Framework is the best design and focuses team attention on translating Inputs into Outcomes.

   2. Project Review is an occasional process that asks managers to step back from the day-to-day work and reassess their approach. It challenges the project design and invites changes in the LogFrame, with emphasis on the Outcome to Purpose link.

   3. Project Evaluation examines impact and cost effectiveness. Project evaluations are often timed as the end of one phase nears and another is about to begin, or after the project is over. Evaluation examines Purpose to Goal linkages.

Other

Why Software Projects Fail

The Pareto principle plays into the common failure (or cost overrun, scope creep, technical debt) of software projects.

Often, development teams prioritize building a minimal viable product (MVP), or delivering the most apparent functionality for a given effort level. The fast achievement of 80% functionality can lead to poor expectations of what’s needed to reach a product that has actual value, i.e. something maintainable and deployable. Clients, project managers, and developers can misunderstand the scope of project if they rank tasks in functionality-first order, without considering the full value chain.

In Short

The Pareto principle is both about the big impact you can have from a few actions (e.g., achieve 80% in 20% of the time), and how easily misled you can be about scope and impact (e.g., forgetting about a necessary 20% that takes 80% of the time).

Exceptional Innovations

The best innovations don’t just provide new value.

They fit within or enable compounding processes, where past innovations keep on providing more and more value as they are built upon. Relatedly, they create more than one opportunity to capture value, i.e. they help expose the business to new opportunities, such as by entering new markets.

They align with the business’ strategic vision (its plan for growth) and reinforces its strategic positioning (how it distinguishes itself from competitors and provides compelling value, despite constraints.)

Why TDD Is Free

Here’s a key assumption I’m making: doing things right the first time is free. If you’re not doing it right the first time, you’ll have to come back to it later anyway. And not doing it right the first time is likely to create many unnecessary costs along the way.

So, how do you do something right the first time? There are 2 parts to this:

1. You need to know what’s the “right” thing you want to do.

2. You need to check that you actually did it right.

Point (2) is testing. You’ll have to test, whether it is at the beginning, throughout, or at the end.

Point (1) is having clear requirements. Sure, you can write down requirements specification in detail and work off of that. But you know what else is a clear requirement? A test case.

You can save time by combining points (1) and (2) together in test cases. Just keep in mind that you’ll have to write tests first in order to satisfy point (1).

So, TDD is free: it’s not doing anything that you wouldn’t have to do anyway, and it’s saving you from extra work now and in the future.

Note that there is a learning curve to TDD. You need to find a TDD workflow that works for you. That takes a bit of time. But afterwards, you are saving time.

This Isn’t a New Idea

You’re already doing TDD:

• In agile development, we use “User Stories” to describe specifications. These are high-level test case descriptions: “given starting point X, I want to do Y to achieve Z.” User stories don’t tell you how to code things - that’s the functional implementation. It’s something you figure out afterwards, once you know what the input looks like, what the function is meant to do, and what the result should look like.

• As mentioned earlier, interactive development is informal TDD. How can you formalize TDD in interactive development, without losing the benefits of interactive development? Simply bring the tests to your interactive development workflow. It can be done by staying organized, or you can use tools like the “ipytest” library for unit testing in Python notebooks.

Next Steps

You’re already doing TDD, but maybe you’re not doing it in the most effective way. If you answer yes to some of the questions below, then it might be worth it to improve your TDD practices:

• Could you save time by catching bugs earlier?
• Could you save time by writing examples/tests, instead of long-form documentation?
• Could you save time by keeping track of the experiments, tests, and examples you use in a notebook as you develop?
• Could you save time by clicking a single button to run all tests in your notebook, instead of backtracking to execute notebook cells one by one?
• Do you often have to go back to fix bugs in your code or other people’s code?

There are lots of guides online about TDD. But remember: you need to create a workflow that works for you. TDD is not about formality, complicated testing, or full-coverage testing. TDD is about speeding up your development and building things right the first time.

TDD Myths

Be careful not to fall into the following traps:

• “All tests need to be written upfront.” No. Your TDD tests only need to cover what you want to code up in the next 5-30 minutes. They’re meant to help you develop, not give you analysis paralysis.
• “Tests can’t change.” No. TDD tests are there to help you develop. Change them as much as you like.
• “I can’t add more test after I’m done implementing.” No. TDD is an iterative process. Create a test, make sure it runs (and generally fails), develop, create more tests, check what fails, develop, and keep going until you are satisfied.
• “I don’t need QA if I do TDD.” No. TDD is all about development. It helps develop faster and better. It’s about you, as a developer, building what you want to build right the first time. But, as often happens, it’s not because something is built right that it is the right thing for your customer!

Practical Example

Here’s what TDD looks like in practice.

Say I want to code a function “fibonacci” that computes the first n numbers of the standard Fibonacci sequence.

# Input
input_n = 5

# Output
expected_output = [1, 2, 3, 5, 8]

def test_fibonacci():
  assert fibonnaci(input_n) == expected_output

def fibonacci(n):
  result = [1, 2]
  while len(result) < n:
    result.append(result[-1], result[-2])
  
  return result

test_fibonacci()

def test_fibonacci_edge_cases():
  assert fibonacci(0) = []
  assert fibonacci(1) = [1]
  # etc 

def fibonacci(n):
  ...

test_fibonacci() # Make sure I didn't break anything
test_fibonacci_edge_cases() # New tests

Don’t manage by numbers.

It’s a bit confusing: Deming encouraged the use of measurement, metrics, data, and statistics, as a key tool for process improvement and quality control. And yet he also painstakingly tried to drive in points like this:

• “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”
• “Eliminate management by numbers and numerical goals.”

How can this be? How can he simultaneously be pro-measurement, pro-data, and against data-driven management?

How can we resolve this false paradox?

As a statistician, Deming was aware how important what you can’t measure is to making valid inferences. Statistics is not about data. It’s about combining data and context to make valid inferences. Data on its own has no meaning. Missing data - including both the data you wish you had and the data you don’t even know you’re missing - is more important than the data you have. A statistician’s work is to help learn about such unknowns. It’s a fallacy to make decisions based only on available data - the McNamara fallacy.

“But when the McNamara discipline is applied too literally, the first step is to measure whatever can be easily measured. The second step is to disregard that which can’t easily be measured or given a quantitative value. The third step is to presume that what can’t be measured easily really isn’t important. The fourth step is to say that what can’t be easily measured really doesn’t exist. This is suicide.” — Daniel Yankelovich, “Interpreting the New Life Styles”, Sales Management (1971)

The problem isn’t data or measurement. In fact, you should aim to measure as much as you can, as often as you can. You should build measurement and observability as core components of your systems and infrastructures. You should work to continually improve your approach to measurement of what matters. And you should have statisticians or data scientists make sense of these numbers through their context, given specific goals.

But here’s the thing: measurement is not management.

As a manager, your job is to create and maintain structures that drive customer value and continuous improvement. To achieve this, you need to think about knowns (i.e., data, metrics) and unknowns. Statisticians or data scientists can help you contextualize data and shed light on unknowns, athough it’s not always an easy process.

In Short

There are many misconceptions surrounding data and its use in management. It is important for all to understand both the importance of data and its limitations. We can do so by learning from resources such as the Deming Institute’s website:

Deming advocated for structures that removed fear in workers, fostered continuous improvement, and enabled taking pride in one’s work.

1.1 Background and Terminology

First, let’s talk about models, parameters and performance evaluation. This is going to be the occasion for me to introduce some terminology and notations.

import pandas as pd
import numpy as np
from sklearn.datasets import fetch_california_housing

dataset = fetch_california_housing(as_frame=True)

X = dataset.data # Covariates
n_features = X.shape[1] # Number of features
y = dataset.target # Median house prices

X

from sklearn.ensemble import GradientBoostingRegressor
from skopt.space import Real, Integer

model = GradientBoostingRegressor(n_estimators=25, random_state=0)

space  = [Integer(1, 8, name='max_depth'),
          Real(0.01, 1, "log-uniform", name='learning_rate'),
          Integer(1, n_features, name='max_features'),
          Integer(1, 50, name='min_samples_leaf')
]

from sklearn.model_selection import cross_val_score

def Rhat(**params):
  model.set_params(**params)
  
  return -np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,
                                  scoring="neg_mean_absolute_error"))

model.fit(X, y)

Rhat()

np.float64(0.5503720160635011)

2 Black-Box Optimization Methods

Black-box hyperparameter optimization algorithms consider the underlying machine algorithm as unknown. We only assume that, given a set of hyperparameters , we can compute the estimated model performance . There is usually variance in , but this is not something that I will talk about in this post. We will therefore consider as a deterministic function to be optimized.

Note: in practice, you need to account for the variance in , as otherwise you could get bad surprises. It’s just not something I’m covering in this post, since I want to focus on a conceptual understanding of the optimization algorithms.

We can use almost any technique to try to optimize , but there are a number of challenges with hyperparameter optimization:

1. is usually rather costly to evaluate.
2. We usually do not have gradient information regarding (otherwise, hyperparameters for which we have gradient information could easily be incorporated as parameters of the underlying ML algorithms).
3. The hyperparameter space is usually complex. It can contain discrete variables and can even be tree-structured, where some hyperparameters are only defined conditionally on other hyperparameters.
4. The hyperparameter space is usually somewhat high-dimensional, with more than just 2-3 dimensions.

These particularities of the hyperparameter optimization problem has led the machine learning community to favor some of the optimization techniques which I discuss below.

from sklearn.model_selection import GridSearchCV

# Budget of 54 evaluations
grid = {
  'max_depth': [1, 3, 5],
  'learning_rate': [0.01, 0.1, 1],
  'max_features': [2, 4, 8],
  'min_samples_leaf': [1, 10]
}

def scoring(estimator, X_test, y_test):
  y_pred = estimator.predict(X_test)
  return -np.mean(np.abs(y_test - y_pred))

results = GridSearchCV(model, grid, cv=3, n_jobs=-1, scoring=scoring).fit(X, y)

-results.best_score_ # Lowest cross-validated mean absolute error

{key:results.best_params_[key] for key in grid.keys()} # Best parameters

{'max_depth': 5,
 'learning_rate': 0.1,
 'max_features': 8,
 'min_samples_leaf': 1}

import matplotlib.pyplot as plt

plt.clf()
p = plt.hist(-results.cv_results_["mean_test_score"], bins=20)
p = plt.title("Score distribution for evaluated hyperparameters", loc="left")
p = plt.xlabel("Cross-validated average absolute error")
plt.show()

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

# Around roughly the same values as for the grid search
param_distribution = {
  'max_depth': range(1, 8),
  'learning_rate': loguniform(0.01, 1),
  'max_features': range(1, 9),
  'min_samples_leaf': range(1, 50)
}

# Budget of 54 evaluations
results = RandomizedSearchCV(model, param_distribution, n_iter=54, cv=3, n_jobs=-1, scoring=scoring, random_state=0).fit(X, y)

-results.best_score_ # Lowest cross-validated mean absolute error

{key:results.best_params_[key] for key in grid.keys()} # Best parameters

{'max_depth': 6,
 'learning_rate': np.float64(0.1453937524243155),
 'max_features': 7,
 'min_samples_leaf': 26}

import matplotlib.pyplot as plt

plt.clf()
p = plt.hist(-results.cv_results_["mean_test_score"], bins=20)
p = plt.title("Score distribution for evaluated hyperparameters", loc="left")
p = plt.xlabel("Cross-validated average absolute error")
plt.show()

from skopt import gp_minimize
from skopt.utils import use_named_args

@use_named_args(space)
def objective(**params):
  model.set_params(**params)
  
  return -np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,
                                  scoring="neg_mean_absolute_error"))

res_gp = gp_minimize(objective, space, n_calls=54, random_state=1)

## 0.46

plt.clf()
p = plt.hist(res_gp.func_vals, bins=20)
p = plt.title("Score distribution for evaluated hyperparameters", loc="left")
p = plt.xlabel("Cross-validated average absolute error")
plt.show()

from skopt.plots import plot_convergence

plot_convergence(res_gp)

3 Summary

This blog post provided a basic overview of hyperparameter optimization and of what can be gained from these techniques. We reviewed grid search, the simplest brute force approach. We reviewed random search, which improves upon grid search when some hyperparameter dimensions are not influencial. Finally, we reviewed sequential model-based optimization, which much more effectively samples models with good performance.

Introduction

Duke’s Graduate and Professional Student Government (GPSG) has been operating a community food pantry for about five years. The pantry provides nonperishable food and basic need items to graduate and professional students on campus. There is a weekly bag program, where students order customized bags of food to be picked up on Saturdays, as well as an in-person shopping program open on Thursdays and Saturdays.

The weekly bag program, which began in May 2018 and is still the most popular pantry offering, provides quite a bit of data regarding pantry customers and their habits. Some customers have ordered more than 80 times in the past 2 years, while others only ordered once or twice. For every bag order, we have the customer’s first name and last initial, an email address (which became mandatory around mid 2018), a phone number in a few cases, an address in some cases (for delivery), we have demographic information some cases, and we have the food order information. Available quasi-identifying information is shown in Table 1 below.

Gaining the most insight from this data requires linking order records from the same customer. Identifying individual customers and associating them with an order history allows us to investigate shopping recurrence patterns and identify potential issues with the pantry’s offering. For instance, we can know who stopped ordering from the pantry after the home delivery program ended. These are people who, most likely, do not have a car to get to the pantry but might benefit from new programs, such as a ride-share program or a gift card program.

This blog post describes the way in which records are linked at the Community Pantry. As we will see, the record linkage problem is not particularly difficult. It is not trivial either, however, and it does require care to ensure that it runs reliably and efficiently, and that it is intelligible and properly validated. This post goes in detail into these two aspects of the problem.

Regarding efficiency and reliability of the software system, I describe the development of a Python module, called GroupByRule, for record linkage at the pantry. This Python module is maintainable, documented and tested, ensuring reliability of the system and the potential for its continued use throughout the years, even as technical volunteers change at the pantry. Regarding validation of the record linkage system, I describe simple steps that can be taken to evaluate model performance.

Before jumping into the technical part, let’s take a step back to discuss the issue of food insecurity on campus.

The Record Linkage Approach

The bag order form contains email addresses which are highly reliable for linkage. If two records have the same email, we know for certain that they are from the same customer. However, customers do not always enter the same email address when submitting orders. Despite the request to use a Duke email address, some customers use personal emails. Furthermore, Duke email addresses have two forms. For instance, my duke email is both ob37@duke.edu and olivier.binette@duke.edu. Emails are therefore not sufficient for linkage. Phone numbers can be used as well, but these are only available for the period when home delivery was available.

First name and last initial can be used to supplement emails and phone numbers. Again, agreement on first name and last initial provides strong evidence for match. On the other hand, people do not always enter their names in the same way.

Combining the use of emails, phone numbers, and names, we may therefore link records which agree on any one of these attributes. This is a simple deterministic record linkage approach which should be reliable enough for the data analysis use of the pantry.

from abc import ABC, abstractmethod


class LinkageRule(ABC):
    """
    Interface for a linkage rule which can be fitted to data.

    This abstract class specifies three methods. The `fit()` method fits the 
    linkage rule to a pandas DataFrame. The `graph` property can be used after 
    `fit()` to obtain a graph representing the linkage fitted to data.  The 
    `groups` property can be used after `fit()` to obtain a membership vector 
    representing the clustering fitted to data.
    """
    @abstractmethod
    def fit(self, df):
        pass

    @property
    @abstractmethod
    def graph(self):
        pass

    @property
    @abstractmethod
    def groups(self):
        pass

rule = Any(Match("name"), Match("email"), Match("phone"))

rule.fit(data)
data.groupby(rule.groups).last() # Get last visit data for all customers.

import pandas as pd
import numpy as np
import itertools
from igraph import Graph

def _groups(rule, df):
    """
    Fit linkage rule to dataframe and return membership vector.

    Parameters
    ----------
    rule: string or LinkageRule
        Linkage rule to be fitted to the data. If `rule` is a string, then this 
        is interpreted as an exact matching rule for the corresponding column.
    df: DataFrame
        pandas Dataframe to which the rule is fitted.

    Returns
    -------
    Membership vector (i.e. integer vector) u such that u[i] indicates the 
    cluster to which dataframe row i belongs. 

    Notes
    -----
    NA values are considered to be non-matching.

    Examples
    --------
    >>> import pandas as pd
    >>> df = pd.DataFrame({"fname":["Olivier", "Jean-Francois", "Alex"], 
      "lname":["Binette", "Binette", pd.NA]})

    Groups specified by distinct first names:
    >>> _groups("fname", df)
    array([2, 1, 0], dtype=int8)

    Groups specified by same last names:
    >>> _groups("lname", df)
    array([0, 0, 3], dtype=int8)

    Groups specified by a given linkage rule:
    >>> rule = Match("fname")
    >>> _groups(rule, df)
    array([2, 1, 0])
    """
    if (isinstance(rule, str)):
        arr = np.array(pd.Categorical(df[rule]).codes, dtype=np.int32) # Specifying dtype avoids overflow issues
        I = (arr == -1)  # NA value indicators
        arr[I] = np.arange(len(arr), len(arr)+sum(I))
        return arr
    elif isinstance(rule, LinkageRule):
        return rule.fit(df).groups
    else:
        raise NotImplementedError()


def _groups_from_rules(rules, df):
    """
    Fit linkage rules to data and return groups corresponding to their logical 
    conjunction.

    This function computes the logical conjunction of a set of rules, operating 
    at the groups level. That is, rules are fitted to the data, membership 
    vector are obtained, and then the groups specified by these membership 
    vectors are intersected.

    Parameters
    ----------
    rules: list[LinkageRule]
        List of strings or Linkage rule objects to be fitted to the data. 
        Strings are interpreted as exact matching rules on the corresponding 
        columns.

    df: DataFrame
        pandas DataFrame to which the rules are fitted.

    Returns
    -------
    Membership vector representing the cluster to which each dataframe row 
    belongs.

    Notes
    -----
    NA values are considered to be non-matching.

    Examples
    --------
    >>> import pandas as pd
    >>> df = pd.DataFrame({"fname":["Olivier", "Jean-Francois", "Alex"], 
      "lname":["Binette", "Binette", pd.NA]})
    >>> _groups_from_rules(["fname", "lname"], df)
    array([2, 1, 0])
    """

    arr = np.array([_groups(rule, df) for rule in rules]).T
    groups = np.unique(arr, axis=0, return_inverse=True)[1]
    return groups

class Match(LinkageRule):
    """
    Class representing an exact matching rule over a given set of columns.

    Attributes
    ----------
    graph: igraph.Graph
        Graph representing linkage fitted to the data. Defaults to None and is 
        instantiated after the `fit()` function is called.

    groups: integer array
        Membership vector for the linkage clusters fitted to the data. Defaults 
        to None and is instantiated after the `fit()` function is called.

    Methods
    -------
    fit(df)
        Fits linkage rule to the given dataframe.

    Examples
    --------
    >>> import pandas as pd
    >>> df = pd.DataFrame({"fname":["Olivier", "Jean-Francois", "Alex"], 
    "lname":["Binette", "Binette", pd.NA]})

    Link records which agree on both the "fname" and "lname" fields.
    >>> rule = Match("fname", "lname")

    Fit linkage rule to the data.
    >>> _ = rule.fit(df)

    Construct deduplicated dataframe, retaining only the first record in each cluster.
    >>> _ = df.groupby(rule.groups).first()
    """

    def __init__(self, *args):
        """
        Parameters
        ----------
        args: list containing strings and/or LinkageRule objects.
            The `Match` object represents the logical conjunction of the set of 
            rules given in the `args` parameter. 
        """
        self.rules = args
        self._update_graph = False
        self.n = None

    def fit(self, df):
        self._groups = _groups_from_rules(self.rules, df)
        self._update_graph = True
        self.n = df.shape[0]

        return self

    @property
    def groups(self):
        return self._groups

# Part of the definition of the `Match` class:
    @property
    def graph(self) -> Graph:
        if self._update_graph:
            # Inverted index
            clust = pd.DataFrame({"groups": self.groups}
                                 ).groupby("groups").indices
            self._graph = Graph(n=self.n)
            self._graph.add_edges(itertools.chain.from_iterable(
                itertools.combinations(c, 2) for c in clust.values()))
            self._update_graph = False
        return self._graph

def pairwise(iterable):
    """
    Iterate over consecutive pairs:
        s -> (s[0], s[1]), (s[1], s[2]), (s[2], s[3]), ...

    Note
    ----
    Current implementation is from itertools' recipes list available at 
    https://docs.python.org/3/library/itertools.html
    """
    a, b = itertools.tee(iterable)
    next(b, None)
    return zip(a, b)


def _path_graph(rule, df):
    """
    Compute path graph corresponding to the rule's clustering: cluster elements 
    are connected as a path.

    Parameters
    ----------
    rule: string or LinkageRule
        Linkage rule for which to compute the corresponding path graph 
        (strings are interpreted as exact matching rules for the corresponding column).
    df: DataFrame
        Data to which the linkage rule is fitted.

    Returns
    -------
    Graph object such that nodes in the same cluster (according to the fitted 
    linkage rule) are connected as graph paths.
    """
    gr = _groups(rule, df)
    
    # Inverted index
    clust = pd.DataFrame({"groups": gr}
                         ).groupby("groups").indices
    graph = Graph(n=df.shape[0])
    graph.add_edges(itertools.chain.from_iterable(
        pairwise(c) for c in clust.values()))

    return graph

class Any(LinkageRule):
    """
    Class representing the logical disjunction of linkage rules.

    Attributes
    ----------
    graph: igraph.Graph
        Graph representing linkage fitted to the data. Defaults to None and is 
        instantiated after the `fit()` function is called.

    groups: integer array
        Membership vector for the linkage clusters fitted to the data. Defaults 
        to None and is instantiated after the `fit()` function is called.

    Methods
    -------
    fit(df)
        Fits linkage rule to the given dataframe.
    """

    def __init__(self, *args):
        """
        Parameters
        ----------
        args: list containing strings and/or LinkageRule objects.
            The `Any` object represents the logical disjunction of the set of 
            rules given by `args`. 
        """
        self.rules = args
        self._graph = None
        self._groups = None
        self._update_groups = False

    def fit(self, df):
        self._update_groups = True
        graphs_vect = [_path_graph(rule, df) for rule in self.rules]
        self._graph = igraph.union(graphs_vect)
        return self

    @property
    def groups(self):
        if self._update_groups:
            self._update_groups = False
            self._groups = np.array(
                self._graph.clusters().membership)
        return self._groups

    @property
    def graph(self) -> Graph:
        return self._graph

Model Evaluation

I cannot share any of the data which we have at the Pantry. However, I can describe general steps to be taken to evaluate model performance in practice.

Final thoughts

There are many ways in which the record linkage approach could be improved. As previously discussed, probabilistic record linkage would allow the consideration of negative evidence and the use of additional quasi-identifying information (such as IP addresses and other responses on the bag order forms). I’m looking forward to building on the GroupByRule Python module to provide a user-friendly and unified interface to more flexible methodology.

However, it is important to ensure that any record linkage approach is intelligible and rooted in a good understanding of the underlying data. In this context, the use of a well-thought deterministic approach can provide good performance, at least as a first step or baseline for comparison. Furthermore, it is important to spend sufficient time investigating the results of the linkage to evaluate performance. I have highlighted simple steps which can be taken to estimate precision and make a good effort at identifying missing links. This is highly informative for model validation, improvement, and for the interpretation of any following results.

Validating function input arguments in R

The easiest way is to manually incorporate checks.

mySum <- function(a, b) {
  if (!is.numeric(a) | !is.numeric(b)) {
    stop("Arguments should be numeric.")
  }
  if (length(a) != length(b)) {
    stop("Arguments should be of the same length.")
  } 
  
  return(a+b)
}

This works well enough, but it takes up a lot of space and you have to manually write up the description of the errors.

mySum <- function(a, b) {
  assert_that(is.numeric(a), is.numeric(b))
    assert_that(length(a) == length(b))
  
  return(a+b)
}

> mySum(1, "1")
        Error: b is not a numeric or integer vector 

source("assert.R")

mySum <- function(a, b) {
  assert(is.numeric(a), is.numeric(b))
    assert(length(a) == length(b))
  
  return(a+b)
}

> mySum(1, "1")
        Error: in mySum(a = 1, b = "1")
        Failed checks: 
            is.numeric(b) 

The framework

We continue in the same theoretical framework as before: is a set of densities on a complete and separable metric space with respect to a -finite measure defined on the Borel -algebra of , is the Hellinger distance defined by and we make use of the Rényi divergences defined by Here we assume that data is generated following a distribution having a density in our model (this assumption could be weakened), and therefore defined the off-centered Rényi divergence where assuming that all this is well defined.

Separation -entropy

We state our concentration results in terms of the separation -entropy. It is inspired by the Hausdorff -entropy introduced in Xing et al. (2009), although the separation -entropy has no relationship with the Hausdorff measure and instead builds upon the concept of -separation of Choi et al. (2008) defined below.

Given a set , we denote by the convex hull of : it is the set of all densities of the form where is a probability measure on .

Definition (-separation). Let be fixed as above. A set of densities is said to be -separated from with respect to the divergence if for every , A collection of sets is said to be -separated from if every $A {A_i}_{i=1}^$ is -separated from .

An important property of -separation, first noted by Walker (2004) and used for the study of posterior consistency, is that it scales with product densities. The general statement of the result is stated in the following lemma.

Lemma (Separation of product densities). Let , , be a sequence of -finite measured spaces where each is a complete and separable locally compact metric space and is the corresponding Borel -algebra. Denote by the set of probability density functions on , fix and let be -separated from with respect to for some . Let where is the product density on defined by . Then is -separated from with respect to where .

We can now define the separation -entropy of a set with parameter as the minimal -entropy of a -separated covering of . When this entropy is finite, we can study the concentration properties of the posterior distribution using simple information-theoretic techniques similar to those used in Bhattacharya (2019) for the study of Bayesian fractional posteriors.

Definition (Separation -entropy). Fix , and let be a subset of . Recall , and fixed as previously. The separation -entropy of is defined as where the infimum is taken over all (measurable) families , , satisfying and which are -separated from with respect to the divergence . When no such covering exists we let , and when we define .

Remark. When , so that , we drop the indicator and denote , to emphasize the fact.

Proposition (Properties of the separation -entropy). The separation -entropy of a set is non-negative and if is -separated from with respect to the divergence . Furthermore, if and , then
and if also , then
For a subset with , we have

and, more generally, if for subsets , then

Posterior consistency

Theorem (Posterior consistency). Let and let be a sequence of independent random variables with common probability density . Suppose there exists such that If satisfies for some , then almost surely as .

Remark. The condition implies in particular that .

Corollary (Well-specified consistency). Suppose that is in the Kullback-Leibler support of . If satisfies for some and for some , then almost surely as .

Corollary (Well-specified Hellinger consistency). Suppose that is in the Kullback-Leibler support of and fix . If there exists a covering of by Hellinger balls of diameter at most satisfying for some , then almost surely as .

Posterior concentration

Following Kleijn et al. (2006) and Bhattacharya et al. (2019), we let be a Kullback-Leibler type neighborhood of (relatively to ) where the second moment of the log likelihood ratio is also controlled.

Theorem (Posterior concentration bound). Let and let . For any and we have that holds with probability at least .

Corollary (Posterior concentration bound, i.i.d. case). Let and let be a sequence of independent random variables with common probability density . For any and we have that holds with probability at least .

References

• Bhattacharya, A., D. Pati, and Y. Yang (2019).Bayesian fractional posteriors.Ann.Statist. 47(1), 39–66.
• Choi, T. and R. V. Ramamoorthi (2008).Remarks on consistency of posterior distributions,Volume Volume 3, pp. 170–186. Beachwood, Ohio, USA: Institute of Mathematical Statistics.
• De Blasi, P. and S. G. Walker (2013). Bayesian asymptotics with misspecified models.StatisticaSinica, 169–187.
• Grünwald, P. and T. van Ommen (2017). Inconsistency of bayesian inference for misspecifiedlinear models, and a proposal for repairing it.Bayesian Anal. 12(4), 1069–1103.
• Kleijn, B. J., A. W. van der Vaart, et al. (2006). Misspecification in infinite-dimensional bayesianstatistics.The Annals of Statistics 34(2), 837–877.
• Ramamoorthi, R. V., K. Sriram, and R. Martin (2015). On posterior concentration in misspec-ified models.Bayesian Anal. 10(4), 759–789.
• Walker, S. (2004). New approaches to Bayesian consistency.Ann. Statist. 32(5), 2028–2043.
• Xing, Y. and B. Ranneby (2009). Sufficient conditions for Bayesian consistency. J. Stat. Plan.Inference 139(7), 2479–2489.

Gibbs posterior distributions

Gibbs Learning approaches this problem from a pseudo Bayesian point of view. While typically a Bayesian approach would require the specification of a full data-generating model, here we replace the likelihood function by the pseudo-likelihood function . Given a prior on , the Gibbs posterior distribution is then given by and satisfies whenever these expressions are well defined.

In the context of integrable pseudo-likelihoods, the above can be re-interpreted as a regular posterior distributions built from density functions and with a prior satisfying However, the reason we cannot apply standard asymptotic theory to the analysis of Gibbs posterior is that the quantity will typically be sample-size dependent. That is, if for i.i.d. random variables and if the loss separates as the sum then . This data-dependent prior, tilting by the function , is what allows Gibbs learning to target general risk-minimizing parameters rather than likelihood Kullback-Leibler minimizers.

Some of my ongoing research, presented as a poster at the O’Bayes conference in Warwick last summer, focused on understand the theoretical behaviour of Gibbs posterior distributions. I studied the posterior convergence and finite sample concentration properties of Gibbs posterior distributions under the large sample regime with additive losses . I’ve attached the poster (joint work with Yu Luo) below and you can find the additional references here.

Note that this is very preliminary work. We’re still in the process of exploring interesting directions (and I have very limited time this semester with the beginning of my PhD at Duke).

Squared error loss

Suppose is a prior on an euclidean parameter space with norm defined through the dot product. Given a likelihood for data , the posterior distribution is defined as

and the mean of the posterior distribution is

If we define the risk of an estimator for the estimation of a parameter as

and if

is the expected risk of with respect to the prior (also called the Bayes risk), then we have that the posterior mean estimate satisfies

for any estimator . That is, the posterior mean estimate minimizes the expected risk.

The proof follows from the fact that

Writing the expected risk as an expected posterior loss, i.e. using the fact that

where has density , and since

we obtain the result.

A few remarks:

1. The expected risk has stability properties. If and are two priors that are absolutely continuous with respect to each other, and if , then

If the risk is uniformly bounded by some constant over , then

This shows how small chances in the prior does not result in a dramatic change in the expected loss of an estimator, as long as the priors have “compatible tails” (i.e. a manageable likelihood ratio).

2. It is sometimes advocated to choose the prior so that the risk is constant over : the resulting estimator is then agnostic, from a risk point of view, to . This may result in a sample-size dependent prior (which is arguably not in the Bayesian spirit), but the fun thing is that it makes the expected risk maximal and the Bayes estimator minimax: . Indeed, in that case we have for any estimator that , from which it follows that is minimax.

3. The idea of minimizing expected risk is not quite Bayesian, since it required us to first average over all data possibilities when computing the risk. One of the main advantage of the Bayesian framework is that it allows us to condition over the observed data, rather than pre-emptively considering all possibilities, and we can try to make use of that. Define the posterior expected loss (or posterior risk) or an estimator , conditionally on , as

• It is clear from the previous computations that the posterior mean estimate minimizes the posterior risk, and hence the two approaches are equivalent. It turns out that, whatever the loss function we consider (under some regularity condition ensuring that stuff is finite and minimizers exist), minimizing the posterior risk is equivalent to minimizing the Bayes risk. In other words, we have that for any loss function (again under some regularity conditions ensuring finiteness and existence of stuff), we have

• This is roughly self-evident if we think about it. An interesting consequence is that any estimator minimizing a Bayes risk is a function of the posterior distribution.

2. Randomized estimation and information risk minimization

Let be a model, let be some data and let be a loss associated with using to fitting the data . For instance, we could have a set of functions, a set of features with associated responses, and the sum of squared loss.

There may be a parameter minimizing the risk , which will then be our learning target. Now we consider randomized estimators taking the form , where is a data-dependent distribution, and the performance of this estimation method can then be evaluated by the empirical risk .

Here we should be raising an eyebrow. There is typically no point in having the estimator being random, i.e. we typically will prefer to take a point mass rather than anything else. But bear with me for a sec. The cool thing is that if we choose

where is the Kullback-Leibler divergence, then this distribution will satisfy

That is, Bayesian-type posteriors arise by minimizing the empirical risk of a randomized estimation scheme penalized by the Kullback-Leibler divergence form prior to posterior (Zhang, 2006).

For the proof, write

which is also equal to and, by properties of the Kullback-Leibler divergence, obviously minimized at .

Is this practically useful and insightful? Possibly. But at least this approach is suited to a general theory, as shown in Zhang (2006) and as I reproduce below.

Let us introduce a Rényi-type generalization error defined, for , by

This is a measure of loss associated with the use of a parameter to fit new data . We also write

for the expected Rényi generalization error when using the randomization scheme .

In order to get interesting bounds on this generalization error, we can follow the approach of Zhang (2006).

3. Online learning, regret and Kullback-Leibler divergence

Following Barron (1998), suppose we sequentially observe data points which are say i.i.d. with common distribution with density . At each time step , the goal is to predict using the data . Our prediction is not a point estimate of , but somewhat similarly as in the randomized estimation scenario we output a density estimate , the goal being that be as large as possible. A bit more precisely, we individually score a density estimate through the risk which is the Kullback-Leibler divergence between and . The regret over times is the sum of the risk over the whole process, i.e.

Formally, this process is equivalent to estimating the distribution of all at once: our density estimate of would simply be

and the regret is, by the chain rule, simply , where is the th independent product of .

Given a prior over a space of distributions for , our problem then to minimize the Bayes risk

This is achieved by choosing the prior predictive density. This is equivalent to using, at each time step , the poterior predictive density .

To see this minimizing property of the Bayes average, it suffices to write

Note that an consequence of this analysis is also that the posterior predictive distribution will minimize the expected posterior risk:

Following section 1, this furthermore means that the posterior predictive distribution minimizes the Bayes risk associated with the Kullback-Leibler loss.