Marco Pinto-Orellana

Optimization models in Python with PuLP

2026-11-07T00:00:00Z

This post is a written version of an invited lecture on Python for optimization and logistics that I taught at the Universidad San Francisco Xavier de Chuquisaca (USFX), in Sucre, Bolivia. The lectures follows four classic problems: placing two hospitals, the p-median problem, the traveling salesman problem and vehicle routing.

Tools

The models use PuLP, a Python library for writing linear programs, which calls GLPK (an open-source mixed integer linear programming solver) behind the scenes.

python -m pip install pulp

PuLP also ships with the CBC solver. If GLPK is not installed, replace GLPK(options=['--mipgap', '0.01']) in the code below with PULP_CBC_CMD(gapRel=0.01). Both stop once the solution is within 1% of the optimum.

A small helper file, utils.py, creates the variables in bulk and runs the solver:

from pulp import *

def create_variable_matrix(i, j, name="x_{i}_{j}", **other_args):
    return [[LpVariable(name.format(i=a, j=b), **other_args) for b in range(j)]
            for a in range(i)]

def create_variable_array(i, name="y_{i}", **other_args):
    return [LpVariable(name.format(i=a), **other_args) for a in range(i)]

def euclidian_distance(point1, point2):
    return ((point1[0] - point2[0]) ** 2 + (point1[1] - point2[1]) ** 2) ** 0.5

def create_distance_matrix(points):
    return [[euclidian_distance(a, b) for b in points] for a in points]

def resolve_problem(problem, show_variables=True):
    GLPK(options=['--mipgap', '0.01']).solve(problem)
    if show_variables:
        for v in problem.variables():
            if v.varValue != 0:
                print(v.name, "=", v.varValue)
    return value(problem.objective)

Note that LpVariable takes three main params: cat, the kind of variable ('Continuous' or 'Integer'); lowBound, its minimum (\(-\infty\) when omitted); and upBound, its maximum (\(+\infty\) when omitted).

An integer variable between 0 and 1 is a yes-or-no decision. A \(5 \times 10\) matrix of them, and a list of 17 non-negative integers can be written as:

f = create_variable_matrix(5, 10, name="F_{i}_{j}", cat='Integer', lowBound=0, upBound=1)
l = create_variable_array(17, name="L_{i}", cat='Integer', lowBound=0)

A model has the class LpProblem. It has overloaded the operator += to allow an interesting syntax for adding constraint: model += constraint:

problem = LpProblem("my_model", LpMinimize)   # or LpMaximize

Model A: two hospitals

Seven neighborhoods need two new hospitals, each built in one of the neighborhoods. Where should they go?

The seven neighborhoods, every possible connection between them, and their distance matrix.

A good solution should keep every neighborhood close to the hospital that serves it, so the objective is the sum of the distances from each neighborhood to its hospital. That needs two kinds of decisions to be modeled:

\(x_{ij} = 1\) if neighborhood \(i\) is served by a hospital in neighborhood \(j\), and \(0\) otherwise, for \(0 \le i, j \le 6\).
\(y_j = 1\) if a hospital is built in neighborhood \(j\), and \(0\) otherwise.

problem = LpProblem("two_hospital_problem", LpMinimize)
x = create_variable_matrix(7, 7, cat='Integer', lowBound=0, upBound=1)
y = create_variable_array(7, name="y_{i}", cat='Integer', lowBound=0, upBound=1)

Where \(d_{ij}\) is the distance between neighborhoods \(i\) and \(j\), the objective is

\[\min \sum_{i=0}^{6} \sum_{j=0}^{6} d_{ij}\, x_{ij} = 0.0\,x_{0,0} + 20.0\,x_{0,1} + 20.0\,x_{0,2} + 28.3\,x_{0,3} + \dots + 14.1\,x_{6,5} + 0.0\,x_{6,6}.\]

This means that the objective function needs to link all possible \(7 * 7\) combinations:

objective = (0.0 * x[0][0] + 20.0 * x[0][1] + 20.0 * x[0][2] + 28.3 * x[0][3]
             + 14.1 * x[0][4] + 22.4 * x[0][5] + 22.4 * x[0][6]
             + 20.0 * x[1][0] + ...)                 # 49 terms in all
problem += objective

There is a trivial solution: \(x=0\), i.e., the solution that minimizes the distance between hospitals and neighborhoods is to not build any hospital at all. So, we need additional constraints to make a solution valid:

Every neighborhood is served by exactly one hospital: \(\sum_{j} x_{ij} = 1\) for every \(i\). For neighborhood 0 that is \(x_{0,0} + x_{0,1} + \dots + x_{0,6} = 1\), and the same holds for the other six.
There are exactly two hospitals: \(\sum_{j} y_j = 2\).
A neighborhood can only be served by a hospital that exists: if \(x_{ij} = 1\), then \(y_j\) must be 1. We can write this as a linear constraint, \(x_{ij} \le y_j\) for every \(i\) and \(j\).

for i in range(7):                              # 1. one hospital per neighborhood
    problem += lpSum(x[i][j] for j in range(7)) == 1
problem += lpSum(y) == 2                        # 2. two hospitals
for i in range(7):                              # 3. only hospitals that exist
    for j in range(7):
        problem += x[i][j] <= y[j]

resolve_problem(problem, show_variables=True)

If we run the code, GLPK prints its progress, then the variables that are not zero:

x_0_4 = 1
x_1_5 = 1
x_2_4 = 1
x_3_5 = 1
x_4_4 = 1
x_5_5 = 1
x_6_4 = 1

We can interpret this as the hospitals should be built in neighborhoods 4 and 5. Neighbourhoods 0, 2, 4 and 6 use the hospital in 4; neighborhoods 1, 3 and 5 use the one in 5. The total distance is 58.2.

Solution drawn in black: two star-shaped groups of connections, centered on neighborhoods 4 and 5

Model B: p-median problem

The two-hospital model is a special case of a location problem. There are \(n\) customers to serve and \(m\) places where a service point could open. The p-median problem opens \(p\) of them and assigns every customer to one open service point, so that the sum of the distances from the customers to their service points is as small as possible.

20 random points joined by every possible connection

The variables are similar to those in model A: \(y_j = 1\) if a service point opens at \(j\), and \(x_{ij} = 1\) if customer \(i\) is served from \(j\). So is the model, with \(p\) in place of 2:

\[\begin{aligned} \min\ & \sum_{i=1}^{n} \sum_{j=1}^{m} d_{ij}\, x_{ij} \\ \text{subject to}\ & \sum_{j=1}^{m} x_{ij} = 1, && i = 1, \dots, n, \\ & \sum_{j=1}^{m} y_j = p, \\ & x_{ij} \le y_j, && i = 1, \dots, n,\ j = 1, \dots, m, \\ & x_{ij}, y_j \in \{0, 1\} \end{aligned}\]

This time we can write this objective function can be written as loop for any number of points and any \(p\). To keep the example simple, every customer is also a candidate place (\(m = n\)):

def p_median_problem(distance_matrix, facilities):
    n = len(distance_matrix)
    problem = LpProblem("p_median_problem", LpMinimize)
    x = create_variable_matrix(n, n, cat='Integer', lowBound=0, upBound=1)
    y = create_variable_array(n, name="y_{i}", cat='Integer', lowBound=0, upBound=1)

    problem += lpSum(distance_matrix[i][j] * x[i][j] for i in range(n) for j in range(n))
    for i in range(n):                          # each customer served once
        problem += lpSum(x[i][j] for j in range(n)) == 1
    problem += lpSum(y) == facilities           # exactly p service points
    for i in range(n):                          # only from open service points
        for j in range(n):
            problem += x[i][j] <= y[j]

    return resolve_problem(problem)

The best three service points for the 20 customers

Model C: the traveling salesman

This is one of the most classical examples in linear programming or optimization. Assume we know \(n\) cities and the distances between them, the traveling salesman problem looks for the shortest route that starts in one city, visits each of the others exactly once and comes back.

20 cities to visit

Let \(x_{ij} = 1\) if the route goes straight from city \(i\) to city \(j\), and \(0\) otherwise. Thus, the objective is the length of the route:

\[\min \sum_{i=1}^{n} \sum_{j=1}^{n} d_{ij}\, x_{ij}.\]

At first sight two constraints are enough and intutive enough: one road arrives at each city, and one road leaves it.

\[\sum_{i=1}^{n} x_{ik} = 1 \quad\text{and}\quad \sum_{j=1}^{n} x_{kj} = 1, \qquad k = 1, \dots, n.\]

for k in range(n):
    problem += lpSum(x[i][k] for i in range(n)) == 1   # one road in
    problem += lpSum(x[k][j] for j in range(n)) == 1   # one road out

Are those two enough? No!. This is a valid solution only using those two constraints:

A solution using only 2 constraints may involve several small loops

The answer is correct for the model: every city has exactly one road in and one road out. Nothing in the model informs that the roads must form a single loop, and several short loops are cheaper than one long one. We often called these loops as subtours.

There are several ways to rule them out. We use the flow formulation (Gavish&Graves, 1978). They solve the subtours issue in a clever way: the idea was to add a new variable: \(f_{ij} \ge 0\), an amount of flow sent along the road from \(i\) to \(j\). This amount is not measured and does not appear in the objective function, but it works as a variable that appears while the model is running (like a temporary variable in a programming language!). Using it, now we can model that city 1 need to send out \(n - 1\) units, and every other city keeps exactly one:

\[\sum_{i=1}^{n} f_{ik} - \sum_{j=1}^{n} f_{kj} = 1, \qquad k = 2, \dots, n.\]

Flow can only travel along roads that the route uses, and since no road ever needs to carry more than \(n - 1\) units,

\[f_{ij} \le (n - 1)\, x_{ij}, \qquad i, j = 1, \dots, n,\ i \ne j.\]

A subtour that does not pass through city 1 has no road bringing flow in from outside, and each of its cities must keep one unit. That is impossible, so every city has to be on the same loop as city 1. For the same reason, a city cannot be its own loop (\(x_{kk} = 1\)), which is why the code does not have to exclude those variables.

def travelling_salesman_problem(distance_matrix):
    n = len(distance_matrix)
    problem = LpProblem("travelling_salesman_problem", LpMinimize)
    x = create_variable_matrix(n, n, cat='Integer', lowBound=0, upBound=1)
    f = create_variable_matrix(n, n, name="f_{i}_{j}", cat='Integer', lowBound=0)

    problem += lpSum(distance_matrix[i][j] * x[i][j] for i in range(n) for j in range(n))
    for k in range(n):
        problem += lpSum(x[i][k] for i in range(n)) == 1          # one road in
        problem += lpSum(x[k][j] for j in range(n)) == 1          # one road out
    for k in range(1, n):                                          # each city keeps one unit
        problem += lpSum(f[i][k] for i in range(n)) - lpSum(f[k][j] for j in range(n)) == 1
    for i in range(n):                                             # flow only on used roads
        for j in range(n):
            if i != j:
                problem += f[i][j] <= (n - 1) * x[i][j]

    GLPK(options=['--mipgap', '0.01', '--tmlim', '30']).solve(problem)
    return value(problem.objective)

The --tmlim 30 option stops GLPK after 30 seconds, so on larger instances the result is the best route found in that time, not necessarily the shortest one.

With the flow constraints, the solution is a single route through every city

Model D: vehicle routing

The vehicle routing problem could be seen as a expansion of the TSP, where we have more than one salesman (or trucks in this case) and the traveler can only bring a limited amount of supply or goods. A depot, point 0, serves \(n\) customers, numbered 1 to \(n\). Customer \(k\) needs \(q_k\) units of goods, and the depot needs none (\(q_0 = 0\)). There are as many vehicles as needed, each carrying at most \(Q\) units. The goal is the cheapest set of routes such that:

every route starts and ends at the depot,
every customer’s demand is delivered, and
no route carries more than \(Q\) units in total.

A depot and 9 customers

Defining \(c_{ij}\) as the cost of traveling from \(i\) to \(j\), the variables \(x_{ij} = 1\) if a vehicle goes from \(i\) to \(j\), and \(f_{ij} \ge 0\), the amount of goods carried from \(i\) to \(j\), the model is then:

\[\begin{aligned} \min\ & \sum_{i=0}^{n} \sum_{j=0}^{n} c_{ij}\, x_{ij} \\ \text{subject to}\ & \sum_{j=0}^{n} x_{kj} = 1, \quad \sum_{i=0}^{n} x_{ik} = 1, && k = 1, \dots, n, \\ & \sum_{j=1}^{n} x_{0j} = \sum_{i=1}^{n} x_{i0}, \\ & \sum_{i=0}^{n} f_{ik} - \sum_{j=0}^{n} f_{kj} = q_k, && k = 1, \dots, n, \\ & f_{ij} \le Q\, x_{ij}, && i, j = 0, \dots, n,\ i \ne j. \end{aligned}\]

Line by line the model can be interpreted as we did before:

One vehicle leaves and one arrives at each customer
As many vehicles come back to the depot as leave it
Each customer keeps exactly its demand from the goods that pass through it
And, no road carries more than a vehicle holds

The flow constraints play the same role as in the TSP model: goods can only start at the depot, so every route must pass through it.

def vehicle_routing_problem(distance_matrix, demand, vehicle_capacity):
    n = len(distance_matrix)
    problem = LpProblem("vehicle_routing_problem", LpMinimize)
    x = create_variable_matrix(n, n, cat='Integer', lowBound=0, upBound=1)
    f = create_variable_matrix(n, n, name="f_{i}_{j}", cat='Integer', lowBound=0)

    problem += lpSum(distance_matrix[i][j] * x[i][j] for i in range(n) for j in range(n))
    for k in range(1, n):
        problem += lpSum(x[k][j] for j in range(n)) == 1          # one vehicle leaves
        problem += lpSum(x[i][k] for i in range(n)) == 1          # one vehicle arrives
    problem += (lpSum(x[0][j] for j in range(1, n))                # depot: as many out as in
                == lpSum(x[i][0] for i in range(1, n)))
    for k in range(1, n):                                          # each customer keeps q_k
        problem += lpSum(f[i][k] for i in range(n)) - lpSum(f[k][j] for j in range(n)) == demand[k]
    for i in range(n):                                             # vehicle capacity
        for j in range(n):
            if i != j:
                problem += f[i][j] <= vehicle_capacity * x[i][j]

    return resolve_problem(problem)

In the example every customer needs one unit and a vehicle carries five, so the nine customers need at least two vehicles. The best plan uses exactly two, which could make their deliveries at the same time.

Optimal route: 2 vehicles with their own loop from the depot

Numerical Hell: When Cross-Language Code Breaks (Part 1)

2025-02-09T00:00:00Z

I believe engineers have an unspoken rule: we trust our tools. When you design an IIR filter or an FIR filter, you expect identical results regardless of your programming language. While I suspected this assumption wouldn’t always hold, I never gave it much thought until recently.

Working with electrical brain signals, my lab analyzes bursts that could help identify seizure-generating tissue. Some of our legacy code runs in MATLAB, but I had my statistical and machine learning pipelines in Python. After having horrible experiences in the past with bugs while connecting MATLAB-Python, I had two options: either use an Python-Octave bridge (oct2py) or rewrite the algorithm in Python. Due to memory efficiency, I decided to port the algorithm entirely.

The plan was simple: implement everything identically and use a unit test battery that can confirm the results are the same. The code looked perfect until I ran the tests. To my surprise, around 20% failed. Well, it is known that some operations can raise sligtly different results on both platforms, as the inverse random transformation. Sometimes, due to the way the algorithm is implemented between MATLAB versions.

After two frustrating days of debugging, I isolated several issues. Let’s start with the simplest and most frustrating one: the FIR filter design functions (fir1 in MATLAB/Octave, firwin in scipy).

Minimal example

Here’s what we obtained as a FIR bandpass filter between 0.2\(f_s\) and 0.4\(f_s\) with a Hamming window (\(f_s\) is the sampling frequency) in Octave, python and R:

% Octave 9.3.0 / signal 1.4.6
pkg load signal
fir1(2, [0.1, 0.2], "bandpass")
% ans =
%    0.06310305169   0.88754953849   0.06310305169

% Matlab 2024b
format longg
fir1(2, [0.1, 0.2], "bandpass")
%ans =
%   0.063016142388235   0.887704412005799   0.063016142388235

# Python 3.9.16 / NumPy 1.25.1 / SciPy 1.11.1
import scipy.signal
scipy.signal.firwin(2+1, [0.1, 0.2], pass_zero="bandpass", window="hamming", scale=1)
# array([0.06301614, 0.88770441, 0.06301614])

# R 4.4.2 / signal 1.8-1
library(signal)
b = signal::fir1(2, c(0.1, 0.2), "pass", window="hamming")
# [1] 0.06310305 0.88754954 0.06310305

You might think “it’s just the Hamming window implementation.” Let’s try with a boxcar window:

% Octave
fir1(2, [0.1, 0.2], "bandpass", "boxcar")
% ans =
%    0.343971272   0.387038706   0.343971272

# Python
scipy.signal.firwin(2+1, [0.1, 0.2], pass_zero="bandpass", window="boxcar", scale=1)
# array([0.34376453, 0.38740713, 0.34376453])

# R
signal::fir1(2, c(0.1, 0.2), "pass", window="boxcar")
# [1] 0.3439713 0.3870387 0.3439713

These differences might seem negligible. But when I applied this filter to detect bursts in a signal, the results were surprisingly different. Assume that we record the following signal:

To be precise, these are the recorded values:

x = [ 1.76405235, 0.40015721, 0.97873798, 2.2408932 , 1.86755799, -0.97727788, 0.95008842, -0.15135721, -0.10321885, 0.4105985 , 0.14404357, 1.45427351, 0.76103773, 0.12167502, 0.44386323, 0.33367433, 1.49407907, -0.20515826, 0.3130677 , -0.85409574, -2.55298982, 0.6536186 , 0.8644362 , -0.74216502, 2.26975462, -1.45436567, 0.04575852, -0.18718385, 1.53277921, 1.46935877, 0.15494743, 0.37816252, -0.88778575, -1.98079647, -0.34791215, 0.15634897, 1.23029068, 1.20237985, -0.38732682, -0.30230275, -1.04855297, -1.42001794, -1.70627019, 1.9507754 , -0.50965218, -0.4380743 , -1.25279536, 0.77749036, -1.61389785, -0.21274028, -0.89546656, 0.3869025 , -0.51080514, -1.18063218, -0.02818223, 0.42833187, 0.06651722, 0.3024719 , -0.63432209, -0.36274117, -0.67246045, -0.35955316, -0.81314628, -1.7262826 , 0.17742614, -0.40178094, -1.63019835, 0.46278226, -0.90729836, 0.0519454 , 0.72909056, 0.12898291, 1.13940068, -1.23482582, 0.40234164, -0.68481009, -0.87079715, -0.57884966, -0.31155253, 0.05616534, -1.16514984, 0.90082649, 0.46566244, -1.53624369, 1.48825219, 1.89588918, 1.17877957, -0.17992484, -1.07075262, 1.05445173, -0.40317695, 1.22244507, 0.20827498, 0.97663904, 0.3563664 , 0.70657317, 0.01050002, 1.78587049, 0.12691209, 0.40198936]

Now, we need to know how many pulses are inferred if we use a detection threshold of 1 mean + 1 standard deviation. We got THREE different results based on the implementation:

Python implementation: 9 bursts
R implementation: 11 bursts
Octave implementation: 12 bursts

The implications are concerning. If such a basic operation produces different results, what other disparities might exist in more complex algorithms? Are other discrepancies due to language and operating systems (I tested the method on the same laptop)? Is our assumption of cross-platform consistency fundamentally flawed?

In the next post, I’ll explore why these differences occur and discuss potential solutions. For now, remember: when porting algorithms between languages, verify every numerical operation, no matter how basic it seems.

Octave code

output_precision (10)
b = fir1(2, [0.1, 0.2], "bandpass")
% b =
%    0.06310305169   0.88754953849   0.06310305169
x_filtered = filtfilt(b, 1, x);
y = x_filtered > mean(x_filtered) + std(x_filtered);
y = [0 y 0];
intervals = [find(diff(y)>0)', find(diff(y)<0)'];
size(intervals)
% ans =
%    12    2
intervals
% intervals =
%     1    2
%     4    6
%    12   13
%    17   18
%    25   26
%    29   31
%    37   39
%    44   45
%    73   74
%    85   88
%    92   93
%    98   99

Python code

b = scipy.signal.firwin(2+1, [0.1, 0.2], pass_zero="bandpass", window="hamming", scale=1)
b
# array([0.06301614, 0.88770441, 0.06301614])
x_filtered = scipy.signal.filtfilt(b, [1], x)
y = np.array(x_filtered) > (np.mean(x_filtered) + 1*np.std(x_filtered))
y = np.concatenate([[0], y, [0]])
interval_start = np.nonzero((np.diff(y)>0))[0]
interval_stop = np.nonzero((np.diff(y)<0))[0]
intervals = np.vstack([interval_start, interval_stop]).T
intervals.shape
# (9, 2)
intervals
# array([[ 5, 7],
#        [16, 18],
#        [31, 33],
#        [46, 47],
#        [49, 50],
#        [52, 53],
#        [57, 58],
#        [74, 75],
#        [88, 89]])

R code

library(signal)
b = signal::fir1(2, c(0.1, 0.2), "pass", window="hamming")
b
#[1] 0.06310305 0.88754954 0.06310305
x_filtered = signal::filtfilt(b, x)
y = x_filtered > (mean(x_filtered) + sd(x_filtered))
y = c(0, y, 0)
intervals = cbind(which(diff(y) > 0), which(diff(y) < 0))
dim(intervals)
# [1] 11  2
intervals
#       [,1] [,2]
#  [1,]    1    2
#  [2,]    3    6
#  [3,]   12   13
#  [4,]   17   18
#  [5,]   25   26
#  [6,]   29   31
#  [7,]   37   39
#  [8,]   44   45
#  [9,]   85   88
# [10,]   92   93
# [11,]   98   99

EDF files' specification

2024-11-11T00:00:00Z

It’s 2024 and ChatGPT have been trained on all the Internet’s available information. So, it should contain inforabout a widely used file format from the 90s, right? In particular, I needed help on a data-file format used in biomedical datasets since 1992: EDF+C (European Data Format for contiguous stored signals). And, several LLMs hallucinated the results, regurgitating that EDF+C was a gzip-compressed format. Because +C, the LLMs inferred that it meant “compression” and not “contiguous”. Given the age of the standard and its popularity among open-access datasets, it surprised me a lot.

But, I needed to known about the EDF+C binary format, and I digged into the specification to explore how this format is implemented. This is the tutorial for dummies like me.

Let’s recall that when signals are recording, we expected to have the following multidimensional pattern:

We should rememeber that this is an abstract representation of the data we will use for our algorithms, and not how the data will be stored in disk. Also, because there are additional information that could be valuable for our analysis such as when this data was recorded, a subject identifies, channel names, and so on, that should be stored somewhere. A proper standard should standardize the way to store such information, while also allowing a fast data-access, and that’s precisely what the EDF format does. In particular, it uses the following structure by adding metadata and partioning the entire dataset into records (in DL nomenclature, this would be equivalent to batches):

Now, a file dataset1.edf can be stored in either, the the EDF1990 or the EDF+2003 standard. The latter, EDF+2003, is almost compatible with EDF1990 with a couple of differences. The discrepancy that is most important to me is that EDF+2003 allows that contiguous records do not need to represent consecutive time intervals. This random recording behaviour is denoted as EDF+D (discontiguous EDF), while the classical contiguous structure is categorized as EDF+C. While, I think that EDF+D involves a huge penalty in computationally efficiency, that’s the standard.

General metadata

The first 256 of the file describes the main metadata: patient, date, time, storage mode, number of channels, records and the duration of each record. Note that classically, EDF names channels as signals, I’ll use channels here, becaue it’s less confusing that “signals”.

There are a few things that needs to be emphasized in this header:

The last field (bits 252-253) contains the total number of channels (or signals) in the file. So, the maximum number of simultaneous channels to be recorded is 9999.
This header uses only strings, so the fields marked with (N) are integers or floats saved as strings, and they will need to be parsed.
The segment Reserved(StorageMode) was supposed to be ignored in the EDF1990 standard. But in EDF+2003 will contain the string “EDF+C” or “EDF+D” to identify if the recordings are contiguous or not.

Channel-level metadata

Then, per each channel another 256 bytes are used to store channel-specific information. We need to emphasize that here, all values are ascii strings, even though that they represent numbers. I assume that the designers were inspired by the HDLC or TCP packet formats.

If the file only contains a single channel, this would be the header:

However, if there are several channels in the EDF file, each field is repeated for all channels before moving to the next field:

For instance, if we have three channels, the 768-byte metadata will have the following layout:

We need to note that each channel could have channels with different number of samples per record.

Sampling frequency?

If you check carefully, the EDF header does not specify the sampling frequency. Thus, how we get it? For example, for a channel \(q\), we check the number of samples per record \(M_p\) (10th field in the channel metadata) and the duration of each record per second \(T_r\) (bytes 244-251 in the general metadata). Then, the sampling frequency, in Hertz, for the channel \(p\) will be:

\[f^{(p)}_s = \frac{1}{M_p T_r}\]

Records

Records, or batches as I prefer to think, came after the channel-level metadata. And, they are also structured as the channel-level metadata: the data from one channel after the data from another channel, and so on.

EDF files organized the data in records, or batches (as I prefer to think about them). The records are saved consecutively:

That means that if we have \(N_{ch}\) channels, the first record (record 0) will start at byte \(256\cdot(1+N_{ch})\), just after the general metadata (first 256 bytes) and the channels’ metadata (next \(256\cdot N_{ch}\) bytes).

Each record has the same size, and it is organized in the following way, the data from one channel followed by the channel of the next one:

Note that each channel can have a different number of bytes assigned. This number (\(M_p\) for a channel \(p\)) is defined in the “samples per record” field in the channels’ metadata.

Thus, as we expected, to get the size of the record, we only need to sum all the sizes for each channel: \(N_{rec} = \sum_{p=0}^{N_{ch}-1} M_p\).

We need to emphasize that all records have the same size, and due to that, it’s possible to move to any record \(q\), because it will be start at the byte \(B_{(q)}\) and it will end exactly at byte \(B_{(q+1)}-1\):

\[B_{(q)} = 256\cdot(1+N_{ch}) + q\cdot N_{rec}\]

Why I stressing the position locations? Because, if you only need a portion of the data, fseek will be your friend (in C-based languages, of course).

Annotations

If you are familiar with biomedical data formats, you have seen that EDF files often contain annotations. But, until now, we only saw records and channels, and nothing else so far. The EDF+2003 standard define that the annotations should be stored in the last channel which should have the case sensitive label EDF Annotations. This implies that the values for this channel in each record are ascii strings and not numbers. So, a parser should be aware of that.

Innocent P-hacking

2023-02-11T00:00:00Z

As a researcher, reading and interpreting academic papers has become unavoidable in my daily routine. Lately, perhaps due to a Baader-Meinhof phenomenon, I have come across several papers exhibiting what I’d call non-malicious p-hacking, often stemming from an insufficient number of simulations.

About p-values

If you are not familiar with uncertainty estimation or p-values, allow me to provide a brief, albeit not entirely accurate, intuition: a p-value represents the probability that a measured (or observed) value is not merely a result of error or chance. Consider the following scenario: you flip a recently forged coin 20 times and get 14 heads (a 70% rate). How do you determine whether this coin is fair or not?

Let’s use a proof by contradiction. Imagine we have a theoretical-ideal fair coin and observe the values using it as a “reference” or null model:

[1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1]
[1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1]
(repeating this process a million times)
[1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1]

In this simulated world where a fair coin is flipped, how many times do we observe a rate equal to or higher than 70%? 57839 times, 5.87%, or 0.0587 (the exact value may vary a bit if you run the simulation yourself). By tradition, we use the values of 0.05 (or 0.01 sometimes) as a cutoff to determine whether something is “significant” or not.

In summary, in a simulated world where a fair coin is flipped, we still see 5.78% with 14 or more heads. Therefore, we cannot claim that what we observed is exceptionally unusual, and we cannot prove the coin is unfair (or, using a more statistical phrasing, we do not have enough statistical evidence to reject the possibility that the coin is fair).

Note that in this peculiar scenario, we know the experiment can be modeled using a binomial distribution. Therefore, we could be tempted to think there is a method to calculate the p-value analytically or in a closed form. However, while the variance of our estimate can be reasonably calculated, the p-value and the confidence intervals are particularly challenging to obtain.

The P-hacking issue

P-hacking is a pejorative term to highlight experiments that manipulate methods to achieve p-values below 0.05, passing them off as “statistically significant.” In our example, 14 out of 20 heads are not highly uncommon for a fair coin; it’s just a fortunate situation. However, the probability of seeing 14 or more heads is dangerously close to 0.05.

The interesting aspect here is that the “p-value” measured through simulations depends on the number of simulations conducted. We may get inconsistent results with fewer simulations, leading to incorrect reporting.

Thus, let’s make a meta-simulation of the simulations with different simulation sizes (assuming that creating a large number of simulations is computationally challenging):

With 10 simulations per scenario, 4506 out of 10000 (45.06%) agree that the coin could be fair. 54.94% will suggest that the coin was rigged!
With 100 simulations per scenario, 6914 out of 10000 (69.14%) agree with our simulations. 30.86% will suggest that the coin was unfair! Almost one-third of the cases!
With 1000 simulations per scenario, 8731 out of 10000 (87.31%) agree that the coin is fair.
With 10000 simulations per scenario, 9998 out of 10000 (99.98%) agree that the coin is fair.

This illustrates a common issue with simulations. Even in simple experiments like coin flipping, estimating p-values through simulations should be cautiously approached, primarily when used as a fixed threshold, because it could lead to diverse conclusions.

For real-world examples of this mistake in academic papers, I recommend Googling these:

EEG p-value "100 surrogate" "significance" site:https://www.ncbi.nlm.nih.gov

EEG p-value "500 surrogate" "significance" site:https://www.ncbi.nlm.nih.gov

x86-assembling, cross-compilation and C

2020-11-23T00:00:00Z

Working in research often means moving code between my laptop and various workstations or servers. This constant migration comes with a familiar headache: managing dependencies and permissions. While I could use Docker containers or write code in Go, I started wondering - could I achieve this using plain C with minimal OS interaction?

I’m specifically interested in raw numerical simulations like Metropolis-Hastings samplers - algorithms that essentially just work with vectors of numbers. These are perfect candidates for stripped-down implementations that should, theoretically, run anywhere. The ideal scenario in my mind is straightforward: compile my C code, upload the binary to a server, and run it with a simple:

chmod +x simulation
./simulation

While cross-compilation from Linux to Windows is possible with GCC, macOS binaries remain a challenge. This got me thinking about the minimal requirements I actually need:

Build dependencies under 10MB - no VMs or huge toolchains like MinGW
Cross-platform binaries that might sacrifice some optimization for portability
Simple compilation process for Windows/Linux deployment
Basic debugging capabilities (GCC/GDB would suffice)

The SmallerC + YASM approach

SmallerC is a stripped-down C compiler that generates 32-bit instructions. It’s not a complete C90 implementation, but it covers enough ground for numerical computing. What makes it particularly useful is its ability to work with various assemblers - this is where YASM comes in, providing the cross-platform compilation capability we need.

After some exploration, I found an interesting solution: using a cross-compiler assembler. Specifically, the combination of SmallerC and YASM caught my attention.

Here’s how to set it up SmallerC:

Clone the GitHub repository SmallerC, or download the ZIP source code. I’ll do it ~/.smlrcc, but you can use your favorite directory.

In order to create binary files in Windows, the files must be combined. Some of the compiler’s low-level instructions can cause anti-malware software to report false positives.

If you don’t trust the code, you can always read, check, and compile it yourself (just type ./configure && make in the root directory and that’s all).

Set the environment variable SMLRC. In Linux/macOS, you can do it by adding this at the end of ~/.profile:

export SMLRC=~/.smlrcc/v0100

There are three folders with the binaries ~/.smlrcc/v0100/binw, ~/.smlrcc/v0100/binl or ~/.smlrcc/v0100/binm in your system PATH. For instance, in macOS you can do it by adding this in ~/.profile:

export PATH="$PATH:~/.smlrcc/v0100/bin"

Then, we can install YASM:

Windows users can download it directly or use MSYS2 (my preferred approach)
On Debian/Ubuntu: sudo apt install yasm
On macOS with MacPorts: sudo port install yasm

Our first Cross-Hello World!

That’s all. Let’s test it with a simple Hello World sample.c:

#include <stdio.h>

int main(int argc, char** argv){
  printf("Hello world!");
  return 0;
}

Now we can generate binaries for each platform:

# Windows binary
smlrcc sample.c -win -o sample-win.exe

# Linux binary
smlrcc sample.c -Linux -o sample-lin

# macOS binary
smlrcc sample.c -macos -o sample-mac

The size differences compared to GCC (gcc -O2 -o GCC-sample-default sample.c) build are interesting:

$ ls -Alh | awk '{print $5, $9}'
357K GCC-sample-default
20K sample-lin
17K sample-mac
26K sample-win.exe
99 sample.c

Additional thoughts

After working with this setup, I’ve found both advantages and limitations worth considering:

The good parts:
- The project has 6 years of development behind it, showing reasonable maturity
- Cross-platform compilation feels surprisingly straightforward
- Binaries are compact with minimal dependencies
- The codebase is well-written and comprehensible, making it fixable if needed
The challenges:
- It only supports a subset of C90. More details here
- No integrated debugger (though GCC/GDB can fill this gap)
- Not quite production-ready for commercial applications
- May require code adjustments based on specific use cases as every user can have different experiences

For my research simulations, this approach has proven valuable. It’s not a universal solution, but for numerical computations that need to run across different platforms with minimal setup, it’s a practical alternative to heavier toolchains.

If you want to explore this setup yourself, here are the key resources:

SmallerC Compiler:
- Binaries: Windows , Linux, MacOS
- Documentation
YASM: Yasm Modular Assembler

Smoothing, exponential averages and digital filters

2020-11-23T00:00:00Z

I always found the wide application of digital filters fascinating. However, I think that their versatility is not always intuitive.

Here, I want to discuss the connection between filters and a commonly used smoothing algorithm.

Let’s consider a stream of data denoted as \(x\), which is extremely noisy. We seek a method to obtain a smoothed version with fewer spikes. A common technique for this purpose is the exponential average. In this method, we assume that the current value \(x(n)\) is too noisy, and we intend to use the streaming history to adjust it:

\[y(n)=\alpha x(n)+(1-\alpha)y(n-1)\]

Think of it this way: if we observe a value of 10, it doesn’t make much sense if all previous values were between 0 and 1. Perhaps a new event occurred, but it is more logical to assume that the increment wasn’t too large and it’s an error in the stream. The forgetting factor \(\alpha\) controls this change; the closer the value to 1, the less we rely on previous values of the system.

Now that we understand the logic behind this technique, let’s delve into a discussion. What does this smoothing algorithm have in common with a frequency filter? At least here, our goal is to clean the data, not select a frequency band.

Interestingly, our minds tend to assume that a clean signal implies a signal that is not changing drastically too often. I want to emphasize the idea of “not changing faster”; this means we associate low-band signals with “smooth signals.”

If the linguistic connection wasn’t clear enough, we can derive the spectrum of \(y\), by obtaining the discrete Fourier transform of \(y\):

\[Y(\omega)=\alpha X(z)+(1-\alpha)e^{j\omega}Y\left(\omega\right)\]

Rearranging the items,

\[Y(\omega)=\frac{\alpha}{1-\alpha e^{j\omega}}X(\omega)=G(\omega)X(\omega)\]

where \(G\) is the filter gain

\[\left\Vert G(e^{j\omega})\right\Vert =\left\Vert \frac{\alpha}{1-\alpha e^{j\omega}}\right\Vert\]

To easily check what kind of filter is G, let’s examine its behavior as \(z\) approaches different values:

Closer to 0Hz:

\[\left\Vert G(\omega\rightarrow0)\right\Vert =\left\Vert G(z\rightarrow0)\right\Vert =\frac{\alpha}{1-\alpha}\]

And closer to the Nyquist frequency (the maximum frequency that we can represent in the stream):

\[\left\Vert G(\omega\rightarrow\pi)\right\Vert =\frac{\alpha}{1+\alpha}\]

Now, let’s unravel the implications when \(\alpha\) takes specific values:

Minimal smoothing: \(\alpha=0.9\) (\(x\) is almost copying \(y\)). In this case, the filter gain goes from 9 (0 Hz) to 0.81 (Nyquist’s frequency), that is, a 9.09% decrease. A minimal decrease for a minimal smoothing.
Strong smoothing: \(\alpha=0.1\). In this case, we trust less in the current stream values \(x\) so, probably, we will tend to over-smooth it. In the spectrum, that means that gain changes from 0.111 at 0Hz plumbetting to 0.091 at the max. frequency, in other words, a significant 81.81% decrease.

So, we can confirm that there an interplay between concepts: smoothing implicates low-pass filtering, even if not consciously perceived in that way.

But, as a consequence, an intriguing question emerges: while “low-pass” is inherently associated with “smoothness” and “high-pass” with “noise”, does a subtle conceptual association extend to “band-pass” processes? At which point, we stop thinking that something is “noisy” or “smooth”?

Spectrum and eigendecomposition

2020-11-23T00:00:00Z

In the vault of things that I get amused every time I remember is the connection between the time series’ spectrum and matrices’ spectrum.

Having been trained as an engineer during my undergraduate years, I instinctively associate a signal’s spectrum with its oscillations. It’s the spectrum that unveils the predominant oscillatory components within the signal under analysis. But as a graduate student learning the dark arts of ~~magic~~ statistics, I’ve often encountered colleagues whose notion of spectrum was completely different from mine: for them, spectrum represents the sum of eigenvalues of the signal’s covariance matrix (as a consequence of the Wiener–Khinchin–Einstein theorem.)

The discussion I show here might seem too detailed for a book yet too basic for a academic paper. Nonetheless, I find it to be a stimulating mental exercise in linking seemingly distant concepts.

Let us start with spectral definitions in linear algebra: the spectrum of a square matrix is the set of its eigenvalues. (I think a different post is needed to discuss the proper natural interpretation of eigenvectors and eigenvalues.)

In the classical definition of a time series spectrum (or power spectrum density, PSD), the spectrum is the Fourier transform of the covariance function:

\[X\left(\omega\right) =\left|\mathcal{F} \left\{ \sigma_{xx}\left(\tau\right) \right\} \left(\omega\right) \right|^{2}\]

However, consider a scenario where we only have \(T\) points sampled from a weakly stationary time series. Then, the spectrum could be defined as

\[X\left(\omega\right)=\sum_{t=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j\omega t}\]

Then, we can construct a covariance matrix \(\Sigma_{xx}\) as

\[\Sigma_{xx}=\left(\begin{array}{cccc} \sigma_{xx}\left(0\right) & \sigma_{xx}\left(1\right) & \cdots & \sigma_{xx}\left(T-1\right)\\ \sigma_{xx}\left(-1\right) & \sigma_{xx}\left(0\right) & \cdots & \sigma_{xx}\left(T-2\right)\\ \vdots & \vdots & \ddots & \vdots\\ \sigma_{xx}\left(-T+1\right) & \sigma_{xx}\left(-T+2\right) & \cdots & \sigma_{xx}\left(0\right) \end{array}\right)\]

The nice trick here is to remember that due to weak stationarity, the correlation at \(-\tau\), \(\sigma_{xx}\left(-\tau\right)\), will be the same at \(\tau\), \(\sigma_{xx}\left(\tau\right)\). This symmetric property makes \(\Sigma_{xx}\) to be a circulant matrix.

Then, let’s proceed with an eigendecomposition of \(\Sigma_{xx}\), where \(Y\) will be the eigenvectors and \(\gamma\) the eigenvalues:

\[\Sigma_{xx}Y=\gamma Y\]

While this expression may seem complex, focusing on the \(k\)-th row of \(\Sigma_{xx}\) simplifies matters:

\[\gamma_{k}Y_{k}=\sum_{\tau=0}^{T-1}\sigma_{xx}\left(\tau-k\right)Y_{\tau}=\sum_{\tau=-k}^{T-1-\tau}\sigma_{xx}\left(\tau\right)Y_{\tau+k}\]

Recall that \(k\) represents the lag index and it can go from \(0\) to \(T-1\): \(\left(0\le k\le T-1\right)\).

Now, a last trick: assume that \(Y_{k}\) has some periodicity. That means that the values \(Y_{k}\) would start repeating after some period. Even if we do not think that condition is true, the period can be made as large as possible so that, in practice, no periodic values would be observed.

Considering the periodicity assumption for \(Y_{k}\), let’s say \(Y_{k}=e^{-j2\pi\frac{k}{T}\tau}\) for some \(\tau\). This allows us to define the \(k\)-th eigenvalue as

\[\gamma_{k}e^{-j2\pi\frac{k^{2}}{T}}=\sum_{\tau=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j2\pi\frac{k}{T}\left(\tau+k\right)}\]

Simplifying further,

\[\gamma_{k}=\sum_{\tau=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j2\pi\frac{k}{T}\tau}\]

Now, replacing \(\omega=2\pi\frac{k}{T}\), we notice striking similarities with the PSD equation:

\[X\left(\omega\right)=\sum_{t=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j\omega t}\]

This expression establishes a deeper connection- the PSD is also the set of eigenvalues of the Fourier transform of the covariance matrix.

In essence, the matrix’s spectrum and a time series’ spectrum share a special link that was unclear to me for a long time. And I would try to explain why it is still amusing to me:

In my mind, I picture a matrix as a fixed object, as something that holds information in rows and columns. This object has nice descriptive elements: the eigenvalues/eigenvectors. We can calculate trace and determinants with them. And we can check if systems will be stable using them too. Yet, a matrix, per se, remains unchanged unless it is decided to be time-varying.
On the other hand, we have a time series, a perpetually fluctuating entity, ever-changing from one moment to the next. And a crucial information of this time-living object is the spectrum.
- In this time series, a matrix emerges when you calculate the average correlation between each point in the signal with any other past value and organize this information into rows and columns. Again, this matrix is a fixed, non-time-varying entity. But, the spectrum of this newly-constructed matrix mirrors the spectrum of the original time series.