Guide to data tools landscape for developers

- Found yourself on a data project and have no idea what they all are talking about? Feel excluded from all the fun discussions in the office kitchen? If only there were a humongous guide going over all the concepts and buzzwords......

Mapping Maritime Traffic with H3 Hexagons

- How I wrangled millions of raw vessel position reports into a structured H3 grid — and what that grid makes possible....

What is psuedo-relevance feedback?

- After retrieving BM25 or any ranked search results you might not realize it but you have new information about the......

Fragments: April 9

- I mostly link to written material here, but I’ve recently listened to two excellent podcasts that I can recommend. Anyone who regularly reads these fragments knows that I’m a big fan of Simon Willison, his (also very fragmentary) posts have earned a regular spot in my RSS...

Session vs Query based search evals

- To evaluate search we typically build a judgment list We transform clickstream data into evaluation data This labels a result......

Bayesian BM25 is cool

- Look at this math and grasp at its majesty: P(R) = P(R | BM25) * P(R | Emb) # Prob(Relevance) = lexical * embedding OK what’s so special about that? That’s an AND. A probabilistic way of combining scores so that when BOTH “things happen”, the final result becomes true. Here when...

Vector search means query understanding!?

- Good vector search means more than embeddings Embeddings don’t know when a result matches doesn’t match Similarity floors don’t work......

Analyse and run simulations on your energy usage

- I’ve been using the Irish energy provider Energia for 5 years or so (as of writing, 2026) and they used to have a useful insights dashboard that let me analyse my power usage. Well, they seem to have removed it so I built a handy dashboard that anyone can use. It’s at...

Ugly hack to force BM25 to 0-1

- Its convenient to have a lexical score normalized from 0 1 Sadly BM25 scores tend to be all over the......

BM25’s other parameter - WTF is k3?

- You may know BM25 lets you tune two parameters k1 how quickly to saturate document term frequency’s contribution b how......

High IDF doesn't always mean relevant search

- Rare terms have high inverse document frequency IDF BM25 scoring treats high IDF terms as more relevant Why We assume......

Constance Crozier: Forecasting s-curves is hard

- There was a famous Covid era chart that I always struggle to find, showing how hard it is to estimate an S curve while living through it. in the early days it seems that everything is exploding as an exponential and you always get hypey essays about how YOU, YOU DUMB DUMB, DONT...

Measure your rater reliability

- In the previous tip we discussed how pointwise 1 5 labels fall apart The expert rater gives only nit picky......

Statistics made simple

- I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand? I mean, sure, you might try to...

Free course: Cheat at Search Essentials

- A free introductory search course for anyone who wants better search without all the hard work...

Under the hood of Canada Spends with Brendan Samek

Highlights from my appearance on the Data Renegades podcast with CL Kao and Dori Wilson

LLM Judges aren’t the shortcut you think

- After the LLM judge hype curve crashes, what will come after?...

Bayesian A/B testing is not immune to peeking

Reasoning boosts search relevance 15-30%

- Kicking the tires on an initial, naive agentic search with some thoughts on how it could be improved further...

My review of Claude's new Code Interpreter, released under a very confusing name

Recreating the Apollo AI adoption rate chart with GPT-5, Python and Pyodide

Wiggling Into Correlation

- Jeff Kaufman shared some data around contra dance attendance as a function of requirements on wearing surgical masks. He compares this data to survey data, which is a useful way to validate in both directions. I found the plot compelling for a different reason – depending on how...

My take on blog analytics

- I recently read You do not need “analytics” for your blog because you are neither a military surveillance unit nor a commodity trading company by Leon Paternoster. It’s a well-argued piece, and I agree with the general thrust… but I also won’t be removing analytics from my site...