About a month ago
- Embeddings take up a lot of space. When I embed the MSMarco passage corpus with the 384 dimension MiniLM model, I quickly get to ~14GB in memory embeddings: 9,000,000 records x 384 dimensions x 4 bytes per float 32 ------- ~ 14GB It doesn’t take much for millions of PDFs turn...
- Found yourself on a data project and have no idea what they all are talking about? Feel excluded from all the fun discussions in the office kitchen? If only there were a humongous guide going over all the concepts and buzzwords......
3 months ago
- How I wrangled millions of raw vessel position reports into a structured H3 grid — and what that grid makes possible....
4 months ago
- After retrieving BM25 or any ranked search results you might not realize it but you have new information about the......
- I mostly link to written material here, but I’ve recently listened to two excellent podcasts that I can recommend. Anyone who regularly reads these fragments knows that I’m a big fan of Simon Willison, his (also very fragmentary) posts have earned a regular spot in my RSS...
5 months ago
- To evaluate search we typically build a judgment list We transform clickstream data into evaluation data This labels a result......
- Look at this math and grasp at its majesty: P(R) = P(R | BM25) * P(R | Emb) # Prob(Relevance) = lexical * embedding OK what’s so special about that? That’s an AND. A probabilistic way of combining scores so that when BOTH “things happen”, the final result becomes true. Here when...
- Good vector search means more than embeddings Embeddings don’t know when a result matches doesn’t match Similarity floors don’t work......
- I’ve been using the Irish energy provider Energia for 5 years or so (as of writing, 2026) and they used to have a useful insights dashboard that let me analyse my power usage. Well, they seem to have removed it so I built a handy dashboard that anyone can use. It’s at...
- Its convenient to have a lexical score normalized from 0 1 Sadly BM25 scores tend to be all over the......
- You may know BM25 lets you tune two parameters k1 how quickly to saturate document term frequency’s contribution b how......
- Rare terms have high inverse document frequency IDF BM25 scoring treats high IDF terms as more relevant Why We assume......
6 months ago
- There was a famous Covid era chart that I always struggle to find, showing how hard it is to estimate an S curve while living through it. in the early days it seems that everything is exploding as an exponential and you always get hypey essays about how YOU, YOU DUMB DUMB, DONT...
- In the previous tip we discussed how pointwise 1 5 labels fall apart The expert rater gives only nit picky......
8 months ago
- I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand? I mean, sure, you might try to...
- A free introductory search course for anyone who wants better search without all the hard work...
9 months ago
10 months ago
- After the LLM judge hype curve crashes, what will come after?...
- Kicking the tires on an initial, naive agentic search with some thoughts on how it could be improved further...
11 months ago
- Jeff Kaufman shared some data around contra dance attendance as a function of requirements on wearing surgical masks. He compares this data to survey data, which is a useful way to validate in both directions. I found the plot compelling for a different reason – depending on how...
Rows per page
