SIMD-friendly algorithms for substring searching (2018) (Score: 150+ in 12 hours)

Link: https://readhacker.news/s/6w7Zb
Comments: https://readhacker.news/c/6w7Zb

Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking (❄️ Score: 150+ in 5 days)

Link: https://readhacker.news/c/6vQEL

Hey HN! We're Shreyash and Bhavnick. We're building Chonkie (https://chonkie.ai), an open-source library for chunking and embedding data.
Python: https://github.com/chonkie-inc/chonkie
TypeScript: https://github.com/chonkie-inc/chonkie-ts
Here's a video showing our code chunker: https://youtu.be/Xclkh6bU1P0.
Bhavnick and I have been building personal projects with LLMs for a few years. For much of this time, we found ourselves writing our own chunking logic to support RAG applications. We often hesitated to use existing libraries because they either had only basic features or felt too bloated (some are 80MB+).
We built Chonkie to be lightweight, fast, extensible, and easy. The space is evolving rapidly, and we wanted Chonkie to be able to quickly support the newest strategies. We currently support: Token Chunking, Sentence Chunking, Recursive Chunking, Semantic Chunking, plus:
- Semantic Double Pass Chunking: Chunks text semantically first, then merges closely related chunks.
- Code Chunking: Chunks code files by creating an AST and finding ideal split points.
- Late Chunking: Based on the paper (https://arxiv.org/abs/2409.04701), where chunk embeddings are derived from embedding a longer document.
- Slumber Chunking: Based on the "Lumber Chunking" paper (https://arxiv.org/abs/2406.17526). It uses recursive chunking, then an LLM verifies split points, aiming for high-quality chunks with reduced token usage and LLM costs.
You can see how Chonkie compares to LangChain and LlamaIndex in our benchmarks: https://github.com/chonkie-inc/chonkie/blob/main/BENCHMARKS....
Some technical details about the Chonkie package: - ~15MB default install vs. ~80-170MB for some alternatives. - Up to 33x faster token chunking compared to LangChain and LlamaIndex in our tests. - Works with major tokenizers (transformers, tokenizers, tiktoken). - Zero external dependencies for basic functionality. - Implements aggressive caching and precomputation. - Uses running mean pooling for efficient semantic chunking. - Modular dependency system (install only what you need).
In addition to chunking, Chonkie also provides an easy way to create embeddings. For supported providers (SentenceTransformer, Model2Vec, OpenAI), you just specify the model name as a string. You can also create custom embedding handlers for other providers.
RAG is still the most common use case currently. However, Chonkie makes chunks that are optimized for creating high quality embeddings and vector retrieval, so it is not really tied to the "generation" part of RAG. In fact, We're seeing more and more people use Chonkie for implementing semantic search and/or setting context for agents.
We are currently focused on building integrations to simplify the retrieval process. We've created "handshakes" – thin functions that interact with vector DBs like pgVector, Chroma, TurboPuffer, and Qdrant, allowing you to interact with storage easily. If there's an integration you'd like to see (vector DB or otherwise), please let us know.
We also offer hosted and on-premise versions with OCR, extra metadata, all embedding providers, and managed vector databases for teams that want a fully managed pipeline. If you're interested, reach out at [email protected] or book a demo: https://cal.com/shreyashn/chonkie-demo.
We're eager to hear your feedback and comments! Thanks!

The Army’s Newest Recruits: Tech Execs From Meta, OpenAI and More (Score: 151+ in 1 day)

Link: https://readhacker.news/s/6w6fM
Comments: https://readhacker.news/c/6w6fM

Google Cloud Incident Report – 2025-06-13 (Score: 150+ in 11 hours)

Link: https://readhacker.news/s/6w8bd
Comments: https://readhacker.news/c/6w8bd

I have reimplemented Stable Diffusion 3.5 from scratch in pure PyTorch (🔥 Score: 156+ in 3 hours)

Link: https://readhacker.news/s/6w8Mn
Comments: https://readhacker.news/c/6w8Mn

Last fifty years of integer linear programming: Recent practical advances (Score: 150+ in 14 hours)

Link: https://readhacker.news/s/6w8bh
Comments: https://readhacker.news/c/6w8bh

Liquid Glass – WWDC25 [video] (❄️ Score: 150+ in 4 days)

Link: https://readhacker.news/s/6vTKE
Comments: https://readhacker.news/c/6vTKE

Waymo rides cost more than Uber or Lyft and people are paying anyway (❄️ Score: 152+ in 2 days)

Link: https://readhacker.news/s/6w2VV
Comments: https://readhacker.news/c/6w2VV

Student discovers fungus predicted by Albert Hoffman (❄️ Score: 150+ in 3 days)

Link: https://readhacker.news/s/6vW8D
Comments: https://readhacker.news/c/6vW8D

Solar Orbiter gets world-first views of the Sun's poles (❄️ Score: 154+ in 2 days)

Link: https://readhacker.news/s/6vZd7
Comments: https://readhacker.news/c/6vZd7

Seven replies to the viral Apple reasoning paper and why they fall short (🔥 Score: 155+ in 3 hours)

Link: https://readhacker.news/s/6w9pM
Comments: https://readhacker.news/c/6w9pM

Infinite Grid of Resistors (Score: 151+ in 10 hours)

Link: https://readhacker.news/s/6w9DF
Comments: https://readhacker.news/c/6w9DF

The Many Sides of Erik Satie (❄️ Score: 150+ in 6 days)

Link: https://readhacker.news/s/6vMMT
Comments: https://readhacker.news/c/6vMMT

Q-learning is not yet scalable (Score: 154+ in 11 hours)

Link: https://readhacker.news/s/6w9RC
Comments: https://readhacker.news/c/6w9RC

Inside the Apollo “8-Ball” FDAI (Flight Director / Attitude Indicator) (Score: 150+ in 21 hours)

Link: https://readhacker.news/s/6w8XD
Comments: https://readhacker.news/c/6w8XD

Datalog in Rust (Score: 150+ in 5 hours)

Link: https://readhacker.news/s/6wat9
Comments: https://readhacker.news/c/6wat9

How to modify Starlink Mini to run without the built-in WiFi router (Score: 151+ in 5 hours)

Link: https://readhacker.news/s/6wayj
Comments: https://readhacker.news/c/6wayj

Canyon.mid (Score: 153+ in 4 hours)

Link: https://readhacker.news/s/6waBb
Comments: https://readhacker.news/c/6waBb

Journalists Wary of Travelling to US Due to Palantir Surveillance (🔥 Score: 156+ in 2 hours)

Link: https://readhacker.news/s/6waMu
Comments: https://readhacker.news/c/6waMu

Show HN: McWig – A modal, Vim-like text editor written in Go (❄️ Score: 150+ in 3 days)

Link: https://readhacker.news/s/6w2Lt
Comments: https://readhacker.news/c/6w2Lt

Hey! Check out my "toy" text editor which I use as my daily driver.
Features
LSP autocomplete, goto definition, hover info
Tree-sitter support
Color themes (borrowed from the Helix text editor)
Lots of bugs
Macro support
Something like Emacs org-mode: Open test.txt, place the cursor at line 15, and press "Ctrl-C Ctrl-C".
This project was written as a "speed run" — not for speed in terms of time, but rather as an exercise to explore the text editor problem space without overthinking or planning ahead. It’s a quick and "dirty" implementation, so to speak.
https://github.com/firstrow/mcwig