7 months ago, you guys gave me feedback on my SQLite wrapper. 374 commits later, it's a real NoSQL e
Posted by cwt114@reddit | Python | View on Reddit | 9 comments
7 months ago, I shared NeoSQLite v1.0.0 here. It was a simple idea: Give SQLite a PyMongo API so we can have the NoSQL experience in Python without the "NoSQL Server" overhead.
The feedback was amazing (and admittedly, a bit brutal). You guys rightly pointed out the flaws and edge cases. So, I went back to the lab. 374 commits later, it's no longer just a "wrapper" falling back to Python loops—it's a full-blown database engine.
What My Project Does
NeoSQLite gives you the complete NoSQL/MongoDB experience in Python without the infrastructure overhead. It turns a standard SQLite database into a MongoDB-compatible engine.
For Python apps, it's a completely serverless, in-process library. But for this release, I also built the "Magic trick": NX-27017, an optional (and permanently experimental) tiny daemon that speaks the actual MongoDB wire protocol. You can point any existing project, GUI tool like MongoDB Compass, or non-Python app at a single SQLite file with zero code changes.
# Terminal 1: nx-27017 --db myapp.db
# Terminal 2:
from pymongo import MongoClient
# This is the real PyMongo client, but it's talking to SQLite!
client = MongoClient('mongodb://localhost:27017/')
db = client.my_app
db.users.insert_one({"name": "Alice", "tags": ["python", "sqlite"]})
Target Audience
This is meant for production use in specific contexts: desktop apps, CLI tools, local development environments, IoT devices, and small-to-medium backend services.
If you are building a massive, horizontally scaled enterprise cluster, use a real server. But if you want a drop-in PyMongo replacement that lives in a single file, this is for you.
I know replacing your database engine sounds terrifying, so to sleep at night, I've built a testing suite of 2,600+ unit tests and an automated "compatibility lab". It runs 377 different complex scenarios against both NeoSQLite and a real MongoDB instance to assert the results are strictly identical. We are sitting at 100% API parity for all comparable features.
Real-World Usage
It's actually being used out in the wild now! For example, Andy Felong recently wrote a full blog post about using NeoSQLite for his astronomy projects across a Raspberry Pi Zero, a headless Ubuntu server, and a Mac:
The fact that I can write an app's database layer once and have it run identically on a Pi Zero, an Ubuntu server, and macOS — all without starting up a single server process — is exactly the kind of pragmatic elegance I love in open-source software.
Comparison
- vs. MongoDB: You get the exact same PyMongo API, but without managing a Docker container, replica sets, or a heavy server process.
- vs. Postgres with JSONB: Postgres is incredible for massive web apps. But if you're building a desktop app, a local CLI tool, or a small service, managing a Postgres server is overkill. NeoSQLite gives you similar JSON querying power with zero infrastructure setup.
- vs. TinyDB / Simple Wrappers: NeoSQLite isn't just a basic dictionary store. I wanted it to be a drop-in replacement for real apps, so it fully supports ACID Transactions (with_transaction), Change Streams (watch()), GridFS, and complex Window Functions ($setWindowFields).
Making it "Production Fast"
In the early days, complex queries were slow because I was evaluating them in Python. I've spent the last few months pushing that logic down into raw SQL:
- Hash Joins: $lookup (joins) used to be O(n*m). It's now O(n+m) using a custom hash-join algorithm implemented in the query engine. It's the difference between a 10-second query and a 10ms one.
- Translation Caching: If you run the same query often, the engine now "learns" the SQL translation and caches the AST. It's about 30% faster for repeated operations.
- JSONB Support: If you're on a modern version of SQLite (3.45+), NeoSQLite automatically detects it and switches to binary JSON (JSONB), which is 2-5x faster across the board.
Try it: pip install neosqlite
GitHub: https://github.com/cwt/neosqlite
I'd love to hear your thoughts. Roast me again, or tell me what feature is keeping you tied to a "real" database server for local dev!
The Boring Stats for those interested: 374 commits since v1.0.0, 460 files changed (+105k lines), 30+ releases.
Python-ModTeam@reddit
Hello from the r/Python moderation team,
We appreciate your contribution but have noticed a high volume of similar projects (e.g. AI/ML wrappers, YouTube scrapers, etc.) or submissions that do not meet our quality criteria. To maintain the diversity and quality of content on our subreddit, your post has been removed.
All showcase, code review, project, and AI generated projects should go into the pinned monthly Showcase Thread.
You can also try reposting in one of daily threads instead.
Thank you for understanding, and we encourage you to continue engaging with our community!
Best, The r/Python moderation team
Full-Definition6215@reddit
374 commits from a wrapper to a full NoSQL engine is a great arc. The "you guys gave me feedback" → iterating for 7 months is how open source should work.
I run a production SaaS on SQLite (WAL mode) — paid article platform with Stripe payments, OAuth, the whole stack. People always ask "why not Postgres?" and the answer is: SQLite handles my traffic fine, backups are just copying one file, and there's zero operational overhead.
Curious about your indexing — how does it handle compound queries across multiple "collections"? That's where document stores usually shine vs rolling your own on top of SQLite.
cwt114@reddit (OP)
It translates MongoDB-style pipelines into optimized SQL. $lookup becomes a LEFT JOIN (using json_group_array for the result), $unionWith becomes a UNION ALL, and $graphLookup uses recursive CTEs. To keep it fast, it creates functional indexes on the permanent JSON data (using json_extract). When a pipeline is too complex for a single query, it breaks things down step-by-step: it creates temporary tables, automatically indexes them to keep joins fast, and drops them when the pipeline finishes. It basically manages the routing between simple joins and temp tables.
Berlibur@reddit
Pretty rad
mobbade@reddit
“it's no longer just a "wrapper" falling back to Python loops—it's a full-blown database engine.”
Holy AI. Might as well just talk to Claude at this point
cwt114@reddit (OP)
You're saying that all 7 months works just mean nothing when I use AI help me draft the post?
mobbade@reddit
You spend my 7 months working on a project and couldn’t write a 3 paragraph summary about it yourself?
cwt114@reddit (OP)
Fair enough.
AliceCode@reddit
Yep. Exactly. How else are people supposed to filter out bullshit?