bitexploder
I started setting up my workflows using Temporal. It
deploys as relatively light weight local app. For an
isolated local installation it uses SQLite. It makes the
process of dealing with API retries and organizing
workflows and tasks really simple. I recommend giving it a
try. It is, philosophically, exactly what this article is
suggesting, but it adds an incredibly rich and flexible
interface for agents to work with. Additionally, the web
UI makes it very easy to inspect workflows, review agent
execution, etc. Temporal also encodes much higher
reliability into your system, almost for free. Distributed
and reliable systems are hard, don't reinvent the wheel
IMO.If you find yourself wanting things like an easy way
to then introspect your SQLite database, figure out what
is happening in the workflow, compose individual tasks,
make workflows trivially callable, etc, give Temporal a
look.Alongside this, I have mostly moved away from files
for agents. Markdown and JSON are great, but also feel
like traps when building out smaller local apps. LLMs are
great at SQLite and you can render anything you want out
of it (Markdown, JSON, etc). It saves a lot of tokens when
an agent can just query a specific row instead of having
to fire up jq or grep through markdown. You get a nice
portable self contained data management system that
encourages agents to be more disciplined about how they
structure their data than a bunch of files. It also
continues to scale into MySQL/Postgres if your little
local projects start to outgrow or become more formal, you
already have schema and discipline around data.
|
> fcarraldo
It sounds like you're running this mostly on a single
machine?
Temporal gets much more complex with scale. Cassandra
isn't fun to manage. Ringpop and TChannel are hard to
debug when things go wrong. The SQL backend support
doesn't support horizontally scaled replicas (just
single instance) due to consistency requirements.
Depending on how your code is written, modifying code
baked into workflows becomes complex, as anything that
modifies the history event ordering breaks determinism
in already-deployed workers.We use it heavily and
everyone who started on it doing simple
scripting/automation all love it, everyone who built
real production systems on top of it all hate it.
Possibly operator error, but my experience hasn't
matched the rosy picture painted in these comments.
|
> > inglor
In the last two years, we built (with a team of
15, now 100) a billion dollar business on top of
Temporal that performs business critical
applications for fortune 500 companies. We
couldn't be happier with temporal.Determinism
sucks, you do have to work hard and make
everything idempotent in activities like we would
for durable software anyway. The language we used
was incorrect (Go) and has a lot of boilerplate
compared to alternatives we later investigated
(Python and TypeScript). Visibility can be slow
and misses information. We needed to write our own
APIs to work effectively with Agents for
root-cause analysis of failures.With all the
caveats - Temporal is amazing, it feels much
better than previous orchestrators I used like
Prefect or Airflow. 100% would adopt again.
|
> > > bitexploder
That is the real truth people are voicing when
they say Temporal is heavy. They are really
saying: Durable, reliable, distributed
workloads are hard and it takes effort to
manage! And that is true. I know of no systems
that make that genuinely easy. It is a hard
discipline. Maybe Temporal makes that harder
than it should be, but I have no experience
there.There are no free lunches in this space.
I have no idea how good or bad Temporal is
since my usage is pretty small and isolated,
but software rarely just works and impresses
me and Temporal for my local machine
orchestrating genuinely did. I think Netflix's
conductor is another cool option, but I ended
up with Temporal due to license.
|
> > > > hmaxdml
Have you looked into DBOS? Same thesis:
durable and reliable workflows are hard to
manage -- it just doesn't have to be as
hard as Temporal makes it be :)
|
> > > > > bitexploder
Have not. For my workflows this was
fine. Good to keep in mind thoUgh. I
don't plan to manage a truly
distributed system with it. Plus my
only reason to do so is professional
and we rolled our own system here due
to our size solutions like DBOS or
Temporal would not work well.
|
> > > UnfitFootprint
Could you share a bit more about your
learnings on go + temporal? That combo was
next in line for us to migrate _to_
|
> > > > inglor
Sure, basically:- Temporal itself is
written in Go and we use Go for our
backend so we expected this to be a
natural fit.
- Temporal makes writing activities in Go
very explicit and boilerplatey
- This in turn makes testing more
difficult than it needs to be often
- Temporal doesn't play well with Go's
concurrency model at all (all stuff like
goroutines needs to go through its special
workflows.Go) a lot more often you have to
write stuff that "appeases" temporal.
- The whole
workflows.ExecuteActivity(...).Get(...) is
weird, having futures in a language
explcitly designed to avoid that is weird.
- All our compute isn't done on temporal
workers anyway, its done (in another AWS
account, owned by the customer) in batch
compute (aws batch, lambda, ec2, whatever)
so our temporal code isn't CPU heavy but
is highly concurrent and needs a very high
reliability guarantee.
- Compare that to temporal with
TypeScript, where it's simple and easy to
use the same code inside or outside of
temporal. Testing is trivial and the code
looks like "regular code".
|
> > > > > UnfitFootprint
Interesting. Perhaps the go sdk is in
alignment with the go-principle of
being explicit.
I'll take care to review examples of
goroutines etc under temporal before
calling any shots.
Thanks
|
> > taberiand
> Depending on how your code is written, modifying
code baked into workflows becomes complex, as
anything that modifies the history event ordering
breaks determinism in already-deployed workers.I
see this as Temporal surfacing inherent complexity
of the domain in a way that forces the developer
to consider it, rather than introducing extra
complexity.If it didn't make workflow determinism
a strict requirement, the requirement would still
exist - it would just hurt much worse in
production when it's broken.See also: Rust
borrowing
|
> > > inglor
Actually Temporal does have a way to avoid
determinism called rainbow deployments.If
you're fine with deploying several versions of
workers (and are on a reasonably new version)
you can just avoid the determinism issue
altogether with their k8s controller.If you do
need to have some long workflows, there is an
explicit hook for "what happens to existing
workflows on version upgrade".But to be fair -
none of the other orchastrators I used (like
AirFlow) made me write
workflows.IsNewCode/IsOldCode like temporal
does. On the other hand AirFlow doesn't even
have the capability to do that in the first
place (or at least it didn't last I used it).
|
> > > > taberiand
Well sure - that's essentially the same as
wrapping the whole workflow in a version
check for each version; copy-paste the
whole lot and change the code wherever.
It's still surfacing an issue that would
otherwise be less visible on a system that
did allow a worker on a new version to
pull an old half-completed workflow
|
> > kumulo
Temporal feels massive, I tried it for a small
workflow embedded on my system, and worked fine,
but when thinking on scaling it, it just didn't
made any sense for my use case.I also have
restate.dev on my reseearch list, which on paper
should scale well and be definitely more
lightweight and simple to setup, worth having a
look.
|
> > > burner_acc001
give conductor a try. runs with sqlite or if
you want to run it on a server mysql or
postgres. I know people running with postgres
with a decent scale and it works just
fine.https://docs.conductor-oss.org/
|
> > bitexploder
Yep. Individual systems with yolo agents doing
stuff in isolation. I could see how it can get
complex. Most distributed systems are. No free
lunches I guess. I am not sure what the
alternatives are at scale.
|
> > cnlwsu
We are a huge production setup where it's
absolutely critical successfully but we use
temporal cloud. Hosting it yourself is what makes
it miserable.
https://temporal.io/resources/on-demand/netflix
|
> > > hmaxdml
That's why their entire business model -- like
Astronomer's -- is geared toward cloud
hosting. The architecture is so complex it
takes a full time team to run it.
|
> > chrisss395
"gets much more complex with scale" feels like the
crux of it. Pick the right solution for your
intended scaleThat said, I appreciate this is hard
in practice. We need to start small to manage the
development rabbit hole risk, while also wanting
to dream big. There is a tension there that I find
hard to balance.
|
> > linkregister
Did I read your comment correctly that Temporal
includes Ringpop in their product? Interesting
choice to say the least.
|
> svara
Word on HN is that you're either paying more money
than you expected for temporal's managed solution or
taking on substantial ops burden ultimately running
their very heavy system yourself.I wouldn't know, I've
not done either, but I'd like to learn more from your
or other's experience.
|
> > bitexploder
I told an agent to set it up for me for some local
stuff. It is written in Go. It has a painless path
to run on a local SQLite DB. My agents use it to
organize and coordinate workflows. It handles
retries and long horizon tasks fine. As far as I
can tell for the core workflows and tasks pieces
it's great. MIT license. Like anything it isn't
free to manage but it offers a lot in return. High
reliability systems are hard. Temporal only solves
some of it. It is far better than rolling it
yourself.I think a genuine problem right now is
people are building agentic work flows and
learning the hard way highly reliable agentic work
flows are hard. Agents are unreliable. They are
both not deterministic and not the backing APIs
have pretty high error rates. Temporal has solved
that pain for me and made it easy to diagnose
problems.I don't have anything really large scale
running. But big enough that it takes billions of
tokens and high reliability to finish.
|
> > > platz
whats an example of things that you have your
agents do that use workflows and sqlite db
|
> > > > bitexploder
Autonomous C to Rust. Automated
penetration testing and vuln validation.
|
> > > > > trueno
you just made me realize how much i
wished people stopped talking in
abstractions and just stated what they
were doing. i hadnt realized how often
i saw things like "workflows" and just
kinda had my eyes glaze over. none of
it ever really clicks until i see the
true descriptor of whats going on.ive
been over here using claude relatively
simply as of recent, just claude code
and i might enter plan mode to do some
bigger like scrap together a test
suite of some sort, or i just have him
scripting and refactoring/reformatting
stuff under my direction. i wrote my
own cli tool (needed to bake in the
snowflake golang driver for external
browser sso propagation) and added it
as a skill so he can talk to our cloud
dbms when im doing analytics things
but for the most part its all pretty
simple. feel like my productivity is
50x but after over a year with claude
ive really backed off on asking him to
do insane stuff and mostly keep him
churning stuff out for me in domains i
know very well.so i read all this
workflow stuff that needs durability
and logging and im kind of astounded
how many people have their AI stuff
just running on their own round the
clock. i didn't realize how much of
peoples day to days needed to be
automated, i don't seem to find myself
surrounded by much that should be
automated. jira is probably the only
thing i need to sit down and automate
because its such a translation tax on
developers just so business people can
feel involved. but outside of that...
guess im behind the times, but i dont
know if its that. i see the big grand
things people use llms for ("im
creating the ultimate knowledge base"
or "ive automated everything under the
sun and im making 10k a week" etc) and
i am feeling either too tired, not
ambitious enough, or unenthused by the
creative and grand ways people are
working with AI. seems like everyone
has their own "perfect way to use AI"
but I can't seem to find the oomph to
go beyond using claude as a utility
anymore. a year ago (maybe more cant
remember anymore its all a blur) with
claude in the sonnet era i was so
amazed the first thing i did was try
to reverse engineer a game using
ghidra. had him building test suites
to verify the math was correct. we
were at this for weeks. my nearby
datacenter probably drained 10 lakes.
that was just one of _many_
over-ambitious projects i selected
because of claude that never saw a
finish line.yesterday i opened beej.us
and just started reading. im young and
i feel like i somehow went from 'damn
this claude shit is pretty cool' to
'AI is whatever its fine' in a year.
like the bell curve meme.
|
> > > > > > tauwauwau
Check out Matt Pocock's coding
workflow. His approach is
repeatable, consistent and is
backed backed by actual theories
in large software development.
|
> > > > > > trueno
thanks for the rec this actually
looks like an interesting way to
maybe prime myself to break a
little further into working with
these things at some larger scale
if I ever find the need. fav'd
this so i could come back to it.
|
> > > > > > simon84
About the same feeling here.
I guess not everything is about
global banking scale.I've tried
clever tricks to get AI produce
unsupervised stuff and came back
from it. The slop and loss of
cognitive knowledge about what it
did was uncomfortable to me... I
cannot understand how you would
hand off critical job to it.
|
> > kenforthewin
Could you expand on the "substantial ops burden"?
Let's say you're using a managed Postgres instance
as the underlying data store, how substantial is
the ops burden in that case? I understand that
temporal is actually a set of 4 or so
microservices on top of a data store, but if
you're already running a distributed system backed
by k8s or something like that, it doesn't seem
like it adds significant incremental ops on top of
that. But I could be wrong.
|
> > > tempest_
As a dev I would tell you its an ops burden.My
devops coworker just shrugs, pumps out some
yaml and helm and away it goes.It really
depends on your experience and tolerance for a
lot of things.Usually maintenance burden
doesent start to make itself known till you
get off the happy path or something breaks.
Sometimes it can be a long while before that
happens, sometimes it happens right away.
|
> > > graerg
I run my own temporal service in my k8s
cluster; this setup is the backbone for almost
all my applications. For simplicity, I opted
for the postgres backend. You still need to
run the 4 (?) other service (history,
matching, frontend, ui, maybe others,
definitely others if you want observability
with prometheus/grafana, and tad bit more
complexity if you want tailscale to get in
there and poke around).They ship Helm charts
so reality is somewhere between "helm deploy"
and "substantial ops burden". I don't have to
touch it very frequently, but that is not to
say I don't have to touch it. There's
occasional releases and there have been times
where (probably due to my inexperience with
helm) I botched an upgrade and lost some data.
And I've been on this journey for years; when
I first started, they didn't have a Python SDK
and it was one of my (many) excuses to learn
Go. But anyway to your point, yes, if you're
comfortable with k8s and Helm then you
shouldn't have much of a problem running
hundreds of thousands of workflows; if you
want to really push the throughput and
optimize cost you probably need to get
creative the individual services and look into
cassandra (maybe? idk).
|
> > > __turbobrew__
I think it depends a lot on the operational
maturity of the company. Some places are
running the LGTM observability stack, sentry
for error reporting, 24/7 on call rotations,
playbooks for all alerts, etc. Those
organizations will have less issues running
systems like temporal because the operational
framework is already there.Other orgs have
never heard of alerts or error reporting and
naturally will not catch issues until they are
catastrophic (for example services that crash
frequently in the background go unnoticed
until the crash frequency causes a
catastrophic failure). In my experience a lot
of issues are pretty simple such as running
out of memory, CPU throttling, crashes caused
by simple bugs (nil panics). If you have good
observability you can catch those issues
early.For example: people rag on Ceph that
their cluster somehow got into a broken state,
but that really only occurs when abuse of the
ceph cluster has went on long enough that the
cluster finally reaches the tipping point
where it is unrecoverable. If you set ceph up,
follow the correct replication rules so
components are spread across failure domains,
and use the metrics and alerts that are
distributed with ceph it is actually quite
hard to break the cluster.
|
> > > mnahkies
In my experience with a relatively modest
number of concurrent workflows (think
hundreds) you'll be pushing several thousand
transactions per second through that postgres
instance.As best I can tell it doesn't do any
batching of it's writes/reads, and it's update
heavy in places rather than append (I suspect
their cloud version might do some of these
things)It's pretty close to "let's make every
function call serialise it's parameters/return
value, go through a postgres table and several
network hops"That said it can be very useful,
but it's a heavy tool that's best suited for
high value/risk workflows where you're earning
enough from the execution that you can afford
the overhead (for example an Uber trip with
several dollars of service fees is probably a
good fit, unsurprisingly since it's roots are
from Uber)
|
> > > > bmurphy1976
At those rates if you aren't sharding your
infra you are doing it wrong.
|
> > edumucelli
Very heavy indeed, people will confuse the
durability that Temporal provide with all the
other properties a distributed system needs. They
will then think that Temporal will solve all their
problems.
|
> > dtech
Their managed solution is pricey and especially
the linear scaling with how much you use it is
very meh. It's comparable with AWS lambda which
also isn't cheap. However it's minor on a typical
cloud bill.Self-hosting is very easy in my
experience, I've done it for 2 years but
management wanted to move to Temporal Cloud. They
have a helm chart which just works including
upgrades. This does assume you have the whole k8s
shebang set up and working in your company. I
never had to touch is outside upgrades which took
maybe 30m including validation.
|
> > parthdesai
use oban and call it a day: https://oban.pro/
|
> bambax
My current client has a forest of 90+ SnapLogic
pipelines that were badly written and maintained even
worse; one of those was completely wrong, in that it
generated wrong accounting data which could eventually
have financial, fiducial and legal repercussions.I
rewrote the pipeline in Python (a correct version of
it) with state management in SQLite and logs in plain
old flat files, and everything has been running
smoothly ever since. In fact this is the only data
flow that has worked without errors or interruptions
in the last six months.Instead of replicating the db
file with Litestream I do a remote backup with Restic
before and after each run; it's not an exact
replacement of Litestream as we could possibly lose a
whole run if the machine died / disappeared at the end
of a run, but it lets one restore any day very easily.
In an ideal world I think we should have both (live
replica + backups).
|
> peterson_lock
This reads like an advertisement for Temporal :)
|
> > switchbak
I'm someone else who has inherited a bunch of
ad-hoc orchestration systems and also used
Temporal quite heavily. The latter does certainly
come with some overhead (not so bad in the age of
LLMs), but it also guides you along a well-trodden
path of good practices. The latter being very
important - it means that when you want to take on
more advanced capabilities, you probably haven't
painted yourself into a corner too badly and can
take that on fairly easily. Think: retries,
multi-tenancy, multi-lang, observability, etc.
|
> > dust42
I just spent the last two weeks digging into
workflow state engines and temporal was one of the
candidates. It is a VC backed fork of Cadence. The
got 0.3B funding and whatever positive I read
about them on the net I take with a big spoon of
salt. Just my 2 cents.
|
> > > pdimitar
I would love to read your analysis somewhere.
It's a current interest of mine, both hobby
and professional -- and limited to Elixir,
Golang and Rust -- and I'm still slowly
scoping the landscape.Recommendations for good
engines? Or just thoughts and analyses?
|
> > baq
Low key amazing tech, kinda like clickhouse -
nobody is bragging it's running their business
|
> > pzduniak
I can vouch for them too, being a super early
adopter. One of the best early bets I've ever
made. Awesome OSS product, glad the team decided
to leave Uber to commercialize it.
|
> > bitexploder
Well, just my experience. I installed it, had my
agents configure it and it immediately solved
problems I had with very little friction. Dealing
with long running, long horizon agentic tasks that
need very high reliability so I don't have to
babysit. I vibed the first version, realized I was
reinventing reliable distributed systems. Stopped
vibing and started surveying for something that
fit :)
|
> > hedgehog
It does, my experience has been that it adds code
complexity, deployment complexity, and performance
problems. There are some observability benefits,
but other ways to solve that. It's possible there
are workloads that fit it but not anything I've
personally worked on.
|
> jawns
Could you give an example of a case where you'd use
SQLite instead of jq or grep through Markdown?
|
> > phamilton
My favorite lens on SQLite is that it is actually
two things:1. A robust durability implementation
2. A library of high performance data structure
and algorithmsThe fact this it's SQL is nice, but
those two attributes are what make it great.For
example, I'm implement an in-process event log
that I want to be durable. I started simple, but
soon saw some edge cases and instead of playing
whackamole I just swapped to using sqlite as an
ordered kv store that gives me ACID.Another
example: ingesting multiple inter related
datasets. Instead of a dozen hash maps in memory,
I load them up into sqlite (no persistence) and
then slice and dice as I need to.It's a super
useful tool.
|
> > > rsalus
mirrors my own experience creating a
persistent event log. I started with JSON,
then JSONL, etc until finally landing on
SQLite.
|
> > chaps
The moment my JSON has any sort of depth and I
need to write a parser for it and potentially
account for unspecified behavior. JSON's nice when
it's nice, but it's terrible when it's terrible.
It's 100x easier to write SQL than writing jq
and... dear god if I have to use grep -A or -B,
I'm doing something wrong. Constraints are
actually a good thing!The underlying database
isn't the most important thing. Just use SQL. Its
namespacing (eg, through CTEs) is good and you're
more likely to have colleagues who know SQL
compared to jq.
|
> > > sofixa
> It's 100x easier to write SQL than writing
jq and... dear god if I have to use grep -A or
-B, I'm doing something wrong. Constraints are
actually a good thing!As an occasional
consumer of JSON/CSV, that's why I really like
DuckDB, it's just SQL for such file formats.
And it manages to be super fast at it too.
|
> > gopalv
> an example of a case where you'd use SQLite
instead of jq or grep through Markdown?Usually we
end up writing a script to incrementally refresh a
data-set I'm analyzing (or have someone send me a
copy after they pull it).I've been using sqlite
for anything which needs an UPDATE - modifying a
row deep inside the data-set with jsonl is a
pain.My github is full of java programs which
update sqlite3 files with threadpools and a single
big lock around the UPDATE (& then I write or have
an agent write code to analyze it).DuckDB is
slowly replacing it in the context of python,
simply because of the ease of pushing a UDF into
the SQL.Also because I really like expressing
things as LEAD/LAG with a UDF on top.
|
> > > dogline
UDF: User Defined Function
|
> > pokstad
SQLite is more efficient for large data sets. A
single markdown or JSON file needs to be streamed
to locate a piece of data O(n). Updating an
existing entry in a sequential file is even worse
because you have to rewrite the file. SQLite has
the data structures to quickly find data in O(log
n) time.
|
> > fragmede
Honest answer is: whenever your markdown or json
files get to be big enough that grep/jq takes long
enough that you get bored waiting for it.
|
> > > embedding-shape
> get to be big enough that grep/jq takes long
enoughOn a modern processor, that's about GBs
of data typically, right?
|
> > > > bitexploder
Practically yes, but much earlier if
agents are touching that data in my
experience. Tens of GB even if you design
well.
|
> rick1290
Interesting about the files vs db approach. I have
been going back and fourth. I landed on db as well.
|
> spwa4
What's wrong with looking up the algorithm called
"work stealing" and just implementing it? Making web
UIs for a working algorithm is something AI is really,
really good at (and I'll actually trust it with that)
|
> password4321
> It saves a lot of tokens when an agent can just
query a specific row instead of having to fire up jq
or grep through markdownJust wanted to make sure no
one missed this point in your comment because
eventually users will be paying the full cost for
tokens instead of VC's paying, with GitHub Copilot's
pricing realignment leading the way.
|
levkk
I don't understand this obsession with SQLite for real,
production apps. SQLite is an embedded database,
completely unsuitable for managing concurrency. This is
what database _servers_ are for, e.g., Postgres, MySQL,
etc. Their entire job is to allow you to modify data from
multiple processes, on different machines, at the same
time.This is a foundational principle of computer science.
It seems to me that the "SQLite for everything" crowd is a
little bit inexperienced.
|
> jph00
You seem to have a rather limited understanding of
what kinds of concurrency exist and how those needs
are best met. Whether something is a server or not is
not very relevant to this discussion.SQLite is an
excellent production db for many real world workloads,
as has been widely documented. It is very different to
Postgres, so requires learning a whole new skill
set.One way to think about it is that SQLite can work
well for the parts of your system where there is
naturally strong partitioning.
|
> > tasuki
> SQLite can work well for the parts of your
system where there is naturally strong
partitioning.Or the parts of your system that
don't have big data and no need for massively
concurrent writes. And that's the vast majority of
systems!
|
> > > MattJ100
You can do big data in SQLite. Concurrent
writes, sure, I'd recommend something else.If
you think the majority of systems require
massively concurrent writes, I think you need
to look a bit harder. SQLite is, after all,
the most widely deployed database system,
ever.
|
> > > > petre
We recently just partitioned the data into
many SQLite databases and got away with
it. It's telemetry data from IoT devices:
one device, one database. Backups are an
easy rsync job now instead of streaming a
multi gigabyte database with compression
that take hours. Reporting will just open
each database and aggregate multi device
data into another database (Duckdb, SQLite
or something else, we'll see). Duckdb is
not readable when locked so it's probably
also going to be SQLite. Even it it's
going to spit out JSON it will go into
SQLite rows instead of many files.
|
> > > > > datadrivenangel
Check out Quack for DuckDB.
|
> > > > > > petre
I saw that but it's a bit too much
friction. We'd rather attach a
SQLite database and copy the
results into that. Also no worries
about version changes and
compatibility, the types are not
too restrictive. But we'll see.
|
> > > > sharts
Internet Explorer 6 was the most widely
deployed awesome piece of software. Those
that hated it need to look a bit harder.
|
> > > > > Gud
It was not really "deployed" by a lot
of people in the same sense.It was
forced upon most of us(not me, I used
BeOS then Debian then FreeBSD).I
deployed phoenix.
|
> > > > > > HighGoldstein
The reason SQLite is the most
deployed is that it's used by
Android.
|
> > > > > > velcrovan
...and iOS, and Windows, and Mac
OS, and Boeing, and Sony, and
Firefox and Chrome and Safari...
|
> > > > > > HighGoldstein
Yes, which goes in line with the
argument that claiming that it's
"the most deployed" as proof of
superiority or suitability for any
use case is equivalent to claiming
the same for Internet Explorer.
It's the most deployed because
it's bundled in a lot of systems,
not because people are
purposefully using it as a DBMS.
|
> > > > > > velcrovan
But it doesn't, because none of
those systems are presenting
SQLite to the user as something
they should be using; they don't
even make SQLite available to the
user at all. Those systems all use
SQLite internally to manage data.
|
> > > > therealdrag0
It's widely deployed as a local DB for
local apps like phones, desktops, and web
browsers. But it's not the most used for
distributed, concurrent web apps, which db
servers were designed for. Maybe people
are talking past each other, but that's
the debate I see.
|
> > > > > kellogah
Not sure why there's a debate at all.
The discussion is on using SQLite
instead of jq and markdown files.
People got lost on a tangent! :)
|
> > > > > > gnabgib
No it's not.. the context of other
threads on this post (which
mention jq) do not apply here. How
poorly coded are you?
|
> > > > > > kellogah
Like how poorly did Jesus code my
DNA? Ask him. By him I mean
ChatGPT Jesus mode.
|
> > > andersmurphy
What's massive amount concurrent writes?
What's big data?
|
> > > > whatisthiseven
Such an exercise is left for the reader.
|
> > leowoo91
What additional skill set do you need to "learn"
for SQLite? Copying files around?
|
> > > Culonavirus
Yeah these are deeply unserious people.
|
> apatheticonion
For me, I have a use case that needs to support a few
thousand users, probably a few hundred
concurrently.The combination of SQLite (libsql, a
concurrent implementation of sqlite) and Rust means I
can do so from a $2/m VPS and a single server
instance.Backups are done via a cron job that uploads
to S3.Does it pass the "Netflix scale" test? NoBut it
doesn't need to. I'm not profiting from the service
and SQLite offers a path to scale if/when ready
because... well it's just SQL and I can literally just
swap `libsql::Connection` with `psql::Connection` in
my repositories.Plus upgrading from a $2/m VPS to a
$10/month VPS quadripples the number of concurrent
users I can support.IMO, you can vertically scale
extraordinary far with SQlite and an efficient server
implementation.I'd wager that 90% of forum websites,
wordpress sites and online shops would be fine with
SQLite.
|
> > Rohansi
> The combination of SQLite (libsql, a concurrent
implementation of sqlite) and Rust means I can do
so from a $2/m VPS and a single server
instance.You can probably do it with regular
SQLite, too. Being limited to a single writer
isn't as devastating as it sounds when they get
processed very quickly. Probably don't need Rust
either but it'll be more efficient than the usual
choices.(Also, it looks like libsql is the same as
SQLite? Only Turso has concurrent writes)
|
> > > apatheticonion
Makes sense. I honestly haven't thought about
it too much. I liked that I could move to
Turso if I needed cloud storage given the free
tier and that the API works with tokio nicely
|
> > andersmurphy
Why do you need libsql? Single writer tends to
scale better than concurent writes.
|
> abtinf
There are many cases where SQLite + concurrent front
end (like a go net/http server) can handle all the
load that a service might ever conceivably have to
handle, especially if allowed to scale up hardware
over time. You can trivially scale up SQLite to, what,
hundreds of thousands of tps?The only thing you really
give up is HA/failover and DR. But there are solutions
to deal with those. And single-server systems are
generally surprisingly robust (since, in the absence
of very complex control planes, uptime goes down with
more systems).
|
> > ummonk
Why go through the trouble of shoehorning SQLite
into a cloud database by getting solutions for
HA/failover and DR, when you can just use Postgres
off the shelf?
|
> > > jappgar
So you can post about it on HN, obviously
|
> > cortesoft
You also have the issue or normal maintenance
(patching, OS upgrades, etc). You can't do those
without downtime if you are using SQLite.
|
> > kellogah
I was thinking of using SQLite on top of
k3s/Longhorn to replicate it. Anyone do something
similar? Folks mention light steam and aws but
Jeff Bezos's biceps are too much for me to handle.
|
> > > andix
A longhorn volume can only be attached to one
node at at time. It can share it with other
nodes over nfs. I don't think this is going to
scale well.Just use Postgres with ro replicas.
|
> > > horsawlarway
I'll echo the other response.I've had pretty
terrible experiences with SQLite and
Longhorn/NFS.It's just not the right database
for pretty much ANY network based filesystem,
where the locking primatives aren't as robust,
and you might get two processes trying to hit
it at the same time.Frankly - they say this
themselves:
https://sqlite.org/howtocorrupt.htmlAs someone
who runs a fairly big personal cluster backed
by a mix of giant NFS storage for media, and
relatively large longhorn SSD drives for
configs/temp data...I avoid sqlite backing
like the plague. It will get corrupted.
Period. It's not the db for this use-case, and
I'll take postgres/maria/mysql/mongo/ANYTHING
else over it.If you do it - back it up ALL THE
TIME, because it's going to get corrupted.
|
> > > > andersmurphy
Yeah sqlite on anything but a directly
attached nvme is a bad time if you're
using it for a web server.
|
> peterspath
That's why there are billions of SQLite databases
right?SQLite is likely used more than all other
database engines combined. Billions and billions of
copies of SQLite exist in the wild. SQLite is found
in:Every Android device
Every iPhone and iOS device
Every Mac
Every Windows 10/11 installation
Every Firefox, Chrome, and Safari web browser
Every instance of Skype
Every instance of iTunes
Every Dropbox client
Every TurboTax and QuickBooks
PHP and Python
Most television sets and set-top cable boxes
Most automotive multimedia systems
Countless millions of other
applicationshttps://sqlite.org/mostdeployed.html
|
> > mr_toad
That's a comprehensive list of single user
devices.
|
> > > ibejoeb
Single-user, a single natural person, doesn't
striclty mean single-accessor though. I don't
think anyone here is suggesting that sqlite is
a viable replacement a for any networked
client/server postgresql system, but it is
certainly capable of handling more than the
most basic 1:1 tasks. Beyond that, the point
is that you only need a file, so when you have
natural data boundaries, a lot of problems
decompose to that single user/single concern
paradigm.
|
> > > larubbio
'production' doesn't equal 'multi-user
concurrent access'. There are production uses
where sqlite is a valid choice even if it may
not be the best choice for multi-user
production use cases.
|
> > > > therealdrag0
strawman? I have seen a dozens of these
debates and never once have I seen someone
questioned the validity of it for embedded
usecases.
|
> > ksd482
levkk is talking about concurrency. The list you
gave doesn't explain high concurrency requirements
for usage.
|
> > > rpdillon
My read is that levkk is conflating
concurrency with "real production apps" and
this whole thread is starting to surface that
"real production apps" and "high concurrency"
are not measuring the same thing at all.Sqlite
is used in real production apps more than any
other database.Sqlite is also weak at any sort
of write concurrency.Both can be true.
|
> > > DANmode
Why doesn't each of your users have a SQLite
database writing up to a main?You can have as
many as you want - and one is often plenty.
|
> > pibaker
GP calls out concurrency as a weakness of SQLite.
Most of the examples here don't experience the
same load even a moderately sized web service
experience day to day.And no, being a part of the
python standard library doesn't means it is being
used by the average python user. These days I'd
say at least half of them are just there for
machine learning.
|
> > > OutOfHere
SQLite is good for read-concurrency, not great
for write-concurrency.
|
> > > > simonw
SQLite requires writes run sequentially.
Most SQLite write operations take single
digit milliseconds or even microseconds.
If your writes are inexpensive (inserting
or updating single small rows) you'll
probably never even notice the queue.
|
> > > > > zarzavat
Exactly, people confuse "doesn't
scale" with "is a bottleneck". There's
many applications whereby hitting the
limits of SQLite is either a physical
impossibility, or implies that the
application has achieved success such
that replacing SQLite is the least of
anyone's problems.I visited a piano
store once that was running everything
off MS Access. If only they had
switched to HA technologies, they
would be able to sell millions of
pianos a day!
|
> > slashdave
I mean... if you count flat files as "databases",
there are a heck of a lot more
|
> > petcat
sqlite is great for the contacts app on your
phone, but that's it.Hipp even said that it is not
a replacement for a real multi-user, concurrent
RDMS. Its primary competitor is "fsync".
|
> > > Rohansi
SQLite is able to handle tens of thousands of
write transactions per second on modern
hardware. That is probably similar to or more
than your real, multi-user, concurrent RDBMS.
|
> > > > andersmurphy
Hundreds of thousands*
|
> franga2000
And I don't understand the obsession with server-based
databases for single apps. Especially in containerised
setups, every "app" gets its own database anyways, and
if the app is further broken down into services, they
usually communicate between each other and not with a
shared database. So in those cases, what do you gain
by pulling the database out of the "process" and onto
the other end of a socket? In most cases, absolutely
nothing. So why bother?Don't get me wrong, I've worked
with plenty of server-based databases, including
proper dedicated database servers. It's great tech and
often the best tool for the job. But not always and
I'd argue not in the majority of uses.
|
> > stingraycharles
"Especially in containerised setups, every "app"
gets its own database anyways, and if the app is
further broken down into services, they usually
communicate between each other and not with a
shared database. "You seem to be talking about a
vastly different use case.Containerized apps
having their own database? What? Aren't these
types of containers stateless? I always very much
try to keep state out of app containers.What kind
of data storage are we talking about?
|
> > > franga2000
If an app needs a database, it gets a database
server container, instead of getting a user
and database on a shared database server as
things used to be done. Every little django
app has its own postgres container. Every
wordpress site gets its own mysql container.
That is the modern way.Those database
containers get a PVC/volume/mount for their
data dirs. The only thing ever connecting to
them is their "owner" application container.
So at that point, why not drop the postgres
container and PVC mount a sqlite directory in
the app container? The result is the same.
|
> > > > jappgar
And when you need to scale to thousands of
instances of your microservice?
|
> > > > > stingraycharles
Yeah this is the part I don't get. It
seems like people are talking about 1
distinct app = 1 container and this is
the new normal? We're back to managing
cows instead of cattle again?
|
> > > > > > jappgar
I just think a lot of people here
haven't ever worked on large scale
systems. They don't know what the
don't know.
|
> > > > > > andersmurphy
I think a lot businesses build
large distributed systems
prematurely. They don't know what
they don't know.
|
> > > > > andersmurphy
What on earth needs thousands of
instances? Are you building a CDN?
Most apps can be a single server or
sharded by business/region.Honestly,
this whole leys run loads of nodes
seems to have sprung up from languages
that are slow oe don't have decent
concurrency.
|
> > > > > graerg
That's the whole thesis; YAGNI.
|
> > > > mxey
Yes if you run a database server like an
embedded application database, then it
won't be very different from an embedded
application database.
|
> > cortesoft
How do you do server maintenance or handle
hardware failure if your database is SQLite? You
are going to have to take downtime, even in the
best scenario.
|
> > > franga2000
1. Proxmox live migration or HA, Ceph
storage2. K8S DaemonSet, PVC backed by
probably Ceph3. Just..don't care? Do
maintenance outside of working hours, fix
issues quickly and explain things nicely to
your customers. Not everything is
google-scale. Most people can deal with some
downtime.And it's not like you won't have
downtime in let's say a postgres-backed app.
But now you have two "servers" to deal with.
|