_______ __ _______
| | |.---.-..----.| |--..-----..----. | | |.-----..--.--.--..-----.
| || _ || __|| < | -__|| _| | || -__|| | | ||__ --|
|___|___||___._||____||__|__||_____||__| |__|____||_____||________||_____|
on Gopher (inofficial)
HTML Visit Hacker News on the Web
COMMENT PAGE FOR:
HTML Caddy compatibility for zeroserve: 3x throughput and 70% lower latency
kennethallen wrote 3 hours 45 min ago:
I read this post and your post introducing zeroserve. One of the main
parts of the original pitch was "no separate config file and scripts".
Now you're adding a config file separate from the scripts. Is the pitch
now that it's like Caddy but runs eBPF scripts in userspace?
stymaar wrote 7 hours 5 min ago:
Can someome enlighten me: What's the point of ârunning eBPF scripts
in userspaceâ? Isn't being run in kernel space the whole point of
eBPF in the first place?
pbohun wrote 16 hours 36 min ago:
I looked into writing an http server based on iouring myself, but all
the resources I could find said iouring is less safe from a
cybersecurity perspective.
Is there a safe way to use iouring for a webserver, or is libuv the
better way to go, even though it has less performance?
athrowaway3z wrote 3 hours 9 min ago:
A webserver shouldn't be calling io_uring internals without doing its
own bound checking and http logic, nor should it be calling io_uring
with remote-controlled crafted fd's or access pattern that might
still have bugs in them.
At the level you'd be exposing io_uring (internals) via external http
requests; it's security is perfectly fine.
bastawhiz wrote 17 hours 28 min ago:
The idea of jit compilation of a web server in a small project is
pretty terrifying to me. The attack surface here is enormous.
And for what? My back end on a single host isn't pumping at 35k qps. If
each request is 500 bytes, 35k qps is nearly 20mbps sustained with zero
other io (in each direction). And this is using only two threads!
I think you'd be hard pressed to find an application where this is
meaningfully useful versus just scaling horizontally. On a box that can
run many threads in parallel, Caddy still vastly exceeds my ability to
respond to pretty much any useful traffic. It's optimizing for a metric
that wasn't a bottleneck in the first place.
cowsandmilk wrote 15 hours 52 min ago:
Why does jit compilation here scare you? eBPF has had a ton of
research on how to limit its resource usage and how to sandbox it.
10000truths wrote 15 hours 53 min ago:
> The idea of jit compilation of a web server in a small project is
pretty terrifying to me. The attack surface here is enormous.
Does Spring Boot terrify you, then? Or Lua scripts in nginx? Or PHP?
All of these use JIT compilation to run code that handles web
requests.
Attack surface is a property of the JIT implementation, not of JIT
itself. And eBPF is specifically designed to be very simple to
implement and audit.
graynk wrote 2 hours 21 min ago:
> Does Spring Boot terrify you, then?
It should.
eptcyka wrote 17 hours 22 min ago:
Except CDNs, where it is.
bastawhiz wrote 16 hours 7 min ago:
CDNs aren't running Caddy
ok123456 wrote 18 hours 23 min ago:
Exposing services that use io_uring is a hard pass. It's only been a
handful of weeks since the last security advisory.
Thaxll wrote 19 hours 25 min ago:
Another vibe coded, dead in 6 month Rust project.
People that trully need performance are not going to use a random
server that has 0 support/ track record.
ianm218 wrote 18 hours 1 min ago:
Isn't there a chicken and egg problem where projects need to start
with 0 track record? Nginx had zero track record at one point as well
chucky_z wrote 17 hours 28 min ago:
Nginx was developed to scale a lot of Russian sites, starting with
Rambler. As a second point, Envoy was made by Lyft. A lot of web
server OSS goes back to big (at the time) corps.
4ndrewl wrote 18 hours 2 min ago:
Never mind cooldowns for dependencies, we need cooldowns for these
adhd vibe projects.
dshat wrote 19 hours 38 min ago:
No thanks
BoingBoomTschak wrote 20 hours 24 min ago:
Interesting. Trying to get some of the performance advantages of
TUX/IIS without as much insecurity makes sense for some big players, I
guess.
The usual 3400 lines lock file and AGENTS.md raise some questions about
the aforementioned security, though.
codingjoe wrote 20 hours 30 min ago:
"Caddy compatible" minus everything that matters, like ACME and
plugins. And NGINX still steals the show. Not everything needs to be
rewritten.
jarym wrote 17 hours 33 min ago:
Agree on lack of ACME but the codebase is far cleaner than nginx. In
theory it'd be easier to audit?
__natty__ wrote 20 hours 25 min ago:
Same thoughts. If I need more performant caddy alternative I'm going
to use nginx at least it has some extras.
1a527dd5 wrote 21 hours 52 min ago:
Anyone else got a really weird Chorme pop-up asking which cert to use
for su3.io:443?
Very bizarre, never seen that before.
Thumbprints:
- 60949a09aab8677f87a0b9eda7099a03ca510fb3
- 1b146798f0dc93773247e86312f1b730c4eeebb3
KronisLV wrote 20 hours 50 min ago:
> Very bizarre, never seen that before.
For my own stuff that's not meant for a wider audience, I sometimes
use mTLS in front of my apps, alongside self-signed certs (my own CA)
that shouldn't show up in certificate transparency logs.
This site also seems to be requesting a certificate from the user.
Normally you probably don't want that for public facing resources.
sunaookami wrote 20 hours 55 min ago:
Same on Firefox
embedding-shape wrote 21 hours 1 min ago:
Here it attempts to read my personal certificate that sits in the
browser that I use for filling my taxes and do government stuff,
suspicious indeed.
naturalmovement wrote 17 hours 51 min ago:
That's literally how client certificates work.
It's not attempting to "read" anything, nor is it the least bit
suspicious or malicious.
Your browser was asked if it would like to present a certificate to
authenticate, and you were prompted to choose one if you please.
You can also hit cancel as client auth can be optional and the
server will either serve you the page or a 401/403.
It's like being asked to show ID to enter a pub, you can either
show one or decline, and they may or may not let you enter based on
that transaction.
TurdF3rguson wrote 14 hours 53 min ago:
It's a little suspicious. Why are they doing something that no
other website in the world does? I was curious about
zero-whatever but not enough to do whatever this is.
solid_fuel wrote 11 hours 28 min ago:
> Why are they doing something that no other website in the
world does?
Clearly other sites do since the user who shared the anecdote
has certificates already configured in their browser? It's
uncommon but pretty easy to understand how this happened.
Bear in mind, this is public/private key crypto so it's not
like the site is asking for your facebook password or
something. The site owner has no way to reuse a certificate to
imitate the user.
denkmoon wrote 12 hours 35 min ago:
Plenty of sites do this, you just don't interact with them.
Corp and govt intranets love this stuff.
TurdF3rguson wrote 11 hours 12 min ago:
Right, I've interacted with them when I had to for work. I
wouldn't post any of them on HN though.
naturalmovement wrote 14 hours 43 min ago:
Bruh it's one line in nginx config.
> that no other website in the world does
That you know of. Anywhere with stringent security it's
everywhere.
mook wrote 20 hours 11 min ago:
That's because the client certificate interface in browsers is
supremely dumb. It always just lists all certificates you have,
with very little context in the UI, and hopes that's good enough. I
believe that's part of the reason client certificates are not
poplar; having actual users deal with that is terrible, and the
browsers (in practice, Chrome because of its overwhelming market
share) isn't incentivized to fix it.
Avamander wrote 17 hours 5 min ago:
Servers can communicate their preference in terms of CAs they
want. But the UX in browsers is unbelievably horrible for no good
reason.
Not only is it difficult for an user to make a proper selection,
it's also hard to fix a wrong one. The error pages are also
terrible. There's no way for the site owner to request that when
the navigation to the (auth) page fails, redirect back. Nope, no
way to do error handling without some really clever iframe stuff
and even then it's way too opaque.
God forbid you have to deal with CORS + mTLS.
elevation wrote 16 hours 2 min ago:
> God forbid you have to deal with CORS + mTLS
As someone who is about to deal with exactly this, what kind of
trouble am I in for?
Avamander wrote 13 hours 35 min ago:
Preflight requests won't be doing mTLS on all browsers.
cmgbhm wrote 20 hours 12 min ago:
Thatâs likely just the side effect of supporting mtls. Mutual TLS
came around at the same time as Microsoft did implicit network
auth. Seemed magical at the time and so hare brained for eons of
problems. The user side tls never caught on in most circles and
still has the ancient sharp edges
iknowstuff wrote 17 hours 55 min ago:
Could probably buff it with passkeys these days
HTML [1]: https://www.passkeyprf.com/
jeroenhd wrote 1 hour 53 min ago:
mTLS supports some protocol level security guarantees that
passkeys don't. Because the keys are exchanged during
connection setup, there's no need for a login screen and
Javascript middleware to begin the authentication process. mTLS
is also easy to implement for APIs, you basically get
authentication for free.
Unfortunately, browsers don't invest into making a good UI for
mTLS. If browsers simply put their foot down and said "we will
not permit websites to ask for a certificate if the request
does not contain the proper requirements" like they do in
passkeys, mTLS would be just as easy to use (and even easier to
manage and rotate!).
When I ran mTLS auth on my intranet, I discovered that a lot of
sites will use mTLS support to do fingerprinting, which means a
lot of pages will open a blocking popup (sometimes multiple
times) when I just want to read an article.
linsomniac wrote 21 hours 13 min ago:
Same on Arc
jorl17 wrote 21 hours 26 min ago:
Same on Zen
tln wrote 21 hours 54 min ago:
No ACME! That is a dealbreaker
HTML [1]: https://github.com/losfair/zeroserve/blob/main/CADDY_COMPAT.md...
codys wrote 21 hours 53 min ago:
Yes, I agree it would be very nice to have a way to integrate ACME
into zeroserve. I'm not sure if zeroserve's plugin system might allow
one to add a plugin to support it?
smallerize wrote 22 hours 8 min ago:
I still think of eBPF as not being Turing-complete. There is still a
complexity limit in the verifier. Even if someone did implement Game of
Life by having the program set a timer to run itself.
HTML [1]: https://isovalent.com/blog/post/ebpf-yes-its-turing-complete/
codys wrote 21 hours 58 min ago:
zeroserve doesn't use the Linux kernel's eBPF runtime to run the eBPF
it uses, so the constraints of the Linux kernel's eBPF runtime
(chosen because of how the Linux kernel thinks about protecting the
Linux kernel from user space) don't apply to zeroserve (or other
tools that use the eBPF instruction set but don't use the Linux
kernel's particular implementation)
augunrik wrote 22 hours 14 min ago:
I am surprised how well nginx holds up?!
miladyincontrol wrote 18 hours 12 min ago:
I mean, nginx dang well should? This is just an incredibly synthetic
http(s)/1.1 test for what its worth.
Like you totally could turn off garbage collection for caddy
especially since this is only testing incredibly short single
response queries that would never need GC. Shockingly you would
actually get better performance than either nginx or zeroserve, but
like the uselessness of this benchmark it'd mean nothing to the real
world usage of these web servers.
chucky_z wrote 17 hours 31 min ago:
Unfortunately that is likely untrue. Try it yourself, Go with GC
disabled isnât some magic bullet for performance as much as Iâd
like it to be.
phillipseamore wrote 21 hours 44 min ago:
Why? It's one of the most optimized HTTP servers ever. Anything that
claims beating nginx in benchmarks should be treated with high
suspicion. I think these zeroserve numbers are likely accurate but it
doesn't have the features and module ecosystem of nginx so the
margins aren't worth it for me.
augunrik wrote 20 hours 37 min ago:
Because it passes more boundaries and stuff.
But hey, I didnât code a Webserver so far - so what do I know. :D
AFAIK eBPF can be hardware offloaded. If you have the use case.
rciorba wrote 5 hours 54 min ago:
This is not in-kernel eBPF, this is a userspace eNPF runtime, so
I don't see why it would pass fewer boundaries
someothherguyy wrote 18 hours 49 min ago:
> But hey, I didnât code a Webserver so far - so what do I know
If you limit the scope, its worth doing and might not take as
much effort as you might think. You could possibly find some
enjoyment and learn a few things doing so.
solid_fuel wrote 11 hours 23 min ago:
In college I had a networks class where the capstone project
was writing a basic HTTP server in C. It's actually shockingly
easy, especially if you're only supporting get, and fetch.
Mine was something like 70 lines, and would just listen on 8080
and fork when it got a connection before checking for the
requested file and sending it or a 404. I was immediately
tempted to try adding something like CGI support but didn't
have the time that semester.
zsoltkacsandi wrote 22 hours 15 min ago:
From a technical standpoint, these are always impressive projects, but
I've always wondered: has anyone ever encountered a use case where the
Caddy was the bottleneck?
tredre3 wrote 11 hours 15 min ago:
In my experience Caddy has worse latency and throughput than nginx.
I've set up a service that frequently sends 600MB/s (~5gbps) with
nginx and the CPU is just chilling at 50%, but Caddy on that machine
bottlenecks at 300MB/s despite using 100% of the CPU. AES hardware
acceleration was enabled and functional on both software. This is
high throughput that most people won't see, but it was also on a far
beefier machine than most people would use. Caddy would definitely be
a bottleneck when serving media from a raspberry pi. My last attempt
was in 2025, Caddy has probably improved since then.
That being said nginx has some terrible defaults so if you're just
naively benchmarking it as a proxy out of the box, you might find
Caddy to be better. For example nginx caches active request bodies
(in and out) to temp files in many scenarios (to block the
backend/upstream as little as possible), whereas Caddy is more of a
transparent proxy.
ksec wrote 42 min ago:
I don't think Caddy being slower is a real problem. But Caddy,
their developers and community refuse to believe Caddy is slow is a
much bigger issue.
jeroenhd wrote 1 hour 49 min ago:
My experience mirrors that, but mostly because Caddy does HTTP/2
and http3 by default, natively. HTTP 1.1 is faster for bulk data
transfers (while being worse at pipelining and latency in general),
so I'm not surprised Caddy is taking more CPU in this case.
In my experience, in terms of latency, Caddy is a lot faster, every
single time. I don't know what modifications I need to do to nginx
to make it comparative but Caddy easily shaves half of the
connection and transfer delays on my local network.
dilyevsky wrote 6 hours 56 min ago:
If it's http2 then Go's stdlib is pretty unoptimized to say the
least. Huffman decoder is really cache unfriendly (pointer chasing)
and I think allocation heavy too. Same probably goes for http1 and
http3.
keynha wrote 18 hours 41 min ago:
For most apps the backend is slower than the proxy by a wide margin,
so Caddy is nowhere near the bottleneck. Where it flips is high
connection churn, since TLS handshakes are the expensive part and a
flood of short lived connections without session resumption burns
proxy CPU well before steady state proxying does. Very high RPS of
tiny responses is the other case, where allocation and header parsing
start to show.
nullstyle wrote 22 hours 29 min ago:
Fudge, I really need to carve out time today to play with zeroserve.
Very cool stuff
DIR <- back to front page