Mailing List Archive

Feedback needed: vmod_reqwest
Hi all,

In January, I wrote here about *vmod_reqwest* and today I'm coming back
with a major update and a request for the community.

Little refresher for those who don't know/remember what vmod_request is
about: https://github.com/gquintard/vmod_reqwest.
In short it does *dynamic backends* and HTTP requests from VCL *(à la
vmod_curl*).
Some random buzzwords to make you click on the link: *HTTPS, HTTP/2, gzip,
brotli, parallel requests, sync/async*, cryptocurrency.

The main benefit of this release is the *probe support.* vmod_reqwest is
now capable of handling probes the same way native backends do, but
combined with dynamic backends, it allows you one pretty neat trick: you
can probe one backend to set the health of another
<https://github.com/gquintard/vmod_reqwest#backend-using-a-probe-to-one-backend-to-determine-anothers-health>
.

The API is fairly complete and ergonomic I believe, but I would love to get
more hands and eyes on this to break it/make it better. If some of you have
opinions and/or want to take it for a spin, there are build explanations in
the README <https://github.com/gquintard/vmod_reqwest#build-and-test>, as
well as a Dockerfile
<https://github.com/gquintard/vmod_reqwest/blob/main/Dockerfile> [1] that
will build onto the official image without polluting it.

Let me know what you think of it!

[1]: thanks @thomersch for the help and push on the Docker front
--
Guillaume Quintard
Re: Feedback needed: vmod_reqwest [ In reply to ]
On Sun, Jul 3, 2022 at 3:16 PM Guillaume Quintard
<guillaume.quintard@gmail.com> wrote:
>
> Hi all,
>
> In January, I wrote here about vmod_reqwest and today I'm coming back with a major update and a request for the community.
>
> Little refresher for those who don't know/remember what vmod_request is about: https://github.com/gquintard/vmod_reqwest.
> In short it does dynamic backends and HTTP requests from VCL (à la vmod_curl).
> Some random buzzwords to make you click on the link: HTTPS, HTTP/2, gzip, brotli, parallel requests, sync/async, cryptocurrency.

I didn't find how to scam people with NFTs in the manual, should I
open a github issue?

> The main benefit of this release is the probe support. vmod_reqwest is now capable of handling probes the same way native backends do, but combined with dynamic backends, it allows you one pretty neat trick: you can probe one backend to set the health of another.
>
> The API is fairly complete and ergonomic I believe, but I would love to get more hands and eyes on this to break it/make it better. If some of you have opinions and/or want to take it for a spin, there are build explanations in the README, as well as a Dockerfile [1] that will build onto the official image without polluting it.
>
> Let me know what you think of it!

I really like the idea of a optional path prefix being automatically
prepended to the value of bereq.url directly at the backend layer
:thumbsup:

In general, I agree, the API looks rather well thought out, even
though it does suffer bloated constructor syndrome. Did you put only
timeout and connect_timeout to lower the number of arguments or
weren't you able to implement ftbo and bbto with reqwest? I suspect
both :p

Also it says this:

> In practice, when contacting a backend, you will need to `unset bereq.http.accept-encoding;`, as Varnish sets it automatically.

Probably a nice spot to mention
https://varnish-cache.org/docs/7.0/reference/varnishd.html#http-gzip-support
to explain why one would be set.

On the other hand, if you disable gzip support you may also be
forwarding the client's accept-encoding header if it survived all the
way to the backend fetch.

I may fork a vext_respounce [1] when extensions become capable of
registering backend implementations ;)

Cheers
Dridi

[1] I won't have time to actually do it
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Feedback needed: vmod_reqwest [ In reply to ]
On Wed, Jul 6, 2022 at 12:07 AM Dridi Boukelmoune <dridi@varni.sh> wrote:

> I didn't find how to scam people with NFTs in the manual, should I
> open a github issue?
>

No no no, it's just to weed out the weak investors, send me 5 bitcoins and
I'll show where it is.


>
> In general, I agree, the API looks rather well thought out, even
> though it does suffer bloated constructor syndrome.


Yes, I realized later that I could use the event method of the backend to
finalize the object, however, I'm not sure this:

new client = reqwest.client();
client.set_base_url("http://www.example.com");
client.set_follow(5);
client.set_brotli(true);
client.set_probe(p1);
client.set_connect_timeout(5s);


is more readable, or practical than:

new client = reqwest.client(
base_url = "http://www.example.com",
follow = 5,
auto_brotli = true,
probe = p1,
connect_timeout = 5s
);


(consider this a question to you all, if you have an opinion, voice it!)

Did you put only
> timeout and connect_timeout to lower the number of arguments or
> weren't you able to implement ftbo and bbto with reqwest? I suspect
> both :p
>

Definitely the latter, once you pass the 6-7 arguments threshod, the sky's
the limit.


>
> Also it says this:
>
> > In practice, when contacting a backend, you will need to `unset
> bereq.http.accept-encoding;`, as Varnish sets it automatically.
>
> Probably a nice spot to mention
>
> https://varnish-cache.org/docs/7.0/reference/varnishd.html#http-gzip-support
> to explain why one would be set.
>
> On the other hand, if you disable gzip support you may also be
> forwarding the client's accept-encoding header if it survived all the
> way to the backend fetch
>

Good points, I can update the docs. I'm wondering though if it's better to
special-case the AE header handling in the vmod and try to be smart, or
just let the user do it in VCL...

--
Guillaume Quintard