Mailing List Archive

varnish and mp4 files
Hi!

I am currently experiencing a memory load problem with video playback.

here is the infrastructure :

client --> haproxy --> varnish --> moodle workers (x5)

a teacher uploaded a 400MB video to Moodle, when we start playing the
video with browser player, Varnish consumes all the memory until it runs
out and oom killer to kill varnishd.i have no configuration for mp4
files in my vcl file, so by default they are not hidden (?). I can't
find a solution :(

I can give my vcl file if necessary.

(I am a beginner on varnish :))

thank you for your support.

Karim
Re: varnish and mp4 files [ In reply to ]
Hello Karim,

You VCL would be useful to debug this (as well as the command line you are
running Varnish with), but it sounds like Varnish is using the Transient
storage (
https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
to store the file, and as the storage isn't bounded, it explodes.
We can fix this in a couple of ways, from storing the file in the regular
cache storage, to using pipe, to waiting a few days for
https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
to be released.

Question is: should that file be cached?

Cheers,

--
Guillaume Quintard


On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari@univ-lyon1.fr>
wrote:

> Hi!
>
> I am currently experiencing a memory load problem with video playback.
>
> here is the infrastructure :
>
> client --> haproxy --> varnish --> moodle workers (x5)
>
> a teacher uploaded a 400MB video to Moodle, when we start playing the
> video with browser player, Varnish consumes all the memory until it runs
> out and oom killer to kill varnishd. i have no configuration for mp4
> files in my vcl file, so by default they are not hidden (?). I can't find
> a solution :(
>
> I can give my vcl file if necessary.
>
> (I am a beginner on varnish :))
>
> thank you for your support.
>
> Karim
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: varnish and mp4 files [ In reply to ]
thank you both for your replies.I forgot the most important thing:the
varnish server has 16 GB of ram and the cache is 1 GB.

the cache never seems full


the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T
localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s
malloc,1g****-p http_max_hdr=96***

I had tried using pipe it didn't work, but Rainer's lines works fine.

this solution should suffice because the videos will soon have to be
hosted on our video platform.

/
/

my vcl (file found on github) :

/
/

/....
/

/sub vcl_recv {//
//
//    # Keep client IP//
//    if (req.restarts == 0) {//
//        if (req.http.x-forwarded-for) {//
//            set req.http.X-Forwarded-For = req.http.X-Forwarded-For +
", " + client.ip;//
//        } else {//
//        unset req.http.X-Forwarded-For;//
//            set req.http.X-Forwarded-For = client.ip;//
//        }//
//    }//
//
//    if (req.http.X-Real-IP) {//
//        set req.http.X-Forwarded-For = req.http.X-Real-IP;//
//    } else {//
//        set req.http.X-Forwarded-For = client.ip;//
//    }//
////
//    # Only deal with "normal" types//
//    if (req.method != "GET" &&//
//      req.method != "HEAD" &&//
//      req.method != "PUT" &&//
//      req.method != "POST" &&//
//      req.method != "TRACE" &&//
//      req.method != "OPTIONS" &&//
//      req.method != "PATCH" &&//
//      req.method != "DELETE") {//
//        /* Non-RFC2616 or CONNECT which is weird. *///
//    /*Why send the packet upstream, while the visitor is using a
non-valid HTTP method? *///
//    return (synth(404, "Non-valid HTTP method!"));//
//      }//
////
//    # Varnish don't mess with healthchecks//
//    if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
"^/healthcheck.php") {//
//        return (pass);//
//    }//
//    # Pipe requests to backup.php straight to backend - prevents
problem with progress bar long polling 503 problem//
//    # This is here because backup.php is POSTing to itself - Filter
before !GET&&!HEAD//
//    if (req.url ~ "^/backup/backup.php")//
//    {//
//        return (pipe);//
//    }//
//
//    # Varnish only deals with GET and HEAD by default. If request
method is not GET or HEAD, pass request to backend//
//    if (req.method != "GET" && req.method != "HEAD") {//
//      return (pass);//
//    }//
//
//    if (req.http.Cookie) {//
//      # Remove any Google Analytics based cookies//
//      set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");//
//      set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");//
//      set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(;
)?", "");//
//      set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
)?", "");//
//      set req.http.Cookie = regsuball(req.http.Cookie,
"__atuv.=[^;]+(; )?", "");//
//      set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");//
//      if (req.http.Cookie ~ "^\s*$") {//
//        unset req.http.Cookie;//
//      }//
//    }//
//
// ### Rules for Moodle ###//
//
//    # Perform lookup for selected assets that we know are static but
Moodle still needs a Cookie//
//    if(  req.url ~ "^/theme/.+\.?" ||//
//     req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||//
//         req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||//
//         req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$"
||//
//     req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"//
//      )//
//    {//
//         # Set internal temporary header, based on which we will do
things in vcl_backend_response//
//         set req.http.X-Long-TTL = "86400";//
//         return (hash);//
//    }//
//    # Requests containing "Cookie" or "Authorization" headers will not
be cached//
//    if (req.http.Authorization || req.http.Cookie) {//
//        return (pass);//
//    }//
//    # Almost everything in Moodle correctly serves Cache-Control
headers, if//
//    # needed, which varnish will honor, but there are some which
don't. Rather//
//    # than explicitly finding them all and listing them here we just
fail safe//
//    # and don't cache unknown urls that get this far.//
//    return (pass);//
//
//}//
//
//sub vcl_backend_response {//
//    # Set backend name//
//    set beresp.http.X-Backend = beresp.backend.name;//
//
//    if (beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
!beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL
header//
//        set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
beresp.http.Pragma;//
//        set beresp.http.X-Orig-Cache-Control =
beresp.http.Cache-Control;//
//        set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";//
//        set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);//
//        unset bereq.http.X-Long-TTL;//
//    }//
//    else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
!beresp.http.WWW-Authenticate ) {//
//        set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
beresp.http.Pragma;//
//        set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";//
//        set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);//
//        unset bereq.http.X-Long-TTL;//
//    }//
//    else { # Don't touch headers if max-age > defined in X-Long-TTL
header//
//        unset bereq.http.X-Long-TTL;//
//    }//
//    # Here we set X-Trace header, prepending it to X-Trace header
received from backend. Useful for troubleshooting//
//    if (beresp.http.x-trace && !beresp.was_304) {//
//        set beresp.http.X-Trace = regsub(server.identity,
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;//
//    }//
//    else {//
//        set beresp.http.X-Trace = regsub(server.identity,
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");//
//    }//
//}//
//
//sub vcl_deliver {//
//
//# Revert back to original Cache-Control header before delivery to client//
//    if (resp.http.X-Orig-Cache-Control)//
//    {//
//        set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;//
//        unset resp.http.X-Orig-Cache-Control;//
//    }//
//    # Revert back to original Pragma header before delivery to client//
//    if (resp.http.X-Orig-Pragma)//
//    {//
//        set resp.http.Pragma = resp.http.X-Orig-Pragma;//
//        unset resp.http.X-Orig-Pragma;//
//    }//
//
//  if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and
the number of hits, disable when not needed//
//    set resp.http.X-Cache = "HIT";//
//  } else {//
//    set resp.http.X-Cache = "MISS";//
//  }//
//    set resp.http.X-Cache-Hits = obj.hits;//
//
//# If desired "Via: 1.1 Varnish-v4" response header can be removed from
response//
//    unset resp.http.Via;//
//    unset resp.http.Server;//
//
//    return (deliver);//
//}//
//
//sub vcl_backend_error {//
//    # More comprehensive varnish error page. Display time, instance
hostname, host header, url for easier troubleshooting.//
//    set beresp.http.Content-Type = "text/html; charset=utf-8";//
//    set beresp.http.Retry-After = "5";//
//    synthetic( {"//
//  <!DOCTYPE html>//
//  <html>//
//    <head>//
//      <title>"} + beresp.status + " " + beresp.reason + {"</title>//
//    </head>//
//    <body>//
//      <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>//
//      <p>"} + beresp.reason + {"</p>//
//      <h3>Guru Meditation:</h3>//
//      <p>Time: "} + now + {"</p>//
//      <p>Node: "} + server.hostname + {"</p>//
//      <p>Host: "} + bereq.http.host + {"</p>//
//      <p>URL: "} + bereq.url + {"</p>//
//      <p>XID: "} + bereq.xid + {"</p>//
//      <hr>//
//      <p>Varnish cache server//
//    </body>//
//  </html>//
//  "} );//
//   return (deliver);//
//}//
//
//sub vcl_synth {//
//    #Redirect using '301 - Permanent Redirect', permanent redirect//
//    if (resp.status == 851) { //
//        set resp.http.Location = req.http.x-redir;//
//        set resp.http.X-Varnish-Redirect = true;//
//        set resp.status = 301;//
//        return (deliver);//
//    }//
//    #Redirect using '302 - Found', temporary redirect//
//    if (resp.status == 852) { //
//        set resp.http.Location = req.http.x-redir;//
//        set resp.http.X-Varnish-Redirect = true;//
//        set resp.status = 302;//
//        return (deliver);//
//    }//
//    #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests,
dont change method on redirected requests//
//    if (resp.status == 857) { //
//        set resp.http.Location = req.http.x-redir;//
//        set resp.http.X-Varnish-Redirect = true;//
//        set resp.status = 307;//
//        return (deliver);//
//    }//
//    #Respond with 403 - Forbidden//
//    if (resp.status == 863) {//
//        set resp.http.X-Varnish-Error = true;//
//        set resp.status = 403;//
//        return (deliver);//
//    }//
//}//
//
//sub vcl_purge {//
//  if (req.method != "PURGE") {//
//    set req.http.X-Purge = "Yes";//
//    return (restart);//
//  }//
//}/

...


Karim Ayari
Université Claude Bernard - Lyon 1
Service ICAP
Ingénieur systèmes

Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
> Hello Karim,
>
> You VCL would be useful to debug this (as well as the command line you
> are running Varnish with), but it sounds like Varnish is using the
> Transient storage
> (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
> to store the file, and as the storage isn't bounded, it explodes.
> We can fix this in a couple of ways, from storing the file in the
> regular cache storage, to using pipe, to waiting a few days for
> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
> to be released.
>
> Question is: should that file be cached?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari
> <karim.ayari@univ-lyon1.fr> wrote:
>
> Hi!
>
> I am currently experiencing a memory load problem with video playback.
>
> here is the infrastructure :
>
> client --> haproxy --> varnish --> moodle workers (x5)
>
> a teacher uploaded a 400MB video to Moodle, when we start playing
> the video with browser player, Varnish consumes all the memory
> until it runs out and oom killer to kill varnishd.i have no
> configuration for mp4 files in my vcl file, so by default they are
> not hidden (?). I can't find a solution :(
>
> I can give my vcl file if necessary.
>
> (I am a beginner on varnish :))
>
> thank you for your support.
>
> Karim
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: varnish and mp4 files [ In reply to ]
Looks like Rainer replied directly to you and not to the list, would you
mind sharing/highlighting the fix for people having the same issue?

Cheers,

--
Guillaume Quintard


On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari@univ-lyon1.fr>
wrote:

> thank you both for your replies. I forgot the most important thing: the
> varnish server has 16 GB of ram and the cache is 1 GB.
>
> the cache never seems full
>
>
> the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T
> localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s
> malloc,1g* *-p http_max_hdr=96*
>
> I had tried using pipe it didn't work, but Rainer's lines works fine.
>
> this solution should suffice because the videos will soon have to be
> hosted on our video platform.
>
>
> my vcl (file found on github) :
>
>
>
> *.... *
>
> *sub vcl_recv {*
>
> * # Keep client IP*
> * if (req.restarts == 0) {*
> * if (req.http.x-forwarded-for) {*
> * set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ",
> " + client.ip;*
> * } else {*
> * unset req.http.X-Forwarded-For;*
> * set req.http.X-Forwarded-For = client.ip;*
> * }*
> * }*
>
> * if (req.http.X-Real-IP) {*
> * set req.http.X-Forwarded-For = req.http.X-Real-IP;*
> * } else {*
> * set req.http.X-Forwarded-For = client.ip;*
> * }*
>
> * # Only deal with "normal" types*
> * if (req.method != "GET" &&*
> * req.method != "HEAD" &&*
> * req.method != "PUT" &&*
> * req.method != "POST" &&*
> * req.method != "TRACE" &&*
> * req.method != "OPTIONS" &&*
> * req.method != "PATCH" &&*
> * req.method != "DELETE") {*
> * /* Non-RFC2616 or CONNECT which is weird. */*
> * /*Why send the packet upstream, while the visitor is using a
> non-valid HTTP method? */*
> * return (synth(404, "Non-valid HTTP method!"));*
> * }*
>
> * # Varnish don't mess with healthchecks*
> * if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
> "^/healthcheck.php") {*
> * return (pass);*
> * }*
> * # Pipe requests to backup.php straight to backend - prevents problem
> with progress bar long polling 503 problem*
> * # This is here because backup.php is POSTing to itself - Filter
> before !GET&&!HEAD*
> * if (req.url ~ "^/backup/backup.php")*
> * {*
> * return (pipe);*
> * }*
>
> * # Varnish only deals with GET and HEAD by default. If request method
> is not GET or HEAD, pass request to backend*
> * if (req.method != "GET" && req.method != "HEAD") {*
> * return (pass);*
> * }*
>
> * if (req.http.Cookie) {*
> * # Remove any Google Analytics based cookies*
> * set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(;
> )?", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
> )?", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(;
> )?", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");*
> * if (req.http.Cookie ~ "^\s*$") {*
> * unset req.http.Cookie;*
> * }*
> * }*
>
> * ### Rules for Moodle ###*
>
> * # Perform lookup for selected assets that we know are static but
> Moodle still needs a Cookie*
> * if( req.url ~ "^/theme/.+\.?" ||*
> * req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||*
> * req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||*
> * req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||*
> * req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"*
> * )*
> * {*
> * # Set internal temporary header, based on which we will do
> things in vcl_backend_response*
> * set req.http.X-Long-TTL = "86400";*
> * return (hash);*
> * }*
> * # Requests containing "Cookie" or "Authorization" headers will not be
> cached*
> * if (req.http.Authorization || req.http.Cookie) {*
> * return (pass);*
> * }*
> * # Almost everything in Moodle correctly serves Cache-Control headers,
> if*
> * # needed, which varnish will honor, but there are some which don't.
> Rather*
> * # than explicitly finding them all and listing them here we just fail
> safe*
> * # and don't cache unknown urls that get this far.*
> * return (pass);*
>
> *}*
>
> *sub vcl_backend_response {*
> * # Set backend name*
> * set beresp.http.X-Backend = beresp.backend.name
> <http://beresp.backend.name>;*
>
> * if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl
> < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
> !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL
> header*
> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
> beresp.http.Pragma;*
> * set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control;*
> * set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";*
> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
> * unset bereq.http.X-Long-TTL;*
> * }*
> * else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
> !beresp.http.WWW-Authenticate ) {*
> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
> beresp.http.Pragma;*
> * set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";*
> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
> * unset bereq.http.X-Long-TTL;*
> * }*
> * else { # Don't touch headers if max-age > defined in X-Long-TTL
> header*
> * unset bereq.http.X-Long-TTL;*
> * }*
> * # Here we set X-Trace header, prepending it to X-Trace header
> received from backend. Useful for troubleshooting*
> * if (beresp.http.x-trace && !beresp.was_304) {*
> * set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;*
> * }*
> * else {*
> * set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");*
> * }*
> *}*
>
> *sub vcl_deliver {*
>
> *# Revert back to original Cache-Control header before delivery to client*
> * if (resp.http.X-Orig-Cache-Control)*
> * {*
> * set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;*
> * unset resp.http.X-Orig-Cache-Control;*
> * }*
> * # Revert back to original Pragma header before delivery to client*
> * if (resp.http.X-Orig-Pragma)*
> * {*
> * set resp.http.Pragma = resp.http.X-Orig-Pragma;*
> * unset resp.http.X-Orig-Pragma;*
> * }*
>
> * if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and
> the number of hits, disable when not needed*
> * set resp.http.X-Cache = "HIT";*
> * } else {*
> * set resp.http.X-Cache = "MISS";*
> * }*
> * set resp.http.X-Cache-Hits = obj.hits;*
>
> *# If desired "Via: 1.1 Varnish-v4" response header can be removed from
> response*
> * unset resp.http.Via;*
> * unset resp.http.Server;*
>
> * return (deliver);*
> *}*
>
> *sub vcl_backend_error {*
> * # More comprehensive varnish error page. Display time, instance
> hostname, host header, url for easier troubleshooting.*
> * set beresp.http.Content-Type = "text/html; charset=utf-8";*
> * set beresp.http.Retry-After = "5";*
> * synthetic( {"*
> * <!DOCTYPE html>*
> * <html>*
> * <head>*
> * <title>"} + beresp.status + " " + beresp.reason + {"</title>*
> * </head>*
> * <body>*
> * <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>*
> * <p>"} + beresp.reason + {"</p>*
> * <h3>Guru Meditation:</h3>*
> * <p>Time: "} + now + {"</p>*
> * <p>Node: "} + server.hostname + {"</p>*
> * <p>Host: "} + bereq.http.host + {"</p>*
> * <p>URL: "} + bereq.url + {"</p>*
> * <p>XID: "} + bereq.xid + {"</p>*
> * <hr>*
> * <p>Varnish cache server*
> * </body>*
> * </html>*
> * "} );*
> * return (deliver);*
> *}*
>
> *sub vcl_synth {*
> * #Redirect using '301 - Permanent Redirect', permanent redirect*
> * if (resp.status == 851) { *
> * set resp.http.Location = req.http.x-redir;*
> * set resp.http.X-Varnish-Redirect = true;*
> * set resp.status = 301;*
> * return (deliver);*
> * }*
> * #Redirect using '302 - Found', temporary redirect*
> * if (resp.status == 852) { *
> * set resp.http.Location = req.http.x-redir;*
> * set resp.http.X-Varnish-Redirect = true;*
> * set resp.status = 302;*
> * return (deliver);*
> * }*
> * #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests,
> dont change method on redirected requests*
> * if (resp.status == 857) { *
> * set resp.http.Location = req.http.x-redir;*
> * set resp.http.X-Varnish-Redirect = true;*
> * set resp.status = 307;*
> * return (deliver);*
> * }*
> * #Respond with 403 - Forbidden*
> * if (resp.status == 863) {*
> * set resp.http.X-Varnish-Error = true;*
> * set resp.status = 403;*
> * return (deliver);*
> * }*
> *}*
>
> *sub vcl_purge {*
> * if (req.method != "PURGE") {*
> * set req.http.X-Purge = "Yes";*
> * return (restart);*
> * }*
> *}*
>
> ...
>
>
> Karim Ayari
> Université Claude Bernard - Lyon 1
> Service ICAP
> Ingénieur systèmes
>
> Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
>
> Hello Karim,
>
> You VCL would be useful to debug this (as well as the command line you are
> running Varnish with), but it sounds like Varnish is using the Transient
> storage (
> https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
> to store the file, and as the storage isn't bounded, it explodes.
> We can fix this in a couple of ways, from storing the file in the regular
> cache storage, to using pipe, to waiting a few days for
> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
> to be released.
>
> Question is: should that file be cached?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari@univ-lyon1.fr>
> wrote:
>
>> Hi!
>>
>> I am currently experiencing a memory load problem with video playback.
>>
>> here is the infrastructure :
>>
>> client --> haproxy --> varnish --> moodle workers (x5)
>>
>> a teacher uploaded a 400MB video to Moodle, when we start playing the
>> video with browser player, Varnish consumes all the memory until it runs
>> out and oom killer to kill varnishd. i have no configuration for mp4
>> files in my vcl file, so by default they are not hidden (?). I can't
>> find a solution :(
>>
>> I can give my vcl file if necessary.
>>
>> (I am a beginner on varnish :))
>>
>> thank you for your support.
>>
>> Karim
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
Re: varnish and mp4 files [ In reply to ]
sorry I thought I noted it in my previous email.

here are the good lines for using pipe :


/if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {//
//   std.log("ispiped:true");//
//   return (pipe);//
//}/

thank you!

Karim.



Le 20/02/2023 à 21:50, Guillaume Quintard a écrit :
> Looks like Rainer replied directly to you and not to the list, would
> you mind sharing/highlighting the fix for people having the same issue?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari
> <karim.ayari@univ-lyon1.fr> wrote:
>
> thank you both for your replies.I forgot the most important
> thing:the varnish server has 16 GB of ram and the cache is 1 GB.
>
> the cache never seems full
>
>
> the command used : */usr/sbin/varnishd -j unix,user=vcache -F
> -a:8080 -T localhost:6082 -f /etc/varnish/moodle.vcl -S
> /etc/varnish/secret -s malloc,1g****-p http_max_hdr=96***
>
> I had tried using pipe it didn't work, but Rainer's lines works fine.
>
> this solution should suffice because the videos will soon have to
> be hosted on our video platform.
>
> /
> /
>
> my vcl (file found on github) :
>
> /
> /
>
> /....
> /
>
> /sub vcl_recv {//
> //
> //    # Keep client IP//
> //    if (req.restarts == 0) {//
> //        if (req.http.x-forwarded-for) {//
> //            set req.http.X-Forwarded-For =
> req.http.X-Forwarded-For + ", " + client.ip;//
> //        } else {//
> //        unset req.http.X-Forwarded-For;//
> //            set req.http.X-Forwarded-For = client.ip;//
> //        }//
> //    }//
> //
> //    if (req.http.X-Real-IP) {//
> //        set req.http.X-Forwarded-For = req.http.X-Real-IP;//
> //    } else {//
> //        set req.http.X-Forwarded-For = client.ip;//
> //    }//
> ////
> //    # Only deal with "normal" types//
> //    if (req.method != "GET" &&//
> //      req.method != "HEAD" &&//
> //      req.method != "PUT" &&//
> //      req.method != "POST" &&//
> //      req.method != "TRACE" &&//
> //      req.method != "OPTIONS" &&//
> //      req.method != "PATCH" &&//
> //      req.method != "DELETE") {//
> //        /* Non-RFC2616 or CONNECT which is weird. *///
> //    /*Why send the packet upstream, while the visitor is using a
> non-valid HTTP method? *///
> //    return (synth(404, "Non-valid HTTP method!"));//
> //      }//
> ////
> //    # Varnish don't mess with healthchecks//
> //    if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
> "^/healthcheck.php") {//
> //        return (pass);//
> //    }//
> //    # Pipe requests to backup.php straight to backend - prevents
> problem with progress bar long polling 503 problem//
> //    # This is here because backup.php is POSTing to itself -
> Filter before !GET&&!HEAD//
> //    if (req.url ~ "^/backup/backup.php")//
> //    {//
> //        return (pipe);//
> //    }//
> //
> //    # Varnish only deals with GET and HEAD by default. If
> request method is not GET or HEAD, pass request to backend//
> //    if (req.method != "GET" && req.method != "HEAD") {//
> //      return (pass);//
> //    }//
> //
> //    if (req.http.Cookie) {//
> //      # Remove any Google Analytics based cookies//
> //      set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$",
> "");//
> //      set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$",
> "");//
> //      set req.http.Cookie = regsuball(req.http.Cookie,
> "__gads=[^;]+(; )?", "");//
> //      set req.http.Cookie = regsuball(req.http.Cookie,
> "__qc.=[^;]+(; )?", "");//
> //      set req.http.Cookie = regsuball(req.http.Cookie,
> "__atuv.=[^;]+(; )?", "");//
> //      set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*",
> "");//
> //      if (req.http.Cookie ~ "^\s*$") {//
> //        unset req.http.Cookie;//
> //      }//
> //    }//
> //
> // ### Rules for Moodle ###//
> //
> //    # Perform lookup for selected assets that we know are static
> but Moodle still needs a Cookie//
> //    if(  req.url ~ "^/theme/.+\.?" ||//
> //     req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||//
> //         req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||//
> //         req.url ~
> "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||//
> //     req.url ~
> "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"//
> //      )//
> //    {//
> //         # Set internal temporary header, based on which we will
> do things in vcl_backend_response//
> //         set req.http.X-Long-TTL = "86400";//
> //         return (hash);//
> //    }//
> //    # Requests containing "Cookie" or "Authorization" headers
> will not be cached//
> //    if (req.http.Authorization || req.http.Cookie) {//
> //        return (pass);//
> //    }//
> //    # Almost everything in Moodle correctly serves Cache-Control
> headers, if//
> //    # needed, which varnish will honor, but there are some which
> don't. Rather//
> //    # than explicitly finding them all and listing them here we
> just fail safe//
> //    # and don't cache unknown urls that get this far.//
> //    return (pass);//
> //
> //}//
> //
> //sub vcl_backend_response {//
> //    # Set backend name//
> //    set beresp.http.X-Backend = beresp.backend.name
> <http://beresp.backend.name>;//
> //
> //    if (beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
> beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
> !beresp.http.WWW-Authenticate ) { # If max-age < defined in
> X-Long-TTL header//
> //        set beresp.http.X-Orig-Pragma = beresp.http.Pragma;
> unset beresp.http.Pragma;//
> //        set beresp.http.X-Orig-Cache-Control =
> beresp.http.Cache-Control;//
> //        set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";//
> //        set beresp.ttl = std.duration(bereq.http.X-Long-TTL +
> "s", 1s);//
> //        unset bereq.http.X-Long-TTL;//
> //    }//
> //    else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL
> && !beresp.http.WWW-Authenticate ) {//
> //        set beresp.http.X-Orig-Pragma = beresp.http.Pragma;
> unset beresp.http.Pragma;//
> //        set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";//
> //        set beresp.ttl = std.duration(bereq.http.X-Long-TTL +
> "s", 1s);//
> //        unset bereq.http.X-Long-TTL;//
> //    }//
> //    else { # Don't touch headers if max-age > defined in
> X-Long-TTL header//
> //        unset bereq.http.X-Long-TTL;//
> //    }//
> //    # Here we set X-Trace header, prepending it to X-Trace
> header received from backend. Useful for troubleshooting//
> //    if (beresp.http.x-trace && !beresp.was_304) {//
> //        set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;//
> //    }//
> //    else {//
> //        set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");//
> //    }//
> //}//
> //
> //sub vcl_deliver {//
> //
> //# Revert back to original Cache-Control header before delivery
> to client//
> //    if (resp.http.X-Orig-Cache-Control)//
> //    {//
> //        set resp.http.Cache-Control =
> resp.http.X-Orig-Cache-Control;//
> //        unset resp.http.X-Orig-Cache-Control;//
> //    }//
> //    # Revert back to original Pragma header before delivery to
> client//
> //    if (resp.http.X-Orig-Pragma)//
> //    {//
> //        set resp.http.Pragma = resp.http.X-Orig-Pragma;//
> //        unset resp.http.X-Orig-Pragma;//
> //    }//
> //
> //  if (obj.hits > 0) { # Add debug header to see if it's a
> HIT/MISS and the number of hits, disable when not needed//
> //    set resp.http.X-Cache = "HIT";//
> //  } else {//
> //    set resp.http.X-Cache = "MISS";//
> //  }//
> //    set resp.http.X-Cache-Hits = obj.hits;//
> //
> //# If desired "Via: 1.1 Varnish-v4" response header can be
> removed from response//
> //    unset resp.http.Via;//
> //    unset resp.http.Server;//
> //
> //    return (deliver);//
> //}//
> //
> //sub vcl_backend_error {//
> //    # More comprehensive varnish error page. Display time,
> instance hostname, host header, url for easier troubleshooting.//
> //    set beresp.http.Content-Type = "text/html; charset=utf-8";//
> //    set beresp.http.Retry-After = "5";//
> //    synthetic( {"//
> //  <!DOCTYPE html>//
> //  <html>//
> //    <head>//
> //      <title>"} + beresp.status + " " + beresp.reason + {"</title>//
> //    </head>//
> //    <body>//
> //      <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>//
> //      <p>"} + beresp.reason + {"</p>//
> //      <h3>Guru Meditation:</h3>//
> //      <p>Time: "} + now + {"</p>//
> //      <p>Node: "} + server.hostname + {"</p>//
> //      <p>Host: "} + bereq.http.host + {"</p>//
> //      <p>URL: "} + bereq.url + {"</p>//
> //      <p>XID: "} + bereq.xid + {"</p>//
> //      <hr>//
> //      <p>Varnish cache server//
> //    </body>//
> //  </html>//
> //  "} );//
> //   return (deliver);//
> //}//
> //
> //sub vcl_synth {//
> //    #Redirect using '301 - Permanent Redirect', permanent redirect//
> //    if (resp.status == 851) { //
> //        set resp.http.Location = req.http.x-redir;//
> //        set resp.http.X-Varnish-Redirect = true;//
> //        set resp.status = 301;//
> //        return (deliver);//
> //    }//
> //    #Redirect using '302 - Found', temporary redirect//
> //    if (resp.status == 852) { //
> //        set resp.http.Location = req.http.x-redir;//
> //        set resp.http.X-Varnish-Redirect = true;//
> //        set resp.status = 302;//
> //        return (deliver);//
> //    }//
> //    #Redirect using '307 - Temporary Redirect', !GET&&!HEAD
> requests, dont change method on redirected requests//
> //    if (resp.status == 857) { //
> //        set resp.http.Location = req.http.x-redir;//
> //        set resp.http.X-Varnish-Redirect = true;//
> //        set resp.status = 307;//
> //        return (deliver);//
> //    }//
> //    #Respond with 403 - Forbidden//
> //    if (resp.status == 863) {//
> //        set resp.http.X-Varnish-Error = true;//
> //        set resp.status = 403;//
> //        return (deliver);//
> //    }//
> //}//
> //
> //sub vcl_purge {//
> //  if (req.method != "PURGE") {//
> //    set req.http.X-Purge = "Yes";//
> //    return (restart);//
> //  }//
> //}/
>
> ...
>
>
> Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
>> Hello Karim,
>>
>> You VCL would be useful to debug this (as well as the command
>> line you are running Varnish with), but it sounds like Varnish is
>> using the Transient storage
>> (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
>> to store the file, and as the storage isn't bounded, it explodes.
>> We can fix this in a couple of ways, from storing the file in the
>> regular cache storage, to using pipe, to waiting a few days for
>> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
>> to be released.
>>
>> Question is: should that file be cached?
>>
>> Cheers,
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari
>> <karim.ayari@univ-lyon1.fr> wrote:
>>
>> Hi!
>>
>> I am currently experiencing a memory load problem with video
>> playback.
>>
>> here is the infrastructure :
>>
>> client --> haproxy --> varnish --> moodle workers (x5)
>>
>> a teacher uploaded a 400MB video to Moodle, when we start
>> playing the video with browser player, Varnish consumes all
>> the memory until it runs out and oom killer to kill
>> varnishd.i have no configuration for mp4 files in my vcl
>> file, so by default they are not hidden (?). I can't find a
>> solution :(
>>
>> I can give my vcl file if necessary.
>>
>> (I am a beginner on varnish :))
>>
>> thank you for your support.
>>
>> Karim
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
RE: varnish and mp4 files [ In reply to ]
Hello,


I'm coming back to this subject because after a moment of calm my problem of memory leak and OOM killer with videos is back :(

we have videos hosted on Moodle behind varnish which are themselves behind haproxy.


version varnish 6.2.1-2ubuntu0.2 amd64


this expression seems to work


if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {
std.log("ispiped:true");
return (pipe);
}


mp4 files are piped :


- VCL_call RECV
- VCL_Log ispiped:true
- VCL_return pipe
- VCL_call HASH
- VCL_return lookup


first graph is memory usage (total of 16GB), the second one requests fetched, we can see 2 restarts after OOM killing at 16h25 and 18h20


[cid:404902a3-6553-4f34-8b67-7b5b9853b3c3]


I don't see any other file extension to add in the expression


thank you for your help


---
Karim Ayari

________________________________
De : varnish-misc <varnish-misc-bounces+karim.ayari=univ-lyon1.fr@varnish-cache.org> de la part de Karim Ayari <karim.ayari@univ-lyon1.fr>
Envoy? : mardi 21 f?vrier 2023 09:05
? : Guillaume Quintard
Cc : varnish-misc@varnish-cache.org
Objet : Re: varnish and mp4 files


sorry I thought I noted it in my previous email.

here are the good lines for using pipe :

if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {
std.log("ispiped:true");
return (pipe);
}

thank you!

Karim.



Le 20/02/2023 ? 21:50, Guillaume Quintard a ?crit :
Looks like Rainer replied directly to you and not to the list, would you mind sharing/highlighting the fix for people having the same issue?

Cheers,

--
Guillaume Quintard


On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari@univ-lyon1.fr<mailto:karim.ayari@univ-lyon1.fr>> wrote:

thank you both for your replies. I forgot the most important thing: the varnish server has 16 GB of ram and the cache is 1 GB.

the cache never seems full

[cid:part1.hll3XDne.buey2s0m@univ-lyon1.fr][cid:part2.3Xb0PaGW.FM1miA0J@univ-lyon1.fr]


the command used : /usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s malloc,1g -p http_max_hdr=96

I had tried using pipe it didn't work, but Rainer's lines works fine.

this solution should suffice because the videos will soon have to be hosted on our video platform.


my vcl (file found on github) :


....

sub vcl_recv {

# Keep client IP
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
unset req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
}
}

if (req.http.X-Real-IP) {
set req.http.X-Forwarded-For = req.http.X-Real-IP;
} else {
set req.http.X-Forwarded-For = client.ip;
}

# Only deal with "normal" types
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "PATCH" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
/*Why send the packet upstream, while the visitor is using a non-valid HTTP method? */
return (synth(404, "Non-valid HTTP method!"));
}

# Varnish don't mess with healthchecks
if (req.url ~ "^/admin/tool/heartbeat" || req.url ~ "^/healthcheck.php") {
return (pass);
}
# Pipe requests to backup.php straight to backend - prevents problem with progress bar long polling 503 problem
# This is here because backup.php is POSTing to itself - Filter before !GET&&!HEAD
if (req.url ~ "^/backup/backup.php")
{
return (pipe);
}

# Varnish only deals with GET and HEAD by default. If request method is not GET or HEAD, pass request to backend
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}

if (req.http.Cookie) {
# Remove any Google Analytics based cookies
set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}
}

### Rules for Moodle ###

# Perform lookup for selected assets that we know are static but Moodle still needs a Cookie
if( req.url ~ "^/theme/.+\.?" ||
req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||
req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||
req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||
req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"
)
{
# Set internal temporary header, based on which we will do things in vcl_backend_response
set req.http.X-Long-TTL = "86400";
return (hash);
}
# Requests containing "Cookie" or "Authorization" headers will not be cached
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}
# Almost everything in Moodle correctly serves Cache-Control headers, if
# needed, which varnish will honor, but there are some which don't. Rather
# than explicitly finding them all and listing them here we just fail safe
# and don't cache unknown urls that get this far.
return (pass);

}

sub vcl_backend_response {
# Set backend name
set beresp.http.X-Backend = beresp.backend.name<http://beresp.backend.name>;

if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) && !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL header
set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma;
set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control;
set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform";
set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);
unset bereq.http.X-Long-TTL;
}
else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL && !beresp.http.WWW-Authenticate ) {
set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma;
set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform";
set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);
unset bereq.http.X-Long-TTL;
}
else { # Don't touch headers if max-age > defined in X-Long-TTL header
unset bereq.http.X-Long-TTL;
}
# Here we set X-Trace header, prepending it to X-Trace header received from backend. Useful for troubleshooting
if (beresp.http.x-trace && !beresp.was_304) {
set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name<http://beresp.backend.name>, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;
}
else {
set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name<http://beresp.backend.name>, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");
}
}

sub vcl_deliver {

# Revert back to original Cache-Control header before delivery to client
if (resp.http.X-Orig-Cache-Control)
{
set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;
unset resp.http.X-Orig-Cache-Control;
}
# Revert back to original Pragma header before delivery to client
if (resp.http.X-Orig-Pragma)
{
set resp.http.Pragma = resp.http.X-Orig-Pragma;
unset resp.http.X-Orig-Pragma;
}

if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
set resp.http.X-Cache-Hits = obj.hits;

# If desired "Via: 1.1 Varnish-v4" response header can be removed from response
unset resp.http.Via;
unset resp.http.Server;

return (deliver);
}

sub vcl_backend_error {
# More comprehensive varnish error page. Display time, instance hostname, host header, url for easier troubleshooting.
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
synthetic( {"
<!DOCTYPE html>
<html>
<head>
<title>"} + beresp.status + " " + beresp.reason + {"</title>
</head>
<body>
<h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
<p>"} + beresp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>Time: "} + now + {"</p>
<p>Node: "} + server.hostname + {"</p>
<p>Host: "} + bereq.http.host + {"</p>
<p>URL: "} + bereq.url + {"</p>
<p>XID: "} + bereq.xid + {"</p>
<hr>
<p>Varnish cache server
</body>
</html>
"} );
return (deliver);
}

sub vcl_synth {
#Redirect using '301 - Permanent Redirect', permanent redirect
if (resp.status == 851) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 301;
return (deliver);
}
#Redirect using '302 - Found', temporary redirect
if (resp.status == 852) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 302;
return (deliver);
}
#Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests, dont change method on redirected requests
if (resp.status == 857) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 307;
return (deliver);
}
#Respond with 403 - Forbidden
if (resp.status == 863) {
set resp.http.X-Varnish-Error = true;
set resp.status = 403;
return (deliver);
}
}

sub vcl_purge {
if (req.method != "PURGE") {
set req.http.X-Purge = "Yes";
return (restart);
}
}

...


Le 20/02/2023 ? 18:55, Guillaume Quintard a ?crit :
Hello Karim,

You VCL would be useful to debug this (as well as the command line you are running Varnish with), but it sounds like Varnish is using the Transient storage (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage) to store the file, and as the storage isn't bounded, it explodes.
We can fix this in a couple of ways, from storing the file in the regular cache storage, to using pipe, to waiting a few days for https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643 to be released.

Question is: should that file be cached?

Cheers,

--
Guillaume Quintard


On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari@univ-lyon1.fr<mailto:karim.ayari@univ-lyon1.fr>> wrote:

Hi!

I am currently experiencing a memory load problem with video playback.

here is the infrastructure :

client --> haproxy --> varnish --> moodle workers (x5)

a teacher uploaded a 400MB video to Moodle, when we start playing the video with browser player, Varnish consumes all the memory until it runs out and oom killer to kill varnishd. i have no configuration for mp4 files in my vcl file, so by default they are not hidden (?). I can't find a solution :(

I can give my vcl file if necessary.

(I am a beginner on varnish :))

thank you for your support.

Karim

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: varnish and mp4 files [ In reply to ]
Hi,

Check the varnishncsa logs to see if something fishy stands out, or even
just the backend logs.
Notably, try
varnishncsa -q 'not VCL_call eq PIPE'
That will give you all the transactions that go through vcl_pipe{} (more
log tips here: https://docs.varnish-software.com/tutorials/vsl-query/)

If that doesn't work we can try to filter big files and whatnot.
--
Guillaume Quintard


On Thu, Oct 26, 2023 at 11:47?PM AYARI KARIM <karim.ayari@univ-lyon1.fr>
wrote:

> Hello,
>
>
> I'm coming back to this subject because after a moment of calm my problem
> of memory leak and OOM killer with videos is back :(
>
> we have videos hosted on Moodle behind varnish which are themselves behind
> haproxy.
>
>
> version varnish
> 6.2.1-2ubuntu0.2 amd64
>
>
> this expression seems to work
>
>
> *if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {*
> * std.log("ispiped:true");*
> * return (pipe);*
> *}*
>
>
> mp4 files are piped :
>
>
>
>
>
>
> *- VCL_call RECV - VCL_Log ispiped:true -
> VCL_return pipe - VCL_call HASH - VCL_return lookup*
>
> first graph is memory usage (total of 16GB), the second one requests
> fetched, we can see 2 restarts after OOM killing at 16h25 and 18h20
>
>
>
>
> I don't see any other file extension to add in the expression
>
>
> thank you for your help
>
>
> ---
> Karim Ayari
>
> ------------------------------
> *De :* varnish-misc <varnish-misc-bounces+karim.ayari=
> univ-lyon1.fr@varnish-cache.org> de la part de Karim Ayari <
> karim.ayari@univ-lyon1.fr>
> *Envoyé :* mardi 21 février 2023 09:05
> *À :* Guillaume Quintard
> *Cc :* varnish-misc@varnish-cache.org
> *Objet :* Re: varnish and mp4 files
>
>
> sorry I thought I noted it in my previous email.
> here are the good lines for using pipe :
>
>
> *if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {*
> * std.log("ispiped:true");*
> * return (pipe);*
> *}*
>
> thank you!
>
> Karim.
>
>
>
> Le 20/02/2023 à 21:50, Guillaume Quintard a écrit :
>
> Looks like Rainer replied directly to you and not to the list, would you
> mind sharing/highlighting the fix for people having the same issue?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari@univ-lyon1.fr>
> wrote:
>
>> thank you both for your replies. I forgot the most important thing: the
>> varnish server has 16 GB of ram and the cache is 1 GB.
>>
>> the cache never seems full
>>
>>
>> the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T
>> localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s
>> malloc,1g* *-p http_max_hdr=96*
>>
>> I had tried using pipe it didn't work, but Rainer's lines works fine.
>>
>> this solution should suffice because the videos will soon have to be
>> hosted on our video platform.
>>
>>
>> my vcl (file found on github) :
>>
>>
>>
>> *.... *
>>
>> *sub vcl_recv {*
>>
>> * # Keep client IP*
>> * if (req.restarts == 0) {*
>> * if (req.http.x-forwarded-for) {*
>> * set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ",
>> " + client.ip;*
>> * } else {*
>> * unset req.http.X-Forwarded-For;*
>> * set req.http.X-Forwarded-For = client.ip;*
>> * }*
>> * }*
>>
>> * if (req.http.X-Real-IP) {*
>> * set req.http.X-Forwarded-For = req.http.X-Real-IP;*
>> * } else {*
>> * set req.http.X-Forwarded-For = client.ip;*
>> * }*
>>
>> * # Only deal with "normal" types*
>> * if (req.method != "GET" &&*
>> * req.method != "HEAD" &&*
>> * req.method != "PUT" &&*
>> * req.method != "POST" &&*
>> * req.method != "TRACE" &&*
>> * req.method != "OPTIONS" &&*
>> * req.method != "PATCH" &&*
>> * req.method != "DELETE") {*
>> * /* Non-RFC2616 or CONNECT which is weird. */*
>> * /*Why send the packet upstream, while the visitor is using a
>> non-valid HTTP method? */*
>> * return (synth(404, "Non-valid HTTP method!"));*
>> * }*
>>
>> * # Varnish don't mess with healthchecks*
>> * if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
>> "^/healthcheck.php") {*
>> * return (pass);*
>> * }*
>> * # Pipe requests to backup.php straight to backend - prevents problem
>> with progress bar long polling 503 problem*
>> * # This is here because backup.php is POSTing to itself - Filter
>> before !GET&&!HEAD*
>> * if (req.url ~ "^/backup/backup.php")*
>> * {*
>> * return (pipe);*
>> * }*
>>
>> * # Varnish only deals with GET and HEAD by default. If request method
>> is not GET or HEAD, pass request to backend*
>> * if (req.method != "GET" && req.method != "HEAD") {*
>> * return (pass);*
>> * }*
>>
>> * if (req.http.Cookie) {*
>> * # Remove any Google Analytics based cookies*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(;
>> )?", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
>> )?", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(;
>> )?", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");*
>> * if (req.http.Cookie ~ "^\s*$") {*
>> * unset req.http.Cookie;*
>> * }*
>> * }*
>>
>> * ### Rules for Moodle ###*
>>
>> * # Perform lookup for selected assets that we know are static but
>> Moodle still needs a Cookie*
>> * if( req.url ~ "^/theme/.+\.?" ||*
>> * req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||*
>> * req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||*
>> * req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$"
>> ||*
>> * req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"*
>> * )*
>> * {*
>> * # Set internal temporary header, based on which we will do
>> things in vcl_backend_response*
>> * set req.http.X-Long-TTL = "86400";*
>> * return (hash);*
>> * }*
>> * # Requests containing "Cookie" or "Authorization" headers will not
>> be cached*
>> * if (req.http.Authorization || req.http.Cookie) {*
>> * return (pass);*
>> * }*
>> * # Almost everything in Moodle correctly serves Cache-Control
>> headers, if*
>> * # needed, which varnish will honor, but there are some which don't.
>> Rather*
>> * # than explicitly finding them all and listing them here we just
>> fail safe*
>> * # and don't cache unknown urls that get this far.*
>> * return (pass);*
>>
>> *}*
>>
>> *sub vcl_backend_response {*
>> * # Set backend name*
>> * set beresp.http.X-Backend = beresp.backend.name
>> <http://beresp.backend.name>;*
>>
>> * if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl
>> < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
>> !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL
>> header*
>> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
>> beresp.http.Pragma;*
>> * set beresp.http.X-Orig-Cache-Control =
>> beresp.http.Cache-Control;*
>> * set beresp.http.Cache-Control = "public,
>> max-age="+bereq.http.X-Long-TTL+", no-transform";*
>> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
>> * unset bereq.http.X-Long-TTL;*
>> * }*
>> * else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
>> !beresp.http.WWW-Authenticate ) {*
>> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
>> beresp.http.Pragma;*
>> * set beresp.http.Cache-Control = "public,
>> max-age="+bereq.http.X-Long-TTL+", no-transform";*
>> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
>> * unset bereq.http.X-Long-TTL;*
>> * }*
>> * else { # Don't touch headers if max-age > defined in X-Long-TTL
>> header*
>> * unset bereq.http.X-Long-TTL;*
>> * }*
>> * # Here we set X-Trace header, prepending it to X-Trace header
>> received from backend. Useful for troubleshooting*
>> * if (beresp.http.x-trace && !beresp.was_304) {*
>> * set beresp.http.X-Trace = regsub(server.identity,
>> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
>> <http://beresp.backend.name>,
>> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;*
>> * }*
>> * else {*
>> * set beresp.http.X-Trace = regsub(server.identity,
>> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
>> <http://beresp.backend.name>,
>> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");*
>> * }*
>> *}*
>>
>> *sub vcl_deliver {*
>>
>> *# Revert back to original Cache-Control header before delivery to client*
>> * if (resp.http.X-Orig-Cache-Control)*
>> * {*
>> * set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;*
>> * unset resp.http.X-Orig-Cache-Control;*
>> * }*
>> * # Revert back to original Pragma header before delivery to client*
>> * if (resp.http.X-Orig-Pragma)*
>> * {*
>> * set resp.http.Pragma = resp.http.X-Orig-Pragma;*
>> * unset resp.http.X-Orig-Pragma;*
>> * }*
>>
>> * if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and
>> the number of hits, disable when not needed*
>> * set resp.http.X-Cache = "HIT";*
>> * } else {*
>> * set resp.http.X-Cache = "MISS";*
>> * }*
>> * set resp.http.X-Cache-Hits = obj.hits;*
>>
>> *# If desired "Via: 1.1 Varnish-v4" response header can be removed from
>> response*
>> * unset resp.http.Via;*
>> * unset resp.http.Server;*
>>
>> * return (deliver);*
>> *}*
>>
>> *sub vcl_backend_error {*
>> * # More comprehensive varnish error page. Display time, instance
>> hostname, host header, url for easier troubleshooting.*
>> * set beresp.http.Content-Type = "text/html; charset=utf-8";*
>> * set beresp.http.Retry-After = "5";*
>> * synthetic( {"*
>> * <!DOCTYPE html>*
>> * <html>*
>> * <head>*
>> * <title>"} + beresp.status + " " + beresp.reason + {"</title>*
>> * </head>*
>> * <body>*
>> * <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>*
>> * <p>"} + beresp.reason + {"</p>*
>> * <h3>Guru Meditation:</h3>*
>> * <p>Time: "} + now + {"</p>*
>> * <p>Node: "} + server.hostname + {"</p>*
>> * <p>Host: "} + bereq.http.host + {"</p>*
>> * <p>URL: "} + bereq.url + {"</p>*
>> * <p>XID: "} + bereq.xid + {"</p>*
>> * <hr>*
>> * <p>Varnish cache server*
>> * </body>*
>> * </html>*
>> * "} );*
>> * return (deliver);*
>> *}*
>>
>> *sub vcl_synth {*
>> * #Redirect using '301 - Permanent Redirect', permanent redirect*
>> * if (resp.status == 851) { *
>> * set resp.http.Location = req.http.x-redir;*
>> * set resp.http.X-Varnish-Redirect = true;*
>> * set resp.status = 301;*
>> * return (deliver);*
>> * }*
>> * #Redirect using '302 - Found', temporary redirect*
>> * if (resp.status == 852) { *
>> * set resp.http.Location = req.http.x-redir;*
>> * set resp.http.X-Varnish-Redirect = true;*
>> * set resp.status = 302;*
>> * return (deliver);*
>> * }*
>> * #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests,
>> dont change method on redirected requests*
>> * if (resp.status == 857) { *
>> * set resp.http.Location = req.http.x-redir;*
>> * set resp.http.X-Varnish-Redirect = true;*
>> * set resp.status = 307;*
>> * return (deliver);*
>> * }*
>> * #Respond with 403 - Forbidden*
>> * if (resp.status == 863) {*
>> * set resp.http.X-Varnish-Error = true;*
>> * set resp.status = 403;*
>> * return (deliver);*
>> * }*
>> *}*
>>
>> *sub vcl_purge {*
>> * if (req.method != "PURGE") {*
>> * set req.http.X-Purge = "Yes";*
>> * return (restart);*
>> * }*
>> *}*
>>
>> ...
>>
>>
>> Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
>>
>> Hello Karim,
>>
>> You VCL would be useful to debug this (as well as the command line you
>> are running Varnish with), but it sounds like Varnish is using the
>> Transient storage (
>> https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
>> to store the file, and as the storage isn't bounded, it explodes.
>> We can fix this in a couple of ways, from storing the file in the regular
>> cache storage, to using pipe, to waiting a few days for
>> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
>> to be released.
>>
>> Question is: should that file be cached?
>>
>> Cheers,
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari@univ-lyon1.fr>
>> wrote:
>>
>>> Hi!
>>>
>>> I am currently experiencing a memory load problem with video playback.
>>>
>>> here is the infrastructure :
>>>
>>> client --> haproxy --> varnish --> moodle workers (x5)
>>>
>>> a teacher uploaded a 400MB video to Moodle, when we start playing the
>>> video with browser player, Varnish consumes all the memory until it
>>> runs out and oom killer to kill varnishd. i have no configuration for
>>> mp4 files in my vcl file, so by default they are not hidden (?). I
>>> can't find a solution :(
>>>
>>> I can give my vcl file if necessary.
>>>
>>> (I am a beginner on varnish :))
>>>
>>> thank you for your support.
>>>
>>> Karim
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
RE: varnish and mp4 files [ In reply to ]
Hi,


Sorry for my late response, but I found another video extension that must have been a problem. It seems ok for 1 month, fingers crossed ;)

thank you Guillaume.


Karim Ayari


________________________________
De : Guillaume Quintard <guillaume.quintard@gmail.com>
Envoyé : vendredi 27 octobre 2023 18:24
À : AYARI KARIM
Cc : varnish-misc@varnish-cache.org
Objet : Re: varnish and mp4 files

Hi,

Check the varnishncsa logs to see if something fishy stands out, or even just the backend logs.
Notably, try
varnishncsa -q 'not VCL_call eq PIPE'
That will give you all the transactions that go through vcl_pipe{} (more log tips here: https://docs.varnish-software.com/tutorials/vsl-query/<https://MailInBlack.univ-lyon1.fr/securelink/?url=https://docs.varnish-software.com&key=eyJsYW5nIjoiRlIiLCJ1cmwiOiJodHRwczovL2RvY3MudmFybmlzaC1zb2Z0d2FyZS5jb20vdHV0b3JpYWxzL3ZzbC1xdWVyeS8iLCJ0b2tlbiI6ImdBQUFBQUJsTy1SYnh6RkpodnhKUTRfVC1JQW90YUNyRGg4MW8taTZIdFlPbTdJTDE4UG9PYVVDOFFkUmhLNWk1aXBvbE1USUFKUjVqV090VFBEOHA0RTBwMUU1eUdGY2tMeVFuWkdHaHN2dEhOTFhCdThvLXJYSEdGS0RxMmE5QjdXV1lHcklsR0RSZy00SUpSNER2c2hFMy00YnJxbHo1TE4yaUs4Nkx0SWp3WWhpRURKX2NfY3laSW85WE9aNVZNNS1TOHRrWFVCZFFWOFotZ0x1WXFNUVN2Wm8wQUZqcTYtdVJpb3pYOGR1RklwNEF5OWU4UV9ETWoxQmlrdlNwRVczQkR0ZEJQVTB1QnVEWU52VTg4dmc0V0tUckJvdmpqUkJrUTZ4MTBYTzBla2RpLWtFQmNtdGVlcmdLTTNfbDN0Qlk2QmcxN25fIn0=>)

If that doesn't work we can try to filter big files and whatnot.
--
Guillaume Quintard


On Thu, Oct 26, 2023 at 11:47?PM AYARI KARIM <karim.ayari@univ-lyon1.fr<mailto:karim.ayari@univ-lyon1.fr>> wrote:

Hello,


I'm coming back to this subject because after a moment of calm my problem of memory leak and OOM killer with videos is back :(

we have videos hosted on Moodle behind varnish which are themselves behind haproxy.


version varnish 6.2.1-2ubuntu0.2 amd64


this expression seems to work


if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {
std.log("ispiped:true");
return (pipe);
}


mp4 files are piped :


- VCL_call RECV
- VCL_Log ispiped:true
- VCL_return pipe
- VCL_call HASH
- VCL_return lookup


first graph is memory usage (total of 16GB), the second one requests fetched, we can see 2 restarts after OOM killing at 16h25 and 18h20


[cid:18b71ec5db3f456b1e53]


I don't see any other file extension to add in the expression


thank you for your help


---
Karim Ayari

________________________________
De : varnish-misc <varnish-misc-bounces+karim.ayari=univ-lyon1.fr@varnish-cache.org<mailto:univ-lyon1.fr@varnish-cache.org>> de la part de Karim Ayari <karim.ayari@univ-lyon1.fr<mailto:karim.ayari@univ-lyon1.fr>>
Envoyé : mardi 21 février 2023 09:05
À : Guillaume Quintard
Cc : varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
Objet : Re: varnish and mp4 files


sorry I thought I noted it in my previous email.

here are the good lines for using pipe :

if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {
std.log("ispiped:true");
return (pipe);
}

thank you!

Karim.



Le 20/02/2023 à 21:50, Guillaume Quintard a écrit :
Looks like Rainer replied directly to you and not to the list, would you mind sharing/highlighting the fix for people having the same issue?

Cheers,

--
Guillaume Quintard


On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari@univ-lyon1.fr<mailto:karim.ayari@univ-lyon1.fr>> wrote:

thank you both for your replies. I forgot the most important thing: the varnish server has 16 GB of ram and the cache is 1 GB.

the cache never seems full

[cid:18b71ec5db3af62603d1][cid:18b71ec5db3c3ba5d52]


the command used : /usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s malloc,1g -p http_max_hdr=96

I had tried using pipe it didn't work, but Rainer's lines works fine.

this solution should suffice because the videos will soon have to be hosted on our video platform.


my vcl (file found on github) :


....

sub vcl_recv {

# Keep client IP
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
unset req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
}
}

if (req.http.X-Real-IP) {
set req.http.X-Forwarded-For = req.http.X-Real-IP;
} else {
set req.http.X-Forwarded-For = client.ip;
}

# Only deal with "normal" types
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "PATCH" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
/*Why send the packet upstream, while the visitor is using a non-valid HTTP method? */
return (synth(404, "Non-valid HTTP method!"));
}

# Varnish don't mess with healthchecks
if (req.url ~ "^/admin/tool/heartbeat" || req.url ~ "^/healthcheck.php") {
return (pass);
}
# Pipe requests to backup.php straight to backend - prevents problem with progress bar long polling 503 problem
# This is here because backup.php is POSTing to itself - Filter before !GET&&!HEAD
if (req.url ~ "^/backup/backup.php")
{
return (pipe);
}

# Varnish only deals with GET and HEAD by default. If request method is not GET or HEAD, pass request to backend
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}

if (req.http.Cookie) {
# Remove any Google Analytics based cookies
set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}
}

### Rules for Moodle ###

# Perform lookup for selected assets that we know are static but Moodle still needs a Cookie
if( req.url ~ "^/theme/.+\.?" ||
req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||
req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||
req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||
req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"
)
{
# Set internal temporary header, based on which we will do things in vcl_backend_response
set req.http.X-Long-TTL = "86400";
return (hash);
}
# Requests containing "Cookie" or "Authorization" headers will not be cached
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}
# Almost everything in Moodle correctly serves Cache-Control headers, if
# needed, which varnish will honor, but there are some which don't. Rather
# than explicitly finding them all and listing them here we just fail safe
# and don't cache unknown urls that get this far.
return (pass);

}

sub vcl_backend_response {
# Set backend name
set beresp.http.X-Backend = beresp.backend.name<https://MailInBlack.univ-lyon1.fr/securelink/?url=http://beresp.backend.name&key=eyJsYW5nIjoiRlIiLCJ1cmwiOiJodHRwOi8vYmVyZXNwLmJhY2tlbmQubmFtZSIsInRva2VuIjoiZ0FBQUFBQmxPLVJiZUxXNHRyQWRqVVI3RDNPNEFJUXkwTVZkWnBvRk9MREQwYWlyOENmYjlUVUVrczBkck5VU05ibTBxZlZVdXpJaEUtenFBT0VLN3FETHJuV2VwZUpyVUlMMFZFN1Ayb2YwbkpVdjh4bWo2N3FTdjFUTUxZeElEdG5jVnZHcVNGVjM2U00tOTRLTF92bzR2eFhrVkdRYkRxSmxQWE5remM5OFB3VjZzSHNPMHZoTDhKekh5ZUxPLWpQdFNXTTZLLVVoelNDVzMycVFKOEFUM256RmswaEkzQVNUQjZONHF3UnBJMV9QMDM3MkU3SU9CRkF2OWtRR18wQ1BTZmtuLVMtNy16UnpTeFZJRHBqWXllUF9oS2VRYXUyalIzbGdTMlBBSU55cGZfbnV1VzJrRjFYUWNKTEpfZURzUkFzQllEeGkifQ==>;

if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) && !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL header
set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma;
set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control;
set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform";
set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);
unset bereq.http.X-Long-TTL;
}
else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL && !beresp.http.WWW-Authenticate ) {
set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma;
set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform";
set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);
unset bereq.http.X-Long-TTL;
}
else { # Don't touch headers if max-age > defined in X-Long-TTL header
unset bereq.http.X-Long-TTL;
}
# Here we set X-Trace header, prepending it to X-Trace header received from backend. Useful for troubleshooting
if (beresp.http.x-trace && !beresp.was_304) {
set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name<https://MailInBlack.univ-lyon1.fr/securelink/?url=http://beresp.backend.name&key=eyJsYW5nIjoiRlIiLCJ1cmwiOiJodHRwOi8vYmVyZXNwLmJhY2tlbmQubmFtZSIsInRva2VuIjoiZ0FBQUFBQmxPLVJiM0x6SzkyRVdNU0xJVzd6SGJ5bVE5UTdUemZ0UnJoemI4R1lIektnSndMY2RLUGw5V3Z2MURHODZuSzlLdW9JNmQ2X0NKb1hfNlpUWi1OZ09URDlfQ2doT1k2U2FVakNaWktwRThBRUF6NFFQR2dsV1JnY29IRml4X1NXZXpXdkZpZDN2QTROQ2RKN0k1V0lGZ0N3UVV3SkQ2c2NxYk1hRnQtUFN2a1d4V01nQ2oyWEp1dE91Q1FDbVAtbXlnWmg1ckp0bl9YcGx5VDE4X210Uk9IV1V5LWF3XzFvNEhONTJtZmNTZ2RJQWlvb0lEUU52NzRpZ2ZrZFFsSGlSTVdTdW01NWVkMl92ZVpEcUFxN3RZbHBGV3lCVjV6ZFVnZjNVZW4xZG43V2JWTjFsc3cxRVIyUHJRQWNLbU5FMTdiZmkifQ==>, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;
}
else {
set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name<https://MailInBlack.univ-lyon1.fr/securelink/?url=http://beresp.backend.name&key=eyJsYW5nIjoiRlIiLCJ1cmwiOiJodHRwOi8vYmVyZXNwLmJhY2tlbmQubmFtZSIsInRva2VuIjoiZ0FBQUFBQmxPLVJiM2doQzZXV2NOd0xYdkdtRk9sZXdtRHI2Q2hwWWwyT0dpaWRNQTJmMVlhdWpQYjRDdG5EMHJEQl9fTUNLRUJFSUd0U3NmckZJczJzcG5WVkowY3d6N213alZ6Z2NPc191UDAzU2R5b1FjUzRIZEJya09SejJMR2xiYklJdWREb0RmX05ZbUI3U25XMzJrbHhkOFVsQ1lVcnhYYlNpMzNvUml4TUNrc2luZzNYV0hzRzRrYm53SmpxbVJrbjdhX0U0Ui02eGlkdWhKd09WY0p4Y1Z5dnEtcmx0T29YYWw2ME1YNG55eWdMNGxFb0hyMmI4ZlpYZHBYc3M0WTFRTGgxZ1JRbkFjYVU0bDNuYlVjZDR3Mmt3YUI3TTA5cFVFWUgxdUNuSWhHb01Wa3BZODhDSFptcU1BWW9uWjZ6aVAtVUQifQ==>, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");
}
}

sub vcl_deliver {

# Revert back to original Cache-Control header before delivery to client
if (resp.http.X-Orig-Cache-Control)
{
set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;
unset resp.http.X-Orig-Cache-Control;
}
# Revert back to original Pragma header before delivery to client
if (resp.http.X-Orig-Pragma)
{
set resp.http.Pragma = resp.http.X-Orig-Pragma;
unset resp.http.X-Orig-Pragma;
}

if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
set resp.http.X-Cache-Hits = obj.hits;

# If desired "Via: 1.1 Varnish-v4" response header can be removed from response
unset resp.http.Via;
unset resp.http.Server;

return (deliver);
}

sub vcl_backend_error {
# More comprehensive varnish error page. Display time, instance hostname, host header, url for easier troubleshooting.
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
synthetic( {"
<!DOCTYPE html>
<html>
<head>
<title>"} + beresp.status + " " + beresp.reason + {"</title>
</head>
<body>
<h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
<p>"} + beresp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>Time: "} + now + {"</p>
<p>Node: "} + server.hostname + {"</p>
<p>Host: "} + bereq.http.host + {"</p>
<p>URL: "} + bereq.url + {"</p>
<p>XID: "} + bereq.xid + {"</p>
<hr>
<p>Varnish cache server
</body>
</html>
"} );
return (deliver);
}

sub vcl_synth {
#Redirect using '301 - Permanent Redirect', permanent redirect
if (resp.status == 851) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 301;
return (deliver);
}
#Redirect using '302 - Found', temporary redirect
if (resp.status == 852) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 302;
return (deliver);
}
#Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests, dont change method on redirected requests
if (resp.status == 857) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 307;
return (deliver);
}
#Respond with 403 - Forbidden
if (resp.status == 863) {
set resp.http.X-Varnish-Error = true;
set resp.status = 403;
return (deliver);
}
}

sub vcl_purge {
if (req.method != "PURGE") {
set req.http.X-Purge = "Yes";
return (restart);
}
}

...


Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
Hello Karim,

You VCL would be useful to debug this (as well as the command line you are running Varnish with), but it sounds like Varnish is using the Transient storage (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage<https://MailInBlack.univ-lyon1.fr/securelink/?url=https://varnish-cache.org&key=eyJsYW5nIjoiRlIiLCJ1cmwiOiJodHRwczovL3Zhcm5pc2gtY2FjaGUub3JnL2RvY3MvdHJ1bmsvdXNlcnMtZ3VpZGUvc3RvcmFnZS1iYWNrZW5kcy5odG1sI3RyYW5zaWVudC1zdG9yYWdlIiwidG9rZW4iOiJnQUFBQUFCbE8tUmIxZnVmSHBWbmpscFhrWDdiMDZDaUNXVGxlVFUyc0Y0YkxMTTBPU3JWYlQzT1kwNEo0cnBBV2tWaEcxdjlaenhTcnBHZWo3cVJ1a1hSbmZYUzFBbWlZb0piRFJBNTdfcGtNY0c4Qjl5c0s2cld2SGZjX3pLVlRtYkY0ckhoVVNIMTBGbXZrOFZORVZHQVkzTEtoTm5WQXl0NHh2ZElPZEItbU9nS2tQQjF5dExCQUlkS05yRDJoZjhGMHQ2Y2p6YW5WTFYwU29qZXJsYU9lZTZxMUJ5Y3RoYzU1aWdhVzJmUkJoUFRKRHVJRGxPT0JoWHR0di1JWnNqdzhvNkw3X0ZfNEtJM2ZVemFnZzJIRkpkRFZZSWpscVRfNUhzRkRhOWlwNjN6NzcyWHloZV9CYTJPQWVaRnJwZnpmbWR5azBVeiJ9>) to store the file, and as the storage isn't bounded, it explodes.
We can fix this in a couple of ways, from storing the file in the regular cache storage, to using pipe, to waiting a few days for https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643 to be released.

Question is: should that file be cached?

Cheers,

--
Guillaume Quintard


On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari@univ-lyon1.fr<mailto:karim.ayari@univ-lyon1.fr>> wrote:

Hi!

I am currently experiencing a memory load problem with video playback.

here is the infrastructure :

client --> haproxy --> varnish --> moodle workers (x5)

a teacher uploaded a 400MB video to Moodle, when we start playing the video with browser player, Varnish consumes all the memory until it runs out and oom killer to kill varnishd. i have no configuration for mp4 files in my vcl file, so by default they are not hidden (?). I can't find a solution :(

I can give my vcl file if necessary.

(I am a beginner on varnish :))

thank you for your support.

Karim

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<https://MailInBlack.univ-lyon1.fr/securelink/?url=https://www.varnish-cache.org&key=eyJsYW5nIjoiRlIiLCJ1cmwiOiJodHRwczovL3d3dy52YXJuaXNoLWNhY2hlLm9yZy9saXN0cy9tYWlsbWFuL2xpc3RpbmZvL3Zhcm5pc2gtbWlzYyIsInRva2VuIjoiZ0FBQUFBQmxPLVJiZUZqNEFLdzBYZ0NGUFowNm9Fa3IwQzVOR1A5cWRPYlB1U0t2S2NWaG1VSGRSRjlYaGt4enNxV095NXVGM1ZwLWRhR0JmZmFRNkc5Qk95c1ZHUktzTVFJY1dPQ0ZqYzlwRlZlNERBUTVFdEtzUWh6YS1vX3hJOUVfYTEwSklhbVkzQ3BhMnZfMnRPbjFkQWF4cWpPY0YxVkR4ZHFFSWl1NWQ1dXZJbW00Z2hLdERoYnlSZk9jVjdHMDJDcTBKWFB1TmRWX3NwZW9vaGMxV3B2a3ZnZlpCb2dTSkEtaU5iaUtRd0Izci1qOUxQQ00zckVfajRUTDdVTUNqczQtS1otUHQtWGt3Tmx5WGNZelJvOTd4ZUR5cEtuUTFUdno4cHVMUjk2NlFVODY3LVVoc1BxcVJseTlxeDA4QVdEcGJlZVkifQ==>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<https://MailInBlack.univ-lyon1.fr/securelink/?url=https://www.varnish-cache.org&key=eyJsYW5nIjoiRlIiLCJ1cmwiOiJodHRwczovL3d3dy52YXJuaXNoLWNhY2hlLm9yZy9saXN0cy9tYWlsbWFuL2xpc3RpbmZvL3Zhcm5pc2gtbWlzYyIsInRva2VuIjoiZ0FBQUFBQmxPLVJiTW4tMVYzSGlGZ3JqSEI3QllsV01jcnp2NG1RSk9Dcy02T2w3eExrcGxwRUduYWlqb0pXdE9aT29FOWxmT2thY01zWDh3U0RONzVxVjFSUWN2NXEzS1pLbWYxY0N5bzhKbURRSV9oSlZQYnBTNEpIb2ZZV2hpYmVDTW5uWGh5QVNncWRTRERmY1FfNHU0S21FMEVRcHlMRVZMcVNHVUFqcWJDaV9TM3dDWEtpT2JRN0hXUjNaMWprVjBlUTFwSGVua3ZJcE5HNEtMelB4eHhvQzgzTUxLOUJJeFkyeXFDSmRENVZ5b3Z2b2w1ck1BVUJyUGM3X2hBeEhDT3BBOUMxaEFOMnNPaW5RYzY2Q0Ewb1ZCOEhSUGh3bnBQV0o3bFRJX0ZUcTRBcmRESVR4YlQySkpHNE9hbDBoX0VPeFhRbnYifQ==>