Mailing List Archive

Migration from 4.1 vcl
Hi,

I am migrating from 4.1 and I am scared that with latest version varnish would break.

What should I take into account when migrating from 4.1?

Regards,

Miguel
Re: Migration from 4.1 vcl [ In reply to ]
On Wed, Oct 9, 2019 at 9:27 AM Miguel Gonzalez
<miguel_3_gonzalez@yahoo.es> wrote:
>
> Hi,
>
> I am migrating from 4.1 and I am scared that with latest version varnish would break.
>
> What should I take into account when migrating from 4.1?

Probably mostly third-party VMODs, since Varnish 6.x still supports
the VCL 4.0 syntax. There may be other non-VCL breaking changes and I
recommend you go through the release notes from 5.0 to 6.x, whichever
you choose (latest is 6.3 and LTS is 6.0).

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Migration from 4.1 vcl [ In reply to ]
On Wed, Oct 9, 2019, at 11:39, Dridi Boukelmoune wrote:
> On Wed, Oct 9, 2019 at 9:27 AM Miguel Gonzalez wrote:
> >
> > I am migrating from 4.1 and I am scared that with latest version varnish would break.
> > What should I take into account when migrating from 4.1?
>
> Probably mostly third-party VMODs, since Varnish 6.x still supports
> the VCL 4.0 syntax.

Any specific info on migration of vmods?
I mostly use the "bundled" vmods[1], but occasionally others like libvmod-sqlite3[2].

If I wanted to convert or upgrade them, do you have any hints or info on what is likely to break and why?
Can these vmods still be built in the same way with 6.0?

Thanks,

--
Cosimo

[1] https://github.com/varnish/varnish-modules
[2] https://github.com/fgsch/libvmod-sqlite3
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Migration from 4.1 vcl [ In reply to ]
> > Probably mostly third-party VMODs, since Varnish 6.x still supports
> > the VCL 4.0 syntax.
>
> Any specific info on migration of vmods?
> I mostly use the "bundled" vmods[1], but occasionally others like libvmod-sqlite3[2].
>
> If I wanted to convert or upgrade them, do you have any hints or info on what is likely to break and why?
> Can these vmods still be built in the same way with 6.0?

The problem with third-party VMODs is that we don't control them, so
if some idiot [1] decides to change a VMOD's API then VCL making use
of it will need some amount of rewriting when upgrading. You shouldn't
have a problem with bundled VMODs, otherwise we made a mistake.

When it comes to third-party VMODS an upgrade from Varnish 4.1 to 6.x
needs to be studied on a case-by-case basis. Worst case scenario the
module is not even available for the Varnish version you with to
upgrade to.

Dridi

[1] https://github.com/dridi/libvmod-querystring
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Migration from 4.1 vcl [ In reply to ]
On Wed, Oct 9, 2019, at 17:53, Dridi Boukelmoune wrote:
> > > Probably mostly third-party VMODs, since Varnish 6.x still supports
> > > the VCL 4.0 syntax.
> >
> > Any specific info on migration of vmods?
> > I mostly use the "bundled" vmods[1], but occasionally others like libvmod-sqlite3[2].
>
> When it comes to third-party VMODS an upgrade from Varnish 4.1 to 6.x
> needs to be studied on a case-by-case basis. Worst case scenario the
> module is not even available for the Varnish version you with to
> upgrade to.

What I meant was if there's any vmod interface/api change between 4.1 and 6.0
that could break third party vmod code, if that makes any sense.

In that case, if a vmod I'd like to use on 6.x doesn't exist or hasn't been
upgraded yet, I could perhaps try and do that myself.

Maybe too much of a generic question :-)

--
Cosimo
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Migration from 4.1 vcl [ In reply to ]
On 10/9/19 18:03, Cosimo Streppone wrote:
>
> What I meant was if there's any vmod interface/api change between 4.1 and 6.0
> that could break third party vmod code, if that makes any sense.

Yes. With just about every new version of Varnish, there's always
something. It may be not be much for each version (for Varnish 6.3.0,
for example, you need to replace WS_Reserve() with WS_ReserveSize(), or
else you get deprecation warnings), but enough so that some kind of fix
is often necessary.

If you're jumping from 4.1 to 6.0, chances are that quite a few of
things accumulated over time.

The What's New/Upgrading docs for the various versions often have a
section about changes for developers of VMODs and other third-party
software, so you might want to look through those. The log in
changes.rst in the git repo also goes into such changes, in more
technical detail.


Good luck,
Geoff
--
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
Re: Migration from 4.1 vcl [ In reply to ]
On 10/09/19 6:12 PM, Geoff Simmons wrote:
> On 10/9/19 18:03, Cosimo Streppone wrote:
>> What I meant was if there's any vmod interface/api change between 4.1 and 6.0
>> that could break third party vmod code, if that makes any sense.
> Yes. With just about every new version of Varnish, there's always
> something. It may be not be much for each version (for Varnish 6.3.0,
> for example, you need to replace WS_Reserve() with WS_ReserveSize(), or
> else you get deprecation warnings), but enough so that some kind of fix
> is often necessary.
>
> If you're jumping from 4.1 to 6.0, chances are that quite a few of
> things accumulated over time.
>
> The What's New/Upgrading docs for the various versions often have a
> section about changes for developers of VMODs and other third-party
> software, so you might want to look through those. The log in
> changes.rst in the git repo also goes into such changes, in more
> technical detail.

Thanks all for answering. I attach the default.vcl hereby. do you
foresee any issue in migrating either to 5.x or 6.x?

#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at
https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

import std;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "XXX.XXX.XXX.XXX";
    .port = "82";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;  

 
}

acl purge {
        "localhost";
        "127.0.0.1";
        "XXX.XXX.XXX.XXX";
}


# This function is used when a request is send by a HTTP client (Browser)
sub vcl_recv {

        # remove ?ver=xxxxx strings from urls so css and js files are
cached.
        # Watch out when upgrading WordPress, need to restart Varnish or
flush cache.
        set req.url = regsub(req.url, "\?ver=.*$", "");

        # The code below makes sure the AJAX "add to cart" function works
        set req.url = regsub(req.url, "add-to-cart=\d+_\d+&", "");
 
        # Remove "replytocom" from requests to make caching better.
        set req.url = regsub(req.url, "\?replytocom=.*$", "");

        #We pass real IP and Port to the backend

        if (req.http.X-Forwarded-Proto == "https" ) {
           set req.http.X-Port = "443";
        } else {
           set req.http.X-Port = "80";
        }

        set req.http.X-Forwarded-For = regsub(req.http.X-Forwarded-For,
"^([^,]+),?.*$", "\1");


    # Normalize the header, remove the port (in case you're testing this
on various TCP ports)

        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

    # Remove has_js and CloudFlare/Google Analytics __* cookies.
    set req.http.Cookie = regsuball(req.http.Cookie,
"(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
    # Remove a ";" prefix, if present.
    set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");


    # Allow purging from ACL
    if (req.method == "PURGE") {
        # If not allowed then a error 405 is returned
        if (!client.ip ~ purge) {
            return(synth(405, "This IP is not allowed to send PURGE
requests."));
        }
        # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()

                if (req.http.X-Purge-Method == "regex") {

                  ban("req.url ~ " + req.url + " && req.http.host ~ " +
req.http.host);

                  return (synth(200, "Banned."));

                } else {

                return (purge);

                }



    }

    # Post requests will not be cached
    #if (req.http.Authorization || req.method == "POST") {
    #    return (pass);
    #}

        # Pass anything other than GET and HEAD directly.
        if (req.method != "GET" && req.method != "HEAD") {
                return( pass );
        }      /* We only deal with GET and HEAD by default */


        #Woocommerce don't cache :
        if (req.url ~
"^/(cart|account|my-account/*|checkout|addons|logout|lost-password|product/*)")
{
          return (pass);
        }
      
        #Woocommerce add to cart pass :
        if (req.url ~ "\?add-to-cart=" ) {
          return (pass);
        }
        if (req.url ~ "/wp-cron.php" || req.url ~ "preview=true") {
          return (pass);
        }
 
        # Woocommerce
        if (req.url ~ "(cart|account|my-account|checkout|addons)") {
          return (pass);
        }
        if ( req.url ~ "\?add-to-cart=" ) {
          return (pass);
        }
 

    # --- WordPress specific configuration
    # Did not cache the admin and login pages
    if (req.url ~
"feed|nocache|cart|account|my-account|checkout|addons|tienda|iniciar-sesion|mi-cuenta|comunidad|carro|carrito|finalizar-compra|producto/*|login|wp-json|wp-admin|wp-(comments-post|login|signup|activate|mail|cron)\.php|preview\=true|admin-ajax\.php|xmlrpc\.php|bb-admin|whm-server-status|server-status|control\.php|bb-login\.php|bb-reset-password\.php|register\.php|colabora|gracias-por-colaborar|tu-colaboracion-ha-fallado")
{
        return (pass);
    }

    if (req.url ~ "(ajax|dynamic|custom)") {
             return(pass);
        }

    # Remove the "has_js" cookie
    set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(;
)?", "");

    # Remove any Google Analytics based cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(;
)?", "");

    # Remove the Quant Capital cookies (added by some plugin, all __qca)
    set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?",
"");

    # Remove the wp-settings-1 cookie
    set req.http.Cookie = regsuball(req.http.Cookie,
"wp-settings-1=[^;]+(; )?", "");

    # Remove the wp-settings-time-1 cookie
    set req.http.Cookie = regsuball(req.http.Cookie,
"wp-settings-time-1=[^;]+(; )?", "");

    # Remove the wp test cookie
    set req.http.Cookie = regsuball(req.http.Cookie,
"wordpress_test_cookie=[^;]+(; )?", "");

    # Remove the wp give session cookie
    set req.http.Cookie = regsuball(req.http.Cookie,
"wp_give_session=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie,
"wp-give_session=[^;]+(; )?", "");

    # Are there cookies left with only spaces or that are empty?

    if (req.http.cookie ~ "^ *$") {
            unset req.http.cookie;
    }

    # Cache the following files extensions
    if (req.url ~ "\.(txt|css|js|png|gif|jp(e)?g|swf|ico)") {
        unset req.http.cookie;
    }

    # Normalize Accept-Encoding header and compression
    # https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
    if (req.http.Accept-Encoding) {
        # Do no compress compressed files...
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
                   unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
                set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
                set req.http.Accept-Encoding = "deflate";
        } else {
            unset req.http.Accept-Encoding;
        }
    }

    # Check the cookies for wordpress-specific items
    if (req.http.Cookie ~ "woocommerce" || req.http.Cookie ~ "wordpress"
|| req.http.Cookie ~ "wp-" || req.http.Cookie ~ "comment_") {
        return (pass);
    }
    if (!req.http.cookie) {
        unset req.http.cookie;
    }

    # --- End of WordPress specific configuration

    # Did not cache HTTP authentication and HTTP Cookie
    if (req.http.Authorization || req.http.Cookie) {
        # Not cacheable by default
        return (pass);
    }

    # Cache all others requests
    return (hash);
}

sub vcl_pipe {
    return (pipe);
}

sub vcl_pass {
    return (fetch);
}

# The data on which the hashing will take place
sub vcl_hash {

     hash_data(req.url);
     if (req.http.host) {
           hash_data(req.http.host);
     } else {
           hash_data(server.ip);
     }

    # If the client supports compression, keep that in a different cache
        if (req.http.Accept-Encoding) {
          hash_data(req.http.Accept-Encoding);
    }

        # Cache the HTTP vs HTTPs separately
        if (req.http.X-Forwarded-Proto) {
          hash_data(req.http.X-Forwarded-Proto);
        }

    return (lookup);
}

# This function is used when a request is sent by our backend (Nginx server)
sub vcl_backend_response {
    # Remove some headers we never want to see
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;

        if (beresp.http.content-type ~
"(text|javascript|application/x-font-woff)") {
          set beresp.do_gzip = true;
        }

    # For static content strip all backend cookies
    if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
        unset beresp.http.cookie;
    }
    # Don't store backend
    if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") {
        set beresp.uncacheable = true;
        set beresp.ttl = 30s;
        return (deliver);
    }

    # Only allow cookies to be set if we're in admin area
        if (!(bereq.url ~
"(wp-login|cart|account|my-account|colabora|gracias-por-colaborar|tu-colaboracion-ha-fallado|checkout|finalizar-compra|addons|tienda|iniciar-sesion|mi-cuenta|comunidad|carrito|carro|producto/*|login|wp-json|wp-admin|preview=true)"))
{
            unset beresp.http.set-cookie;
    }

    # don't cache response to posted requests or those with basic auth
    if ( bereq.method == "POST" || bereq.http.Authorization ) {
            set beresp.uncacheable = true;
        set beresp.ttl = 120s;
        return (deliver);
        }

        # don't cache search results
    if ( bereq.url ~ "\?s=" ){
        set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
    }

    # only cache status ok
    if ( beresp.status != 200 ) {
        set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
    }

    # A TTL of 24h
    set beresp.ttl = 24h;
    # Define the default grace period to serve cached content
    #set beresp.grace = 30s;
    set beresp.grace = 1h;

    return (deliver);
}

# The routine when we deliver the HTTP request to the user
# Last chance to modify headers that are sent to the client
sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "cached";
    } else {
        set resp.http.x-Cache = "uncached";
    }

    # Remove some headers: PHP version
    unset resp.http.X-Powered-By;

    # Remove some headers: Apache version & OS
    unset resp.http.Server;

    # Remove some heanders: Varnish
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_init {
     return (ok);
}

sub vcl_fini {
     return (ok);
}





--
This email has been checked for viruses by AVG.
https://www.avg.com
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc