It seems that no response codes will deter linux package-management tools
and the amount of traffic we had beating up on the project server that
way was totally swamping any other traffic.
That's not a problem of course, we run varnish.
But it is a LOT of users not getting good service.
I spent some hours trying to find out where to send them on packagecloud.io
and came up with this:
if (req.url ~ "repomd.xml") {
set req.http.urlx = regsub(req.url,
".*varnish-([345])[.](.)/el(.*)",
"/varnishcache/varnish\1\2/el/\3");
return (synth(301,"packagecloud"));
}
if (req.url ~ "^/(ubuntu|debian)/dists/[^/]*/I*n*Release$") {
/*
* We don't seem to have files on packageio which lists all
* releases, so point these at release5.2
*/
set req.http.urlx = regsub(req.url,
"/(ubuntu|debian)/dists/(.*)/(I*n*Release)",
"/varnishcache/varnish52/\1/dists/\2/\3");
return (synth(301,"packagecloud"));
}
I'm still seeing a lot of "deep" urls being requested (see below) but
I'm hoping these will migrate to packagecloud now that the top-level
URLs point them there.
I would appreciate if somebody with more linux package clue than me
(ie: any non-zero amount of clue will be helpful) can sanity-check
what I've done.
Poul-Henning
12.47 ReqURL /ubuntu/dists/trusty/InRelease
10.23 ReqURL /redhat/varnish-3.0/el5/x86_64/repodata/repomd.xml
8.45 ReqURL /redhat/varnish-4.0/el6/repodata/repomd.xml
6.69 ReqURL /redhat/varnish-4.1/el6/x86_64/repodata/0db82826c0003726ea45ca2ff822454afb55e209-primary.xml.gz
6.57 ReqURL /redhat/varnish-3.0/el6/x86_64/repodata/repomd.xml
6.54 ReqURL /
6.07 ReqURL /redhat/varnish-3.0/el6/repodata/repomd.xml
5.51 ReqURL /redhat/varnish-3.0/el5/x86_64/repodata/46939bc1312d4686f78d539762f83d0d3bb5f5cf-primary.xml.gz
5.15 ReqURL /redhat/varnish-3.0/el6/x86_64/repodata/0e86f304b9f8c0e214d431f95100ad4668e0ad9e-primary.xml.gz
4.63 ReqURL /ubuntu/dists/lucid/Release
4.60 ReqURL /ubuntu/dists/lucid/Release.gpg
4.45 ReqURL /redhat/varnish-4.0/el6/x86_64/repodata/repomd.xml
4.30 ReqURL /debian/dists/wheezy/Release
4.30 ReqURL /debian/dists/wheezy/Release.gpg
3.84 ReqURL /redhat/varnish-4.1/el6/x86_64/repodata/repomd.xml
3.78 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.gz
3.77 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.lzma
3.74 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.bz2
3.48 ReqURL /redhat/varnish-4.0/el6/x86_64/repodata/b3cc34523cf82349fbdce79b0588b471b842a408-primary.xml.gz
3.47 ReqURL /ubuntu/dists/precise/Release
3.46 ReqURL /ubuntu/dists/precise/Release.gpg
3.18 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en
3.17 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.gz
3.17 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages
3.16 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.lzma
3.14 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.xz
3.13 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.xz
3.12 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.bz2
3.10 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.diff/Index
3.01 ReqURL /debian/dists/jessie/InRelease
2.97 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages
2.96 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en
2.96 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.gz
2.96 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.gz
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.lzma
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.lzma
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.xz
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.xz
2.90 ReqURL /ubuntu/GPG-key.txt
2.89 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.bz2
2.89 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.bz2
2.87 ReqURL /debian/GPG-key.txt
2.78 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.diff/Index
2.53 ReqURL /redhat/varnish-4.0/el7/repodata/repomd.xml
2.44 ReqURL /redhat/varnish-4.0/el7/x86_64/repodata/repomd.xml
2.40 ReqURL /redhat/varnish-4.0/el7/x86_64/repodata/0ffac79c8b21f896830f3e6a81b4e9f8430199d3-primary.xml.gz
2.37 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.gz
2.36 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.lzma
2.36 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages
2.34 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.bz2
2.33 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.xz
2.31 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.diff/Index
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
and the amount of traffic we had beating up on the project server that
way was totally swamping any other traffic.
That's not a problem of course, we run varnish.
But it is a LOT of users not getting good service.
I spent some hours trying to find out where to send them on packagecloud.io
and came up with this:
if (req.url ~ "repomd.xml") {
set req.http.urlx = regsub(req.url,
".*varnish-([345])[.](.)/el(.*)",
"/varnishcache/varnish\1\2/el/\3");
return (synth(301,"packagecloud"));
}
if (req.url ~ "^/(ubuntu|debian)/dists/[^/]*/I*n*Release$") {
/*
* We don't seem to have files on packageio which lists all
* releases, so point these at release5.2
*/
set req.http.urlx = regsub(req.url,
"/(ubuntu|debian)/dists/(.*)/(I*n*Release)",
"/varnishcache/varnish52/\1/dists/\2/\3");
return (synth(301,"packagecloud"));
}
I'm still seeing a lot of "deep" urls being requested (see below) but
I'm hoping these will migrate to packagecloud now that the top-level
URLs point them there.
I would appreciate if somebody with more linux package clue than me
(ie: any non-zero amount of clue will be helpful) can sanity-check
what I've done.
Poul-Henning
12.47 ReqURL /ubuntu/dists/trusty/InRelease
10.23 ReqURL /redhat/varnish-3.0/el5/x86_64/repodata/repomd.xml
8.45 ReqURL /redhat/varnish-4.0/el6/repodata/repomd.xml
6.69 ReqURL /redhat/varnish-4.1/el6/x86_64/repodata/0db82826c0003726ea45ca2ff822454afb55e209-primary.xml.gz
6.57 ReqURL /redhat/varnish-3.0/el6/x86_64/repodata/repomd.xml
6.54 ReqURL /
6.07 ReqURL /redhat/varnish-3.0/el6/repodata/repomd.xml
5.51 ReqURL /redhat/varnish-3.0/el5/x86_64/repodata/46939bc1312d4686f78d539762f83d0d3bb5f5cf-primary.xml.gz
5.15 ReqURL /redhat/varnish-3.0/el6/x86_64/repodata/0e86f304b9f8c0e214d431f95100ad4668e0ad9e-primary.xml.gz
4.63 ReqURL /ubuntu/dists/lucid/Release
4.60 ReqURL /ubuntu/dists/lucid/Release.gpg
4.45 ReqURL /redhat/varnish-4.0/el6/x86_64/repodata/repomd.xml
4.30 ReqURL /debian/dists/wheezy/Release
4.30 ReqURL /debian/dists/wheezy/Release.gpg
3.84 ReqURL /redhat/varnish-4.1/el6/x86_64/repodata/repomd.xml
3.78 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.gz
3.77 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.lzma
3.74 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.bz2
3.48 ReqURL /redhat/varnish-4.0/el6/x86_64/repodata/b3cc34523cf82349fbdce79b0588b471b842a408-primary.xml.gz
3.47 ReqURL /ubuntu/dists/precise/Release
3.46 ReqURL /ubuntu/dists/precise/Release.gpg
3.18 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en
3.17 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.gz
3.17 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages
3.16 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.lzma
3.14 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.xz
3.13 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.xz
3.12 ReqURL /ubuntu/dists/lucid/varnish-3.0/i18n/Translation-en.bz2
3.10 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-i386/Packages.diff/Index
3.01 ReqURL /debian/dists/jessie/InRelease
2.97 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages
2.96 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en
2.96 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.gz
2.96 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.gz
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.lzma
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.lzma
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.xz
2.92 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.xz
2.90 ReqURL /ubuntu/GPG-key.txt
2.89 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.bz2
2.89 ReqURL /debian/dists/wheezy/varnish-3.0/i18n/Translation-en.bz2
2.87 ReqURL /debian/GPG-key.txt
2.78 ReqURL /debian/dists/wheezy/varnish-3.0/binary-amd64/Packages.diff/Index
2.53 ReqURL /redhat/varnish-4.0/el7/repodata/repomd.xml
2.44 ReqURL /redhat/varnish-4.0/el7/x86_64/repodata/repomd.xml
2.40 ReqURL /redhat/varnish-4.0/el7/x86_64/repodata/0ffac79c8b21f896830f3e6a81b4e9f8430199d3-primary.xml.gz
2.37 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.gz
2.36 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.lzma
2.36 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages
2.34 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.bz2
2.33 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.xz
2.31 ReqURL /ubuntu/dists/lucid/varnish-3.0/binary-amd64/Packages.diff/Index
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev