Mailing List Archive

$domain_data not populated, caching problem?
When I try to use $domain_data from a redirect router
the value seems to unpopulated:

aliases:
driver = redirect
domains = +local_domains
data = ${lookup{$local_part}lsearch*{CONFDIR/aliases/$domain_data}}
qualify_preserve_domain
allow_fail
allow_defer
forbid_file
include_directory = CONFDIR
pipe_transport = address_pipe

Debug output:

233304 --------> aliases router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 checking domains
233304 cached yes match for +local_domains
233304 cached lookup data =
233304 wikimedia.org in "+local_domains"? yes (matched
"+local_domains" - cached)
233304 calling aliases router
233304 rda_interpret (string):
'${lookup{$local_part}lsearch*{/etc/exim4/aliases/$domain_data}}'
233304 search_open: lsearch "/etc/exim4/aliases/"
233304 aliases router: defer for lollipopman@wikimedia.org
233304 message: failed to expand
"${lookup{$local_part}lsearch*{/etc/exim4/aliases/$domain_data}}":
/etc/exim4/aliases/ is not a regular file (lsearch lookup)

Is this possibly a caching bug like this one, mentioned many years ago,
https://exim-users.exim.narkive.com/MUiNGYww/bug-in-domain-data-caching?

Or is there something silly that I am missing?

Thanks, Jesse

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On 08/04/2022 17:33, Jesse Hathaway via Exim-users wrote:
> Or is there something silly that I am missing?

Telling us what version of Exim.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On 09/04/2022 10:33, Jeremy Harris via Exim-users wrote:
> On 08/04/2022 17:33, Jesse Hathaway via Exim-users wrote:
>> Or is there something silly that I am missing?
>
> Telling us what version of Exim.

On thinking more - it's far more likely that the
lookup never returned any data (so that's what was cached).

--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On Sat, Apr 9, 2022 at 4:44 AM Jeremy Harris via Exim-users
<exim-users@exim.org> wrote:
> On 08/04/2022 17:33, Jesse Hathaway via Exim-users wrote:
> > Or is there something silly that I am missing?
>
> Telling us what version of Exim.

Sorry, that was silly, 4.94 packaged by Debian.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On Sat, Apr 9, 2022 at 8:03 AM Jeremy Harris via Exim-users
<exim-users@exim.org> wrote:
> On thinking more - it's far more likely that the
> lookup never returned any data (so that's what was cached).

hmm, I'm a bit confused, I thought it wouldn't proceed down the router
if there was not a match
for domains? Also, the domains did match as seen in the below debug output:

233304 data from lookup saved for cache for +local_domains: key
'wikimedia.org' value ''
233304 wikimedia.org in "! +local_domains"? no (matched "! +local_domains")
233304 accept: condition test failed in ACL "acl_check_rcpt"
233304 processing "require" (/etc/exim4/exim4.conf 145)
233304 message: Relay not permitted
233304 check domains = +local_domains : +relay_domains
233304 cached yes match for +local_domains
233304 cached lookup data =
233304 wikimedia.org in "+local_domains : +relay_domains"? yes
(matched "+local_domains" - cached)
233304 check set acl_m_relayed = yes
233304 require: condition test succeeded in ACL "acl_check_rcpt"
233304 processing "defer" (/etc/exim4/exim4.conf 150)
233304 message: Administratively set to defer
233304 check domains = +defer_domains
233304 search_open: lsearch "/etc/exim4/defer_domains"
233304 search_find: file="/etc/exim4/defer_domains"
233304 key="wikimedia.org" partial=-1 affix=NULL starflags=0 opts=NULL
233304 LRU list:
233304 :/etc/exim4/defer_domains
233304 :/etc/exim4/wikimedia_domains
233304 End
233304 internal_search_find: file="/etc/exim4/defer_domains"
233304 type=lsearch key="wikimedia.org" opts=NULL
233304 file lookup required for wikimedia.org
233304 in /etc/exim4/defer_domains
233304 lookup failed
233304 wikimedia.org in "lsearch;/etc/exim4/defer_domains"? no (end of list)
233304 wikimedia.org in "+defer_domains"? no (end of list)
233304 defer: condition test failed in ACL "acl_check_rcpt"
233304 processing "accept" (/etc/exim4/exim4.conf 155)
233304 check local_parts = postmaster : abuse
233304 lollipopman in "postmaster : abuse"? no (end of list)
233304 accept: condition test failed in ACL "acl_check_rcpt"
233304 processing "require" (/etc/exim4/exim4.conf 160)
233304 check verify = recipient
233304 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
233304 Verifying lollipopman@wikimedia.org
233304 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
233304 Considering lollipopman@wikimedia.org
233304 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
233304 routing lollipopman@wikimedia.org
233304 --------> wiki_mail router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 wiki_mail router skipped: verify 2 0 0
233304 --------> donate router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 checking domains
233304 wikimedia.org in "donate.wikimedia.org :
civicrm.wikimedia.org"? no (end of list)
233304 wikimedia.org in "+donate_domains"? no (end of list)
233304 donate router skipped: domains mismatch
233304 --------> dnslookup router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 checking domains
233304 cached yes match for +local_domains
233304 cached lookup data =
233304 wikimedia.org in "! +local_domains"? no (matched "!
+local_domains" - cached)
233304 dnslookup router skipped: domains mismatch
233304 --------> mw_verp_api router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 checking domains
233304 wikimedia.org in "wikimedia.org"? yes (matched "wikimedia.org")
233304 data from lookup saved for cache for +verp_domains: key
'wikimedia.org' value 'wikimedia.org'
233304 wikimedia.org in "+verp_domains"? yes (matched "+verp_domains")
233304 checking "condition" "${if
match{$local_part}{\N^wiki-\w+-\w+-\w+-[+/\w]+$\N}{true}{false}}"...
233304 mw_verp_api router skipped: condition failure
233304 --------> eat router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 checking domains
233304 cached yes match for +local_domains
233304 cached lookup data =
233304 wikimedia.org in "+local_domains"? yes (matched
"+local_domains" - cached)
233304 checking local_parts
233304 lollipopman in "no-reply-notifications"? no (end of list)
233304 eat router skipped: local_parts mismatch
233304 --------> system_aliases router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 checking domains
233304 cached no match for +system_domains
233304 cached lookup data = NULL
233304 wikimedia.org in "+system_domains"? no (end of list)
233304 system_aliases router skipped: domains mismatch
233304 --------> aliases router <--------
233304 local_part=lollipopman domain=wikimedia.org
233304 checking domains
233304 cached yes match for +local_domains
233304 cached lookup data =
233304 wikimedia.org in "+local_domains"? yes (matched
"+local_domains" - cached)
233304 calling aliases router
233304 rda_interpret (string):
'${lookup{$local_part}lsearch*{/etc/exim4/aliases/$domain_data}}'
233304 search_open: lsearch "/etc/exim4/aliases/"
233304 aliases router: defer for lollipopman@wikimedia.org
233304 message: failed to expand
"${lookup{$local_part}lsearch*{/etc/exim4/aliases/$domain_data}}":
/etc/exim4/aliases/ is not a regular file (lsearch lookup)

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On 09/04/2022 22:14, Jesse Hathaway via Exim-users wrote:
> On Sat, Apr 9, 2022 at 8:03 AM Jeremy Harris via Exim-users
> <exim-users@exim.org> wrote:
>> On thinking more - it's far more likely that the
>> lookup never returned any data (so that's what was cached).
>
> hmm, I'm a bit confused, I thought it wouldn't proceed down the router
> if there was not a match
> for domains?

That's correct, but not relevant to $domain_data.

The "domains=" condition is satisfied if the lookup succeeds;
and that essentially means "was the lookup key found".

What goes into $domain_data (for a matching key) is
the *data* for that key. So, what is the actual definition
of your "local_domains" domainlist? If it is doing a lookup
in a file, what sort of lookup, and what is the line you expect
to match?

Also, the domains did match as seen in the below debug output:
>
> 233304 data from lookup saved for cache for +local_domains: key
> 'wikimedia.org' value ''

See that "value" there?
It's empty. I'm betting it was empty when the cache entry
was created, because the value from the original lookup
was empty - probably because your file has no value next
to the key (but here I'm guessing what sort of lookup
local_domains does.

--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On Sat, Apr 9, 2022 at 4:29 PM Jeremy Harris via Exim-users
<exim-users@exim.org> wrote:
> Also, the domains did match as seen in the below debug output:
> >
> > 233304 data from lookup saved for cache for +local_domains: key
> > 'wikimedia.org' value ''
>
> See that "value" there?
> It's empty. I'm betting it was empty when the cache entry
> was created, because the value from the original lookup
> was empty - probably because your file has no value next
> to the key (but here I'm guessing what sort of lookup
> local_domains does.

Ah, that makes sense, and was part of my confusion, local_domains is defined as:

domainlist system_domains = @
domainlist local_domains = +system_domains : +wikimedia_domains :
+legacy_mailman_domains : +verp_domains
domainlist wikimedia_domains = lsearch;CONFDIR/wikimedia_domains
domainlist verp_domains = wikimedia.org
domainlist relay_domains = +wikimedia_domains

I assumed that with a domainlist, the value would also be the key. So
what should I
use in this case, instead of $domain_data?

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On 09/04/2022 22:42, Jesse Hathaway via Exim-users wrote:
> Ah, that makes sense, and was part of my confusion, local_domains is defined as:
>
> domainlist system_domains = @
> domainlist local_domains = +system_domains : +wikimedia_domains :
> +legacy_mailman_domains : +verp_domains
> domainlist wikimedia_domains = lsearch;CONFDIR/wikimedia_domains
> domainlist verp_domains = wikimedia.org
> domainlist relay_domains = +wikimedia_domains
>
> I assumed that with a domainlist, the value would also be the key. So
> what should I
> use in this case, instead of $domain_data?

$domain_data is the thing to be using, but with a local_domains having
that many different sorts of elements you may be needing to run a
more-recent version of Exim. There have been relatively recent
updates providing the facilities needed across the various types
of element, and 4.94 may well be too old.

I see types "@", "lsearch", "literal" at least -
and something undefined "legacy_mailman_domains".

Check the documentation for 4.94 for each of those types against
4.95 (and, possibly, against the current git master).
Chapter 10 is a good place to start.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: $domain_data not populated, caching problem? [ In reply to ]
On Sat, Apr 9, 2022 at 5:17 PM Jeremy Harris via Exim-users
<exim-users@exim.org> wrote:
> Check the documentation for 4.94 for each of those types against
> 4.95 (and, possibly, against the current git master).
> Chapter 10 is a good place to start.

Thanks for the help Jeremy,

After adding ret=key to the lsearch lookup it appears $domain_data is
then properly populated for all lookup types while running 4.94. So it
appears the other lookup types default to substituting the value for the
key, whereas lsearch does not, so the additional config option was
necessary.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/