Mailing List Archive: expansion error in OAuth2 client authenticator

expansion error in OAuth2 client authenticator

Mar 11, 2023, 2:21 PM

Post #1 of 15 (433 views)

Hello.

# uname -sr
FreeBSD 13.1-RELEASE-p6

# pkg info -E exim
exim-4.96

exim compiled with all the patches from exim4_4.96-14
https://packages.debian.org/source/sid/exim4

I try to setup OAuth2 client authenticator:

client_oauth2:
driver = plaintext
public_name = XOAUTH2
client_send = ${lookup{$host<@>$sender_address<@>oauth2}\
lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}

When I sent a test message, I got an error message in the
/var/log/exim/mainlog:

2023-03-11 23:03:13 [2340] 1pb6Mz-0000bk-0v <= v.corvax.test@gmail.com
U=root P=local S=610
2023-03-11 23:03:13 [2341] 1pb6Mz-0000bk-0v == oauth2-test@mx.org.ua
R=mailertable_smtp T=remote_smtp defer (-48): expansion of "${expand"
failed in client_oauth2 authenticator: "${expand" is not a known
operator (or a } is missing in a variable reference)

But value of client_send is correct. I checked it with exim -be.

without ${expand:

# exim -be
'${lookup{smtp.gmail.com<@>v.corvax.test@gmail.com<@>oauth2}lsearch*{/usr/local/etc/exim/smtpauthdb}{$value}fail}'
${run{/bin/sh -c
/usr/local/bin/get_oauth2_access_token-v.corvax.test@gmail.com.sh}{$value}{$value}}

with ${expand:

# exim -be
'${lookup{smtp.gmail.com<@>v.corvax.test@gmail.com<@>oauth2}lsearch*{/usr/local/etc/exim/smtpauthdb}{${expand:$value}}fail}'
user=v.corvax.test@gmail.comauth=Bearer ya29..............FQ0166

I obfuscated part of the result for obvious reasons.

If I replace ${lookup directly with ${run, then everything works properly.

client_oauth2:
driver = plaintext
public_name = XOAUTH2
client_send = ${run{/bin/sh -c
/usr/local/bin/get_oauth2_access_token-v.corvax.test@gmail.com.sh}{$value}{$value}}

/var/log/exim/mainlog:

2023-03-11 23:20:52 [11092] 1pb6e4-0002su-2w <= v.corvax.test@gmail.com
U=root P=local S=610
2023-03-11 23:20:54 [11093] 1pb6e4-0002su-2w => corvax@mx.org.ua
R=mailertable_smtp T=remote_smtp H=smtp.gmail.com [172.253.116.109]
I=[185.187.81.20] X=TLS1.3:TLS_AES_256_GCM_SHA384:256 CV=yes
A=client_oauth2 C="250 2.0.0 OK 1678569654
f4-20020a5d4dc4000000b002c56179d39esm3317181wru.44 - gsmtp"
2023-03-11 23:20:54 [11093] 1pb6e4-0002su-2w Completed QT=1s

So I think it's a bug.

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 11, 2023, 9:55 PM

Post #2 of 15 (431 views)

Permalink

On 2023-03-11 Victor Ustugov via Exim-users <exim-users@exim.org> wrote:
[...]
> client_send = ${lookup{$host<@>$sender_address<@>oauth2}\
> lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}
[...]

Hello,

<Quote>
| |client_send|Use: plaintext|Type: string*|Default: unset|
| +-------------------------------------------------------+
|
| The string is a colon-separated list of authentication data strings. Each
| string is independently expanded before being sent to the server.
</Unquote>

See Chapter 6 - The Exim runtime configuration file "20. List
construction" and "21. Changing list separators" for ways to fix your
configuration.

cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 3:51 AM

Post #3 of 15 (431 views)

Permalink

Andreas Metzler via Exim-users wrote on 12.03.2023 08:55:
> On 2023-03-11 Victor Ustugov via Exim-users <exim-users@exim.org> wrote:
> [...]
>> client_send = ${lookup{$host<@>$sender_address<@>oauth2}\
>> lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}
> [...]
>
> Hello,
>
> <Quote>
> | |client_send|Use: plaintext|Type: string*|Default: unset|
> | +-------------------------------------------------------+
> |
> | The string is a colon-separated list of authentication data strings. Each
> | string is independently expanded before being sent to the server.
> </Unquote>
>
> See Chapter 6 - The Exim runtime configuration file "20. List
> construction" and "21. Changing list separators" for ways to fix your
> configuration.

So is it enough to replace

client_send =
${lookup{$host<@>$sender_address<@>oauth2}lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}

with

client_send =
"${lookup{$host<@>$sender_address<@>oauth2}lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}"

?

> cu Andreas
>

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 4:06 AM

Post #4 of 15 (431 views)

Permalink

> Andreas Metzler via Exim-users wrote on 12.03.2023 08:55:
>> On 2023-03-11 Victor Ustugov via Exim-users <exim-users@exim.org> wrote:
>> [...]
>>> client_send = ${lookup{$host<@>$sender_address<@>oauth2}\
>>> lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}
>> [...]
>>
>> Hello,
>>
>> <Quote>
>> | |client_send|Use: plaintext|Type: string*|Default: unset|
>> | +-------------------------------------------------------+
>> |
>> | The string is a colon-separated list of authentication data strings. Each
>> | string is independently expanded before being sent to the server.
>> </Unquote>
>>
>> See Chapter 6 - The Exim runtime configuration file "20. List
>> construction" and "21. Changing list separators" for ways to fix your
>> configuration.

I reacted to the quoted text from the doc and did not pay attention to
the advice to change the list separator.

In any case, OAuth2 authentication is already working.

Thank you.

> So is it enough to replace
>
> client_send =
> ${lookup{$host<@>$sender_address<@>oauth2}lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}
>
> with
>
> client_send =
> "${lookup{$host<@>$sender_address<@>oauth2}lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}"
>
> ?
>
>
>> cu Andreas
>>
>
>

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 4:38 AM

Post #5 of 15 (431 views)

Permalink

On Sun, 12 Mar 2023, Victor Ustugov via Exim-users wrote:

> Hello.
>
> # uname -sr
> FreeBSD 13.1-RELEASE-p6
>
> # pkg info -E exim
> exim-4.96
>
> exim compiled with all the patches from exim4_4.96-14
> https://packages.debian.org/source/sid/exim4
>
> I try to setup OAuth2 client authenticator:
>
> client_oauth2:
> driver = plaintext
> public_name = XOAUTH2
> client_send = ${lookup{$host<@>$sender_address<@>oauth2}\
> lsearch*{CONFDIR/smtpauthdb}{${expand:$value}}fail}
>
> When I sent a test message, I got an error message in the
> /var/log/exim/mainlog:
>
> 2023-03-11 23:03:13 [2340] 1pb6Mz-0000bk-0v <= v.corvax.test@gmail.com
> U=root P=local S=610
> 2023-03-11 23:03:13 [2341] 1pb6Mz-0000bk-0v == oauth2-test@mx.org.ua
> R=mailertable_smtp T=remote_smtp defer (-48): expansion of "${expand"
> failed in client_oauth2 authenticator: "${expand" is not a known
> operator (or a } is missing in a variable reference)
>
> But value of client_send is correct. I checked it with exim -be.
>
> without ${expand:
>
> # exim -be
> '${lookup{smtp.gmail.com<@>v.corvax.test@gmail.com<@>oauth2}lsearch*{/usr/local/etc/exim/smtpauthdb}{$value}fail}'
> ${run{/bin/sh -c
> /usr/local/bin/get_oauth2_access_token-v.corvax.test@gmail.com.sh}{$value}{$value}}

Can you tell me more about this get_oauth2_access_token script
- I do not see it in https://packages.debian.org/source/sid/exim4 ?

Thanks,

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 6:10 AM

Post #6 of 15 (431 views)

Permalink

Andrew C Aitchison via Exim-users wrote on 12.03.2023 13:38:

>> without ${expand:
>>
>> # exim -be
>> '${lookup{smtp.gmail.com<@>v.corvax.test@gmail.com<@>oauth2}lsearch*{/usr/local/etc/exim/smtpauthdb}{$value}fail}'
>>
>> ${run{/bin/sh -c
>> /usr/local/bin/get_oauth2_access_token-v.corvax.test@gmail.com.sh}{$value}{$value}}
>>
>
> Can you tell me more about this get_oauth2_access_token script
> - I do not see it in https://packages.debian.org/source/sid/exim4 ?
>
> Thanks,
>

This script is not a part of https://packages.debian.org/source/sid/exim4

I wrote about https://packages.debian.org/source/sid/exim4 because Exim
4.95 is still in the FreeBSD ports tree. I had to create the exim 4.96
port for FreeBSD myself. I added my patches and many patches from
https://packages.debian.org/source/sid/exim4 to it. I mean 75_*.patch

Script get_oauth2_access_token-v.corvax.test@gmail.com.sh is based on
https://github.com/pcw11211/exim4-oauth2/blob/main/get_bearer_tocken.sh

This is just a prototype. I'm going to cache responses to requests to
get an access_token, taking into account the expires_in value, in order
to reduce the number of HTTP requests.

Client id, client secret and refresh token will be stored in the
database, so there will be no need to create a script to get an access
token for each user.

Later, I'm going to make requests to get an access token using
${readsocket. In this case, I won't need to call ${run and curl.

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 9:25 AM

Post #7 of 15 (431 views)

Permalink

Victor Ustugov via Exim-users wrote on 12.03.2023 15:10:

> Later, I'm going to make requests to get an access token using
> ${readsocket. In this case, I won't need to call ${run and curl.

Is it possible to use SNI with ${readsocket?

In other words, is it possible to make HTTPS requests to a non-default
HTTPS virtualhost?

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 10:09 AM

Post #8 of 15 (431 views)

Permalink

On 12/03/2023 16:25, Victor Ustugov via Exim-users wrote:
> Is it possible to use SNI with ${readsocket?
No.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 10:31 AM

Post #9 of 15 (431 views)

Permalink

Jeremy Harris via Exim-users wrote on 12.03.2023 19:09:
> On 12/03/2023 16:25, Victor Ustugov via Exim-users wrote:
>> Is it possible to use SNI with ${readsocket?
> No.

Do you plan to implement this functionality?

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 11:06 AM

Post #10 of 15 (431 views)

Permalink

On 12/03/2023 17:31, Victor Ustugov via Exim-users wrote:
> Jeremy Harris via Exim-users wrote on 12.03.2023 19:09:
>> On 12/03/2023 16:25, Victor Ustugov via Exim-users wrote:
>>> Is it possible to use SNI with ${readsocket?
>> No.
>
> Do you plan to implement this functionality?

It's not currently on the radar. Glancing round the
code, it could be implemented with a bit of a hack.
Choosing a syntax would also be needed.

How badly do you need it?

Testing is an issue. I think you mentioned building
a FreeBSD port for yourself; does that mean you
could take a patch and test that?
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 11:46 AM

Post #11 of 15 (431 views)

Permalink

On Sun, Mar 12, 2023 at 9:17?PM Jeremy Harris via Exim-users <
exim-users@exim.org> wrote:

> On 12/03/2023 17:31, Victor Ustugov via Exim-users wrote:
> > Jeremy Harris via Exim-users wrote on 12.03.2023 19:09:
> >> On 12/03/2023 16:25, Victor Ustugov via Exim-users wrote:
> >>> Is it possible to use SNI with ${readsocket?
> >> No.
> >
> > Do you plan to implement this functionality?
>
> It's not currently on the radar. Glancing round the
> code, it could be implemented with a bit of a hack.
> Choosing a syntax would also be needed.
>
> How badly do you need it?
>
> Testing is an issue. I think you mentioned building
> a FreeBSD port for yourself; does that mean you
> could take a patch and test that?
>

Seeing as FreeBSD port got stuck at 4.95, I am also willing to test if this
is going in the direction of getting 4.96 in the ports.

--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(?)_/¯ :-)
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 2:32 PM

Post #12 of 15 (431 views)

Permalink

Jeremy Harris via Exim-users wrote on 12.03.2023 20:06:
> On 12/03/2023 17:31, Victor Ustugov via Exim-users wrote:
>> Jeremy Harris via Exim-users wrote on 12.03.2023 19:09:
>>> On 12/03/2023 16:25, Victor Ustugov via Exim-users wrote:
>>>> Is it possible to use SNI with ${readsocket?
>>> No.
>>
>> Do you plan to implement this functionality?
>
> It's not currently on the radar. Glancing round the
> code, it could be implemented with a bit of a hack.
> Choosing a syntax would also be needed.
>
> How badly do you need it?

The cause of the problem that I encountered when updating the OAuth2
access token turned out to be not in the making of HTTPS requests to
non-default virtual hosts. Therefore, the implementation of SNI has
become non-critical for me.

In any case, I would get around the problem by proxying requests through
the nearest nginh.

> Testing is an issue. I think you mentioned building
> a FreeBSD port for yourself;

Yes

> does that mean you
> could take a patch and test that?

Yes, of course.

I've been building exim from the FreeBSD ports tree with my patches for
years. So for testing it is not at all necessary to build a port from
scratch yourself.

This is also true for building exim for rpm and deb based distributions.

I can test the SNI patch on at least FreeBSD, CentOS, Ubuntu.

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 12, 2023, 2:51 PM

Post #13 of 15 (431 views)

Permalink

Victor Ustugov via Exim-users wrote on 12.03.2023 23:32:

>>>>> Is it possible to use SNI with ${readsocket?
>>>> No.
>>>
>>> Do you plan to implement this functionality?
>>
>> It's not currently on the radar. Glancing round the
>> code, it could be implemented with a bit of a hack.
>> Choosing a syntax would also be needed.
>>
>> How badly do you need it?
>
> The cause of the problem that I encountered when updating the OAuth2
> access token turned out to be not in the making of HTTPS requests to
> non-default virtual hosts. Therefore, the implementation of SNI has
> become non-critical for me.

Rather, the lack of SNI support does not prevent me from getting
response to access token refresh request. But Exim puts certificate
verification error message into the logs.

> In any case, I would get around the problem by proxying requests through
> the nearest nginh.
>
>> Testing is an issue. I think you mentioned building
>> a FreeBSD port for yourself;
>
> Yes
>
>> does that mean you
>> could take a patch and test that?
>
> Yes, of course.

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 13, 2023, 3:00 PM

Post #14 of 15 (431 views)

Permalink

On 12/03/2023 21:51, Victor Ustugov via Exim-users wrote:
> Rather, the lack of SNI support does not prevent me from getting
> response to access token refresh request. But Exim puts certificate
> verification error message into the logs.

Having found a way of doing basic functionality testing
of it, pushed 6fdf76d0eae4.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: expansion error in OAuth2 client authenticator [ In reply to ]

exim-users at exim

Mar 13, 2023, 4:39 PM

Post #15 of 15 (431 views)

Permalink

Jeremy Harris via Exim-users wrote on 14.03.2023 00:00:
> On 12/03/2023 21:51, Victor Ustugov via Exim-users wrote:
>> Rather, the lack of SNI support does not prevent me from getting
>> response to access token refresh request. But Exim puts certificate
>> verification error message into the logs.
>
> Having found a way of doing basic functionality testing
> of it, pushed 6fdf76d0eae4.

Great.

FreeBSD 13.1, exim 4.96 without patch:

# exim -be '${readsocket{inet:oauth2.googleapis.com:443}{GET /
HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nConnection:
close\r\n\r\n}{20s:tls=yes}{\n}{socket failure}}' 2>&1 | perl -n -e
'print $_ if (1.../^\r?\n$/)'
2023-03-14 01:33:58 [14476] [NULL] SSL verify error: depth=0 error=self
signed certificate cert=/OU=No SNI provided; please fix your
client./CN=invalid2.invalid
2023-03-14 01:33:58 [14476] [NULL] SSL verify error: certificate name
mismatch: DN="/OU=No SNI provided; please fix your
client./CN=invalid2.invalid" H="oauth2.googleapis.com"
HTTP/1.1 404 Not Found
Date: Mon, 13 Mar 2023 23:33:58 GMT
Content-Type: text/html; charset=UTF-8
Server: ESF
Content-Length: 1561
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Connection: close

FreeBSD 13.1, exim 4.96 with patch:

# exim -be '${readsocket{inet:oauth2.googleapis.com:443}{GET /
HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nConnection:
close\r\n\r\n}{20s:tls=yes:sni=oauth2.googleapis.com}{\n}{socket
failure}}' 2>&1 | perl -n -e 'print $_ if (1.../^\r?\n$/)'
HTTP/1.1 404 Not Found
Date: Mon, 13 Mar 2023 23:34:06 GMT
Content-Type: text/html; charset=UTF-8
Server: ESF
Content-Length: 1561
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Connection: close

Thanks a lot.

Apparently there is no need to check the patch for CentOS and Ubuntu.

--
Best wishes Victor Ustugov
mailto:victor@corvax.kiev.ua
public GnuPG/PGP key: https://victor.corvax.kiev.ua/corvax.asc

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/