Mailing List Archive

https://bugs.exim.org/show_bug.cgi?id=3037

--- Comment #2 from Zakaria <hi@zakaria.website> ---
Hi Jeremy,

Thanks for getting back to me.

--Please don't tell me
--it hacks on Exim's spool file; this is not a public interface for Exim
--and you should not do that. A transport filter should operate
stdin=to-stdout
--as a stream processor.

I'm afraid to tell you, with all due trying my bestest to abide by the rules
and reading this notice https://github.com/Exim/exim/wiki/Q1601, this is just
what I'm doing. Since it's only my personal email server and testing on some
features but if there is any official mail server which requires some legal
notices and warning, please rest assured I'm going to address it all lawfully.

Yet, I couldn't access message body nor headers from bash script? and I had to
fetch it from spool files while I don't edit spool files at all but I found
calling echo with concatenated headers and body does the job, as transport
filter, takes this and replace it with existing one.

I read the documentation, here
https://www.exim.org/exim-html-current/doc/html/spec_html/ch-generic_options_for_transports.html,
it claimed message body and headers it's present somewhere in the stdin but
couldn't figure out which argument? would it be passed to modifyemail bash
script and under which name?

Also, just to be accurate, it seems as you stated true has nothing to do with
ARG_MAX, but I noticed any small email sent without attachment and pass through
my modifyemail.sh in transport filter, works just fine and doesn't throw broken
pipe?

Any feedback would be very much appreciated.

Looking forward, with thanks.

--
You are receiving this mail because:
You are on the CC list for the bug.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-dev.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-dev-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Tue, 10 Oct 2023, Exim Bugzilla via Exim-dev wrote:

> https://bugs.exim.org/show_bug.cgi?id=3037
>
> --- Comment #2 from Zakaria <hi@zakaria.website> ---
> Hi Jeremy,
>
> Thanks for getting back to me.
>
> --Please don't tell me
> --it hacks on Exim's spool file; this is not a public interface for Exim
> --and you should not do that. A transport filter should operate
> stdin=to-stdout
> --as a stream processor.
>
> I'm afraid to tell you, with all due trying my bestest to abide by the rules
> and reading this notice https://github.com/Exim/exim/wiki/Q1601, this is just
> what I'm doing. Since it's only my personal email server and testing on some
> features but if there is any official mail server which requires some legal
> notices and warning, please rest assured I'm going to address it all lawfully.
>
> Yet, I couldn't access message body nor headers from bash script? and I had to
> fetch it from spool files while I don't edit spool files at all but I found
> calling echo with concatenated headers and body does the job, as transport
> filter, takes this and replace it with existing one.

Ah, so you script is asking echo to print the whole of the spool files ?
That mean that the argument(s) to echo will be exceeding ARG_MAX.

If you can freeze a large message in the mail queue and then from the command line
run
/usr/exim/modifyemail.sh message-id
(with the correct message-id, not those letters) I believe you will see the same error.

> I read the documentation, here
> https://www.exim.org/exim-html-current/doc/html/spec_html/ch-generic_options_for_transports.html,
> it claimed message body and headers it's present somewhere in the stdin but
> couldn't figure out which argument? would it be passed to modifyemail bash
> script and under which name?

The message header and body can be read from /dev/stdin, or in bash with
the 'read' command amongst other ways.
They are not available sseperately.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-dev.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-dev-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

https://bugs.exim.org/show_bug.cgi?id=3037

--- Comment #3 from Zakaria <hi@zakaria.website> ---
Update,

I spent the last many days trying to find out the cause of the broken pipe and
whether it's a bug or just transport filter doesn't work the way I want it to
work, and here are my findings and in short I started to believe it's a bug in
transport filter whatever cat or echo I use, if the number of lines exceeds
certain limit beyond 46KB total size of the email content I echo from bash
script to transport filter matching the original content, i.e. this time I
don't modify the email at all, including headers and body and just going
through transport filter. Also, I noted base46 attachment are more likely to
yield broken pipe in comparison to inline attachment given inline uses one line
and won't exceed the lines limit yet throws DKIM long line limit and I think
it's worth investigating this bug further for EXIM dev purposes and Also, I
would appreciate it very much, as for now I have to disable my transport filter
if email size exceeds 46KB.

In details, when I send email with 46KB attachment, it works fine and beyond
this it fails, and throws either broken pipe or DKIM: signing failed:
LONG_LINE, which requires disabling dkim strict signing mode.

Also, I noticed there are two ways of attaching file to email, either using
base64 attachment disposition, or using octet-stream inline disposition.

When I use inline disposition, which can be both reproduced using Apple Mail
client, through attaching an SVG file for inline disposition or photo for
base64 attachment from keyboard.

All this never happens when I remove transport filter from smtp_transport and I
can still send all these emails without any problem whatsoever.

Looking forward, with thanks.

Zakaria.

--
You are receiving this mail because:
You are on the CC list for the bug.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-dev.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-dev-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

https://bugs.exim.org/show_bug.cgi?id=3037

--- Comment #4 from Zakaria <hi@zakaria.website> ---
Oh, more points, I think they could be helpful for anyone else who wants to
find resolve to this issue.

As Jeremy suggested, running cat in transport filter like the following:-

transport_filter = /bin/bash -c '/bin/cat'

renders no point for me of using transport filter since it doesn't modify nor
offer me any ability to filter anything, and yet it works fine and the broken
pipe issue doesn't happen but if I run it from inside bash script with certain
or identical email content, it throws broken pipe error.

Lastly, it seems EXIM takes limited segment from the cat or echo content or
it's not just the first call to cat if it was outputted from bash script since
I tested calling cat only once for concatenated content, and anyways ends with
throwing broken pipe? other ways to mitigate this?

Also, in case anyone wants to review my bash script, this is it's content, and
not sure if I can run it from inside EXIM configuration file directly given I
use several commands and all are being executed when ran from bash as I tested
if it reaches exit 0.

#!/bin/sh

msgid=$1
bpath=$(/usr/bin/find /var/spool/exim/input -name "*$msgid*-D")
hpath=$(/usr/bin/find /var/spool/exim/input -name "*$msgid*-H")
result=$(/usr/prependpng.sh "$bpath" "$hpath")
if [[ $result != "isntchunked" ]]; then
END=$result
for ((i=1;i<=END;i++)); do
cat "${bpath}-modified${i}"
cat "${bpath}-modified${i}" >> /etc/exim/prependtest
rm -f "${bpath}-modified${i}"
done
else
cat ${bpath}-modified
#cat "${bpath}-modified" > /etc/exim/prependtest
rm -f "${bpath}-modified"
fi
exit 0


Looking forward, with thanks.

Zakaria.

--
You are receiving this mail because:
You are on the CC list for the bug.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-dev.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-dev-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Thu, 12 Oct 2023, Exim Bugzilla via Exim-dev wrote:

> https://bugs.exim.org/show_bug.cgi?id=3037
>
> --- Comment #4 from Zakaria <hi@zakaria.website> ---
> Oh, more points, I think they could be helpful for anyone else who wants to
> find resolve to this issue.
>
> As Jeremy suggested, running cat in transport filter like the following:-
>
> transport_filter = /bin/bash -c '/bin/cat'
>
> renders no point for me of using transport filter since it doesn't modify nor
> offer me any ability to filter anything, and yet it works fine and the broken
> pipe issue doesn't happen but if I run it from inside bash script with certain
> or identical email content, it throws broken pipe error.
>
> Lastly, it seems EXIM takes limited segment from the cat or echo content or
> it's not just the first call to cat if it was outputted from bash script since
> I tested calling cat only once for concatenated content, and anyways ends with
> throwing broken pipe? other ways to mitigate this?
>
> Also, in case anyone wants to review my bash script, this is it's content, and
> not sure if I can run it from inside EXIM configuration file directly given I
> use several commands and all are being executed when ran from bash as I tested
> if it reaches exit 0.

Can we see /usr/prependpng.sh too please ?

> #!/bin/sh
>
> msgid=$1
> bpath=$(/usr/bin/find /var/spool/exim/input -name "*$msgid*-D")
> hpath=$(/usr/bin/find /var/spool/exim/input -name "*$msgid*-H")
> result=$(/usr/prependpng.sh "$bpath" "$hpath")
> if [[ $result != "isntchunked" ]]; then
> END=$result
> for ((i=1;i<=END;i++)); do
> cat "${bpath}-modified${i}"
> cat "${bpath}-modified${i}" >> /etc/exim/prependtest
> rm -f "${bpath}-modified${i}"
> done
> else
> cat ${bpath}-modified
> #cat "${bpath}-modified" > /etc/exim/prependtest
> rm -f "${bpath}-modified"
> fi
> exit 0
>
>
> Looking forward, with thanks.
>
> Zakaria.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>
> --
> ## subscription configuration (requires account):
> ## https://lists.exim.org/mailman3/postorius/lists/exim-dev.lists.exim.org/
> ## unsubscribe (doesn't require an account):
> ## exim-dev-unsubscribe@lists.exim.org
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-dev.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-dev-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

• Exim Bugzilla via Exim-dev [2023-10-12 18:45]:
> https://bugs.exim.org/show_bug.cgi?id=3037
>
> --- Comment #4 from Zakaria <hi@zakaria.website> ---
> Oh, more points, I think they could be helpful for anyone else who wants to
> find resolve to this issue.
>
> As Jeremy suggested, running cat in transport filter like the following:-
>
> transport_filter = /bin/bash -c '/bin/cat'
>
> renders no point for me of using transport filter since it doesn't modify nor
> offer me any ability to filter anything, and yet it works fine and the broken
> pipe issue doesn't happen but if I run it from inside bash script with certain
> or identical email content, it throws broken pipe error.
>
> Lastly, it seems EXIM takes limited segment from the cat or echo content or
> it's not just the first call to cat if it was outputted from bash script since
> I tested calling cat only once for concatenated content, and anyways ends with
> throwing broken pipe? other ways to mitigate this?
>
> Also, in case anyone wants to review my bash script, this is it's content, and
> not sure if I can run it from inside EXIM configuration file directly given I
> use several commands and all are being executed when ran from bash as I tested
> if it reaches exit 0.

the scripts seems to ignore the input which exim is sending to stdin,
no? could that be the cause for the error?

and even if the script starts handling stdin properly, the pipe
transport filter seems like a wrong approach for me, even it _might_
work: why not do a delivery using a normal pipe transport, and do
necessary transformations there, and then make that pipe transport call
exim again with -oMr option with some value and resend the modified
message this way? -oMr sents $received_protocol, which can be used to
know when input has been "filtered" already.

> #!/bin/sh
>
> msgid=$1
> bpath=$(/usr/bin/find /var/spool/exim/input -name "*$msgid*-D")
> hpath=$(/usr/bin/find /var/spool/exim/input -name "*$msgid*-H")
> result=$(/usr/prependpng.sh "$bpath" "$hpath")
> if [[ $result != "isntchunked" ]]; then
> END=$result
> for ((i=1;i<=END;i++)); do
> cat "${bpath}-modified${i}"
> cat "${bpath}-modified${i}" >> /etc/exim/prependtest
> rm -f "${bpath}-modified${i}"
> done
> else
> cat ${bpath}-modified
> #cat "${bpath}-modified" > /etc/exim/prependtest
> rm -f "${bpath}-modified"
> fi
> exit 0
>
>
> Looking forward, with thanks.
>
> Zakaria.
>

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-dev.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-dev-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/