Mailing List Archive

Hi Andrew,

> You lot are all experts at Catalyst and Perl,

I am no expert in Catalyst or Perl, but I have learned a thing or two
about email in my career.

I have found the best way for people to send butt-loads of emails to
members is to use mailing list software. This generally involves also
running a mail server, so it's not a solution from within catalyst or
perl, but it is almost certainly possible to have a catalyst app manage
subscribing/unsubscribing/verifying users from a mailing list, and then
when it comes time to send a mail to everybody, the app only needs to
send a mail to one list address, and let the mail server do what it does
best: send mail...

Just a suggestion...

> and because I'm using Perl to code a Catalyst program, I wanted to ask
> you about a problem I've encountered.
> The Catalyst app I'm coding is to replace a PHP website, which presently
> allows an admin to manage sending out an email to all members. The site
> has 3000 odd members. From what I could tell, the PHP code was simply
> using the PHP mail function to send email (although it did in my
> experience typically take two days for everyone to receive an email sent
> with it).
> I've also been asking other Perl programmers when I've had the chance -
> and the only concerns raised, were about IP blacklisting or emails
> bouncing back - not about actual code working.
> So could sending 3000 emails be as simple as Matt's old CGI form mail?
> I had a look on CPAN, and found the simple looking Email::Stuffer,
> and wrote a script, sending an email to myself, and it worked.
> I then modified the script to pick five email addresses from a MySQL
> database, and email them, and again - it worked.
> Okay - one final change - throwing 3000 email addresses at it.
> How does that do....?
> Starts to work - then throws up an Internal Server Error in the browser
> window,
> and I find out my Fast CGI Plack Up server thingie isn't up, nor running
> my Catalyst app anymore.
> I have to type plackup and my details, etc, again from the command prompt.
> A log in to Web Host Manager (this is an apache2 VPS with CPanel) and a
> check of the "View Sent Summary" shows 1020 emails sent - 611
> successful, 642 deferrals and 329 failures...curious, as that adds up to
> more than 1020, =S. If I click through for more details, it shows 1582
> records - some with green ticks, and others with amber or red
> exclamation marks, ^_^.
> I can worry about IPs and failure rates later....
> ....I just want to know why the whole FastCGI Plack loaded Catalyst app
> came down half way through,
> and how I can have a Perl script process sending 3000 emails without
> that happening every time, =S.
> Do I need to use a queue within Perl? Or does the postfix/sendmail on
> the server, automatically queue everything for me? If so - why did my
> code crash the Catalyst app, and not send all the emails? >_<.
> I'm guessing this is a common problem - how does a website email all its
> signed up members - and so there must surely be a common solution
> among Perl / Catalyst users....?
> Any help, appreciated!
> Yours,
> Andrew.
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

A queue in Perl is better when there is a need of sending thousand messages.
Without a queue, if the Catalyst-based code just sends the messages directly and a browser is waiting for a page to load after the web app sent them, it may time-out. But otherwise it should work and not crash the web app.
3000 messages should be sent pretty fast if they are sent to the local SMTP server, but not fast enough for a pleasant experience.

Regarding the success of those messages... if you don't want them to reach in the spam folder, it helps to sign them by using SPF/DKIM. If you'll find this complicated, a better solution might be to create an account on a site like mailgun.com or mandrill.com and send the messages using their servers. They offer APIs that you can use in your app very easy. mailgun.com allows sending 10,000 messages/month for free and after that limit the prices are pretty cheap. You will have reports with the list of email messages that bounced that you can get programaticly, plus a few other helpful features.

--Octavian

----- Original Message -----
From: Andrew
To: The elegant MVC web framework
Sent: Thursday, November 26, 2015 5:32 PM
Subject: [Catalyst] Sending 3000 emails.


Hello,

You lot are all experts at Catalyst and Perl,
and because I'm using Perl to code a Catalyst program, I wanted to ask you about a problem I've encountered.

The Catalyst app I'm coding is to replace a PHP website, which presently allows an admin to manage sending out an email to all members. The site has 3000 odd members. From what I could tell, the PHP code was simply using the PHP mail function to send email (although it did in my experience typically take two days for everyone to receive an email sent with it).
I've also been asking other Perl programmers when I've had the chance - and the only concerns raised, were about IP blacklisting or emails bouncing back - not about actual code working.

So could sending 3000 emails be as simple as Matt's old CGI form mail?

I had a look on CPAN, and found the simple looking Email::Stuffer,
and wrote a script, sending an email to myself, and it worked.
I then modified the script to pick five email addresses from a MySQL database, and email them, and again - it worked.
Okay - one final change - throwing 3000 email addresses at it.
How does that do....?
Starts to work - then throws up an Internal Server Error in the browser window,
and I find out my Fast CGI Plack Up server thingie isn't up, nor running my Catalyst app anymore.
I have to type plackup and my details, etc, again from the command prompt.
A log in to Web Host Manager (this is an apache2 VPS with CPanel) and a check of the "View Sent Summary" shows 1020 emails sent - 611 successful, 642 deferrals and 329 failures...curious, as that adds up to more than 1020, =S. If I click through for more details, it shows 1582 records - some with green ticks, and others with amber or red exclamation marks, ^_^.
I can worry about IPs and failure rates later....

....I just want to know why the whole FastCGI Plack loaded Catalyst app came down half way through,
and how I can have a Perl script process sending 3000 emails without that happening every time, =S.

Do I need to use a queue within Perl? Or does the postfix/sendmail on the server, automatically queue everything for me? If so - why did my code crash the Catalyst app, and not send all the emails? >_<.

I'm guessing this is a common problem - how does a website email all its signed up members - and so there must surely be a common solution among Perl / Catalyst users....?

Any help, appreciated!

Yours,
Andrew.



------------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Maybe not the answer you're looking for but I'd go with an email service
such as sendgrid which takes most if not all of the pain out of sending
emails. They have a simple REST API you can POST your email to and provides
callbacks for delivery notifications etc as well...

I'm not advocating sendgrid over any other service (I'm sure there are
plenty others) - it's just the one I have experience with... They
apparently have a free tier as well....

/L

On Thu, Nov 26, 2015 at 5:53 PM, Octavian Rasnita <orasnita@gmail.com>
wrote:

> A queue in Perl is better when there is a need of sending thousand
> messages.
> Without a queue, if the Catalyst-based code just sends the messages
> directly and a browser is waiting for a page to load after the web app sent
> them, it may time-out. But otherwise it should work and not crash the web
> app.
> 3000 messages should be sent pretty fast if they are sent to the local
> SMTP server, but not fast enough for a pleasant experience.
>
> Regarding the success of those messages... if you don't want them to reach
> in the spam folder, it helps to sign them by using SPF/DKIM. If you'll find
> this complicated, a better solution might be to create an account on a
> site like mailgun.com or mandrill.com and send the messages using their
> servers. They offer APIs that you can use in your app very easy.
> mailgun.com allows sending 10,000 messages/month for free and after that
> limit the prices are pretty cheap. You will have reports with the list of
> email messages that bounced that you can get programaticly, plus a few
> other helpful features.
>
> --Octavian
>
> ----- Original Message -----
> *From:* Andrew <catalystgroup@unitedgames.co.uk>
> *To:* The elegant MVC web framework <catalyst@lists.scsys.co.uk>
> *Sent:* Thursday, November 26, 2015 5:32 PM
> *Subject:* [Catalyst] Sending 3000 emails.
>
> Hello,
>
> You lot are all experts at Catalyst and Perl,
> and because I'm using Perl to code a Catalyst program, I wanted to ask you
> about a problem I've encountered.
>
> The Catalyst app I'm coding is to replace a PHP website, which presently
> allows an admin to manage sending out an email to all members. The site has
> 3000 odd members. From what I could tell, the PHP code was simply using the
> PHP mail function to send email (although it did in my experience typically
> take two days for everyone to receive an email sent with it).
> I've also been asking other Perl programmers when I've had the chance -
> and the only concerns raised, were about IP blacklisting or emails bouncing
> back - not about actual code working.
>
> So could sending 3000 emails be as simple as Matt's old CGI form mail?
>
> I had a look on CPAN, and found the simple looking Email::Stuffer,
> and wrote a script, sending an email to myself, and it worked.
> I then modified the script to pick five email addresses from a MySQL
> database, and email them, and again - it worked.
> Okay - one final change - throwing 3000 email addresses at it.
> How does that do....?
> Starts to work - then throws up an Internal Server Error in the browser
> window,
> and I find out my Fast CGI Plack Up server thingie isn't up, nor running
> my Catalyst app anymore.
> I have to type plackup and my details, etc, again from the command prompt.
> A log in to Web Host Manager (this is an apache2 VPS with CPanel) and a
> check of the "View Sent Summary" shows 1020 emails sent - 611 successful,
> 642 deferrals and 329 failures...curious, as that adds up to more than
> 1020, =S. If I click through for more details, it shows 1582 records - some
> with green ticks, and others with amber or red exclamation marks, ^_^.
> I can worry about IPs and failure rates later....
>
> ....I just want to know why the whole FastCGI Plack loaded Catalyst app
> came down half way through,
> and how I can have a Perl script process sending 3000 emails without that
> happening every time, =S.
>
> Do I need to use a queue within Perl? Or does the postfix/sendmail on the
> server, automatically queue everything for me? If so - why did my code
> crash the Catalyst app, and not send all the emails? >_<.
>
> I'm guessing this is a common problem - how does a website email all its
> signed up members - and so there must surely be a common solution
> among Perl / Catalyst users....?
>
> Any help, appreciated!
>
> Yours,
> Andrew.
>
>
> ------------------------------
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>

I'll add that you should look into a transactional email partner like
sendgrid, mailgun, etc. They'll help you stay out of the black lists by
showing you what not to do, ans how to set up DKIM, etc. for authentication
of your email.


I use sendgrid professionally, with one of my cusotmers that sends a lot of
email.

If your volume is under 10k emails per month, sendgrid is free, and they
have a templating feature which allows you to upload a template
(parameterizable) and let them to the mail merge on their servers, for up
to one thousand addresses per tx.

So instead of generating 3000 emails and pushing them through your smtp
server, you'd upload the template file, and push three 1000-name lists to
to sendgrid and let them take care of delivery. Something to think about.

Len.


On Thu, Nov 26, 2015 at 12:38 PM, Lasse Makholm <lasse@unity3d.com> wrote:

> Maybe not the answer you're looking for but I'd go with an email service
> such as sendgrid which takes most if not all of the pain out of sending
> emails. They have a simple REST API you can POST your email to and provides
> callbacks for delivery notifications etc as well...
>
> I'm not advocating sendgrid over any other service (I'm sure there are
> plenty others) - it's just the one I have experience with... They
> apparently have a free tier as well....
>
> /L
>
>
> On Thu, Nov 26, 2015 at 5:53 PM, Octavian Rasnita <orasnita@gmail.com>
> wrote:
>
>> A queue in Perl is better when there is a need of sending thousand
>> messages.
>> Without a queue, if the Catalyst-based code just sends the messages
>> directly and a browser is waiting for a page to load after the web app sent
>> them, it may time-out. But otherwise it should work and not crash the web
>> app.
>> 3000 messages should be sent pretty fast if they are sent to the local
>> SMTP server, but not fast enough for a pleasant experience.
>>
>> Regarding the success of those messages... if you don't want them to
>> reach in the spam folder, it helps to sign them by using SPF/DKIM. If
>> you'll find this complicated, a better solution might be to create an
>> account on a site like mailgun.com or mandrill.com and send the messages
>> using their servers. They offer APIs that you can use in your app very
>> easy. mailgun.com allows sending 10,000 messages/month for free and
>> after that limit the prices are pretty cheap. You will have reports with
>> the list of email messages that bounced that you can get programaticly,
>> plus a few other helpful features.
>>
>> --Octavian
>>
>> ----- Original Message -----
>> *From:* Andrew <catalystgroup@unitedgames.co.uk>
>> *To:* The elegant MVC web framework <catalyst@lists.scsys.co.uk>
>> *Sent:* Thursday, November 26, 2015 5:32 PM
>> *Subject:* [Catalyst] Sending 3000 emails.
>>
>> Hello,
>>
>> You lot are all experts at Catalyst and Perl,
>> and because I'm using Perl to code a Catalyst program, I wanted to ask
>> you about a problem I've encountered.
>>
>> The Catalyst app I'm coding is to replace a PHP website, which presently
>> allows an admin to manage sending out an email to all members. The site has
>> 3000 odd members. From what I could tell, the PHP code was simply using the
>> PHP mail function to send email (although it did in my experience typically
>> take two days for everyone to receive an email sent with it).
>> I've also been asking other Perl programmers when I've had the chance -
>> and the only concerns raised, were about IP blacklisting or emails bouncing
>> back - not about actual code working.
>>
>> So could sending 3000 emails be as simple as Matt's old CGI form mail?
>>
>> I had a look on CPAN, and found the simple looking Email::Stuffer,
>> and wrote a script, sending an email to myself, and it worked.
>> I then modified the script to pick five email addresses from a MySQL
>> database, and email them, and again - it worked.
>> Okay - one final change - throwing 3000 email addresses at it.
>> How does that do....?
>> Starts to work - then throws up an Internal Server Error in the browser
>> window,
>> and I find out my Fast CGI Plack Up server thingie isn't up, nor running
>> my Catalyst app anymore.
>> I have to type plackup and my details, etc, again from the command prompt.
>> A log in to Web Host Manager (this is an apache2 VPS with CPanel) and a
>> check of the "View Sent Summary" shows 1020 emails sent - 611 successful,
>> 642 deferrals and 329 failures...curious, as that adds up to more than
>> 1020, =S. If I click through for more details, it shows 1582 records - some
>> with green ticks, and others with amber or red exclamation marks, ^_^.
>> I can worry about IPs and failure rates later....
>>
>> ....I just want to know why the whole FastCGI Plack loaded Catalyst app
>> came down half way through,
>> and how I can have a Perl script process sending 3000 emails without that
>> happening every time, =S.
>>
>> Do I need to use a queue within Perl? Or does the postfix/sendmail on the
>> server, automatically queue everything for me? If so - why did my code
>> crash the Catalyst app, and not send all the emails? >_<.
>>
>> I'm guessing this is a common problem - how does a website email all its
>> signed up members - and so there must surely be a common solution
>> among Perl / Catalyst users....?
>>
>> Any help, appreciated!
>>
>> Yours,
>> Andrew.
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
>> _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>


--
Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
www.lenjaffe.com
Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
@CodeJamCMH <https://www.twitter.com/CodeJamCMH>
Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
Aggregation of Online Advent Calendars.

I also read on the sendgrid website that:

"With SMTP, 100 messages can be sent with each connection"
and
"Customers should utilize SMTPAPI if this is an option. As with SMTP, 100 messages can be sent with each connection, but there can be 1000 recipients for each message."

...given that my attempt to send 3000 emails with nothing but my own VPS server, and Cpan's Email::Stuffer, only sent around 1000, before the plack/fastcgi/catalyst app crashed.... could I have been running into a similar limitation? I'm now thinking of altering my code to split the attempts to send email into six batches of 500 emails, as a short term fix, until I've had time to research which transactional email partner to use, and how their APIs work, or integrate with Catalyst apps.

----- Original Message -----
From: Len Jaffe
To: The elegant MVC web framework
Sent: Sunday, November 29, 2015 1:31 AM
Subject: Re: [Catalyst] Sending 3000 emails.


I'll add that you should look into a transactional email partner like sendgrid, mailgun, etc. They'll help you stay out of the black lists by showing you what not to do, ans how to set up DKIM, etc. for authentication of your email.




I use sendgrid professionally, with one of my cusotmers that sends a lot of email.


If your volume is under 10k emails per month, sendgrid is free, and they have a templating feature which allows you to upload a template (parameterizable) and let them to the mail merge on their servers, for up to one thousand addresses per tx.


So instead of generating 3000 emails and pushing them through your smtp server, you'd upload the template file, and push three 1000-name lists to to sendgrid and let them take care of delivery. Something to think about.


Len.




On Thu, Nov 26, 2015 at 12:38 PM, Lasse Makholm <lasse@unity3d.com> wrote:

Maybe not the answer you're looking for but I'd go with an email service such as sendgrid which takes most if not all of the pain out of sending emails. They have a simple REST API you can POST your email to and provides callbacks for delivery notifications etc as well...


I'm not advocating sendgrid over any other service (I'm sure there are plenty others) - it's just the one I have experience with... They apparently have a free tier as well....



/L




On Thu, Nov 26, 2015 at 5:53 PM, Octavian Rasnita <orasnita@gmail.com> wrote:

A queue in Perl is better when there is a need of sending thousand messages.
Without a queue, if the Catalyst-based code just sends the messages directly and a browser is waiting for a page to load after the web app sent them, it may time-out. But otherwise it should work and not crash the web app.
3000 messages should be sent pretty fast if they are sent to the local SMTP server, but not fast enough for a pleasant experience.

Regarding the success of those messages... if you don't want them to reach in the spam folder, it helps to sign them by using SPF/DKIM. If you'll find this complicated, a better solution might be to create an account on a site like mailgun.com or mandrill.com and send the messages using their servers. They offer APIs that you can use in your app very easy. mailgun.com allows sending 10,000 messages/month for free and after that limit the prices are pretty cheap. You will have reports with the list of email messages that bounced that you can get programaticly, plus a few other helpful features.

--Octavian

----- Original Message -----
From: Andrew
To: The elegant MVC web framework
Sent: Thursday, November 26, 2015 5:32 PM
Subject: [Catalyst] Sending 3000 emails.


Hello,

You lot are all experts at Catalyst and Perl,
and because I'm using Perl to code a Catalyst program, I wanted to ask you about a problem I've encountered.

The Catalyst app I'm coding is to replace a PHP website, which presently allows an admin to manage sending out an email to all members. The site has 3000 odd members. From what I could tell, the PHP code was simply using the PHP mail function to send email (although it did in my experience typically take two days for everyone to receive an email sent with it).
I've also been asking other Perl programmers when I've had the chance - and the only concerns raised, were about IP blacklisting or emails bouncing back - not about actual code working.

So could sending 3000 emails be as simple as Matt's old CGI form mail?

I had a look on CPAN, and found the simple looking Email::Stuffer,
and wrote a script, sending an email to myself, and it worked.
I then modified the script to pick five email addresses from a MySQL database, and email them, and again - it worked.
Okay - one final change - throwing 3000 email addresses at it.
How does that do....?
Starts to work - then throws up an Internal Server Error in the browser window,
and I find out my Fast CGI Plack Up server thingie isn't up, nor running my Catalyst app anymore.
I have to type plackup and my details, etc, again from the command prompt.
A log in to Web Host Manager (this is an apache2 VPS with CPanel) and a check of the "View Sent Summary" shows 1020 emails sent - 611 successful, 642 deferrals and 329 failures...curious, as that adds up to more than 1020, =S. If I click through for more details, it shows 1582 records - some with green ticks, and others with amber or red exclamation marks, ^_^.
I can worry about IPs and failure rates later....

....I just want to know why the whole FastCGI Plack loaded Catalyst app came down half way through,
and how I can have a Perl script process sending 3000 emails without that happening every time, =S.

Do I need to use a queue within Perl? Or does the postfix/sendmail on the server, automatically queue everything for me? If so - why did my code crash the Catalyst app, and not send all the emails? >_<.

I'm guessing this is a common problem - how does a website email all its signed up members - and so there must surely be a common solution among Perl / Catalyst users....?

Any help, appreciated!

Yours,
Andrew.



------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/



_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/





_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/







--

Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe www.lenjaffe.com

Host of Code Jam Columbus - @CodeJamCMH
Curator of Advent Planet - An Aggregation of Online Advent Calendars.




------------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

On Mon, Nov 30, 2015 at 12:29 PM, Andrew <catalystgroup@unitedgames.co.uk>
wrote:

> I also read on the sendgrid website that:
>
> "With SMTP, 100 messages can be sent with each connection"
> and
> "Customers should utilize SMTPAPI if this is an option. As with SMTP, 100
> messages can be sent with each connection, but there can be 1000 recipients
> for each message."
>
> ...given that my attempt to send 3000 emails with nothing but my own VPS
> server, and Cpan's Email::Stuffer, only sent around 1000, before the
> plack/fastcgi/catalyst app crashed.... could I have been running into a
> similar limitation? I'm now thinking of altering my code to split the
> attempts to send email into six batches of 500 emails, as a short term fix,
> until I've had time to research which transactional email partner to use,
> and how their APIs work, or integrate with Catalyst apps.
>
>

That really depends on whether you're sending directly form your Perl
processes (having the perl process connect tot he remote SMTP server, or
queuing the messages locally for delivery by a local SMTP demon. In my
case, my code hands them to local SMTP, and the local SMPT delivers them
upstream (called smarthosting) to sendgrid for delivery to the far corners
to the net.





> ----- Original Message -----
> *From:* Len Jaffe <lenjaffe@jaffesystems.com>
> *To:* The elegant MVC web framework <catalyst@lists.scsys.co.uk>
> *Sent:* Sunday, November 29, 2015 1:31 AM
> *Subject:* Re: [Catalyst] Sending 3000 emails.
>
> I'll add that you should look into a transactional email partner like
> sendgrid, mailgun, etc. They'll help you stay out of the black lists by
> showing you what not to do, ans how to set up DKIM, etc. for authentication
> of your email.
>
>
> I use sendgrid professionally, with one of my cusotmers that sends a lot
> of email.
>
> If your volume is under 10k emails per month, sendgrid is free, and they
> have a templating feature which allows you to upload a template
> (parameterizable) and let them to the mail merge on their servers, for up
> to one thousand addresses per tx.
>
> So instead of generating 3000 emails and pushing them through your smtp
> server, you'd upload the template file, and push three 1000-name lists to
> to sendgrid and let them take care of delivery. Something to think about.
>
> Len.
>
>
> On Thu, Nov 26, 2015 at 12:38 PM, Lasse Makholm <lasse@unity3d.com> wrote:
>
>> Maybe not the answer you're looking for but I'd go with an email service
>> such as sendgrid which takes most if not all of the pain out of sending
>> emails. They have a simple REST API you can POST your email to and provides
>> callbacks for delivery notifications etc as well...
>>
>> I'm not advocating sendgrid over any other service (I'm sure there are
>> plenty others) - it's just the one I have experience with... They
>> apparently have a free tier as well....
>>
>> /L
>>
>>
>> On Thu, Nov 26, 2015 at 5:53 PM, Octavian Rasnita <orasnita@gmail.com>
>> wrote:
>>
>>> A queue in Perl is better when there is a need of sending thousand
>>> messages.
>>> Without a queue, if the Catalyst-based code just sends the messages
>>> directly and a browser is waiting for a page to load after the web app sent
>>> them, it may time-out. But otherwise it should work and not crash the web
>>> app.
>>> 3000 messages should be sent pretty fast if they are sent to the local
>>> SMTP server, but not fast enough for a pleasant experience.
>>>
>>> Regarding the success of those messages... if you don't want them to
>>> reach in the spam folder, it helps to sign them by using SPF/DKIM. If
>>> you'll find this complicated, a better solution might be to create an
>>> account on a site like mailgun.com or mandrill.com and send the
>>> messages using their servers. They offer APIs that you can use in your app
>>> very easy. mailgun.com allows sending 10,000 messages/month for free
>>> and after that limit the prices are pretty cheap. You will have reports
>>> with the list of email messages that bounced that you can get
>>> programaticly, plus a few other helpful features.
>>>
>>> --Octavian
>>>
>>> ----- Original Message -----
>>> *From:* Andrew <catalystgroup@unitedgames.co.uk>
>>> *To:* The elegant MVC web framework <catalyst@lists.scsys.co.uk>
>>> *Sent:* Thursday, November 26, 2015 5:32 PM
>>> *Subject:* [Catalyst] Sending 3000 emails.
>>>
>>> Hello,
>>>
>>> You lot are all experts at Catalyst and Perl,
>>> and because I'm using Perl to code a Catalyst program, I wanted to ask
>>> you about a problem I've encountered.
>>>
>>> The Catalyst app I'm coding is to replace a PHP website, which presently
>>> allows an admin to manage sending out an email to all members. The site has
>>> 3000 odd members. From what I could tell, the PHP code was simply using the
>>> PHP mail function to send email (although it did in my experience typically
>>> take two days for everyone to receive an email sent with it).
>>> I've also been asking other Perl programmers when I've had the chance -
>>> and the only concerns raised, were about IP blacklisting or emails bouncing
>>> back - not about actual code working.
>>>
>>> So could sending 3000 emails be as simple as Matt's old CGI form mail?
>>>
>>> I had a look on CPAN, and found the simple looking Email::Stuffer,
>>> and wrote a script, sending an email to myself, and it worked.
>>> I then modified the script to pick five email addresses from a MySQL
>>> database, and email them, and again - it worked.
>>> Okay - one final change - throwing 3000 email addresses at it.
>>> How does that do....?
>>> Starts to work - then throws up an Internal Server Error in the browser
>>> window,
>>> and I find out my Fast CGI Plack Up server thingie isn't up, nor running
>>> my Catalyst app anymore.
>>> I have to type plackup and my details, etc, again from the command
>>> prompt.
>>> A log in to Web Host Manager (this is an apache2 VPS with CPanel) and a
>>> check of the "View Sent Summary" shows 1020 emails sent - 611 successful,
>>> 642 deferrals and 329 failures...curious, as that adds up to more than
>>> 1020, =S. If I click through for more details, it shows 1582 records - some
>>> with green ticks, and others with amber or red exclamation marks, ^_^.
>>> I can worry about IPs and failure rates later....
>>>
>>> ....I just want to know why the whole FastCGI Plack loaded Catalyst app
>>> came down half way through,
>>> and how I can have a Perl script process sending 3000 emails without
>>> that happening every time, =S.
>>>
>>> Do I need to use a queue within Perl? Or does the postfix/sendmail on
>>> the server, automatically queue everything for me? If so - why did my code
>>> crash the Catalyst app, and not send all the emails? >_<.
>>>
>>> I'm guessing this is a common problem - how does a website email all its
>>> signed up members - and so there must surely be a common solution
>>> among Perl / Catalyst users....?
>>>
>>> Any help, appreciated!
>>>
>>> Yours,
>>> Andrew.
>>>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> List: Catalyst@lists.scsys.co.uk
>>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>>> Searchable archive:
>>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>>> Dev site: http://dev.catalyst.perl.org/
>>>
>>>
>>> _______________________________________________
>>> List: Catalyst@lists.scsys.co.uk
>>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>>> Searchable archive:
>>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>>> Dev site: http://dev.catalyst.perl.org/
>>>
>>>
>>
>> _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
>
>
> --
> Len Jaffe - Information Technology Smoke Jumper -
> lenjaffe@jaffesystems.com
> 614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
> www.lenjaffe.com
> Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
> @CodeJamCMH <https://www.twitter.com/CodeJamCMH>
> Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
> Aggregation of Online Advent Calendars.
>
> ------------------------------
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>


--
Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
www.lenjaffe.com
Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
@CodeJamCMH <https://www.twitter.com/CodeJamCMH>
Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
Aggregation of Online Advent Calendars.

On 2015-11-30 9:29 AM, Andrew wrote:
> ...given that my attempt to send 3000 emails with nothing but my own VPS server,
> and Cpan's Email::Stuffer, only sent around 1000, before the
> plack/fastcgi/catalyst app crashed.... could I have been running into a similar
> limitation? I'm now thinking of altering my code to split the attempts to send
> email into six batches of 500 emails, as a short term fix, until I've had time
> to research which transactional email partner to use, and how their APIs work,
> or integrate with Catalyst apps.

If you are trying to send the emails directly from the process serving your web
API, then this is absolutely the wrong approach, and you should give up
immediately on trying to do that. Your API process should just queue the
messages to be sent by a separate background process, and then return
immediately such as with a message "3000 messages queued to be sent soon, click
here to check queue progress" or such. Sending in-process may be reasonable for
a single email, eg email a copy of the form the user just submitted, but that's
it. -- Darren Duncan


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Hi Len and Darren,
Atm I'm using Email::Stuffer to send email from within a perl script, in the
Controller folder of my Catalyst app.
I'm assuming this results in it being queued to be sent by the local SMTP
server.
My VPS server has Cpanel, and uses Exim, which I'm guessing is what the
scripts are using to send the email.

As noted, it all works until about 1020 emails in, at which point the
Catalyst app crashes (although this has only happened once - I do not wish
to try again and risk another crash). I am using Email::Stuffer in a for
loop, iterating over an array of 3000 email addresses. For my next step I
will alter my code to only attempt to send to 500 email addresses at any one
time, since below 1000 everything seems to work fine.

You seem to be mentioning using a background process, Darren. Is Exim
already the background process that sends email?

If the long term solution is to use mailgun or sendgrid - before I delve
into all their documentation - do you lot have any tips I should consider,
on how to integrate them with a Catalyst app, or is it all pretty straight
forward once I read the documentation?





----- Original Message -----
From: "Darren Duncan" <darren@darrenduncan.net>
To: "The elegant MVC web framework" <catalyst@lists.scsys.co.uk>
Sent: Monday, November 30, 2015 11:41 PM
Subject: Re: [Catalyst] Sending 3000 emails.


On 2015-11-30 9:29 AM, Andrew wrote:
> ...given that my attempt to send 3000 emails with nothing but my own VPS
server,
> and Cpan's Email::Stuffer, only sent around 1000, before the
> plack/fastcgi/catalyst app crashed.... could I have been running into a
similar
> limitation? I'm now thinking of altering my code to split the attempts to
send
> email into six batches of 500 emails, as a short term fix, until I've had
time
> to research which transactional email partner to use, and how their APIs
work,
> or integrate with Catalyst apps.

If you are trying to send the emails directly from the process serving your
web
API, then this is absolutely the wrong approach, and you should give up
immediately on trying to do that. Your API process should just queue the
messages to be sent by a separate background process, and then return
immediately such as with a message "3000 messages queued to be sent soon,
click
here to check queue progress" or such. Sending in-process may be reasonable
for
a single email, eg email a copy of the form the user just submitted, but
that's
it. -- Darren Duncan


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

On 12/2/2015 10:10 AM, Andrew wrote:
> As noted, it all works until about 1020 emails in, at which point the
> Catalyst app crashes (although this has only happened once - I do not wish
> to try again and risk another crash). I am using Email::Stuffer in a for
> loop, iterating over an array of 3000 email addresses. For my next step I
> will alter my code to only attempt to send to 500 email addresses at any one
> time, since below 1000 everything seems to work fine.

Are you sending those emails as 3,000 individual messages? Are you
keeping the Email::Stuffer objects around after doing ->send on them?
With your Catalyst app crashing I wonder if you aren't hitting a memory
resource limit or memory leak bug.

--[Lance]


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

On Wed, Dec 2, 2015 at 10:10 AM, Andrew <catalystgroup@unitedgames.co.uk>
wrote:

> Hi Len and Darren,
> Atm I'm using Email::Stuffer to send email from within a perl script, in
> the
> Controller folder of my Catalyst app.
> I'm assuming this results in it being queued to be sent by the local SMTP
> server.
> My VPS server has Cpanel, and uses Exim, which I'm guessing is what the
> scripts are using to send the email.
>

According to the docs
http://search.cpan.org/~rjbs/Email-Stuffer-0.012/lib/Email/Stuffer.pm
unless you explicitly set the transport, Email::Stuffer defers to
Email::Sender for the default transport,
which appears to be sendmail, which means you'll be handing it off locally
for delivery.


As noted, it all works until about 1020 emails in, at which point the
> Catalyst app crashes (although this has only happened once - I do not wish
> to try again and risk another crash). I am using Email::Stuffer in a for
> loop, iterating over an array of 3000 email addresses. For my next step I
> will alter my code to only attempt to send to 500 email addresses at any
> one
> time, since below 1000 everything seems to work fine.
>

If you can consistently to 500, but 1000 consistently crashes you, and you
don't want to
spend the time on forensics, then just go with 500. But be aware that if
you're not freeing
memory, then it's possible to hit the resource usage limits if e.g. you
start sending a bigger message.

You seem to be mentioning using a background process, Darren. Is Exim
> already the background process that sends email?
>

It is a web service trope now that one does not send any email directly as
the result of a browser interaction,
but rather one notes that an email needs to be sent (queues a job request),
and a demon or cron job (a queue
processor) creates ans sends the mail. Thus the web infrastructure can
finish serving the web request sooner,
which is the greatest good at web scale.


If the long term solution is to use mailgun or sendgrid - before I delve
> into all their documentation - do you lot have any tips I should consider,
> on how to integrate them with a Catalyst app, or is it all pretty straight
> forward once I read the documentation?

I would not explore them unless:
1) you start sending a lot more mail.
2) you start ending up in spam control black lists.

As for tips, my usage of sendmail involves having my local sendmail (I use
postfix) hand off to
sendgrid for delivery. I'd like to make use of their
thousand-addresses-at-a-time capabilities, but
my client doesn't feel the need to optimize their delivery processes yet.



Len.



> ----- Original Message -----
> From: "Darren Duncan" <darren@darrenduncan.net>
> To: "The elegant MVC web framework" <catalyst@lists.scsys.co.uk>
> Sent: Monday, November 30, 2015 11:41 PM
> Subject: Re: [Catalyst] Sending 3000 emails.
>
>
> On 2015-11-30 9:29 AM, Andrew wrote:
> > ...given that my attempt to send 3000 emails with nothing but my own VPS
> server,
> > and Cpan's Email::Stuffer, only sent around 1000, before the
> > plack/fastcgi/catalyst app crashed.... could I have been running into a
> similar
> > limitation? I'm now thinking of altering my code to split the attempts to
> send
> > email into six batches of 500 emails, as a short term fix, until I've had
> time
> > to research which transactional email partner to use, and how their APIs
> work,
> > or integrate with Catalyst apps.
>
> If you are trying to send the emails directly from the process serving your
> web
> API, then this is absolutely the wrong approach, and you should give up
> immediately on trying to do that. Your API process should just queue the
> messages to be sent by a separate background process, and then return
> immediately such as with a message "3000 messages queued to be sent soon,
> click
> here to check queue progress" or such. Sending in-process may be
> reasonable
> for
> a single email, eg email a copy of the form the user just submitted, but
> that's
> it. -- Darren Duncan
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>



--
Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
www.lenjaffe.com
Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
@CodeJamCMH <https://www.twitter.com/CodeJamCMH>
Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
Aggregation of Online Advent Calendars.

On Wed, Dec 2, 2015 at 10:59 AM, Lance A. Brown <lance@bearcircle.net>
wrote:

> On 12/2/2015 10:10 AM, Andrew wrote:
> > As noted, it all works until about 1020 emails in, at which point the
> > Catalyst app crashes (although this has only happened once - I do not
> wish
> > to try again and risk another crash). I am using Email::Stuffer in a for
> > loop, iterating over an array of 3000 email addresses. For my next step I
> > will alter my code to only attempt to send to 500 email addresses at any
> one
> > time, since below 1000 everything seems to work fine.
>
> Are you sending those emails as 3,000 individual messages? Are you
> keeping the Email::Stuffer objects around after doing ->send on them?
> With your Catalyst app crashing I wonder if you aren't hitting a memory
> resource limit or memory leak bug.
>

copious logging will be your friend.
also, strace and Devel::NYTProf output might shed interesting light.







>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>



--
Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
www.lenjaffe.com
Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
@CodeJamCMH <https://www.twitter.com/CodeJamCMH>
Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
Aggregation of Online Advent Calendars.

one does not send any email directly as the result of a browser interaction,
but rather one notes that an email needs to be sent (queues a job request),
and a demon or cron job (a queue processor) creates ands sends the mail.

----> So let's do this. How do we go about it?
Scenario - users want to be alerted via email when a new job on a job board is posted.
So we've a website that's a Catalyst app.
At some point in the app, - when a new job post is being created - we note that emails need to be sent to everyone who wanted a head's up on this event.
So what am I doing?
Am I using Email::Stuffer to send via sendmail, which on my system appears to be handled by exim (which can be installed in place of sendmail)...? Or are you saying using Email::Stuffer doesn't queue a job request - it just sends instantly - in which case - how do you queue? Do I just write the source code for the email, and the mysql statement to get the user's emails, to a text file somewhere, in some directory, and let a Perl script running at set times via cron, read in the files, and process them later?
Or should I be looking for certain CPAN mail queuing modules...?

On Fri, Dec 4, 2015 at 11:35 AM, Andrew <catalystgroup@unitedgames.co.uk>
wrote:

>
>
> one does not send any email directly as the result of a browser
> interaction,
> but rather one notes that an email needs to be sent (queues a job request),
> and a demon or cron job (a queue processor) creates ands sends the mail.
>
> ----> So let's do this. How do we go about it?
> Scenario - users want to be alerted via email when a new job on a job
> board is posted.
> So we've a website that's a Catalyst app.
>
>

> At some point in the app, - when a new job post is being created - we note
> that emails need to be sent to everyone who wanted a head's up on this
> event.
> So what am I doing?
>
>

> Am I using Email::Stuffer to send via sendmail, which on my system appears
> to be handled by exim (which can be installed in place of sendmail)...? Or
> are you saying using Email::Stuffer doesn't queue a job request - it just
> sends instantly - in which case - how do you queue?
>
> If you're handing off the mail to exim, you're fine. Your implementation
is correct for your use case

The initial admonishment about queue is all about not making a web client
wait for your app to send an email as a result of some action by the
browser For example, if the sign up for your service, queue a "send them a
welcome email" job, and let the web request request return. Then a separate
process sees that an email needs to be sent, generates the email and hands
it off to exim. It's a different use case that the one you describe, and
mostly it allows you to server more web requests by making the requests
lighter.

So given that you're batch processing the email, we're back to resource
exhaustion as the primary likely cause of error. So you can do less work
per cycle (500 vs 3000), and/or debug the heck out of the process.
If you can get away with multiple 500s instead of 3000 all at once, then
definitely go for it.

But you still might want to put a few scopes on the process to see if it's
leaking memory ro doiong something else egregiously.



> Do I just write the source code for the email, and the mysql statement to
> get the user's emails, to a text file somewhere, in some directory, and let
> a Perl script running at set times via cron, read in the files, and process
> them later?
> Or should I be looking for certain CPAN mail queuing modules...?
>
>
Nah, you're good.



>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>


--
Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
www.lenjaffe.com
Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
@CodeJamCMH <https://www.twitter.com/CodeJamCMH>
Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
Aggregation of Online Advent Calendars.

Or in simpler terms....

....my web app could just set a flag.
And I could have another perl script going as a cron job, that just checks the flag every hour, to see if it needs to send out the emails or not.

I'm curious in that case, how such a separate script gets bundled in with your Cataylst App, so you've got something you could move to another server easily if ever necessary.
Best to keep it simple for now of course, without complicating the matter.

Have never written a perl script to run as a cron job before.
I assume it's just like writing a perl cgi script - put your shebang at the top, and code like normal. The only difference is it's executed either as a cron job, or just whenever you fancy by typing the command line to run it.

Just thinking out loud.

Thoughts will eventually settle into a course of action, I'm sure, ^_^.

----- Original Message -----
From: Andrew
To: The elegant MVC web framework
Sent: Friday, December 04, 2015 4:35 PM
Subject: Re: [Catalyst] Sending 3000 emails.



one does not send any email directly as the result of a browser interaction,
but rather one notes that an email needs to be sent (queues a job request),
and a demon or cron job (a queue processor) creates ands sends the mail.

----> So let's do this. How do we go about it?
Scenario - users want to be alerted via email when a new job on a job board is posted.
So we've a website that's a Catalyst app.
At some point in the app, - when a new job post is being created - we note that emails need to be sent to everyone who wanted a head's up on this event.
So what am I doing?
Am I using Email::Stuffer to send via sendmail, which on my system appears to be handled by exim (which can be installed in place of sendmail)...? Or are you saying using Email::Stuffer doesn't queue a job request - it just sends instantly - in which case - how do you queue? Do I just write the source code for the email, and the mysql statement to get the user's emails, to a text file somewhere, in some directory, and let a Perl script running at set times via cron, read in the files, and process them later?
Or should I be looking for certain CPAN mail queuing modules...?




------------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

If you're handing off the mail to exim, you're fine. Your implementation is correct for your use case

---> Of course, I'm guessing I'm handing off to exim - I should probably do something to check!


if the sign up for your service,
queue a "send them a welcome email" job,
and let the web request request return.

Then a separate process:
sees that an email needs to be sent,
generates the email,
and hands it off to exim.

----> So what does the separate process have to be? A process apart from my Catalyst app - like an entirely different perl script that could be a cron job or something - or can I return a response to the browser within my catalyst subroutine, and then call a private catalyst subroutine to handle the email sending?

---> And yes, after the quick fix, it wouldn't hurt to debug and see if there are memory issues.

On Fri, Dec 4, 2015 at 2:04 PM, Andrew <catalystgroup@unitedgames.co.uk>
wrote:

>
>
> If you're handing off the mail to exim, you're fine. Your implementation
> is correct for your use case
>
> ---> Of course, I'm guessing I'm handing off to exim - I should probably
> do something to check!
>
> Look for Email::Sender configuration. It probably defaults to sendmail,
which is what you want.


>
> if the sign up for your service,
> queue a "send them a welcome email" job,
> and let the web request request return.
>
> Then a separate process:
> sees that an email needs to be sent,
> generates the email,
> and hands it off to exim.
>
> ----> So what does the separate process have to be? A process apart from
> my Catalyst app - like an entirely different perl script that could be a
> cron job or something - or can I return a response to the browser within my
> catalyst subroutine, and then call a private catalyst subroutine to handle
> the email sending?
>
> What triggers the sending of the 3000?

I use a table as a queue, and I write a program that loops over the queue
until no more work to do:

1. read the least-recently-processed job from the queue that has work
to do
1. determine work-to-do as unassigned. (PID is null)
2. mark it as assigned
1. i keep a pid column and update the pid into the record
3. do the work
1. the whole job of some slice
4. if done,
1. perform post job bookkeeping
5. else
1. update in-progress status (num_complete)
2. update PID = NULL
1. so the job processor can come back too it.


I use this algorithm to send 500 emails at a time (in RoR). After 500, the
job goes to the end of the queue, and the next job is processed.

My processor is a stand-alone script that is run from cron.









>
> ---> And yes, after the quick fix, it wouldn't hurt to debug and see if
> there are memory issues.
>
> That's the fun part.





>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>


--
Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
www.lenjaffe.com
Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
@CodeJamCMH <https://www.twitter.com/CodeJamCMH>
Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
Aggregation of Online Advent Calendars.

Exim itself should be the queue and processor shouldn't it?

----- Original Message -----
From: Len Jaffe
To: The elegant MVC web framework
Sent: Friday, December 04, 2015 8:48 PM
Subject: Re: [Catalyst] Sending 3000 emails.






On Fri, Dec 4, 2015 at 2:04 PM, Andrew <catalystgroup@unitedgames.co.uk> wrote:


If you're handing off the mail to exim, you're fine. Your implementation is correct for your use case

---> Of course, I'm guessing I'm handing off to exim - I should probably do something to check!
Look for Email::Sender configuration. It probably defaults to sendmail, which is what you want.




if the sign up for your service,
queue a "send them a welcome email" job,
and let the web request request return.

Then a separate process:
sees that an email needs to be sent,
generates the email,
and hands it off to exim.

----> So what does the separate process have to be? A process apart from my Catalyst app - like an entirely different perl script that could be a cron job or something - or can I return a response to the browser within my catalyst subroutine, and then call a private catalyst subroutine to handle the email sending?
What triggers the sending of the 3000?


I use a table as a queue, and I write a program that loops over the queue until no more work to do:

1.. read the least-recently-processed job from the queue that has work to do
1.. determine work-to-do as unassigned. (PID is null)
2.. mark it as assigned
1.. i keep a pid column and update the pid into the record
3.. do the work
1.. the whole job of some slice
4.. if done,
1.. perform post job bookkeeping
5.. else
1.. update in-progress status (num_complete)
2.. update PID = NULL
1.. so the job processor can come back too it.


I use this algorithm to send 500 emails at a time (in RoR). After 500, the job goes to the end of the queue, and the next job is processed.


My processor is a stand-alone script that is run from cron.
















---> And yes, after the quick fix, it wouldn't hurt to debug and see if there are memory issues.
That's the fun part.








_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/







--

Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe www.lenjaffe.com

Host of Code Jam Columbus - @CodeJamCMH
Curator of Advent Planet - An Aggregation of Online Advent Calendars.




------------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Am reading some interesting stuff now.

There's an odq argument you can pass to sendmail, which adds stuff to the mail queue without waiting to send,
and there's a tutorial on Perl Maven for sending to multiple addresses using Email::Stuffer without wasting memory creating multiple objects...or so I gather from an initial skim read...


----- Original Message -----
From: Andrew
To: The elegant MVC web framework
Sent: Friday, December 04, 2015 9:03 PM
Subject: Re: [Catalyst] Sending 3000 emails.


Exim itself should be the queue and processor shouldn't it?

----- Original Message -----
From: Len Jaffe
To: The elegant MVC web framework
Sent: Friday, December 04, 2015 8:48 PM
Subject: Re: [Catalyst] Sending 3000 emails.






On Fri, Dec 4, 2015 at 2:04 PM, Andrew <catalystgroup@unitedgames.co.uk> wrote:


If you're handing off the mail to exim, you're fine. Your implementation is correct for your use case

---> Of course, I'm guessing I'm handing off to exim - I should probably do something to check!
Look for Email::Sender configuration. It probably defaults to sendmail, which is what you want.




if the sign up for your service,
queue a "send them a welcome email" job,
and let the web request request return.

Then a separate process:
sees that an email needs to be sent,
generates the email,
and hands it off to exim.

----> So what does the separate process have to be? A process apart from my Catalyst app - like an entirely different perl script that could be a cron job or something - or can I return a response to the browser within my catalyst subroutine, and then call a private catalyst subroutine to handle the email sending?
What triggers the sending of the 3000?


I use a table as a queue, and I write a program that loops over the queue until no more work to do:

1.. read the least-recently-processed job from the queue that has work to do
1.. determine work-to-do as unassigned. (PID is null)
2.. mark it as assigned
1.. i keep a pid column and update the pid into the record
3.. do the work
1.. the whole job of some slice
4.. if done,
1.. perform post job bookkeeping
5.. else
1.. update in-progress status (num_complete)
2.. update PID = NULL
1.. so the job processor can come back too it.


I use this algorithm to send 500 emails at a time (in RoR). After 500, the job goes to the end of the queue, and the next job is processed.


My processor is a stand-alone script that is run from cron.
















---> And yes, after the quick fix, it wouldn't hurt to debug and see if there are memory issues.
That's the fun part.








_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/







--

Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe www.lenjaffe.com

Host of Code Jam Columbus - @CodeJamCMH
Curator of Advent Planet - An Aggregation of Online Advent Calendars.




----------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/



------------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

On Fri, Dec 4, 2015 at 4:03 PM, Andrew <catalystgroup@unitedgames.co.uk>
wrote:

> Exim itself should be the queue and processor shouldn't it?
>
> exim is not the job processor and queue. It is the Mail Transfer Agent
(MTA). The job
processor generates my email, and hands it off to exim for delivery.




--
Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe <https://www.twitter.com/lenJaffe>
www.lenjaffe.com
Host of Code Jam Columbus <http://www.meetup.com/techlifecolumbus/> -
@CodeJamCMH <https://www.twitter.com/CodeJamCMH>
Curator of Advent Planet <http://www.lenjaffe.com/AdventPlanet/> - An
Aggregation of Online Advent Calendars.

On Thu, Nov 26, 2015 at 4:32 PM, Andrew <catalystgroup@unitedgames.co.uk> wrote:
> ....I just want to know why the whole FastCGI Plack loaded Catalyst app came
> down half way through,
> and how I can have a Perl script process sending 3000 emails without that
> happening every time, =S.
>

As pointed out already the main problem here is that you are trying to
do stuff (sending email) within the app thread, that is you are
breaking the app design and should use a batch/background thread to do
intensive work (even reading all the emails from database could be a
problem, depending on how you do the query and how optimized is the
database).
I would pick up at least an email alias, that can be easibly
auto-generated and updated each time a new user joins. This way you
are at least hiding the 3000+ receivers behind a single email, letting
the email software to do the stuff it has been built for.
The next step would be to use a mailing list, internal or external.
The only reason I could see for looping thru every email address could
be to store in the database the result of the sending process, which
is in my opinion worthless.

Luca

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Your talk of an email alias has intrigued me, =).
I'll try and read up on them.

I managed to solve the problem of the Catalyst app crashing, using some of the advice given.
- To queue the emails in the background,
- To try not to waste memory.

I stopped trying to be all modern perl, and gave up using Email::Stuffer.
Instead, I went back to oldskool perl, and used:

my $mailprog= 'usr/lib/sendmail -i -t -odq -f myreturn@emailaddress.com';
open(MAIL,"|$mailprog");
print MAIL $email;
close MAIL;

Using the "-odq" flag, adds it to the exim4 mail queue without waiting for it to send.

When I was using Email::Stuffer in a loop, I gather it was creating an object...so it could have been creating 3000 objects - maybe that caused memory issues, which could have been responsible for the crash.

So this time, I had my email source code as a text file, that was just read in once at the start of the script,
and the only thing left to do in the foreach loop (now iterating over a list of userids to email), was to add in the TO line - with the name and email address.
Rather than do a find and replace (via pattern matching) of the email source 3000 times to substitute in a TO address, I merely concatenated the TO line onto the rest of the email source, during each iteration... in the hope it would reduce processing.

my $toline='To: "'.$id_fullname{$userid}.'" <'.$id_email{userid}.'>';
my $email=$toline."\n".$emailsource;

So what was the result?

The result was - all emails were successfully queued.
Because my last attempt had sent the first 1100 or so emails, I was only sending around 1900 emails this time.
The Catalyst app did not crash - the plackup process stayed up.
However - the Catalyst app was unable to give any response to a web browser until all messages had been queued - which took about 8 minutes to be fair (going by the time given in the dates in the mail queue - where the first email has a time of 3:31pm and the last 3:39pm).
Even other subroutines, mapped to different urls, would not give a response to a web browser, until the exim mail queue had been fully populated.

It would seem then, although I succeeded in not crashing the server or Catalyst App,
the Catalyst App doesn't seem to want to give responses to web browsers while it's doing this stuff.
I was hoping merely queuing emails instead of sending them, meant there was nothing much to wait on.
It seems I will indeed need a separate script in the background to send these things, leaving the main Catalyst App free to handle all the web browser requests.





----- Original Message -----
From: "Luca Ferrari" <fluca1978@infinito.it>
To: "The elegant MVC web framework" <catalyst@lists.scsys.co.uk>
Sent: Friday, December 11, 2015 11:12 AM
Subject: Re: [Catalyst] Sending 3000 emails.


On Thu, Nov 26, 2015 at 4:32 PM, Andrew <catalystgroup@unitedgames.co.uk> wrote:
> ....I just want to know why the whole FastCGI Plack loaded Catalyst app came
> down half way through,
> and how I can have a Perl script process sending 3000 emails without that
> happening every time, =S.
>

As pointed out already the main problem here is that you are trying to
do stuff (sending email) within the app thread, that is you are
breaking the app design and should use a batch/background thread to do
intensive work (even reading all the emails from database could be a
problem, depending on how you do the query and how optimized is the
database).
I would pick up at least an email alias, that can be easibly
auto-generated and updated each time a new user joins. This way you
are at least hiding the 3000+ receivers behind a single email, letting
the email software to do the stuff it has been built for.
The next step would be to use a mailing list, internal or external.
The only reason I could see for looping thru every email address could
be to store in the database the result of the sending process, which
is in my opinion worthless.

Luca

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

I would pick up at least an email alias, that can be easibly
auto-generated and updated each time a new user joins.

---> While I understand the principles you are recommending, I'm not sure
exactly what is being suggested, as regards to implementation. Are we
talking exim, vexim, a custom perl script, sendgrid, mailshot, mailman, ....
I get the principle - the Catalyst app just sends to one email address -
that email address sends a copy to all 3000 emails. How to setup that one
email though, I'm not sure.

This way you
are at least hiding the 3000+ receivers behind a single email, letting
the email software to do the stuff it has been built for.

---> Yes...and that's the question: What email software? ^_^

The next step would be to use a mailing list, internal or external.

---> Perhaps this is a clue - are you recommending mailing list software to
do the job?

The only reason I could see for looping thru every email address could
be to store in the database the result of the sending process, which
is in my opinion worthless.

---> Well, at the moment I'm looping through every email address to
construct a "To" line, to add to the email source, which then gets sent
using sendmail - as described in my last email to the group. Given I
mentioned 8 minutes of unresponsiveness when it comes to webpage requests
from a browser - this obviously isn't ideal, ^_^.




_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

----- Original Message -----
From: Len Jaffe
To: The elegant MVC web framework
Sent: Friday, December 04, 2015 10:14 PM
Subject: Re: [Catalyst] Sending 3000 emails.






On Fri, Dec 4, 2015 at 4:03 PM, Andrew <catalystgroup@unitedgames.co.uk> wrote:

Exim itself should be the queue and processor shouldn't it?


exim is not the job processor and queue. It is the Mail Transfer Agent (MTA). The job
processor generates my email, and hands it off to exim for delivery.

---> Yes, that happens in your case. I was hoping I could get exim to be the thing running in the background that handles the mail, and my experiments showed it was able to be the queue for me. However, my Catalyst app, while no longer crashing, does fail to serve responses to a browser, until exim has everything in its queue, and as such, it seems I do indeed need a process, separate from my web app, to be passing those emails to the queue.






--

Len Jaffe - Information Technology Smoke Jumper - lenjaffe@jaffesystems.com
614-404-4214 @LenJaffe www.lenjaffe.com

Host of Code Jam Columbus - @CodeJamCMH
Curator of Advent Planet - An Aggregation of Online Advent Calendars.




------------------------------------------------------------------------------


_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

---> Trying to think of ways to KISS (Keep It Simple Stupid)....

What triggers the sending of the 3000?

---> There's nothing in my Catalyst app that needs to trigger this. I just only know how to code websites, so I put the code in a subroutine, that was triggered by visiting a url when I want to send it, and after I've sent it, I change the subroutine to a private subroutine so nothing else visits the url or triggers it.



My processor is a stand-alone script that is run from cron.

---> Instead of having my script talk to a separate processor, I could literally, just take my subroutine out of the Catalyst app, and make it a standalone script. Then just execute it on the odd occasions I need to send emails out.