Hi. I’m interested to find out if anyone else has written some frequently used code that generates at least 100,000 personalised emails at a time? e.g. a mailing list. Have you done anything in particular to optimise the process such as sharing the work between multiple servers or splitting the task into multiple smaller threads?
All of the recipients are in a query and I’m using <cfmail query=”#query#”> to loop through it. There are several cfif/cfelse parts to provide some personalisation based on what’s already in the query.
Issues:
- Slow to generate 100K+ emails (it’s single threaded)
- Have to use <cfsetting requestTimeout=”1 hour”> to remove risk of timeout
- If an error is thrown then all remaining emails fail to generate (no damage limitation or recovery)
Have you found a better way to generate emails internally for sending to external subscribers/customers?
The final issue is to do with the recipients’ email providers who periodically mistake genuinely requested email for spam. Obviously that’s not a CF issue but one many developers encounter. Has anyone implemented DMARC and noticed an improvement in emails getting through to inboxes?
Thanks.
Newbie
Gary, what I have used in the past is this: Write the emails to a spool table (which generally goes pretty fast), then sort by domain and send in batches on a 5-minute interval so as not to clog the network. The spool record included a sent flag and a counter, so it would only try to resend twice. We could then check records that failed and update accounts as applicable. (This was a few years back, using just ColdFusion and a database server.)
Sorting by domains would ensure that the SMTP server would not need to make a new connection for each address; only for each domain.
One thing my current employer is doing is to offload the email to a messaging queue (they use Apache’s ActiveMQ; a former employer used RabbitMQ) so the application could dump the email records to the messaging queue and move on to other tasks, and let the messaging queue do the same functionality as above, sending out messages without affecting the application server performance. The messaging queue can be used with a REST email service, or wired directly into an SMTP server.
The only caveat with mail services is that some specialize in transactional email (based on customer request) ONLY. So using them to send bulk email can get you in trouble.
Hey Gary,
Definitely compare SendGrid to SparkPost.
In my experience, I’ve queued up emails in a outbound table using SendGrid as the backbone and use cfmail tags to transmit the message along with tags to connect metrics back to the outbound message. We batch 10,000 emails every 2 minutes.
James, thanks for your reply. Interesting idea about Sparkpost and offloading the pain of email creation and SMTP elsewhere. But if they have clients who create spammy emails or don’t clean their mailing lists properly, wouldn’t your emails get tarred with the same brush if the reputation of their SMTP IPs suffer? i.e. blacklisting
if “they” = SparkPost, no we haven’t had any issues regarding deliverability. Sparkpost requires DKIM usage and using your own sub-domain name for bounces (optionally tracking domain too). More often, our clients would use outdated subscriber lists or receive too many spam complaints and we’d get contacted. They recently required us to process two of our clients through kickbox.io to identify & remove bad, role & risky email addresses. Some of our clients have come up with better ways to label their sign-up processes and have added double-opt in as a result. It’s been a really good experience and our primary mailserver, for our clients, hasn’t been impacted since switching over to this.
SparkPost isn’t the only game in town. Here are some others. I recommend using any of these over sending bulk email via CFMail:
http://socialcompare.com/en/comparison/transactional-emailing-providers-mailjet-sendgrid-critsend
I took a look at the price for Sparkpost to send our emails: ~$800/month. Eeek! We send a lot but that price is uneconomical. Thanks for linking to a comparison table, it show more competitive companies.
I think I’ll try your previous method of putting recipients in a table and running cfmail against them in small batches, but it might need a table column to mark which CF server is processing them so the load can be shared out… and implement DKIM of course.
We’ve set up individual accounts for each of our sending clients and they personally fund what they consume, but most are on the free plan. We switched to SparkPost from Mandrill when they were closing their API. They were offering 100k messages/mo for free at that time. They’ve since changed their rates and now only have a free 15k/mo Developer plan.
Hopefully you have a method to process the delayed bounces & feedback loop complaints from mail providers. We always thought our bulk messages were being delivered when we sent them locally w/cfmail, but they were often accepted & deleted by mail providers. CFMail doesn’t provide a unique identifer when sending email, so it can be sometimes difficult to match up bounced messages with the original recipient.
We get cfmail to insert an extra header into emails (a GUID) so we can match up bounced emails with both the intended recipient and the specific message that triggered the email to bounce.
I wrote code to process 100’s of bounced mails per hour. It’s all logged in a db and CF will refuse to send further emails to recipients who’ve had a recent email “permanently” bounce.
All that and understanding how and why emails fail is almost a dark art! But dealing with the big email providers who bounce or delay our solicited emails is a nightmare!
We loop over a recipient table and send in small batches. We also have a simple SMTP server (which connects to our primary mail server) installed on the same server as ColdFusion so that CF-spooled messages can be removed from the process as soon as possible. Previously we had lots of issues with mail not being processed at all and it seemed that CF was just dropping the messages. (Messages weren’t even listed in the mail log, but the script would run & records would be cleared.) We’ve also found that the file system gets extremely slow (Windows) as all messages are being spooled.
We switched to a service called SparkPost that uses a REST API service. (They also offer SMTP connections, but the API offers more features and their SMTP IP changes and ColdFusion tends to eternally cache DNS lookup data.) If we want to send 100k messages now, we just generate multiple JSON files (w/20k addresses each) and post to their endpoint. This can often be performed in a single ColdFusion request and is much faster than having to generate thousands of local CFMAIL files. SparkPost can add optional tracking so that you can see how many have received, opened and clicked on links. They also have a webhook that can be configured so that any bounce/reject/spam/unsubscribe reports by users can be immediately reported back to your server via web post. This is important because the reasons and recipient information isn’t consistent when a message bounces.
DMARC along with feedback loops will help you a lot. We have to keep on top of all them and it keeps your rep up. SPF records are a must and DKIM helps tons as well.
In terms of the emails, we do a similar thing. It still takes a while to go out, around the 1 hour mark but we find that if there is content that is the same for all users except for personalisation (i.e. name etc) then we generate the content first, store this in a variable or in the database and then just insert it where required. This way each email is not doing the same logic over and over. Making sure to use CFTRY as well so that if an error is encountered its just skipped and carries on.
Generally generating the email is fairly quick, we found its more the mail server that slows things up. We use our own which means we can specify how many threads etc that its using so when Coldfusion passes the email out its up to the mail server to do a good quick job.
Thanks for your reply, Toby. I would have pressed LIKE if this forum provided such a button!
Adobe: Can we have a LIKE button please for individual replies?
Hi Gary and Adobe,
+1 for reply-level Like button.
Thanks!,
-Aaron
And another reply-level button to report spam. There is already spam in this brand new community. It devalues what the Adobe CF team are trying to do.
Thanks.
You must be logged in to post a comment.
23 Nov
10AM Pacific
Online
