VERP on Rails

We sent newsletter campaigns from our Rails app. One of the main requirement of such campaign is how many emails bounced?. We need to track all bounced emails and map them again to a specific campaign because there will be multiple campaigns going on all the time.

To track bounced emails efficiently, we need to set unique return path for every recipient. This technique is called Variable Envelope Return Path or VERP

Postfix supports VERP with -V switch. For eg.

sendmail.postfix -V -f bounced

will generate return path for anyone@example.com asbounced+anyone=example.com@yourdomain.com.

If you want more control and more information like from which campaign this email was sent, we need to send more information in the return-path.

So it is best to generate our own return path pattern and parse it once it is received. We generated a pattern where return-path will generated from all the required information for tracking. This pattern was given to mail method as follows:

    mail(
         from:        'hello@example.com',
         to:          'client@example.com',
         return_path: generate_verp_pattern
        )

So now the first problem was solved. We were sending unique return-path for every email that was going out from the system.

Next part is to track it once it bounces and update database.

Postfix allows piping incoming email to a particular address to a script. So if we pipe all incoming emails to bounce@yordomain.com to our script then we can parse the incoming address and update database.

For this, we have to edit /etc/aliases (or /etc/postfix/aliases)file as follows:

bounced: "|/path/to/your/script"

This means all incoming emails to bounced@yourdomain.com will be piped to our script. This actually means that the whole email with body, headers, attachments etc is forwarded to our script.

The /etc/aliases file is a text file that is used by postfix as a table to redirect mail for local recipients. To rebuild this table after a change, we need to run newaliases command

newaliases

This will rebuild the table for postfix. Now we have completed the second part of the process.

The shell script will get the bounced email content now. There can be multiple bounced emails generated at the same time. So we can’t directly pass them to rails runner scripts or rake tasks. Because that will kill our server by launching multiple rails instances. Instead we need to use some background tasks mechanism.

We were already using resque, so decided to use it for bounced emails also. So a resque worker will actually update the database. Our script just has to enqueue the job for resque.

We broke this enqueuing process into two parts.

First - A shell script which will use correct RVM Ruby version and call the ruby script.

Second - A Ruby script which will enqueue the job to resque.

So the shell script looked like -

   #!/bin/bash
   rvm use ruby_version@ruby_gemset
   ruby /path/to/ruby/script

And Ruby script looked like

    require 'rubygems'
    require 'resque'

    # Adds the incoming bounced_email to background job
    class BouncedEmail
      def initialize(content)
      Resque.enqueue_to(:bounced_email_receiver, 'BouncedEmailReceiver', content)
    end

    BouncedEmail.new($stdin.read)

We had to go in 2 steps here because we had multiple apps using multiple rubies on same server. If you have only one ruby then you can make a executable ruby script instead of shell script which decides which ruby to use.

Now its upto resque worker to parse the content and update database.

For that, we used bounced_email gem which detects lot of things such as bounced code, reason, type of failure etc. As it is integrated with mailgem, we got the recipient address(which was unique pattern generated by us only) and were able to parse it to update the database. With bounced_email we got some more relevant information for free :)

References :

http://keakaj.com/wisdom/2007/08/08/verp-on-rails/
http://blog.sosedoff.com/2011/08/10/processing-emails-with-postfix-and-rails/
https://github.com/mitio/bounce_email