Fri 11 May 2007
White list For Exim Sender Verify Callout
Posted by jsupport-user1 under Computers , PersonalDigg This Entry , [44] Comments
Hello,
This is a re-write with a few additions of a solution that I found on 2 BLOG’s and in the cPanel Forum. My sources are: johnhesch.com, yamzy.net and forums.cpanel.net.
I was having some issues with a few clients and their email. A client would call me and say, “A vendor says that they cannot send email to me. What’s going on?”
I’d chime back, “Did they give you any more information? If you can ask them to fax you the bounce message or email it to my comcast account I will look into it.”
Eventually I’d receive the error the message. It would read something like:
Error 451: Deferred sender callout cannot be verified.
or
Error 550: Verify sender callout failed.
If you look in your exim Logs /var/log/exim_mainlog you might find something like:
could not complete sender verify callout
Exim by default, will check the senders email address and send a callback to the sending server to check and see if the users email address actually exists. In this case the senders email server was not verifying the email address actually exists and so the email was being rejected. In some cases the sending server does not wait long enough for the check to complete. Most of the time this is an issue with the sending servers configuration. It is not RFC compliant. It is not always possible to contact the senders server admin to alert them of their server issue. You may want to just make a concession on your end.
In cPanel or more specifically “WHM -> Service Configuration -> Exim Configuration Editor” there are 2 setting that help keep SPAM down “Verify the existence of email senders.” and “Use callouts to verify the existence of email senders.” These Exim directives tell Exim to perform the checks. I tried to turn them off for about 4 months. My server mail queue was loaded with over 3000 emails. The queue ages 7 days then deletes but still something was wrong. Then I got on an RBL list and that was the straw that started the search for a solution. I enabled both “Verify the existence of email senders.” and “Use callouts to verify the existence of email senders.” while I looked for a solution. In 7 days my queue dropped to just 40 emails. Now I still had a clients that needed to communicate with their vendors.
After Googling I found my solution on johnhesch.com. I nearly lost it. When I finally confirmed that what was posted there was worth trying the link was broken. I contacted John via email to ask about it and he sent me back the info I needed. I later found what looks like a copy of John’s posting here yamzy.net.
So it turns out what I needed was a white list. Now Starts the “How To” Create a file that will be the actual white list. In this example it is /etc/exim_whitelist_senders – the addresses need to be listed one entry per line, either the email address or use the wildcard to do an entire domain. The Following supports cPanel 10.
- SSH into your server and as root or using SUDO or SU run this command:
touch /etc/exim_whitelist_senders - In WHM, got to “WHM -> Service Configuration -> Exim Configuration Editor.”
In the top most edit box add (if there is anything else in the text box add this bellow it):
addresslist whitelist_senders = wildlsearch;/etc/exim_whitelist_senders - Still in WHM. scroll down to where there are three text boxes together. This is the begin ACL section. In the middle box scroll down until you find:
#sender verifications are required for all messages that are not sent to lists
require verify = sender/callout
accept domains = +local_domains
endpassIncPanel 11 look for:
[% ACL_RBL_BLOCK %]
require verify = sender/callout=60s - and change it to:
#sender verifications are required for all messages that are not sent to lists
deny
!verify = sender/callout=30s,defer_ok,maxwait=60s
!senders = +whitelist_senders
accept domains = +local_domains
endpass - Save and exit. Now try to send and receive email to make sure everything is still working. If all is ok add the address in question to the white list and see if it works.
- Put the sender addresses in the file /etc/exim_whitelist_senders, one per line, e.g. someone@domain1.tld
*@domain2.tld
If you do not want an RFC compliant email server make this change too. When I made this change it broke my setup. Verifying the header can cause valid email to fail this check since some valid email does not come from users but is created by the automated systems, like a server. I WOULD NOT MAKE THIS CHANGE. It took me 5 day to figure out this was the part that broke the above setup.
- Still in the middle box scroll down to the end and change:
#!!# ACL that is used after the DATA command
check_message:
# Enabling this will make the server non-rfc compliant
#require verify = header_sender
accept - and change it to:
#!!# ACL that is used after the DATA command
check_message:
deny
!verify = header_sender
!senders = +whitelist_senders
accept
It did not really break it but for some reason beyond me it was not working with this section active. Disabling it made my white list work like a charm.
Sincerely,
Mike
[…] Michael Brandonisio » White list For Exim Sender Verify Callout […]
Thanks for posting this, this is exactly what I need to do on my new Cpanel server since I have a client who is getting all her craigslist confirmations & NYTimes daily e-mails blocked because of the the callback failures.
However my Exim configuration is a little different than what you describe above so I am hesitant to make the changes you recommend since I’m not sure where I should put them exactly.
For example mine reads
“require verify = sender/callout=60s”
instead of what you mention above and I also have “accept domains = +relay_domains” instead of ‘local_domains’ as you reference above.
I’m running Exim 4.
Any suggestions would be appreciated.
Thanks
-Matt
Hi Matt,
The important thing to do is to compare the items in the same sections of the exim.conf file. Where you have:
require verify = sender/callout=60s
means the callout should wait up to 60 seconds. That is fine. Just comment your entry out with a pound # at the beginning of the line. so for you it might look like this:
deny
!verify = sender/callout=60s
!senders = +whitelist_senders
accept domains = +local_domains
accept domains = +relay_domains
endpass
The accept part is where you are white listing. Local_domains those that are hosted on that server that are accepting mail on the server. relay_domains is a list of domains that your server is acting as the gateway to forward on. The relay_domains does not contain domains where the server is the final destination. Only domains that will be forwarded on. So if you make the changes you see here your server will continue to function as before with a white list for senders.
If you get an error that local_domains is an invalid alias then go back and comment it out. It possible that your server forward all mail to another for final delivery.
Sincerely,
Mike
Hello! Good Site! Thanks you! Anonymous
What about cPanel 11? The new ACLs don’t look the same.
Hi Kevin,
In cPanel 11 look for:
[% ACL_RBL_BLOCK %]
require verify = sender/callout
and replace it with step #4. I went back to check my servers and those that were upgraded from cpanel 10 to 11 have the same exim.conf entries as described above. However, I recently added this to a new server that had a new installation of cpanel 11. I believe that the entry below [% ACL_RBL_BLOCK %] was as stated here or may have been:
require verify = sender
only. I don’t have a copy of the original conf file to review. Maybe you can check yours and confirm or deny?
Sincerely,
Mike
Hi,
Thanks for the above.
Have implemented as per instructions and works a treat.
After a few days though, this seems to stop working with some domains and also will not bypass specifically whitelisted addresses (either in the form of user@domain.com or *@domain.com).
Has anyone else experienced similar?
This is on a cpanel 11 machine with a very vanilla exim.conf
Any assistance is greatly appreciated.
please ignore above, I removed the same code that broke your setup, mine now works a treat.
Just taling the rejectlog for any real senders I need to whitelist now.
Thanks for a great piece of info.
Hey Judd,
I was just getting ready to respond to you this morning. I’m glad you have got it working now. I’m going to update the post to let people know that last part can break the whitelist as well.
Mike
Hi Michael,
Removing the second part looks to have fixed MOST of my issues. I still have some persistent domains that are having verification problems even when whitelisted or even when sender verifaction and also verification by callout are COMPLETELY turned off (via cpanel anyway).
After turning on extended logging, the domains in question seem to be failing with the follwoing two errors:
incomplete transaction (QUIT) from
incomplete transaction (RSET) from
anyone have further thoughts? exim seems to still be performing verification somewhere that the above whitelisting method seems not to cover.
?
OK, further investigation done.
it seems the above whitelisting solution works well on most domains for me. Some whitelisted domains seem to bypass the whitelisting chek/processs though for some reason I cannot work out and never pass sender verify.
If I comment out all in the acl_rbl_block and add an accept (as below):
#deny
#!verify = sender/callout
#!senders = +whitelist_senders
#accept domains = +local_domains
#endpass
accept
then those specific domains that were whitelisted but still failing verification now are fine and succed in delivery.
So it seems the specific issue is in the above few lines for me.
Any thoughts?
Has anyone else been watching their reject log very closesly to see if ALL whitelisted are succesfully sending mail to you?
as in:
# tail -f /var/log/exim_rejectlog | grep verify
Hi Judd,
Try changing this:
!verify = sender/callout
to this:
!verify = sender/callout=30s,defer_ok,maxwait=60s
see if that makes a difference.
Mike
Awesome, works a treat for the specific domains that were having the issues.
Nice one!
Hi Judd,
I’ve changed the post to reflect the timing parameters set.
Mike
Your steps do not work for cpanel 11. The config editor reports “Invalid syntax”.
Hi Monty,
These steps do work. You have to place them in the correct location in the advanced editor as described above. If your cPanel was an upgrade to 11 then you have to restore default and then make the above changes. You would also have make any other custom changes you have had in the previous conf file.
Mike
Anyone found that enabling the above stops spam assasin from checking e-mails on the server?
I turned on whitelisting earlier via the above, but then spotted that there’s no mention of SA in the e-mail headers or in the exim log. Take that out and hey presto, it reappears.
My only guess is that it’s stopping when it reaches the end of the whitelist and not doing the SA stuff which is below (or at least a lot of it is mentioned below). Then again, I don’t understand enough about Exim to fully figure the files.
Cpanel 11.6.0 and WHM 11.2.0
Hi Trev,
I saw your post on the cPanel forum too. I am running the same configuration on 7 servers, Cpanel 11.6.0 and WHM 11.2.0, and SA is running fine all 7 servers. I’d try resetting to defaults in the exim configuration editor. Then select “use Old style transport”. Then add the above items in. That is pretty much how I set mine up.
Mike
Great posts.
I am running into the same thing that Trev is. Where if I use a whitelist Spam Assassin doesn’t run correctly (no mention in logs or headers) However, If I remove the whitelist checking, Spamassassin works fine. I tried to set it up to use Old Style Transport as you suggested Mike, but still no luck. I am curious to know if you got it to work Trev?
Hi Tam,
I’m very curious to know too. If either you or Trev figure out what or why please post back here. So that makes 2 that get a broken SA and 1 who is getting a Conf error when saving in WHM.
Thanks,
Mike
Not had a chance to play with it again properly, but hopefully soon. Also ran into problems the day after when Cpanel wanted to update Exim and complained bitterly that I’d dared to alter the config so went back to normal. 🙂
I’m using the new I think they call it ACL system where it checks at SMTP time to save resources and not tried with the “old style” one.
Only very vauge thing I’m wondering – has anyone who’s got this to work changed to the new “maildir” mail folder system instead of whatever the old one was? I’ve not on any servers, but not sure if that’d make any odds.
Suspecting that there’s something like that interfering with it. Either that or it’s stopping reading the config file for some reason just above SA.
Trev
Hi Trev,
I’m using the old style transport with “Maildir” storage. I converted from mbox about a year ago. On my servers when I was using the new ACL’s or the default configuration for exim, no email marked with x_status was making it users inboxes and subjects were not being rewritten. Only when I set my configuration to use “old style” transport did my system start working again like in cPanel 10.
Mike
Hi,
Matt Carey from cPanel has read this entry in my BLOG and recommends that those users having issues with SA open a ticket with cPanel. Here is link: https://tickets.cpanel.net/submit/index.cgi?reqtype=tickets
Mike
Okay, I have cPanel 11 and here’s what I did that appears to work:
via ssh: touch /etc/exim_whitelist_senders
in whm first box:
addresslist whitelist_senders = wildlsearch;/etc/exim_whitelist_senders
after [% ACL_RBL_BLOCK %], replaced:
require verify = sender/callout=60s
with:
deny
!verify = sender/callout=30s,defer_ok,maxwait=60s
!senders = +whitelist_senders
And saved. I did NOT alter or add any of the other lines, as my +local_domains uses more complex code in my config than in the examples (e.g. “warn domains = ! ${primary_hostname} : +local_domains” … etc.), and it already has the +relay_domains and other checks that follow as well.
Also, I did NOT switch it to old style transport.
Testing, it is still doing SA checks, and it is still doing sender verify checks, according to the exim logs. I added an email address to my new exim_whitelist_senders file and it allowed that email to come through sucessfully.
I hope this helps other cPanel 11 users!
Jamie
Hi Jamie,
Thank you for taking time to share your findings. To get it working for you left out:
accept domains = +local_domains
Have tried to put it in and see of it breaks when it is there?
Mike
Just testing. I have not been able to post.
can I use an IP address in the whitelist file instead of domains? the company who has caused me to search for this tool has a ton of domains that they send mail from to one of my clients, but they all come out of the same server.
also, am I to assume that when you reference the changes to exim configuration you’re using the “Advanced Editor” found in WHM? because all I see on my screen are checkboxes 🙂 I’ve been using WHM/cPanel for a couple of years now but am a noob when it comes to toying with it. thanks in advance.
Hi,
The entries have to be domains since the envelope and from in the headers do not have the IP of the server in it.
Yes “WHM -> Service Configuration -> Exim Configuration Editor -> “Advanced Editorâ€.
Mike
What effect should leaving out
accept domains = +local_domains
cause then? I seem to be able to send email from one local domain to another, if that’s what it does.
Hi,
Not sure. I think It means that even local domains are subject the check rather than bypassing the check.
Mike
Ah okay. You had said to get it to work that I needed that, so I thought you knew something particular that had an effect on.
Looking at the original cpanel post, he never explains what that does either, and I wonder if the “accept domains = +local_domains endpass” stuff might just have been quoting what he already had in his config, not something he added. If so, and since it is different in my cpanel 11, then that wouldn’t apply perhaps.
As best I can tell from the exim docs
http://www.exim.org/exim-html-4.10/doc/html/spec_7.html#SECT7.2
is that that means “if the domain is a local domain, accept the message with no further checking, otherwise pass control to the next ACL statement.”
And I tried reposting without the backslashes, but it still didn’t like it, so I’m summarize and you can update it with the one I emailed if like:
–(start message)–
Well my config already has this by default right after it:
warn domains = ! ${primary_hostname} : +local_domains
condition = […]
set acl_m0 = 1
set acl_m1 = […]
warn domains = ${primary_hostname}
condition = […]
set acl_m0 = 1
set acl_m1 = $local_part
accept domains = +relay_domains
deny message = $sender_fullhost is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
Should that already take care of it?
can this be used for rdns failure as well, i have a important client who can’t send me emails since his mail server fails rdns check.
can i use the above method to whitelist domains which fail rdns check as well??
Hi Thesaint2,
I have not tried it for that specific issue, But it may. I’d try it and see if it works. Fixing rDNS is really a configuration issue with the email server IP. Your client should ask their ISP providing the IP to set rDNS to the client domain. I have never had one refuse this if you are paying for static IP service. If they are using dynamic DNS then I can see how this may not work as the rDNS would not change to match the new dynamic IP.
Give this a try and post your results.
Mike
Hello,
Just like to say: I originally tried Michael’s ‘how to’ and successfully set-up the whitelist but it also appeared to disable SpamAssassin.
However, JamieW’s post fixed my problem. Now both SA and the whitelist are working. Thanks to all.
WHM manager just got upgraded today to 11.11.0 for the stable release set.
I’ve implemented the changes and will comment again on if it still works.
thanks for the whitelist technique
Hi Granville,
When WHM goes through a major update like that I like to let the exim configuration tool reset to defaults and reapply this technique. This way I should pick all of the exim.conf changes.
Mike
It seems that another WHM update occurred, it’s at 11.15.0 and there have been updates to the exim configuration page.
They now have access lists where you can click on the edit and add hosts to some but IP addresses to others.
If I’m correct this will negate the need to use this fix for those that are updated to this release.
The last item in the list is a Whitelist for trusted mail hosts/ip blocks to bypass all smtp checks except recipient verification. I hope this works the same as this particular how-to
Thanks for the great work on it.
Hi Granville,
You are correct. Since this last update. I have removed these manual entries. I now use the exim configuration screen in WHM. The best part is that I do not have to re-add anything to the configuration when WHM updates.
Yes “Whitelist for trusted mail hosts/ip” yes it does. I put all the entries in my whitelist_senders file in this list and mails flows from those domains into our server.
Mike
Does this work if I’m not using cPanel
Hi Jeremy,
It should, but really depends on your exim configuration. cPanel’s configure may be different from your’s so where you place the entries may be slightly different.
Mike
Sender verification is broken by design, since it does not recognise that it makes SMTP errors of its own – and then rejects the message. It can only blacklist individual senders, but it is incorrectly used as a whitelist.
[…] of the stuff I found online related to solving this problem on a server running WHM (here and here) explain how to modify exim.conf to add special whitelist rules. Luckily, my server is running WHM […]
I would like to do the sender verify callouts but instead of denying, I would like exim to insert a header tag with the result so I can then use spam assassin rules to assign points to it. This would let the newsletters through but marked as spam so that users could individually white list the senders in their private whitelist and it wouldn’t be my job.
Any ideas how to make this add a header line to the email?
Hi Dave,
I hope a viewer can weigh in on this. I don not have a solution for you. Have you looked at any of the examples at http://www.exim.org?
Mike