Self-hosted SMTP server for high volume sending?
Posted by Shanga_Ubone@reddit | sysadmin | View on Reddit | 58 comments
Hi folks! My org sends about 16 million emails a month of largely transactional emails from a variety of systems located in our data centers. Currently we're using a commercial email security gateway in a cluster configuration that is primarily intended to provide inbound email protection and also happens to handle outbound email, but the gateway doesn't support SMTP-Auth so we're looking to replace it with a self-hosted solution that does.
Other volume, our needs are pretty standard in that we need the server to support DKIM signing, SMTP-Auth and logging/reportability (e.g. largest senders, transaction log, forward to external logging, etc.)
Has anyone worked with a high-volume sender who could advise what worked well in that environment?
OtisMilburn-15@reddit
For high-volume self-hosted SMTP with SMTP-Auth, DKIM, and strong logging/reporting, consider SMTPget or Postfix. Both scale well, support DKIM, SPF, and DMARC, and offer good integration with external logging systems
ffiene@reddit
Linux + Postfix + openDKIM + SASL-AUTH + External Syslog-server like Graylog.
GiraffeNo7770@reddit
Uh, oh! That sounds like it's going to be cost-neutral, can run even on discarded hardware, will be lightweight, will be a great project for building employee skills, and have massive positive impact on workflow.
My employer will NEVER go for it.
bbqwatermelon@reddit
The MSP I was at did not go for it but the NP I joined is where I got to implement 😎
Glass_Call982@reddit
15 years ago when most SMB still ran their own exchange or SBS you had to have some tech skills to be an MSP. Now any douchebag sales person can start one and tack on a bunch of click ops web admin crap under the guise of knowing what any of it does.
ffiene@reddit
I have implemented outgoing MTA with this and a set of MX with the same priority with Linux + Postfix + RSPAMD and several AV solutions attached. This is running since over 20 years (only exception was replacing amavis and spamassassin with rspamd). Company is happy, users are happy for not reading too many AV digest (we reject directly at the MX). Just the rest is covered by M365, where rejecting is not possible. Nowadays I am teaching our trainees how to implement this kind of systems.
bikergeekx@reddit
Are you a Windows shop?
You can easily set up a linux server and use postfix to send out emails.
The trick is to make sure you have it set up correctly, including DKIM signatures.
Shanga_Ubone@reddit (OP)
Thanks! We're an anything shop, so could run postfix. The catch there is that although postfix would definitely handle the mail transfer, it doesn't seem like there's a straightforward way to get the reporting we need. Or is there?
homercles89@reddit
> straightforward way to get the reporting we need.
https://linux.die.net/man/1/pflogsumm Get this and then run the reports nightly/weekly from cron.
weehooey@reddit
You might want to take a look at Proxmox Mail Gateway (PMG). Under the hood it is Postfix but it has a web GUI to manage it.
It is designed for inbound spam and malware protection but it can be used for outbound.
Some disclosure and experience…
We are a Proxmox partner focused on the virtualization product so had not initially looked at PMG. We used to use vanilla Postfix for over 200,000 outbound transactional emails per month.
We decided to test PMG and found its features useful for outbound.
Just to be fully upfront, we are a Proxmox partner and do sell subscriptions with PMG support in addition to Proxmox VE. However, it is open source so you can download and use the full product from Proxmox without a subscription.
nosimsol@reddit
What reporting do you need? Maybe you could turn the logging and have something parse logs for your use case?
Shanga_Ubone@reddit (OP)
Thanks! We monitor the mail flow for anomalies to identify problematic senders and signs of possible compromise. Top senders, volume reports, bounces, etc. We route all the logs into a separate log aggregator (Elastic), although currently we're able to do the day to day monitoring in the cluster itself. Elastic is more for if we lose the cluster somehow and to comply with policy.
I freaking LOVE Elastic BTW. Not for this but M365 audit log analysis is *chef's kiss *
GraemMcduff@reddit
pflogsumm is a simple log analysis tool for postfix that will give you all of those stats. Nothing fancy. It just outputs a report in plain text, but easy enough to set up a cron job that emails you the output from pflogsumm at regular intervals.
mats_o42@reddit
what do you need. You can add own scripts to many stages in postfix and maybe create the data you need.
GiraffeNo7770@reddit
We run Postfix, restrict the allowed senders by IP, and the emails themselves aren't reported on -- they are what populates data into systems that do reporting, like RequestTracker. You can use those outgoing emails to send into a logging server.
The details will depend entirely on your use case -- does it "really" need DKIM if it's internal only, and IP-restricted? Do you need to run a log parser on the email logs, or do you need to run a logging server that ingests the emails and turns them into alerts?
Rather than start with restrictions like DKIM and reporting, start by mapping out exactly what the data flow will be, and what you need out of it. That makes the tech stack selection easier.
Shanga_Ubone@reddit (OP)
Just to add, we're looking at a dedicated sending setup. We don't need/use the inbound protection any longer!
JurgenVM@reddit
https://docs.postalserver.io/
JurgenVM@reddit
postal mail server
https://docs.postalserver.io/
kao1985@reddit
Not sure if this is the correct tool for you but I used to work at an email marketing company that sent about 2 billion emails each month and they used PowerMTA.
kapetans@reddit
yes, altenatively mailerq
andr0m3da1337@reddit
kumoMTA?
kapetans@reddit
yes, if you know how to config it
Toasty_Grande@reddit
Your biggest hurdle will be reputation, and that volume level will likely be problematic unless you use a bulk sender such as sendgrid. I would recommend sendgrid over roll-your-own for that reason.
If you are a M365 shop, MS has their own bulk sending service now, and would be worth looking into.
kerubi@reddit
”HVE will support exclusively internal (within the tenant) messaging capabilities. As a result, the ability to send email to external recipients will be removed in June 2025.”
https://learn.microsoft.com/en-us/exchange/mail-flow-best-practices/high-volume-mails-m365
sharpshout@reddit
He's probably talking about Azure Communication Services which can send bulk email similar to Sendgrid, MailChimp, etc
kerubi@reddit
Could be! That would not require M365, though. Entra, yes, M365 no.
Toasty_Grande@reddit
Yeah, Azure Communication Services. I should have said, if you are already in the MS Azure/365 environment, this is an easy add. If one has no investment, it's a toss up if it's worth it vs an established company like sendgrid.
radoslav_stefanov@reddit
I would stay away from self hosted for transactional emails. Too much work for too little gain. Been there.
Why not SES? For marketing it is a no no, but it is perfect for transactional use case.
Or if you have the budget pick some random enterprise provider.
mahsab@reddit
Little gain? 16 million emails with any commercial service is going to cost a lot
radoslav_stefanov@reddit
It depends what you value and what you are after.
With AWS SES thats like what $2.5k per month? Even a single engineer in some underdeveloped country costs more. Now add hardware cost (or renting), managing failures, dealing with blocked IPs, server reputation etc.
Assuming this is a legitimate use case and not some spam machine he can estimate the cost of deliverability he needs. Then he can make a decision if self hosting is ok for him.
In general self hosting is always more expensive, because reliability will never be better than a managed service. There is a reason big businesses choose AWS over DYI.
You cant have all. Pick being cheap or being reliable.
mahsab@reddit
That's assuming you start from zero. OP said they already have data centers (plural).
Once set up, dealing with blocked IPs, reputation etc, takes very little extra time, definitely nowhere near a full-time job.
radoslav_stefanov@reddit
It is even more true if you have established business, because instead of investing in R&D to improve your main product you waste resources on brainless grunt work.
Or you think your 2 man team will have enough time to manage and develop the same quality analytics system that you get from folks like Sendgrid?
It depends on how you value deliverability rates, reliability and downtime costs.
SMTP is one of the things I would never DYI again. I used to manage large scale email infra in my sysadmin days.
Shanga_Ubone@reddit (OP)
Thanks for all the ideas and suggestions everyone! There are a lot of really good tips here.
The way we'll likely go is to keep the existing cluster but set up a smart host in front just for the subset of senders that need SMTP auth. Since that will still route through the existing cluster that means the smart host can be something lightweight like postfix since all it needs to handle is the auth.
dab_penguin@reddit
Hurricane MTA is also pretty nifty
Shanga_Ubone@reddit (OP)
This looks like a real possibility - thanks!
ThriveCTO@reddit
We have a customer who has used this for years successfully. It’s amazing and scary what it can do.
OmegaNine@reddit
Man, that many emails is going to be a headache to keep your reputation in tact. I would pay someone like sendgrid or mailchimp to deal with that for you. Even with them its easy to get on a black list, but they will be able to help you if you do and tell you why it happened.
CyberHouseChicago@reddit
A Linux server with direct admin control panel and crossbox is what we run works well for us , not doing high volume but you done need a lot of hardware for sending emails
Vivid_Mongoose_8964@reddit
Do NOT self host this, you're crazy! sendgrid, mailersend, smtp2go are all way better options.
AntranigV@reddit
16 million emails a month is about 5 emails a second. that's not high volume.
A simple email server with OpenSMTPd or Postfix can handle that.
For DKIM out, I'd suggest using rspamd, and if you need to also receive emails, maybe pass it through rspamd as well, to filter the spam coming from the internet.
All you need is OpenSMTPd+rspamd, I'd run it on FreeBSD or OpenBSD, and then parse the logs by sending them into a logging system.
Nothing complicated here.
sexbox360@reddit
Mimecast can do smtp Auth. They only support port 587/SSL but it works really well. I have all of our enterprise software using it. Simple username and password to authenticate.
For printers and old devices that don't do SSL I set up a SMTP relay docker container. Very easy and free.
Substantial_Tough289@reddit
You said self hosted but have you considered something like smtp2go? We use it for "mass" mailing (not spamming) withing our organization, works great and is super cheap.
autogyrophilia@reddit
The requirements of CPU aren't not very onerous, specially if you don't filter spam outbound.
The complicated part it's not getting your IP blacklisted. Shouldn't be an issue considering you already have mail traffic.
I'm very fond of OpenSMTPD for it's ease of use, but yours may be the case where it genuinely doesn't scale. In which case, Postfix will.
Load balancing multiple SMTP instances is also very easy to do, either behind a load balancer or lazily via DNS should the need arise (unlike IMAP which is hard .
Have you considering simply setting a smart relay for authentication before your security gateway ?
Shanga_Ubone@reddit (OP)
Holy crap no I have not. A smart relay is a GREAT idea, and also gives more flexibility given the senders that need SMTP auth are just a small portion of our mail flow. We could spin that up easily using postfix or whatever. Thanks!
rmeman@reddit
You want to host yourseldf or use a service ?
Avas_Accumulator@reddit
A large question is if these transactional mails are for all internal use or if you expect them to work towards say customers. The latter is way harder because you'll be scrutinized heavily if building something yourself instead of going with an SMTP provider like Microsoft ACS which is priced around $4K month for that amount. We use Mailgun, and are happy enough with that.
Shanga_Ubone@reddit (OP)
Yes that's the catch. These are mostly customer emails, so we need to be pretty bulletproof. You hit the nail on the head!
WaaaghNL@reddit
Build up your sending reputation. And handle bouncers. Dont send mail to a non existing mail adres over and over again.
Shanga_Ubone@reddit (OP)
Thanks! We currently send about 16 million messages a month without difficulty but these are good tips!
Minimum_Sell3478@reddit
There is proxmox mail gateway it supports dkim etc and can be clustered
Shanga_Ubone@reddit (OP)
This looks promising! I'll check it out- thanks!
autogyrophilia@reddit
Doesn't support authentication. As most gateways, they are intended to be used as proxies from your mail server to the internet.
Shanga_Ubone@reddit (OP)
Nuts. The issue we are running into is that some of our senders are using systems that require SMTP auth - not a lot, but enough that it's becoming a problem and is causing them to roll their own solutions instead of using our main infrastructure.
Minimum_Sell3478@reddit
We use it at work🙂😉
sryan2k1@reddit
Unless you had a dedicated team and tons of public IPs you'll never get a reliable system yourself. Use SES or send grid or smtp2go
petarian83@reddit
Take a look at Xeams, it supports DKIM, SMTP-Auth, logging, and high-volume outbound.
titlrequired@reddit
What is the trigger for sending an email? What is the budget for self hosting a server?
I would be looking at commercial sending systems like sendgrid.
oliland1@reddit
Send grid works well but I’m nowhere near your volume so I’m not sure if it will make sense pricing wise