After 15 years, I have finally reached the point where I use Outlook as my build pipeline

Posted by codingtofreedom@reddit | programming | View on Reddit | 65 comments

I had an annoying problem at work where we had to work around some less-than-optimal firewall rules, and because we are working on a tool that requires quite frequent testing (like several times per hour we work on it), it was really adding up to take these five-minutes per file that it took to get them from my coworker's machine to mine, and I quickly designed a server workflow that could take the encoded file and save it to its own file system.
Because we could not find a good way to bring these files over, we figured it was actually a good idea to send it via email to an inbox with a specific subject line, and then have a Python script on my end checking the inbox every minute to then take the attached file and send it to the endpoint.

This has to be one of the hackiest solutions I have come up with over the years for a problem that wouldn't have to be one if my coworker was simply allowed to call the endpoint from his code, but I found it interesting how simple it actually is to connect from a Python script to a local Outlook client and parse the results. Less than 100 lines of code, and we have a "build pipeline" going that reduces our manual overhead significantly.

[-]

TadpoleNo1549@reddit

this is such a classic “it shouldn’t work but it does” solution, honestly love how you just worked around the system instead of fighting it, turning email into a mini pipeline is peak developer creativity, sometimes the hacky solutions are the ones that actually save the most time

[-]

TadpoleNo1549@reddit

this is such a classic “it shouldn’t work but it does” solution 😭 honestly love how you just worked around the system instead of fighting it, turning email into a mini pipeline is peak developer creativity, sometimes the hacky solutions are the ones that actually save the most time

[-]

Spiritual_Cycle_7881@reddit

Outlook was a part of the data pipeline in my ETL config 😆 An org had an app adopted and the app required additional fee to enable API and export feature. No money, dear! Invent something. Direct db access forbidden (security!)

Okay, the app allows you to generate xlsx reports and email them on a schedule. Data volume is tiny (10k rows, 5M) - why not? I created a service account and used python requests to scan inbox via MS Graph API (no client libs because it was 20 sloc in pure requests). Worked like a charm, saved some subscription money.

The most difficult part was setting up the auth process properly: removing 2FA for the service acc and limiting senders list using allow/block lists or something. Spent a day or two on this part.

[-]

UnderstandingFun2119@reddit

If you’re going to keep Outlook-as-pipeline, treat the mailbox like a queue with strict auth, filtering, and idempotency.

What’s worked for me: use Graph application permissions with a cert (app-only) so you don’t have to weaken 2FA on a user; subscribe to /messages/delta so you only process new stuff. Whitelist sender addresses and attachment MIME types, hard-cap size, and reject anything else to a Quarantine folder. Dedupe with internetMessageId + a hash of the attachment and store a tiny state table (sqlite/Redis) so replays don’t double-run. Move processed mail to a Processed folder and send failures to a Poison folder after N retries. If you can add a cloud step, have Power Automate drop attachments to Blob/SharePoint, then an Azure Function validates and forwards to your endpoint. I’ve also used Postman to mock this flow, and when we needed downstream tools to read the results, DreamFactory exposed a read-only REST API from SQL Server without extra glue.

Net: mailbox as a queue, app-only auth, delta queries, idempotency, and strict allowlists.

[-]

Spiritual_Cycle_7881@reddit

Yep, probably a cert or something instead of push notifications, I cannot recall tbh. Other things were much easier, I didn't bother with all these things: request emails filtered by the subject line for the last 24 hours, must be signed verified (I guess some cert as well), push attachments into DWH LZ. Done. Let the downstream pipeline to sort the things out - it was a tiny SCD type 1 as a destination.

It was a "temporary solution until finance sorts things out how to budget the API subscription" haha, worked like this for a year or so until moved to another data provider

[-]