It’s entirely possible that they intentionally deleted things (maliciously) but even if they did, there should be an alternate archive system.

All too often backups get abused into trying to be archives, and it just does not work.  This is so common tat many backup systems have bent to this.  It’s so common that one that doesn’t gets asked so often they usually write a FAQ about it as AMANDA has done.

So folks, if you need archives, get a proper archiving system and quit trying to bastardize perfectly good backup software into doing it.  There are many archiving systems out here.  And no they’re not cheap.  Why?  Because a database archive, an e-mail archive, and a file archive all have very different properties.

 
Return-Path:
Received: from jobe.wgops.com ([unix socket])
by jobe (Cyrus v2.2.13-Debian-2.2.13-10) with LMTPA;
Fri, 09 May 2008 16:19:24 -0600
X-Sieve: CMU Sieve 2.2
Received: by jobe.wgops.com (Postfix, from userid 65534)
id 716481E5C7; Fri, 9 May 2008 16:19:24 -0600 (MDT)
X-Spam-ASN: AS26282 216.82.240.0/22
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on jobe.wgops.com
X-Spam-Level:
X-Spam-Status: No, score=-5.2 required=5.0 tests=BAYES_00,MIME_QP_LONG_LINE,
RCVD_IN_DNSWL_MED autolearn=ham version=3.2.3
X-Greylist: Passed host: 216.82.241.67
Received: from mail95.messagelabs.com (mail95.messagelabs.com [216.82.241.67])
by jobe.wgops.com (Postfix) with SMTP id 4E0A21E5C2
for ; Fri, 9 May 2008 16:19:17 -0600 (MDT)
X-VirusChecked: Checked
X-Env-Sender: DelRey@randomhouse.com
X-Msg-Ref: server-8.tower-95.messagelabs.com!1210371155!5055952!1
X-StarScan-Version: 5.5.12.14.2; banners=randomhouse.com,-,-
X-Originating-IP: [170.171.1.201]
Received: (qmail 30380 invoked from network); 9 May 2008 22:12:35 -0000
Received: from ares.randomhouse.com (HELO ares.randomhouse.com) (170.171.1.201)
by server-8.tower-95.messagelabs.com with SMTP; 9 May 2008 22:12:35 -0000
Received: from NYXSRV-6.us.randomhouse.com ([10.105.2.64]) by ares.randomhouse.com with Microsoft SMTPSVC(5.0.2195.6713);
Fri, 9 May 2008 18:12:35 -0400
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Douglas Adams - The Ultumate Hitchhikers Guide to The Galaxy w/ missing pages....
Date: Fri, 9 May 2008 18:12:35 -0400
Message-ID: <16FD8EBC81C0B84E90C2F133D179E6B60F6E5E9A@NYXSRV-6.us.randomhouse.com>
In-Reply-To: <05060AE1A8F8D12C69AC8B82@ZOP-MACTEL-2.local>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: Douglas Adams - The Ultumate Hitchhikers Guide to The Galaxy w/ missing pages....
Thread-Index: AciyITJNNZ5li5r4ROO42upgSD9aBwAAFFkA
From: "Del Rey"
To: "Michael Loftis"
X-OriginalArrivalTime: 09 May 2008 22:12:35.0789 (UTC) FILETIME=[C91DAFD0:01C8B221]
X-Virus-Scanned: ClamAV version 0.92.1, clamav-milter version 0.92.1 on jobe.wgops.com
X-Virus-Status: Clean

Michael,

Sorry, I just took over the Del Rey inbox and there were a bunch of
unanswered emails (obviously from quite a long time ago). I thought the
"defective book" deserved a response regardless of its moldy age.

But if you give me your address I would be happy to send you a complimentary
book anyway ...


Return-Path:
Received: from jobe.wgops.com ([unix socket])
by jobe (Cyrus v2.2.13-Debian-2.2.13-10) with LMTPA;
Fri, 09 May 2008 15:27:46 -0600
X-Sieve: CMU Sieve 2.2
Received: by jobe.wgops.com (Postfix, from userid 65534)
id 8C7D81E59B; Fri, 9 May 2008 15:27:46 -0600 (MDT)
X-Spam-ASN: AS26282 216.82.252.0/22
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on jobe.wgops.com
X-Spam-Level:
X-Spam-Status: No, score=-5.2 required=5.0 tests=BAYES_00,MIME_QP_LONG_LINE,
RCVD_IN_DNSWL_MED autolearn=ham version=3.2.3
X-Greylist: Passed host: 216.82.253.163
Received: from mail166.messagelabs.com (mail166.messagelabs.com [216.82.253.163])
by jobe.wgops.com (Postfix) with SMTP id 5DBDD1E58F
for ; Fri, 9 May 2008 15:27:38 -0600 (MDT)
X-VirusChecked: Checked
X-Env-Sender: DelRey@randomhouse.com
X-Msg-Ref: server-6.tower-166.messagelabs.com!1210368057!17460964!1
X-StarScan-Version: 5.5.12.14.2; banners=randomhouse.com,-,-
X-Originating-IP: [170.171.1.201]
Received: (qmail 2868 invoked from network); 9 May 2008 21:20:57 -0000
Received: from ares.randomhouse.com (HELO ares.randomhouse.com) (170.171.1.201)
by server-6.tower-166.messagelabs.com with SMTP; 9 May 2008 21:20:57 -0000
Received: from NYXSRV-6.us.randomhouse.com ([10.105.2.64]) by ares.randomhouse.com with Microsoft SMTPSVC(5.0.2195.6713);
Fri, 9 May 2008 17:20:56 -0400
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Douglas Adams - The Ultumate Hitchhikers Guide to The Galaxy w/ missing pages....
Date: Fri, 9 May 2008 17:20:56 -0400
Message-ID: <16FD8EBC81C0B84E90C2F133D179E6B60F6E5E88@NYXSRV-6.us.randomhouse.com>
In-Reply-To: <15C69C3E48402048B6120DA27673DDAD02DAC265@nyxsrv-2.us.randomhouse.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: Douglas Adams - The Ultumate Hitchhikers Guide to The Galaxy w/ missing pages....
Thread-Index: AcT0iGSBkC1Nk6KmSN+8CVLswYHQMYd7JDjQ
From: "Del Rey"
To: "Michael Loftis"
X-OriginalArrivalTime: 09 May 2008 21:20:56.0992 (UTC) FILETIME=[9216CA00:01C8B21A]
X-Virus-Scanned: ClamAV version 0.92.1, clamav-milter version 0.92.1 on jobe.wgops.com
X-Virus-Status: Clean

Dear Michael,

We sincerely apologize, and will replace the book as soon as possible. Please
send us the defective book so that we can give it to our production
department to prevent future errors. Please send the book to:

Del Rey Books
c/o Defective Book
1745 Broadway
New York, NY 10019

Wow. Time warp, MUCH? What queue did that get stuck in? Seriously!?  I’m not even sure if i still have the book, and honestly I completely forgot.  There again it could be a good old post-humous DA prank.

Seriously though.  Three years?

I don’t know about anyone else, but I sure as heck have seen that before.

What would I like to see?  Well…It’d be nice if they used a SpamCop like process to find the ACTUAL source of an email and inform some sort of blacklist of that source, that other places can query.  (DNSBL’s are great for this, REALLY!)

The problem is then that the small percentage of spam in this particular case causes the majority of email to be undeliverable.  AND PEOPLE ARE REQUESTING THIS STUFF GET FORWARDED.

This is part of why we’re trying to take steps to reduce the amount of junk we take in, I’m still researching (or rather it’s on the ToDo list) how we can effectively deploy greylisting.

I’m also working on redesigning the mail system so that we can do more processing “up front” and for forwarded addresses.  Doing this requires more horsepower in our mail front end’s (MFE’s) and requires we take some steps to reduce the amount of obvious junk before we start doing heavy work.  Right now we do some blacklist lookups, then go through AV processing, then pass it along for final delivery, either for local delivery, or offsite delivery.  Local delivery benefits from being processed by SpamAssassin, remote delivery can’t.  Our current SA setup is bound by which mailbox the final delivery goes to for selecting rules/scores, so no local mailbox, no way to go.  It’s also directly coupled with final delivery presently.

What I want to do, is for forwarded addresses allow a “discard” threshold of some sort to be set.  Set it so it can’t be turned off completely and can’t be raised above a certain score threshold (say 8 or 10) and either discard or locally deliver anything above the threshold.

The real problem is of course the rather dull way in which we all handle this mess.  If more providers blocked the crap up front there’d be a lot less to deal with.

I hear everyone clamoring but it’s so haaaaaard.  Seriously now, REALLY, how many of your customers NEED port 25 access?  I bet almost none.  They should be using your mail servers ISPs!  Heck even just having a voluntary opt-in-to-port-25 access will solve a huge chunk of the issue, even if the ISP does no verification.

Maybe this is the end of all forwarding service.  As that seems to cause most of the problem, uneducated users who refuse to listen and stop reporting spam on their forwarded email is what lead to the removal of AOL forwarding.

Wow.  Just WOW.  In a nutshell Trend is suing Barracuda.  Why?  Because they can.  Because the USPTO gave them a BS patent, like they’ve given so many others.  Wow.  Just.  Wow.

If Trend manages to win this it could open up all other AV software, gateway scanning software, and similar products to litigation.

Output from one of our mail processing frontends.  The script runs mailq and collates the output, sorted by recipient domain.  Redacted of course to protect the people with the bad idea.

mfe3:~# /opt/bin/top-mailq-to-domains
   2947 *REDACTED*.com
    158 *REDACTED*.co.uk
    117 *REDACTED*.org
    116 mailbox.com

This is why sending yourself an email for every error on your site is A BAD IDEA.  Make a log.  Process the log.

Kthnxbai.

Right off I had an issue with a false positive rate that was killing delivery from a few less-than-perfectly-well-behaved newsletter senders that shall remain nameless.  After I cleared them up I then hit some false positives unrelated to greylisting, and related to recipient limiting.  Moral of that is you should use simpler software, not more complex.  It took me a while to figure out why a few things were getting blocked.

The results have been pretty dramatic.  I’ve only been running it about a week but the only spam I’m receiving now is stuff that comes in via relay through “approved” hosts.  Places like my administrative contact addresses for mailing lists that aren’t filtered by some form of challenge-response (I’ll talk about my feelings on challenge-response later…they’re not very positive) or greylisting mechanism themselves.

In time I’m pretty sure the spammers will adapt and get their crap back into my inbox though.  My next step is to figure out how to deploy the greylisting technique to our large, busy mail cluster at work, reliably.