System Adminsitrators: Disregard “Reputation” when fighting SPAM

Envelope with an @ symbol

Preface

A number of companies offer a service where they provide “reputation” scores for various domains and IP addresses based on reports of SPAM originating from those networks. These “reputations” are meaningless however, and E-mail Server Administrators should completely disregard them, for a number of reasons.

One: False Reporting

Probably the most significant issue is that the majority of E-mail reported as SPAM, does not actually fit the definition. SPAM is clearly defined as Unsolicited Bulk E-mail. The first requirement is it must be unsolicited. If you do business with a company, you sign up on their website or you order a thing or you just send feedback to an E-mail address of theirs, you have opted in to receive E-mail from that company. Under the rules of the CAN-SPAM act that company has to include in it’s marketing E-mails a link which allows you to easily unsubscribe from those newsletters, but because you initiated contact with that company it is by definition not “Unsolicited”.

The second requirement is that the E-mail has to be bulk E-mail. This means it has to be sent to a lot of people, and is not targeting specific individuals or businesses. If you post a comment on a website forum about gardening, and another reader of that forum builds a mailing list which includes your name and then sends you individual marketing information about his gardening products, that is not SPAM. It may be unsolicited, but it does not qualify as bulk, because he is targeting you as an individual.

Most users don’t understand these requirements. The average person in our society believes that SPAM is any E-mail they don’t explicitly want to receive. And they will often click the “SPAM” button in their E-mail clients when they should instead be clicking the “Unsubscribe” button in those E-mails. For this reason, the vast majority of SPAM reports, which “reputation scores” are based on, are false.

Two: Maintenance

It is practically impossible for the operators of reputation lists to maintain those lists with anything resembling accuracy. A lot of spammers will rent cheap servers from legitimate providers (those with a zero tolerance for abusive customers such as spammers), send millions of SPAM emails from their cheap server, and then when they get booted off they order a new server under a new identity. A lot of SPAM is also sent using servers which have been compromised because their owners are not keeping them secure. The result of this is that the IP addresses of those servers or even their parent networks get a lower reputation. Digital Ocean is a great example, they have a strict zero tolerance policy toward SPAM, but Outlook.com will often block entire subnets of the Digital Ocean IP range, because of the momentary behavior of a few unrelated servers on their network. The people selling these “reputation” lists are not checking up on individual IPs every week to see if the SPAM has stopped or if the operators of those IPs have been booted off the network. The information is almost always going to be outdated.

Three: It hurts legitimate businesses

One of the greatest things about The Internet is how it empowers any person to create small businesses from nothing. It’s extremely cheap to set up a website and start operating a business. It is extremely frustrating when you are a small business owner, to discover you can’t email one of your customers because they use Outlook.com for their E-mail, and you use a respectable hosting provider who just happens to be blocked by Outlook.com’s idiotic reputation list.

Four: There are better ways

There is an accepted “best practice” for E-mail Server Administrators to deal with the threat of SPAM. E-mail which is suspicious should be shuffled into a “Junk” folder in the recipient’s inbox, where it can still be reviewed. There are a number of tools available to help identify suspicious E-mail. A sender’s DNS information should include an SPF record, which tells recipients which IP addresses are authorized to send E-mail on behalf of that domain name. It should include a DKIM public key, which is used to verify the authenticity of each individual E-mail by comparing it against the DKIM signature which should be in the headers of all outgoing E-mails, and it should include a DMARC record which instructs recipients on precisely what steps should be taken when an E-mail fails either the SPF or DKIM tests. These methods are effective at identifying the majority of SPAM. For example any SPAM sent from hijacked servers is going to fail both the SPF and DKIM tests.

In addition to this, there are public “Blacklists”. Unlike “Reputation Lists”, the publically maintained domain blacklists are actually trusted. They contain lists of Domains which have contributed obscenely to the SPAM problem. Furthermore there are steps which can be taken to have a domain removed from a blacklist. It’s possible to check the status of your domain name and see if it is on any of the public blacklists, because they are not commercial services. Reputation Lists are commercial products and so a provider of those services is not necessarily going to let you see your own score unless you pay them, which makes it impossible for a domain owner to petition for changes. On top of this, some “Reputation Lists” are known to take bribes to “whitelist” your domain name. The public and trusted blacklists cannot be paid off.

Conclusion: Reputation Lists are garbage

So to summarize, if you are an E-mail Server Administrator, do not use “Reputation Scores” to identify SPAM. Use Blacklists, ensure that your server is performing tests against SPF and DKIM, and following recommended behavior in DMARC records. Configure your server to label suspicious E-mails as potential SPAM, and dump those into a “Junk” sub-folder of your users’ Inbox. Encourage your users to try the “Unsubscribe” link instead of reporting an E-mail as SPAM.

If you try SPAM filtering methods which are too aggressive, you are going to lose users. As an E-mail Server Administrator, your first priority is to ensure that 100% of legitimate E-mails sent to your users are reaching their destination. Everything else is secondary, and if you can’t provide that then your users are going to find someone else who will.

Google’s “NoCaptcha ReCaptcha” product is slave labor.

US Flag - Distress

Google are Crooks.

Google has been in trouble over their ReCaptcha product in the past, and they are bound to face the fire again. Their “NoCaptcha” service advertises itself as being user friendly, by presenting users’ with a single check box they click to pass the test. However it very rarely works like this, especially if you use any sort of adblocker or privacy protecting addons in your web browser. Most of the time it presents the user with a photograph, or a series of photographs, in a 16 square grid. It then asks the users to click on each square which matches a specific description.

If you don’t answer the captcha to Google’s satisfaction, the challenge becomes more annoying. For example after clicking a square you might have to wait for that square to reload. The time it takes that square to reload is adjusted by Google, based on how much they “trust” you. So while some users might only have to click 3 squares, others might have to click 6, and wait for as many as 10 seconds for other squares to reload. And then when you perform the tasks they demand and you click ‘verify’, it will often start the entire process over again, for no explicable reason as you answered everything correctly.

What is really going on?

What is actually going on here? Is the system broken? No. What is happening is Google has a massive database of images, billions and billions of images, and they are using the unpaid labor of millions of computer users to add digital tags to those images. “This Image contains a Car” or “This Image contains a Mountain” and so on. The entire system is automated. On it’s rosy surface it would appear as if the labor of those users is being used to improve the captcha system, but that’s not what it’s for. It exists to force millions of users to do the work that Google is required by law to pay people to perform.

This is slavery.

This is slavery. We haven’t tolerated slavery in this nation in nearly 300 years, but Google thinks they can get away with anything they want because they think the world depends on them. It doesn’t. We don’t need their stupid search engine, there are a number of 100% equally useful search engines such as Bing and DuckDuckGo, in fact some have found those engines provide better results for their searches. We don’t need their free E-mail service, as there are countless providers of that same type of service, none of whom use it to spy on their users the way Google has done for years. The truth is Google does not provide ANY valuable services to the world, and they are raping the information market in new and disgusting ways to make a quick profit.

Stop Google.

It’s time we put a stop to it. A class action lawsuit should be raised against Google again for their “NoCaptcha ReCaptcha” service, and the FTC needs to step in and force them to stop.

Choosing a webhost, why are they all lying to us?

Red circle with a line through it, crossing out the word "LIES"

Preface

If you have a website you have 3 options to put your site on the internet. You can self-host, you can rent hosting space from someone else, or you can use a free hosting service. Self-hosting is the best, if you can afford it. You need a computer and a reliable internet connection, typically not residential internet but that is changing thanks to services like Google Fiber. Free hosting services are great, if you don’t need a private domain name or special server side applications. For people who want “Mydomain.com” however, renting a hosting service is usually the best choice.

Nearly every hosting provider is lying

That seems straightforward enough, so this should be easy, you visit a few websites, compare some prices, and purchase your hosting. This shouldn’t demand more than an hour of your day, but it’s not that easy. Nearly every large hosting company is outright lying about the services they offer. There’s no regulation in the hosting market and the prices are as low as they can realistically get, so the companies have resorted to competing with lies instead.

The devaluation of the word “Managed”

For example, let’s look at the word “Managed”. Every hosting provider in the world offers “Managed Hosting” if their websites are to be believed, but what does that really mean? Well not much as it turns out. The word Managed starts with “Man” because it’s supposed to mean a human being is overseeing operations, but with large hosts that have 100,000 customers how is that even possible? It’s not, they are lying. They use completely automated systems and if you require any personal support it will come from underskilled phone-operators, or you’ll pay extra for it.

Unlimited is never unlimited

The other popular buzzword in the hosting industry is “Unlimited”. Nearly every hosting provider offers “Unlimited” service, but computer hardware has a finite amount of storage space, processing power, and memory, so how can they offer “Unlimited” service? They can’t. Every hosting package in the world is limited. What’s really happening is they’ve stopped being up front about the details of their service. This makes it especially difficult to compare the value of different services because most of the details are hidden from you.

Full Disclosure

I own a technology consulting business and one of the services I offer is webhosting. I’m not writing this article to advertise my business however. I was unaware of the problems in this marketplace until I entered it, and I am disgusted by how nearly all of my competitors operate. I say nearly all, because there are a good number of honest businesses who are up front about what service you receive and for what price.

What can we do about it?

The simple answer is stop supporting hosting services which behave in dishonest ways. If their sales page doesn’t provide full details of the service you’ll receive for the prices they advertise, or if they claim to offer “Unlimited” storage/memory/CPU, or if they hide extra costs and features on sub-pages of their website, shop elsewhere.

PowerDNS Script to add new zones using pdnsutil

vertical server racks and clean cabling

I recently began using PowerDNS, and it has a really nice command line utility called “pdnsutil” which is sadly under-documented, so I am sharing my bash script which creates a new zone and a reverse dns zone, using pdnsutil.

This script is extremely basic, and I welcome any efforts to fork and improve it.

Be sure to specify your own nameservers where I have placed ns1.nameserver.com and ns2.nameserver.com

The DKIM record is based on a ‘standard’ DKIM configuration that I use, yours may vary.

Encryption Keys

Matrix.org Device IDs

ID:
Description:
FAHUQEBZOT
Mobile
UCJELIGUCV
Desktop

PGP Public Key

Fingerprint: 32FF 61C4 563D 7008 2239 598B D9C6 E8A1 E28C 16F4
64-Bit: D9C6 E8A1 E28C 16F4


The “Internet of Things” must be stopped.

Preface

The Internet of Things is an idiotic idea dreamed up in a marketing lab at Apple and other corporations. Wide-eyed executives with no real grasp of technology saw an emerging market where they could capitalize and make billions of dollars, and they rushed to dive in without thought toward the consequences.

What is the Internet of Things?

The IoT or Internet of Things is the name given to a world filled with devices that have embedded operating systems which are internet-capable. On paper, in a fictional utopian paradise, it presents some pretty cool ideas. Sharing of data between apps, remote control, etc.

Where did the Internet of Things go wrong?

The IoT has been a complete fucking disaster. Because the devices are being rushed to market and made by the lowest bidders in chinese code-factories, the software is easily exploitable. The IoT has regressed internet security by 30 years in a handful of months.

What is the evidence for this disaster?

A recent DDoS attack, that is a “Distributed Denial of Service”, was launched using a botnet which is comprised at least in part by millions of “Internet of Things” devices. This was the largest recorded DDoS attack in the history of the internet and it won’t be the last. Think of this like a country testing a nuclear bomb. A black-hat hacker-for-hire group is displaying it’s capability. Next they will sell their service to anyone willing to pay.

What is the solution?

The Internet of Things needs to die, right now. Boycott all embedded devices which do not have robust security controls. Device owners need the ability to upgrade the software and install their own security controls which could be superior to the factory settings. This is absolutely required to bring embedded devices up to speed with the rest of the computerized world.

What if we do nothing?

Selling unstoppable DDoS attacks will become a common practice. With the IoT growing at a ridiculous rate, it will become trivial for anyone with some technical know-how to own networks of millions of bots. A huge black-market industry will emerge. There will be attacks on corporations, on non-profits, on news organizations, on government systems. Giant CDNs like Cloudflare will be destroyed by botnets. The internet will become a wasteland. The benefits we’ve seen in recent decades, improvements to human rights for example, will be lost.

Summary

This is a pivotal moment in human development. We have a choice to nudge our own technological evolution forward in a responsible manner, or push everyone off a cliff in a fool’s gold-rush lead by absolute idiots who are drunk on greed and have no idea how the technology works.

Let’s Encrypt IS the panacea to all our HTTPS woes.

This is a response to a blogger’s post located here: “Here’s how broken today’s web will feel in Chrome’s ‘secure by default’ future” discussing the future of encryption on the World Wide Web.

I was banned from commenting on Mr. Hunt’s page because I made one critical comment about one of his articles. So much for free expression and the exchange of ideas. I don’t feel like re-writing my opinions so when you read it, bear in mind it was originally intended as a comment.

My Comment:

I can’t entirely tell what your intention with this article is, perhaps just to discuss the subject, which is great cause it needs discussing.

There’s no debate to have, encryption is the future, it should have been the standard by now. A discussion needs to happen to wake the average user up. Encryption isn’t only about protecting privacy, it’s about security, as well as simple standards of living.

SSL is trivially easy today. You said that Let’s Encrypt is not the cure, but you are wrong. Let’s Encrypt _solves_ the problem completely. Any remaining difficulty in adapting 100% SSL is the fault of lazy engineers and developers, and nothing else.

SSL is not a luxury product. SSL _was_ a luxury product. And just like Operating Systems, Digital Audio Workstations, Anti-Virus, E-Mail, and so many other applications, the generosity and effort of computer purists have given us free tools capable of replacing proprietary models. SSL as a Luxury product is going away.

The internet is changing the world, just as people predicted it would. Corporations find a niche where they can profit and they refuse to innovate, preferring to maximize their profits by taking advantage of peoples’ ignorance for as long as possible. What has happened with Let’s Encrypt, and other technologies, is that information-freedom-fighters have liberated the masses from the control of those corporations.

The future of the internet is bright, probably more so than 100% of ‘meat-space’.


FIN.