URL Filter

The URL Filter allows message filtering using URL or Domain-based Block List Providers (also known as URI DNSBL or SURBL service). The concept is similar to IP Block List Providers but instead of validating the source IP address, text-based DNS domain names are validated. DNS Domain names from the following sources are screened against one or more URL Block List Providers:

• The domain provided in the SMTP HELO/EHLO command
• The domain part of the email address provided in the SMTP FROM command.
• The domain part of the email address provided in the From message header.
• The domain part of the email address provided in the Reply-To message header.
• The domain part of any hyperlinks or image links in the body of an HTML message.
• The domain part of any URLs in the body text of a message.

If any of the above domains are listed by a URL Block List Provider, there is an extremely high chance that the message is spam or contains malicious content and it will be rejected.

The URL filter is an additional filter provided by QSS Exchange Anti-Spam Toolkit as Exchange does not support Domain-based Block List Providers with any of the included anti-spam agents.

Enabling the URL filter will generate additional DNS queries, so it is important to ensure that the Exchange server has an appropriate DNS configuration. Preferably there should be an internal DNS server on the same network segment, with appropriate resources and caching enabled.

Allowed Domains

Domains on the list of Allowed Domains will be excluded from URL validation, primarily to reduce unnecessary DNS lookups. In contrast to built-in Exchange filters, Allowed Domains is not a whitelist and it does not cause messages to skip any other types of anti-spam filtering. If a message contains both Allowed Domains and non-excluded domains, the non-excluded domains will still be validated and the message can still be rejected if any of the non-excluded domains are listed by a URL Block List Provider.

The Allowed Domains list is pre-populated with well-known domains which are known to be trustworthy and are recommended to be excluded by URL Block List Provider services. It is recommended that the pre-populated domains are not removed.

URL Shorteners

URL Shorteners are treated differently to Allowed Domains. When an URL Shortener is detected, the URL Filter will attempt to follow the URL Shortener redirection to find the destination (actual) URL. The domain of destination URL will be validated by the URL filter, although the destination URL will not be followed even if it also would result in a redirection (to avoid effectively accessing spam links and confirming to a spammer that the message was delivered). As long as only legitimate URL Shortener domains are added to this list, this should not result in any additional spam risk.

Understanding URL Block List Providers

It is useful to understand the process by which the QSS URL Blocklist Agent (or another mail server) queries an URL Block List Provider.

In this example, the domain being queried is example.com and the provider being queried is dbl.spamhaus.org.

• The provider is queried by performing a special DNS query.
• A DNS lookup is performed for the address example.com.zen.spamhaus.org.

The response to the DNS query from the URL Block List Provider determines the action to be taken:

• If a DNS entry is not found (no response), then the domain is not considered to be listed by that provider and message processing will continue (by checking other domains in that message and then other URL Block List Providers, if necessary).
• If a DNS entry is found, a response code will be returned in the format of an IP address or multiple IP addresses.
• The responses are not normally valid IP addresses. They are usually special loopback IP addresses (such as 127.0.0.2) which correlate to specific response codes. The response codes are different for each provider and need to be checked by referring to the documentation or usage guidelines for that provider. We have provided the details for commonly-used providers below.
• The presence of a response does not necessarily mean that the domain is listed. The response code needs to be interpreted to determine whether the domain is actually listed.
• If the response code returned matches one of the values defined as IP Addresses Match for that provider (or if Any Match is enabled), then QSS URL Filter Agent will consider the domain to be listed by that provider.
• If the response code does not match one of the values defined as IP Addresses Match for that provider and Any Match is disabled, message processing will continue (by checking other domains in that message and then other URL Block List Providers, if necessary).

You can use the Windows command nslookup to manually query a specific URL Block List Provider. Many providers have special codes which can be queried to allow you to test your configuration.

For example, the command nslookup dbltest.com.dbl.spamhaus.org will respond with the return code 127.0.1.2, which means a listing for a spam domain.

Non-authoritative answer:
Name:    dbltest.com.dbl.spamhaus.org
Address:  127.0.1.2

Adding URL Block List Providers

The Name, Lookup Domain and either Any Match or IP Addresses Match are mandatory for every URL Block List Provider.

Each return code which should be considered as a listing needs to be added to the IP Addresses Match list.

The Rejection Response will be returned in the SMTP session, together with an SMTP error code, when a domain is listed by an URL Block List Provider. {0} can be used as a placeholder for the domain name in the rejection response message.

Strip Sub-Domains

Some URL Block List Providers only list Publicly-Registered Domain Names (top-level) and require sub-domains to be removed before the list is queried. uribl.com is an example which requires sub-domain stripping.

Enabling Strip Sub-Domains will cause sub-domains to be stripped from the domain name for queries to the specific URL Block List provider. QSS URL Blocklist Agent uses the Public Suffix List to determine the highest-level publicly-registered domain name which does not contain sub-domains.

Most URL Block List Providers do not require sub-domain stripping and enabling this setting should not be enabled as it will render the URL Block List Provider ineffective.

URL Block List Provider Configuration Examples

The following example configurations are provided to assist with configuration of URL Block List Providers. These are provided for your reference only and we encourage you to refer to each provider's documentation to ensure that the IP Addresses Match setting is configured accurately and appropriately for your environment.

URL Block List Provider Name	Lookup Domain	Strip Sub-Domains	IP Addresses Match
Spamhaus DBL	dbl.spamhaus.org	No	127.0.1.2 127.0.1.4 127.0.1.5 127.0.1.6	127.0.1.102 127.0.1.103 127.0.1.104 127.0.1.105 127.0.1.106
SURBL	multi.surbl.org	No	127.0.0.2 127.0.0.8 127.0.0.16 127.0.0.24 127.0.0.64 127.0.0.72	127.0.0.80 127.0.0.88 127.0.0.128 127.0.0.136 127.0.0.144 127.0.0.192
URIBL	black.uribl.com	Yes (required)	127.0.0.2 127.0.0.4 127.0.0.6	127.0.0.8 127.0.0.14 127.0.0.20
Spamhaus ZRD (Zero-Reputation Domains)	Registration required (user-specific URL)	No	127.0.2.2 127.0.2.4 127.0.2.5 127.0.2.6	127.0.2.102 127.0.2.103 127.0.2.104 127.0.2.105 127.0.2.106

Logging

QSS URL Blocklist Agent generates logs of all messages which are rejected due to URL filtering. Allowed Domains which are bypassed are also logged. The log files are in the same format as other Exchange logs and are stored in the TransportRoles\Logs folder.

By default, the path to the logs is %ProgramFiles%\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\UrlBlocklistAgent.