FEATURE: Implement check the availability of website / detection of parked & for sale domains

[keczuppp]

Description

I have been thinking about it even before I've found this: StevenBlack/hosts#1613 (comment) @AdKiller

As many as dead (non-existent domain etc) domains, there are many parked / for sale zombie domains, which still do exist but are dead at the same time...: "This domain is for sale", "Website is no longer available", currenlty PyFunceble marks such domain as ACTIVE...but PyFunceble could have an option to check whether a real website exists on the domain and mark it "REAL" if not, then "ZOMBIE" label.

Possible Solution

Such feature:

• would require downloading a domain's main body and searching in the body for text phrases like: "for sale", "no longer available", "rent domain" etc,
• could lead to some false positives
• a pity there is no status code for it: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes, would not require downloading the whole site's body and searching for text phrases

Screenshoot

[spirillen]

The trick is to setup your own DNS recursor and then import (Setup, configure) it with the RPZ zone pirated.mypdns.cloud

The reason is you then will be blocking through the .rpz-nsdname The next thing you have to do is to disable the http status code and whois check and purely relay on the DNS test.

You can now do 2 things with the results file domain/INACTIVE/list

1. Add them as pirtared domain and block them
2. use a removal tool like sed and remove them from your source

This is a much safer approach than than trying to keep a up to date set of rules, it will also add the ip to the bot lists.

Give it a spin, there are some getting started config for Power dns Recursor here: https://mypdns.org/rpz/dns-rpz-integration/-/tree/master/PowerDNS-Recursor

If you having trouble getting it to work, please do open a issue or a discussion

[funilrys]

The idea is good @keczuppp.

My problem right now is that the same webpage as in the screenshot gives me the text in German (IP-based).
Let's keep this open until I find a way to implement it - somehow ...

[keczuppp]

spirillen : #255 (comment)

I don't know much about these things, I would have to study all that stuff first

funilrys : #255 (comment): text in German (IP-based).

• in this case it's based not purely on IP, but on language of the browser in the first place, by one of the methods, have not checked so far, whether the website will fallback to IP-based localize detection when not using a browser to get a body, but yeah, if the website will fall back, then it complicates searching for text phrases, there is just too many languages, but the phrase search method will work on many other sites, which don't provide localized messages, so we should not resign from phrase search method
• also another problem is that in this case, the example text phrase "for sale", regardless of the site's language, is not even present in raw source code of the page, but is generated from javascript file, when loading and parsing by the browser, so searching for "for sale" text phrase in the source code won't work in this case (returns 0 results), but on many other parked sites will still work of course, so we should not resign from text search method
• we shoud search body for some other unique text identifiers / links as well, as most of parked domains have some unique links to the main parking server, example: https://publicwww.com/websites/parking.bodiscdn.com/ , going trough the results, all are parked zombies, no functional websites, almost 10 000...there could be created a list of such parking-links, if a parking-link is present in the body's source, that means this is a parked / for sale domain

[keczuppp]

Another idea, from jawz101 : easylist/easylist#2374 (comment) :

OP here- I just want to say this is very impressive work.

@ ryanbr @ felix-22

I have a suggestion and I may propose it to @ funilrys for the PyFunceble utility. I think I have done so before. For the past couple of years I've been using the Cisco Umbrella (formerly OpenDNS) Top 1 Million daily DNS lookup reports they publish here to evaluate the adaway list.

If you are unfamiliar with OpenDNS, it is a public DNS service which has been around for longer than most other public resolvers which allowed for content filtering and malware/phishing protection. They make lists of the top 1 million name lookups records publicly available. "The OpenDNS Global Network processes an estimated 100 billion DNS queries daily from 85 million users through 25 data centers worldwide."

So, regardless of if a domain is registered- these are the actual queries made by us in circulation. If a domain is parked, it's going to be valid but nothing is pointing to it so you'll never see it used. I will download, say, the past 3 months of logs and if no one has tried to lookup a domain, I pull it from the adaway list.

edit- crazy. I just did it with 2 days top 1million files and of the 6,999/25,556 were on the top 1 million lists.

[spirillen]

@keczuppp wrote in #255 (comment)

Another idea, from jawz101 : easylist/easylist#2374 (comment) :

Touching #128

@keczuppp wrote in #255 (comment)

I don't know much about these things, I would have to study all that stuff first

You should do that as it will enhance your system performances significantly: you should read Performance test of Hosts file vs DNS-Recursors 😉