afshari
[shameless plug] I made a tool to look into BGP data as
well as other networky things at https://god.ad. Can be
added to homescreen as a PWA
app.https://god.ad/view/AS15169
|
> motohagiography
this is great, thank you.
|
> satyamkapoor
Thanks, looks very llmish
|
> > satyamkapoor
Doesn't mean it's bad. Thanks for the work
|
> > afshari
Front-end was indeed done with Claude, with many
iterations and tweaks though
|
faangguyindia
does anyone know where can i buy rDNS records? purpose is
find all domains hosted on specific ip for a monthly
price? I think crawling all my own is big environment
waste of resources, i'd be happy to pay for a subscription
or somethign
|
> yjftsjthsd-h
> does anyone know where can i buy rDNS records?
purpose is find all domains hosted on specific ip for
a monthly priceAre rdns records always populated? I
thought they were optional and many sites didn't
bother
|
> > faangguyindia
i am talking about this data:
https://host.io/lgyy.cc
|
> > > zamadatix
They mean anyone will only ever be able to get
some rather than all because of how DNS works.
Even that site is unable to find all such
mappings for every IP. I.e. the first option
to build such a dataset is to use rDNS
lookups, but it's optional to configure those
and many don't bother (opting to just have the
forward lookup). The other way about it is
collecting domains and seeing which IPs they
resolve to, but this is only ever partial
(there is no such thig as a conplete lists of
domains in the world) and has additional
problems with CDNs or things hosted in more
than one place at once (you need to do a
lookup from every possible location each
domain could conceivably be hosted to be sure
you get all of the reverse mappings for that
domain).As an example: Unless you know my
domains ahead of time you'll never be able to
come up with what domains are hosted on my IPs
because I don't bother to configure rDNS. So
those IPs will look like they host no servers
(or only some if you only had a partial list
of my domains) rather than all of them on
those IPs.Anyways, for free data sources
trying to get a partial view of this you can
check out Rapid7's Sonar or Common Crawl. Each
should have the pieces needed to construct
this kind of view from the data.
|
> > > > faangguyindia
Sonar does seem to have data here:
https://opendata.rapid7.com/But it seems
you've to go through their sales team and
all for the data.I can't find CommonCrawl
making their DNS resolution available.
|
> > > > > zamadatix
Common Crawl doesn't publish raw DNS
separately, you have to pull the
information out of the aggregate
database. The WARC-IP-Address header
should record the IP Common Crawl
connected to for the site.
|
> > > > > > ccgreg
Good timing, I'm about to release
that dataset.
|
> basilikum
Do you mean rDNS records or a mapping of an IP address
to all ([semi] publicly known) DNS records that point
to an IP address? Cause these are two different
things. Just because a.com points to 1.2.3.4 does not
mean 1.2.3.4 has an rDNS record to a.com.
|
> > faangguyindia
You are right, i'd need both actually. I can't
find any service which provides weekly/monthly
data download for a flat subscription price, so
that i can perform local fast lookups.
|
> > > basilikum
How granular do you need this data? Domains
are ultimately not publicly known, especially
when it comes to subdomains. Do you just need
domains that get a certain amount of public
traffic or are you trying to get as much
domains as possible? In the latter case
scraping this data would require the logs of
public DNS servers to even know of the
existence of many domains (although
certificate transparency logs could get one
half the way).
|
Bender
Just to have a compressed static file that contains all
the routes one could download from time to time I think
would be interesting to analyze. Same goes for the anycast
root DNS servers. To have a full dump from them could be
interesting. Not to be confused with the root.zone [1] I
mean the whole kit and caboodle.[1] -
https://www.internic.net/domain/root.zone
|
> icehawk
RIPE keeps this
https://ris.ripe.net/docs/mrt/#name-and-location
|
> > Bender
The collectors are browsable nice, Thankyou!The
collectors seem to be overloaded so made a
temporary mirror of the one for AMS In AMS. [1] I
only mirrored the latest files. Feel free to beat
it up. The .gz files decompress to just under
4GB[1] -
https://ams.nochan.net/data.ris.ripe.net/rrc00/
|
> > > Bender
No interest in this mirror, gonna nuke it.
|
> johnhtodd
This is conflating BGP routes and DNS.DNS data:
Root server data is available via AXFR ("dig . AXFR
@f.root-servers.net") but this isn't what you're
referencing.Second level TLD server data is available
is available at CZDS. (https://czds.icann.org/home)
but some TLDs don't participate, but this also isn't
what you're referencing.What I think you want: There
is no canonical list of all zones that exist - there
is no "central repository" once you pass the root zone
downwards - that's a feature, not a bug. Some
organizations have partial views based on large
recursive resolver data (DomainTools, Google, Cisco,
Cloudflare, Quad9) but access to that data is limited
to vetted researchers or more typically only available
at a cost (disclaimer: I work for Quad9.) Smaller
versions of recursive data sets exist, but are usually
significantly limited by geography and demographics of
the user community that generates the data set.BGP
route data:
This exists in many forms in realtime like the site
referenced above, though historic data is difficult to
track. No matter what the source or latency, there is
bias in the data set because BGP pathing is unique to
each ASN that collects it - no two views of the table
are identical, and any data set is as "best guess" at
state conditions at that time.Here are some possible
data sets for BGP:Packet Clearing House (PCH) provides
a set of snapshots going back 20+ years (though it
seems to be offline at the moment):
https://www.pch.net/resources/Routing_Data/Cymru has a
live version you can query via various APIs (including
ironically via DNS):
https://www.team-cymru.com/ip-asn-mappingRouteviews
from University of Oregon also has a data set that is
widely used by researchers:
https://www.routeviews.org/routeviews/
|
> > Bender
This is conflating BGP routes and DNS.No I was
fantasizing that I could get these two things.
They are two separate things. If I had a nickle
for every time I said something off topic I could
by everyone a cup of coffee.
|
> toast0
> Just to have a compressed static file that contains
all the routes one could download from time to time I
think would be interesting to analyze.When I needed
this for some work stuff, it was pretty easy to find
table dumps and work with them? I don't remember where
it was, but I'm sure you can find some. After an
acquisition, we had our own ASN and I was able to get
table dumps from our own infrastructure.> Same goes
for the anycast root DNS servers. To have a full dump
from them could be interesting. Not to be confused
with the root.zone [1] I mean the whole kit and
caboodle.What do you think the root servers have
that's not in the root.zone? I think you can AXFR from
the root servers, too, but it should be the same thing
as from the https site.
|
> > Bender
What do you think the root servers have that's not
in the root.zone?Every glue record for every
domain name. The glue is the valuable goo. Without
the goo everyone goes back to /etc/hosts which I
am not opposed to.
|
> > > toast0
The root servers don't have the glue record
for every domain name though. All they have is
the glue records for the TLDs. (which is in
the zone file you linked)
|
> > > > Bender
Every time I stand up new name servers I
have to add the glue records into the root
servers or the name servers do not exist.
In fairness to me I keep forgetting the
trend is to shove everything into big
centralized DNS servers as it is something
I would never do at least not as a
primary.
|
> > > > > toast0
Afaik, those glue records are held at
your TLD's registry and served by the
tld nameservers, not the root
servers.It might be nice to get a zone
transfer for every tld, but that's not
possible for the public. (I understand
there's some way to get many of them,
but $$$$)
|
> > > > > > Bender
I run my own name servers. I never
use the name servers of a
registry. I can see the glue
records of my name servers in the
root servers. In fact the reason I
left NetworkSolutions (web.com)
was that their interface to update
the root servers broke and there
was nobody left that knew how to
fix it. I'm sure they must have
fixed it by now but I was being
impatient only waiting 3 weeks.I
should add that I have been adding
name servers to the root servers
since 1998. I've just never
managed one of the root servers
and I guess nobody on HN has
either.
|
> > > > > > toast0
I'm pretty sure you're confused.If
you query your domain at the root
servers, they will refer you to
the tld servers, run by your
registry.Then when you query your
domain at the tld servers, they
return your selected nameservers
along with their addresses, if in
their bailiwick.
|
> > > > > > Bender
Im not confused at all. For your
recursive servers to know how to
get the name servers they have to
have a hints file, that tells them
what to use for the tld, then from
the group of root servers
associated with that TLD you get
the glue (name servers)
assocociated with the domain, then
you query the authoritative name
servers for that domains resources
you were requesting. Every name
server has a glue record in the
root servers. I can manually walk
you through it with dig if you
would like.I need to clean up the
formatting a bit but this is a
walk through. [1] I think I know
where your confusion is but I will
wait for your reply.[1] -
https://blawg.nochan.net/b/Interne
t-Crap/20260530-DNS-Recurs...
|
> > > > > > toast0
> from the group of root servers
associated with that TLD you get
the glue (name servers)
assocociated with the domainThose
are not root servers. They are tld
servers, run by the registry for
the tld your domain is under.The
root servers serve the root. Root
hints just tell you where those
are. The zone file you listed has
the whole root
zone.a.root-servers.net is a root
server.a.gtld-servers.net is a tld
server (controlled by verisign,
iirc?). several tlds use
gtld-servers, but that doesn't
make them root servers.
|
> > > > > > Bender
TLD servers are just another level
of root servers. Each TLD does
have a primary registrar that
manages them however they provide
access to thousands of registrars
to manage the glue records at that
layer. They are still collectively
part of the root servers even if
they are not under the domain
root-servers.net. My registrar
does not manage them. Nobody here
is on a regisrar that manages the
TLD root servers. The only
permission my registrar and most
registrars have is to update glue
records for the domains their
domain clients own and to add
their domains. I and every DNS
admin I've ever know collectively
refers to all levels of the root
and TLD servers as root servers.
Even if that is technically
incorrect that is how we have
always referred to them. I think
that distinction would primarily
mater for people that worked at
one of the high level registrars
that manages the anycast clusters
of tld root servers. At least I
think that is where the confusion
started. This was fun we should do
this again. I updated the blawg to
add (TLD) in reference to root
servers.
|
> > > > > > toast0
If every level is a root, then the
word root means nothing. I guess
you could say that the root
servers point to the tld root
servers that point to your
domain's root servers. But then
root server just means dns
server.I never mentioned
registrars, only registries.>
Nobody here is on a regisrar that
manages the TLD root servers.I
have a .is domain. For .is and
some of the other country code
tlds, you contract directly with
the entity managing the TLD and
its servers.
|
> > > > > > Bender
Are you secretly the admin of
archive.is? He's made accounts on
here before but they were all
temporary. I want to believe he
has an account that has been here
a long time.
|
> > > > > > toast0
No, I live on an island, and is is
a usps acceptable abbreviation for
island ... I was trying to
convince my spouse to leave yahoo
mail for mail hosted on one of our
domains by getting a domain that's
less out there than enslaves.us,
so I have islandname.is (well not
literally that).Tldr, a cool
domain name isn't enough to make a
switch happen.
|
tiernano
Seems to be from 6 years ago.
|
johnea
Ignorant question here: Isn't this an open port for BGP
poisoning?
|
> ronsor
Only if they haven't enabled filtering, which would be
ludicrous.
|