Building a node crawler to map network



Summary:

A Java developer started building a node crawler for Bitcoin to map its network and provide useful statistics. Currently, the crawler stores version message and results of getaddr for each node which can be connected to but the developer is seeking input on more useful information that can be extracted from a node. The main issue is how to connect since most nodes do not accept incoming connections due to lack of upnp. Active crawl approach is thus not ideal as it would result in hopelessly distorted data. The developer suggests a honeypot approach where a node is made attractive to other nodes to connect to in order to capture non-listening nodes. If this is not possible, another approach could be building an easy-to-deploy crawler node that many volunteers could run and upload collected data to a central repository. Lastly, the developer seeks clarification on the getaddr message - where does the list of 1000 addresses come from and does it include some nodes discovered by IRC/DNS?


Updated on: 2023-06-04T18:55:58.601845+00:00