Published on: 2011-09-06T15:25:09+00:00
The discussion revolves around the scalability of Bitcoin and how to handle an increase in popularity. The issue is that as more users migrate to using lightweight clients, it can lead to running out of sockets. Lightweight clients cannot verify or relay, so they use "heard from lots of nodes" as a proxy for validity. This leads to them being socket-intensive and potentially causing problems with full nodes.One solution proposed is to use frontend proxies, which would handle things like outbound tx broadcasts, deduping inbound broadcasts, connection setup, relaying recently found blocks, etc. This would free up the bitcoinds time for verification and wallet manipulation. A well-written proxy could support tens of thousands of simultaneous clients.Steve questions the use case of increasing the default max connections for bitcoind without fundamentally altering the concurrency model. The issue arises in the context of scalability, as lightweight clients would be used more frequently if Bitcoin's popularity continued to grow. These clients cannot verify transactions and should not relay them, thus making them socket intensive. This could result in running out of sockets, especially since lightweight clients rely on "heard from lots of nodes" as a proxy for validity in order to show transactions immediately.One solution to this problem is to use frontend proxies, which has been used by companies like Jabber and Google in the past. If someone is already running a full node, they could increase their client capacity by bringing up a frontend proxy that can handle outbound tx broadcasts, inbound broadcasts, connection setup, relaying recently found blocks, etc. A well-written proxy could support tens of thousands of simultaneous clients, freeing up bitcoinds time for verification and wallet manipulation.Rick has developed minna patches for nio based on bitcoinj and is working on a DNS seed service as well as some weather reports. He is happy to start a branch when the committers are ready. On the other hand, Steve started building a node crawler to map out the bitcoin network and provide useful statistics using a mutilated bitcoinj to connect. He is hoping to get input into what would be useful as well as strategy for getting the maximum possible connections without distorted data.Steve says that the vast majority of nodes do not accept incoming connections due to the lack of upnp, so it seems the active crawl approach is not ideal for the purpose. A honeypot approach would probably be better if there was some way to make a node 'attractive' to other nodes to connect to. If there is a way to influence other nodes to connect to the crawler node that solves the problem. If there isn't, then perhaps another approach is to build an easy-to-deploy crawler node that many volunteers could run and that could then upload collected data to a central repository.Steve is also asking questions regarding the getaddr message and how most nodes return about 1000 addresses in response to this message. He wonders where this list comes from, whether it's a mixture of addresses obtained from other nodes somehow sorted by timestamp, and whether it includes some nodes discovered by IRC/DNS or if those are only used to find the first nodes to connect to. Finally, he is looking for input to build something that's useful for the network.In an email exchange, Mike Hearn suggests to a developer that they could experiment with writing a proxy that sits in front of bitcoind and multiplexes connections. The goal is to reduce socket exhaustion as users move to lightweight clients. BitCoinJ uses thread-per-connection so wouldn't do a good job of that right now, but allowing it to use a mix of async io and multi-threading would be a nice improvement. It'd need some changes to bitcoind as well for a really good effort, to allow for IPs to be forwarded.Later on, the developer responds saying that they are not confident enough to submit patches for bitcoinj yet, but building a pool protocol using protobufs and netty could potentially lead to implementing the kind of multiplexing proxy that Mike is talking about. However, the developer is unsure of the use case and whether the goal is to increase the default max connections of 8 without fundamentally altering the bitcoind concurrency model, or to provide capacity for a more hub/client oriented network. If the latter is the case, then this functionality should ideally be native to the client in the long term in the form of NIO.The person commenting is looking for a post that they cannot seem to find. They mention having some free time and are waiting for several pools to finish testing and move into production. They express an interest in contributing patches to BitCoinJ or experimenting with writing a proxy to sit in front of bitcoind and multiplex connections, something Gavin has been concerned about recently. The commenter notes that BitCoinJ uses thread-per-connection and allowing it to use a mix of async io and multi-threading would be a nice improvement, but changes would need to be made to bitcoind as well.
Updated on: 2023-08-01T02:24:08.437369+00:00