Published on: 2015-12-04T13:30:33+00:00
In a Bitcoin developer mailing list, concerns were raised about the potential complexity and attack surface that may come with implementing the Lempel-Ziv (LZ) family of compressors. Despite the minor compression gain of 15 to 27%, it was suggested that syncing over slow or restrictive connections could be better addressed by a sync-over-http-via-cdn protocol. However, arguments were made in favor of compression, stating that larger data sizes benefit from better compression and that adding compression could be easily turned off if necessary.The discussion also touched on the limitations of the LZ family of compressors for binary data like the Bitcoin wire protocol. It was proposed that a custom, Bitcoin-aware compressor could achieve significantly higher compression ratios of 2X or more in some cases. However, concerns were raised about the maintainability and potential difficulties in changing the wire format later on. Despite these concerns, the idea of compressing Bitcoin was seen as an opportunity for free improvements and additional throughput without any cost.A programming challenge/contest was suggested as one way to find the best possible, Bitcoin-specific compressor. This would not only lead to potential improvements but also bring new programmers into the ecosystem. The idea of building a standardized dataset using real data from the network to test against was also discussed, allowing for measurements of proposed optimizations.Another email conversation proposed the idea of adding compression directly accessible to the network on financial software. While there were concerns about security issues, it was argued that the LZO compressor has been around for 20 years with few problems. The performance improvement through compression was shown to increase as block sizes increased, making it beneficial for transmission but not for post-processing and validating blocks.Further discussions highlighted the potential benefits of a custom, Bitcoin-specific compressor compared to the existing LZ family of compressors. While LZ compressors assign unique numbers to subsequence encounters, they are not as efficient for binary data like Bitcoin. Building a custom compressor could lead to higher compression ratios, although there are considerations regarding maintainability and potential difficulties in changing the wire format. The idea of a programming challenge/contest to find the best possible Bitcoin-specific compressor was seen as an effective way to discover the limits of compressibility and bring in new programmers.The email exchanges also addressed concerns about the security of compression libraries and the possibility of exposing them to potential attackers. Various suggestions were made, such as isolating the decompression phase and verifying block hashes through external processes or daemons.In summary, there is ongoing discussion among Bitcoin developers regarding the implementation of compression for Bitcoin data transmission. While there are concerns about complexity, attack surface, and potential security issues, there is also recognition of the potential benefits in terms of improved performance and additional throughput. The idea of a custom, Bitcoin-specific compressor has been proposed, along with the suggestion of a programming challenge/contest to find the best possible solution. Overall, the goal is to find the most efficient and secure method of compressing Bitcoin data to enhance network efficiency and scalability.
Updated on: 2023-08-01T16:57:46.530241+00:00