Merge bitcoin/bitcoin#31553: cluster mempool: add TxGraph reorg functionality

1632fc1 txgraph: Track multiple potential would-be clusters in Trim (improvement) (Pieter Wuille)
4608df3 txgraph: add Trim benchmark (benchmark) (Pieter Wuille)
9c436ff txgraph: add fuzz test scenario that avoids cycles inside Trim() (tests) (Pieter Wuille)
938e86f txgraph: add unit test for TxGraph::Trim (tests) (glozow)
a04e205 txgraph: Add ability to trim oversized clusters (feature) (Pieter Wuille)
eabcd0e txgraph: remove unnecessary m_group_oversized (simplification) (Greg Sanders)
19b14e6 txgraph: Permit transactions that exceed cluster size limit (feature) (Pieter Wuille)
c4287b9 txgraph: Add ability to configure maximum cluster size/weight (feature) (Pieter Wuille)

Pull request description:

  Part of cluster mempool (#30289).

  During reorganisations, it is possible that dependencies get added which would result in clusters that violate policy limits (cluster count, cluster weight), when linking the new from-block transactions to the old from-mempool transactions. Unlike RBF scenarios, we cannot simply reject the changes when they are due to received blocks. To accommodate this, add a `TxGraph::Trim()`, which removes some subset of transactions (including descendants) in order to make all resulting clusters satisfy the limits.

  Conceptually, the way this is done is by defining a rudimentary linearization for the entire would-be too-large cluster, iterating it from beginning to end, and reasoning about the counts and weights of the clusters that would be reached using transactions up to that point. If a transaction is encountered whose addition would violate the limit, it is removed, together with all its descendants.

  This rudimentary linearization is like a merge sort of the chunks of the clusters being combined, but respecting topology. More specifically, it is continuously picking the highest-chunk-feerate remaining transaction among those which have no unmet dependencies left. For efficiency, this rudimentary linearization is computed lazily, by putting all viable transactions in a heap, sorted by chunk feerate, and adding new transactions to it as they become viable.

  The `Trim()` function is rather unusual compared to the `TxGraph` functionality added in previous PRs, in that `Trim()` makes it own decisions about what the resulting graph contents will be, without good specification of how it makes that decision - it is just a best-effort attempt (which is improved in the last commit). All other `TxGraph` mutators are simply to inform the graph about changes the calling mempool code decided on; this one lets the decision be made by txgraph.

  As part of this, the "oversized" property is expanded to also encompass a configurable cluster weight limit (in addition to cluster count limit).

ACKs for top commit:
  instagibbs:
    reACK 1632fc1
  glozow:
    reACK 1632fc1 via range-diff
  ismaelsadeeq:
    reACK 1632fc1 🛰️

Tree-SHA512: ccacb54be8ad622bd2717905fc9b7e42aea4b07f824de1924da9237027a97a9a2f1b862bc6a791cbd2e1a01897ad2c7c73c398a2d5ccbce90bfbeac0bcebc9ce

Author: glozow

src/bench/CMakeLists.txt

@@ -48,6 +48,7 @@ add_executable(bench_bitcoin
   sign_transaction.cpp
   streams_findbyte.cpp
   strencodings.cpp
+  txgraph.cpp
   util_time.cpp
   verify_script.cpp
   xor.cpp

src/bench/txgraph.cpp

@@ -0,0 +1,123 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <bench/bench.h>
+#include <random.h>
+#include <txgraph.h>
+#include <util/feefrac.h>
+
+#include <cassert>
+#include <cstdint>
+
+namespace {
+
+void BenchTxGraphTrim(benchmark::Bench& bench)
+{
+    // The from-block transactions consist of 1000 fully linear clusters, each with 64
+    // transactions. The mempool contains 11 transactions that together merge all of these into
+    // a single cluster.
+    //
+    // (1000 chains of 64 transactions, 64000 T's total)
+    //
+    //      T          T          T          T          T          T          T          T
+    //      |          |          |          |          |          |          |          |
+    //      T          T          T          T          T          T          T          T
+    //      |          |          |          |          |          |          |          |
+    //      T          T          T          T          T          T          T          T
+    //      |          |          |          |          |          |          |          |
+    //      T          T          T          T          T          T          T          T
+    //  (64 long)  (64 long)  (64 long)  (64 long)  (64 long)  (64 long)  (64 long)  (64 long)
+    //      |          |          |          |          |          |          |          |
+    //      |          |         / \         |         / \         |          |         /
+    //      \----------+--------/   \--------+--------/   \--------+-----+----+--------/
+    //                 |                     |                           |
+    //                 B                     B                           B
+    //
+    //  (11 B's, each attaching to up to 100 chains of 64 T's)
+    //
+    /** The maximum cluster count used in this test. */
+    static constexpr int MAX_CLUSTER_COUNT = 64;
+    /** The number of "top" (from-block) chains of transactions. */
+    static constexpr int NUM_TOP_CHAINS = 1000;
+    /** The number of transactions per top chain. */
+    static constexpr int NUM_TX_PER_TOP_CHAIN = MAX_CLUSTER_COUNT;
+    /** The (maximum) number of dependencies per bottom transaction. */
+    static constexpr int NUM_DEPS_PER_BOTTOM_TX = 100;
+    /** Set a very large cluster size limit so that only the count limit is triggered. */
+    static constexpr int32_t MAX_CLUSTER_SIZE = 100'000 * 100;
+
+    /** Refs to all top transactions. */
+    std::vector<TxGraph::Ref> top_refs;
+    /** Refs to all bottom transactions. */
+    std::vector<TxGraph::Ref> bottom_refs;
+    /** Indexes into top_refs for some transaction of each component, in arbitrary order.
+     *  Initially these are the last transactions in each chains, but as bottom transactions are
+     *  added, entries will be removed when they get merged, and randomized. */
+    std::vector<size_t> top_components;
+
+    InsecureRandomContext rng(11);
+    auto graph = MakeTxGraph(MAX_CLUSTER_COUNT, MAX_CLUSTER_SIZE);
+
+    // Construct the top chains.
+    for (int chain = 0; chain < NUM_TOP_CHAINS; ++chain) {
+        for (int chaintx = 0; chaintx < NUM_TX_PER_TOP_CHAIN; ++chaintx) {
+            int64_t fee = rng.randbits<27>() + 100;
+            FeePerWeight feerate{fee, 1};
+            top_refs.push_back(graph->AddTransaction(feerate));
+            // Add internal dependencies linking the chain transactions together.
+            if (chaintx > 0) {
+                 graph->AddDependency(*(top_refs.rbegin()), *(top_refs.rbegin() + 1));
+            }
+        }
+        // Remember the last transaction in each chain, to attach the bottom transactions to.
+        top_components.push_back(top_refs.size() - 1);
+    }
+
+    // Make the graph linearize all clusters acceptably.
+    graph->GetBlockBuilder();
+
+    // Construct the bottom transactions, and dependencies to the top chains.
+    while (top_components.size() > 1) {
+        // Construct the transaction.
+        int64_t fee = rng.randbits<27>() + 100;
+        FeePerWeight feerate{fee, 1};
+        auto bottom_tx = graph->AddTransaction(feerate);
+        // Determine the number of dependencies this transaction will have.
+        int deps = std::min<int>(NUM_DEPS_PER_BOTTOM_TX, top_components.size());
+        for (int dep = 0; dep < deps; ++dep) {
+            // Pick an transaction in top_components to attach to.
+            auto idx = rng.randrange(top_components.size());
+            // Add dependency.
+            graph->AddDependency(/*parent=*/top_refs[top_components[idx]], /*child=*/bottom_tx);
+            // Unless this is the last dependency being added, remove from top_components, as
+            // the component will be merged with that one.
+            if (dep < deps - 1) {
+                // Move entry top the back.
+                if (idx != top_components.size() - 1) std::swap(top_components.back(), top_components[idx]);
+                // And pop it.
+                top_components.pop_back();
+            }
+        }
+        bottom_refs.push_back(std::move(bottom_tx));
+    }
+
+    // Run the benchmark exactly once. Running it multiple times would require the setup to be
+    // redone, which takes a very non-negligible time compared to the trimming itself.
+    bench.epochIterations(1).epochs(1).run([&] {
+        // Call Trim() to remove transactions and bring the cluster back within limits.
+        graph->Trim();
+        // And relinearize everything that remains acceptably.
+        graph->GetBlockBuilder();
+    });
+
+    assert(!graph->IsOversized());
+    // At least 99% of chains must survive.
+    assert(graph->GetTransactionCount() >= (NUM_TOP_CHAINS * NUM_TX_PER_TOP_CHAIN * 99) / 100);
+}
+
+} // namespace
+
+static void TxGraphTrim(benchmark::Bench& bench) { BenchTxGraphTrim(bench); }
+
+BENCHMARK(TxGraphTrim, benchmark::PriorityLevel::HIGH);

src/test/CMakeLists.txt

@@ -106,6 +106,7 @@ add_executable(test_bitcoin
   transaction_tests.cpp
   translation_tests.cpp
   txdownload_tests.cpp
+  txgraph_tests.cpp
   txindex_tests.cpp
   txpackage_tests.cpp
   txreconciliation_tests.cpp