feat: handle on-chain TLC settlement for force-closed channels

[doitian]

Closes #1222

Summary

When a force-closed channel settles an HTLC on-chain, the watchtower already records whether the claim included a preimage or not. This PR wires those signals into channel and payment state so off-chain flows can complete after an on-chain settlement.

• Success path (preimage revealed on-chain): On every MaintainChannelTlcs tick, settle_onchain_fulfilled_tlcs looks up watchtower-discovered preimages, marks the local TLC fulfilled, relays fulfillment upstream for forwarded TLCs, notifies the payer payment session, and marks the payee invoice paid when fulfilled received TLCs satisfy the invoice amount.
• Fail path (settled without preimage): Expired forwarded TLCs are failed only when is_tlc_settled_on_chain is set and no on-chain preimage was discovered — the two signals are mutually exclusive.
• Network fallback: CheckChannels uses collect_onchain_fulfillable_upstream_tlcs to fulfill upstream forwarded TLCs for force-closed channels whose actor is no longer live (e.g. after settlement is finalized).
• Restart / partial fulfill: sync_already_fulfilled_onchain_tlcs reconciles TLC status and invoice state for TLCs already marked fulfilled (e.g. after a partial off-chain fulfill on a closing channel).
• Store / watchtower: Split channel-store access into is_tlc_settled_on_chain and get_on_chain_discovered_preimage; add WatchtowerStore::is_tlc_settled and watchtower logging plus unit tests for preimage vs without-preimage on-chain claims.

Test plan

• cargo nextest run -p fnn -p fiber-bin
• test_closed_channel_upstream_fulfillment_from_onchain_preimage — forwarding node fulfills upstream TLC from watchtower preimage
• test_payer_payment_success_from_onchain_preimage — payer payment session reaches Success
• test_payee_invoice_paid_from_onchain_preimage — payee invoice status updated to Paid
• test_onchain_settlement_restart_restores_upstream_waiting_commitment_actor — node restart after downstream force-close still drives upstream fulfillment
• Watchtower unit tests for on-chain preimage discovery and without-preimage settlement markers

[gpBlockchain]

The current test has passed, but whether the results of list_channel and get_invoice need to be synchronizedly modified ？

lnd0->lnd1->fiber1->fiber2

1. fiber2 successfully settle on chain
2. fiber2.get_invoice -> Received

{"jsonrpc": "2.0", "id": 42, "result": {"invoice_address": "fibd1000001py803aefk6mczxzat8lhtajtd96je3ufftyatpn5tj7g7u3c933f8hzwguvwzu6575f6072fjah0gup4axf4ctf4lfjxnk4ztc7wteyymdrar0arre2sxqt5nljy4dfwnjutmhfuj6phrljsm6dn0cw3ws8atfv57zz7uk8kuqcv0su70celywz829ngu6wg44rpxpqsqdpsm2eckevnta9mqgylugwmaukud5unepswgpc9h8xfs2myah34qyykymt92hctzaw3z2z7zpn6j7m9ms8ldrhmjyn5l0585f000qx5g548092e30d98j473tmws9zkez9x7yx7d7aytd3zzxejzu6728t4qr07yxu85qgz6rzpuvxtjulafasqvx8nywfjj0c927g58lft489mqdxefshnu4faasz57gymh8d9s2rpaxrm4ua2cv2ga84agg9g0rzjjvu6phhxpdaq4kgz567hqaw6jcqp6vkkhr", "invoice": {"currency": "Fibd", "amount": "0x186a0", "signature": "0c060713040e0912120f18050a1e0814071f090b1507051b000d0619091017131c15091d1d1002141e08041b17070d05100a03011d06031b151c1d0a180c0a081d07151d080805080f030212120c1c1a01171706010d1d0015160802141a1e17001d0e1a12180001", "data": {"timestamp": "0x19d6bcc50e6", "payment_hash": "0xb00ce15c1a83174e2ea4576a4d7cfbcbe79ce1a6c364501c2910403ff635a116", "attrs": [{"description": "already settled invoice"}, {"final_htlc_minimum_expiry_delta": "0x927c00"}, {"udt_script": "0x55000000100000003000000031000000102583443ba6cfe5a3ac268bbb4475fb63eb497dce077f126ad3b148d4f4f8f8012000000032e555f3ff8e135cece1351a6a2971518392c1e30375c1e006ad0ce8eac07947"}, {"hash_algorithm": "sha256"}, {"payee_public_key": "03362ada447f3b1334fb1ee5786641c8deaba43e1dcc8e44eca3e2118872d1aba7"}]}}, "status": "Received"}}

3. fiber2.list_channel->"status": {"Inbound": "LocalRemoved"}}

{"jsonrpc": "2.0", "id": 42, "result": {"channels": [{"channel_id": "0xb346fe2c42093cec58eb420b66530feddbfd0003098ebd51c3b0e30dbd91a316", "is_public": true, "is_acceptor": false, "is_one_way": false, "channel_outpoint": "0xde25fb461c194c04237afb7380ca37a77a8a997927976ee49947a0136c28ef8d00000000", "pubkey": "02fd4cfea10f398c5e3a27c5f383ee9aa4d11fd82f35f28a3d9ce2994fe4b7c98f", "funding_udt_type_script": {"code_hash": "0x102583443ba6cfe5a3ac268bbb4475fb63eb497dce077f126ad3b148d4f4f8f8", "hash_type": "type", "args": "0x32e555f3ff8e135cece1351a6a2971518392c1e30375c1e006ad0ce8eac07947"}, "state": {"state_name": "Closed", "state_flags": "UNCOOPERATIVE_REMOTE"}, "local_balance": "0x19968ceb00", "offered_tlc_balance": "0x0", "remote_balance": "0x174876e800", "received_tlc_balance": "0x186a0", "pending_tlcs": [{"id": "0x0", "amount": "0x186a0", "payment_hash": "0xb00ce15c1a83174e2ea4576a4d7cfbcbe79ce1a6c364501c2910403ff635a116", "expiry": "0x19d94b5f5ff", "forwarding_channel_id": null, "forwarding_tlc_id": null, "status": {"Inbound": "LocalRemoved"}}], "latest_commitment_transaction_hash": "0x46cb82891d8bef1a1abe805614edd0b5c2221f72dcfd7852bd49168cd2e298f1", "created_at": "0x19d6bcc154e", "enabled": true, "tlc_expiry_delta": "0xdbba00", "tlc_fee_proportional_millionths": "0x3e8", "shutdown_transaction_hash": "0x9865f79e0da81cb20a73be1e99d04335e3ff10666cc35514e050f16c818e3922", "failure_detail": null}]}}

4. fiber1.list_channel -> "status": {"Outbound": "Committed"}

{"jsonrpc": "2.0", "id": 42, "result": {"channels": [{"channel_id": "0xb346fe2c42093cec58eb420b66530feddbfd0003098ebd51c3b0e30dbd91a316", "is_public": true, "is_acceptor": true, "is_one_way": false, "channel_outpoint": "0xde25fb461c194c04237afb7380ca37a77a8a997927976ee49947a0136c28ef8d00000000", "pubkey": "03362ada447f3b1334fb1ee5786641c8deaba43e1dcc8e44eca3e2118872d1aba7", "funding_udt_type_script": {"code_hash": "0x102583443ba6cfe5a3ac268bbb4475fb63eb497dce077f126ad3b148d4f4f8f8", "hash_type": "type", "args": "0x32e555f3ff8e135cece1351a6a2971518392c1e30375c1e006ad0ce8eac07947"}, "state": {"state_name": "Closed", "state_flags": "UNCOOPERATIVE_LOCAL"}, "local_balance": "0x174876e800", "offered_tlc_balance": "0x186a0", "remote_balance": "0x19968ceb00", "received_tlc_balance": "0x0", "pending_tlcs": [{"id": "0x0", "amount": "0x186a0", "payment_hash": "0xb00ce15c1a83174e2ea4576a4d7cfbcbe79ce1a6c364501c2910403ff635a116", "expiry": "0x19d94b5f5ff", "forwarding_channel_id": null, "forwarding_tlc_id": null, "status": {"Outbound": "Committed"}}], "latest_commitment_transaction_hash": "0x3a965b46b376ab99d592dad3ef02e1fb9107174cfc5d9b6bfa3d7f4c826abb28", "created_at": "0x19d6bcc1552", "enabled": true, "tlc_expiry_delta": "0xdbba00", "tlc_fee_proportional_millionths": "0x3e8", "shutdown_transaction_hash": "0x9865f79e0da81cb20a73be1e99d04335e3ff10666cc35514e050f16c818e3922", "failure_detail": null}]}}

[doitian]

TODO: check the invoice is fully paid.

[Copilot]

Pull request overview

Fixes a long-standing edge case in Fiber’s payment lifecycle where a peer’s RemoveTlc(Fulfill) can arrive after a channel has been force-closed and the channel actor is already gone, leaving the sender (and downstream CCH tracking) stuck in Inflight. The PR adds network-level handling to recover fulfillment from persisted channel state, plus tests to validate one-hop and two-hop on-chain settlement flows and a new watchtower Bruno e2e scenario.

Changes:

• Intercept RemoveTlc(Fulfill) for force-closed channels in NetworkActor::handle_peer_message, recover TLC info from persisted channel state, and propagate fulfillment upstream / to the payment actor.
• Ensure invoices are marked Paid when a fulfilled received-TLC is locally removed but the commitment round-trip never completes (common around force-closes).
• Add unit tests and register a new Bruno e2e watchtower scenario in CI.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
crates/fiber-lib/src/fiber/network.rs	Adds fallback handling for fulfill removes on closed channels and additional invoice-status reconciliation in periodic channel checks.
crates/fiber-lib/src/store/store_impl/mod.rs	Emits StoreChange::PutPreimage when inserting watchtower preimages so downstream watchers (e.g. CCH) can react.
crates/fiber-lib/src/fiber/tests/payment.rs	Adds one-hop and two-hop unit tests covering on-chain settlement after force-close.
.github/workflows/e2e.yml	Registers the new Bruno e2e scenario in CI.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/01-node1-connect-node2.bru	New e2e step: connect Node1↔Node2.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/02-node2-connect-node3.bru	New e2e step: connect Node2↔Node3.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/03-node1-node2-open-channel.bru	New e2e step: open Node1→Node2 channel.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/04-node2-get-auto-accepted-channel.bru	New e2e step: discover N1–N2 channel id.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/05-ckb-generate-blocks.bru	New e2e step: mine epochs for N1–N2 channel confirmation.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/06-node2-node3-open-channel.bru	New e2e step: open Node2→Node3 channel.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/07-node3-get-auto-accepted-channel.bru	New e2e step: discover N2–N3 channel id.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/08-ckb-generate-blocks.bru	New e2e step: mine epochs for N2–N3 channel confirmation.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/09-get-node1-funding-script.bru	New e2e step: read Node1 funding lock script.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/10-get-node3-funding-script.bru	New e2e step: read Node3 funding lock script.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/11-get-node1-balance.bru	New e2e step: snapshot Node1 chain balance.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/12-get-node3-balance.bru	New e2e step: snapshot Node3 chain balance.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/13-node3-gen-invoice.bru	New e2e step: create hold invoice / preimage pair for the scenario.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/14-node1-send-payment-with-invoice.bru	New e2e step: Node1 initiates payment via invoice.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/15-node2-force-close-channel.bru	New e2e step: Node2 force-closes N2–N3 to force on-chain resolution path.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/16-node2-disconnect-node3.bru	New e2e step: disconnect Node2 from Node3.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/17-ckb-generate-blocks-for-force-close-tx.bru	New e2e step: mine epochs for force-close tx confirmation.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/18-node3-remove-tlc.bru	New e2e step: Node3 removes TLC with preimage (fulfill).
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/19-ckb-generate-blocks-for-settlement-tx-preimage.bru	New e2e step: mine epochs for settlement tx/preimage discovery.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/20-ckb-generate-blocks-for-final-settlement-tx.bru	New e2e step: mine epochs to reach final settlement.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/21-ckb-generate-blocks-for-final-settlement-tx-commit.bru	New e2e step: mine epochs to commit final settlement.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/22-check-channel1-balance.bru	New e2e assertion: N1–N2 channel balance reflects claimed amount.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/23-check-payment-status.bru	New e2e assertion: payment status becomes Success.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/24-disconnect.bru	New e2e teardown: disconnect peers.

[gpBlockchain]

• need add test for: On one channel, both TLC transactions were successfully processed on the blockchain.

import time

from framework.basic_fiber import FiberTest
from framework.util import ckb_hash


class TestForceCloseMultiPayments(FiberTest):
    """
    Regression coverage for one force-closed channel with multiple pending TLCs.

    A sends multiple hold-invoice payments to B on the same channel. A then
    force-closes that channel. After B settles all invoices, every payment
    should reach Success and every invoice should reach Paid.
    """

    start_fiber_config = {"fiber_watchtower_check_interval_seconds": 3}

    def _wait_force_close_unlock(self, timeout=600):
        if len(self.get_commit_cells()) == 0:
            raise Exception("No commit cells found")
        self.node.getClient().generate_epochs("0x1", wait_time=0)
        for _ in range(timeout // 10):
            if len(self.get_commit_cells()) == 0:
                return
            time.sleep(10)
        assert len(self.get_commit_cells()) == 0

    def _get_tlc_status(self, fiber, remote_pubkey, channel_id, payment_hash):
        channels = fiber.get_client().list_channels(
            {"pubkey": remote_pubkey, "include_closed": True}
        )["channels"]
        for channel in channels:
            if channel["channel_id"] != channel_id:
                continue
            for tlc in channel.get("pending_tlcs", []):
                if tlc["payment_hash"] == payment_hash:
                    return tlc["status"]
        raise AssertionError(f"TLC {payment_hash} not found in channel {channel_id}")
    def test_one_channel_multiple_payments_force_close(self):
        self.open_channel(self.fiber1, self.fiber2, 1000 * 100000000, 0)

        payments = []
        for i in range(2):
            preimage = self.generate_random_preimage()
            payment_hash = ckb_hash(preimage)
            invoice = self.fiber2.get_client().new_invoice(
                {
                    "amount": hex(1 * 100000000),
                    "currency": "Fibd",
                    "description": f"one channel multiple payments {i}",
                    "payment_hash": payment_hash,
                    "allow_mpp": True,
                    "allow_trampoline_routing": True,
                }
            )
            payment = self.fiber1.get_client().send_payment(
                {
                    "invoice": invoice["invoice_address"],
                    "max_fee_rate": hex(1000000000000000),
                }
            )
            assert payment["payment_hash"] == payment_hash
            payments.append((payment_hash, preimage))

        for payment_hash, _ in payments:
            self.wait_payment_state(self.fiber1, payment_hash, "Inflight")
            self.wait_invoice_state(self.fiber2, payment_hash, "Received")

        channels = self.fiber1.get_client().list_channels(
            {"pubkey": self.fiber2.get_pubkey()}
        )["channels"]
        assert len(channels) > 0
        channel_id = channels[0]["channel_id"]
        self.fiber1.get_client().shutdown_channel(
            {"channel_id": channel_id, "force": True}
        )

        time.sleep(10)
        for payment_hash, preimage in payments:
            self.fiber2.get_client().settle_invoice(
                {"payment_hash": payment_hash, "payment_preimage": preimage}
            )
        self._wait_force_close_unlock()

        for payment_hash, _ in payments:
            self.wait_payment_state(self.fiber1, payment_hash, "Success", timeout=360)
            self.wait_invoice_state(self.fiber2, payment_hash, "Paid", timeout=360)
            assert self._get_tlc_status(
                self.fiber1,
                self.fiber2.get_pubkey(),
                channel_id,
                payment_hash,
            ) == {"Outbound": "RemoteRemoved"}
            assert self._get_tlc_status(
                self.fiber2,
                self.fiber1.get_pubkey(),
                channel_id,
                payment_hash,
            ) == {"Inbound": "LocalRemoved"}

test reuslt : Only one transaction successfully changed its status

[gpBlockchain]

• Restarting the node causes the status not to be updated

    def test_one_hop_payer_restart_before_payee_settle_invoice(self):
        """
        A -> B CKB payment, payer restarts after force-close.

        A force-closes the channel and then restarts before B reveals the
        preimage. The final Success status should still be recovered from
        persisted channel/payment state, not in-memory channel actor state.
        The closed channel should show RemoteRemoved on A and LocalRemoved on B.
        """
        self.open_channel(self.fiber1, self.fiber2, 1000 * 100000000, 0)

        preimage = self.generate_random_preimage()
        payment_hash = ckb_hash(preimage)
        invoice = self.fiber2.get_client().new_invoice(
            {
                "amount": hex(1 * 100000000),
                "currency": "Fibd",
                "description": "pr1254 p2 payer restart hold invoice",
                "payment_hash": payment_hash,
                "allow_mpp": True,
                "allow_trampoline_routing": True,
            }
        )
        payment = self.fiber1.get_client().send_payment(
            {
                "invoice": invoice["invoice_address"],
                "max_fee_rate": hex(1000000000000000),
            }
        )
        assert payment["payment_hash"] == payment_hash
        self.wait_payment_state(self.fiber1, payment_hash, "Inflight")
        self.wait_invoice_state(self.fiber2, payment_hash, "Received")

        channels = self.fiber1.get_client().list_channels(
            {"pubkey": self.fiber2.get_pubkey()}
        )["channels"]
        assert len(channels) > 0
        channel_id = channels[0]["channel_id"]
        self.fiber1.get_client().shutdown_channel(
            {"channel_id": channel_id, "force": True}
        )

        time.sleep(10)
        self._restart_fiber(self.fiber1)

        self.fiber2.get_client().settle_invoice(
            {"payment_hash": payment_hash, "payment_preimage": preimage}
        )
        self._wait_unlock()

        self.wait_payment_state(self.fiber1, payment_hash, "Success", timeout=300)
        self.wait_invoice_state(self.fiber2, payment_hash, "Paid", timeout=300)
        self._assert_sender_remote_removed(
            self.fiber1, self.fiber2, channel_id, payment_hash
        )
        self._assert_receiver_local_removed(
            self.fiber2, self.fiber1, channel_id, payment_hash
        )
        self._assert_success_and_paid(self.fiber1, self.fiber2, payment_hash)

[doitian]

I'm not sure is it too heavy to run these in CheckChannels. Also the code contains many logic similar to existing code blocks with nuance.

[doitian]

This is a test/bench command before and is promoted to release environment here to synchronize the state in case the actor is still alive.

[doitian]

Turn to draft since there are complex edge cases not fixed yet. See test cases post by @gpBlockchain above.

[doitian]

Need to investigate the performance regression Done