feat: sync from unfinalized checkpoint state #8527

nflaig · 2025-10-13T12:08:45Z

Motivation

This was a feature we developed for rescuing Holesky as part of #7501 to quickly sync nodes to head during a period of long non-finality (~3 weeks). While it's unlikely we will have such a long period of non-finality on mainnet, this feature is still useful to have for much shorter periods and testing purposes on devnets.

It is now part of Ethereum protocol hardening mitigations described here

Ordinary checkpoint sync begins from the latest finalized checkpoint (block and/or state). As an escape hatch during non-finality, it is useful to have the ability to checkpoint sync from an unfinalized checkpoint. A client implementing this mitigation MUST support checkpoint sync from an arbitrary non-finalized checkpoint state.

We will support this with the exception that our checkpoint state needs to be an epoch boundary checkpoint.

Description

The main feature of this PR is to allow initializing a node from an unfinalized checkpoint state either retrieved locally or from a remote source.

This behavior is disabled by default but can be enabled by either adding

the --lastPersistedCheckpointState flag to load from the last safe persisted checkpoint state stored locally
or --unsafeCheckpointState to provide a file path or url to an unfinalized checkpoint state to start syncing from which can be used with new endpoint GET /eth/v1/lodestar/persisted_checkpoint_state to sync from a remote node or by sharing states from checkpoint_states folder

Both of these options are not safe to use on a network that recently finalized an epoch and must only be considered if syncing from last finalized checkpoint state is unfeasible.

An unfinalized checkpoint state persisted locally is only considered to be safe to boot if

it's the only checkpoint in it's epoch to avoid ambiguity from forks
its last processed block slot is at an epoch boundary or last slot of previous epoch
state slot is at an epoch boundary
state slot is equal to epoch * SLOTS_PER_EPOCH

But even if these criteria are met, there is chance that the node will end up on a minority chain as it will not be able to pivot to another chain that conflicts with the checkpoint state it was initialized from.

Other existing flags (like --checkpointState) are unchanged by this PR and will continue to expect a finalized checkpoint state.

Previous PRs #7509, #7541, #7542 not merged to unstable are included.

Closes #7963

cc @twoeths

**Motivation** - we already have local checkpoint states in either db or file, this PR use one of them to save syncing time the next time the node is restarted - see #7504 (comment) **Description** 2 new init unfinalized state options: - new `--lastPersistedCheckpointState`, true by default to load from the last safe persisted checkpoint state. And the last safe persisted checkpoint state is considered unfinalized - use `--checkpointState` option: this is old option and support finalized state only, this PR supports booting from unfinalized state similar to `--lastPersistedCheckpointState` ==> to quickly sync a new node, we can use this option. Then remove it and the next time node will use `--lastPersistedCheckpointState` option by default - we can configure one of our nodes with `chain.nHistoricalStatesFileDataStore = true` - then feed other nodes with a persisted "safe checkpoint state" from there in `~/checkpoint_states` folder - a persisted checkpoint state is consider to be safe to boot if - it should be the checkpoint state that's unique in its epoch - its last processed block slot should be at epoch boundary or last slot of previous epoch - state slot should be at epoch boundary - state slot should be equal to epoch * SLOTS_PER_EPOCH other options to boot from `stateArchived` or `checkpointSyncUrl` are considered finalized states including `wssCheckpoint`. It's not possible to use `wssCheckpoint` option with unfinalized state for now. **TODO** - this is for `holesky-rescue`, consider supporting this for `unstable` branch too - update document in that case --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>

**Motivation** - implement an api to get a node synced asap **Description** - new api: `eth/v1/lodestar/persisted_checkpoint_state` to return a state based on an optional `rootHex:epoch` param - if not specified, return the latest safe checkpoint state - a node need to specify `--checkpointState` from the previous PR #7509 **Test** - [x] `curl -H "Accept: application/octet-stream" http://localhost:9596/eth/v1/lodestar/persisted_checkpoint_state -o latest_checkpoint_state.ssz` - [x] `curl -H "Accept: application/octet-stream" http://localhost:9596/eth/v1/lodestar/persisted_checkpoint_state?checkpoint_id=0x4f4d4c1b81141fe77a4a1c6a376dbe64ed9baa8f123664195e4f710c9fc4238d:118936 -o state_epoch_118936.ssz` --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com>

…ied (#7542) We have added a new api in #7541 to serve latest checkpoint states. However the problem is that `--checkpointState` right now assumes that we receive a ssz-serialized state bytes, so we also need to ensure that the REST API only returns SSZ responses by applying the correct Accept header. This PR includes the following changes - Ensure we only receive SSZ responses if REST API is queried - Consider `--forceCheckpointSync` flag and if set always prioritze remote checkpoint from `--checkpointState` over latest safe checkpoint in db - Minor tweaks to logs to make it easier to understand what the node is doing - Misc formatting changes

codecov · 2025-10-13T12:25:29Z

Codecov Report

❌ Patch coverage is 17.47126% with 359 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.98%. Comparing base (21fbdc9) to head (858013d).
⚠️ Report is 4 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #8527      +/-   ##
============================================
- Coverage     52.18%   51.98%   -0.20%     
============================================
  Files           852      852              
  Lines         65317    65703     +386     
  Branches       4793     4800       +7     
============================================
+ Hits          34083    34156      +73     
- Misses        31165    31478     +313     
  Partials         69       69

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-10-13T13:05:05Z

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: `227ad29`	Previous: `21fbdc9`	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	961.85 us/op	1.0463 ms/op	0.92
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	34.908 us/op	37.791 us/op	0.92
BLS verify - blst	1.2890 ms/op	988.95 us/op	1.30
BLS verifyMultipleSignatures 3 - blst	1.4634 ms/op	1.3144 ms/op	1.11
BLS verifyMultipleSignatures 8 - blst	2.4642 ms/op	1.9449 ms/op	1.27
BLS verifyMultipleSignatures 32 - blst	7.2448 ms/op	5.7441 ms/op	1.26
BLS verifyMultipleSignatures 64 - blst	10.008 ms/op	9.9465 ms/op	1.01
BLS verifyMultipleSignatures 128 - blst	16.683 ms/op	18.357 ms/op	0.91
BLS deserializing 10000 signatures	642.14 ms/op	715.58 ms/op	0.90
BLS deserializing 100000 signatures	6.4376 s/op	7.0529 s/op	0.91
BLS verifyMultipleSignatures - same message - 3 - blst	895.50 us/op	1.0645 ms/op	0.84
BLS verifyMultipleSignatures - same message - 8 - blst	1.0899 ms/op	1.2256 ms/op	0.89
BLS verifyMultipleSignatures - same message - 32 - blst	1.7009 ms/op	1.7380 ms/op	0.98
BLS verifyMultipleSignatures - same message - 64 - blst	2.5108 ms/op	2.9402 ms/op	0.85
BLS verifyMultipleSignatures - same message - 128 - blst	4.3824 ms/op	4.5769 ms/op	0.96
BLS aggregatePubkeys 32 - blst	20.397 us/op	20.005 us/op	1.02
BLS aggregatePubkeys 128 - blst	66.089 us/op	71.419 us/op	0.93
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	58.881 ms/op	54.205 ms/op	1.09
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	43.243 ms/op	56.620 ms/op	0.76
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	44.243 ms/op	40.723 ms/op	1.09
getSlashingsAndExits - default max	68.856 us/op	81.006 us/op	0.85
getSlashingsAndExits - 2k	294.85 us/op	318.12 us/op	0.93
isKnown best case - 1 super set check	194.00 ns/op	201.00 ns/op	0.97
isKnown normal case - 2 super set checks	193.00 ns/op	200.00 ns/op	0.96
isKnown worse case - 16 super set checks	193.00 ns/op	200.00 ns/op	0.96
InMemoryCheckpointStateCache - add get delete	2.2700 us/op	2.3550 us/op	0.96
validate api signedAggregateAndProof - struct	2.3974 ms/op	1.3769 ms/op	1.74
validate gossip signedAggregateAndProof - struct	2.5987 ms/op	2.5849 ms/op	1.01
batch validate gossip attestation - vc 640000 - chunk 32	115.89 us/op	113.48 us/op	1.02
batch validate gossip attestation - vc 640000 - chunk 64	101.74 us/op	99.776 us/op	1.02
batch validate gossip attestation - vc 640000 - chunk 128	93.246 us/op	93.839 us/op	0.99
batch validate gossip attestation - vc 640000 - chunk 256	92.890 us/op	97.967 us/op	0.95
pickEth1Vote - no votes	920.38 us/op	1.0295 ms/op	0.89
pickEth1Vote - max votes	6.4938 ms/op	5.9589 ms/op	1.09
pickEth1Vote - Eth1Data hashTreeRoot value x2048	11.340 ms/op	11.820 ms/op	0.96
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	30.764 ms/op	19.033 ms/op	1.62
pickEth1Vote - Eth1Data fastSerialize value x2048	419.98 us/op	424.30 us/op	0.99
pickEth1Vote - Eth1Data fastSerialize tree x2048	3.4378 ms/op	2.3873 ms/op	1.44
bytes32 toHexString	342.00 ns/op	396.00 ns/op	0.86
bytes32 Buffer.toString(hex)	230.00 ns/op	272.00 ns/op	0.85
bytes32 Buffer.toString(hex) from Uint8Array	358.00 ns/op	340.00 ns/op	1.05
bytes32 Buffer.toString(hex) + 0x	276.00 ns/op	252.00 ns/op	1.10
Object access 1 prop	0.11400 ns/op	0.11800 ns/op	0.97
Map access 1 prop	0.11800 ns/op	0.12500 ns/op	0.94
Object get x1000	5.7550 ns/op	6.1070 ns/op	0.94
Map get x1000	6.1380 ns/op	6.4420 ns/op	0.95
Object set x1000	26.871 ns/op	31.242 ns/op	0.86
Map set x1000	18.347 ns/op	20.369 ns/op	0.90
Return object 10000 times	0.27530 ns/op	0.29070 ns/op	0.95
Throw Error 10000 times	4.0400 us/op	4.5271 us/op	0.89
toHex	125.70 ns/op	142.73 ns/op	0.88
Buffer.from	113.98 ns/op	134.81 ns/op	0.85
shared Buffer	75.226 ns/op	87.564 ns/op	0.86
fastMsgIdFn sha256 / 200 bytes	2.0720 us/op	2.3590 us/op	0.88
fastMsgIdFn h32 xxhash / 200 bytes	207.00 ns/op	227.00 ns/op	0.91
fastMsgIdFn h64 xxhash / 200 bytes	252.00 ns/op	266.00 ns/op	0.95
fastMsgIdFn sha256 / 1000 bytes	7.0900 us/op	7.2990 us/op	0.97
fastMsgIdFn h32 xxhash / 1000 bytes	333.00 ns/op	352.00 ns/op	0.95
fastMsgIdFn h64 xxhash / 1000 bytes	326.00 ns/op	343.00 ns/op	0.95
fastMsgIdFn sha256 / 10000 bytes	65.727 us/op	65.902 us/op	1.00
fastMsgIdFn h32 xxhash / 10000 bytes	1.7780 us/op	1.8200 us/op	0.98
fastMsgIdFn h64 xxhash / 10000 bytes	1.1900 us/op	1.2060 us/op	0.99
send data - 1000 256B messages	15.163 ms/op	18.494 ms/op	0.82
send data - 1000 512B messages	18.654 ms/op	19.882 ms/op	0.94
send data - 1000 1024B messages	26.355 ms/op	28.026 ms/op	0.94
send data - 1000 1200B messages	23.905 ms/op	26.732 ms/op	0.89
send data - 1000 2048B messages	24.736 ms/op	24.819 ms/op	1.00
send data - 1000 4096B messages	29.690 ms/op	28.585 ms/op	1.04
send data - 1000 16384B messages	46.285 ms/op	43.809 ms/op	1.06
send data - 1000 65536B messages	111.32 ms/op	113.31 ms/op	0.98
enrSubnets - fastDeserialize 64 bits	844.00 ns/op	989.00 ns/op	0.85
enrSubnets - ssz BitVector 64 bits	319.00 ns/op	331.00 ns/op	0.96
enrSubnets - fastDeserialize 4 bits	124.00 ns/op	136.00 ns/op	0.91
enrSubnets - ssz BitVector 4 bits	330.00 ns/op	353.00 ns/op	0.93
prioritizePeers score -10:0 att 32-0.1 sync 2-0	223.99 us/op	238.56 us/op	0.94
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	250.52 us/op	268.27 us/op	0.93
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	360.77 us/op	378.91 us/op	0.95
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	678.48 us/op	705.83 us/op	0.96
prioritizePeers score 0:0 att 64-1 sync 4-1	815.23 us/op	853.89 us/op	0.95
array of 16000 items push then shift	1.5564 us/op	1.6224 us/op	0.96
LinkedList of 16000 items push then shift	6.9530 ns/op	7.3620 ns/op	0.94
array of 16000 items push then pop	72.039 ns/op	77.305 ns/op	0.93
LinkedList of 16000 items push then pop	6.8920 ns/op	7.2270 ns/op	0.95
array of 24000 items push then shift	2.2960 us/op	2.4017 us/op	0.96
LinkedList of 24000 items push then shift	6.9800 ns/op	7.5460 ns/op	0.92
array of 24000 items push then pop	95.725 ns/op	115.00 ns/op	0.83
LinkedList of 24000 items push then pop	6.8380 ns/op	7.7840 ns/op	0.88
intersect bitArray bitLen 8	6.1690 ns/op	6.5420 ns/op	0.94
intersect array and set length 8	34.644 ns/op	38.009 ns/op	0.91
intersect bitArray bitLen 128	28.849 ns/op	29.772 ns/op	0.97
intersect array and set length 128	589.45 ns/op	629.94 ns/op	0.94
bitArray.getTrueBitIndexes() bitLen 128	996.00 ns/op	1.0510 us/op	0.95
bitArray.getTrueBitIndexes() bitLen 248	1.7720 us/op	2.0310 us/op	0.87
bitArray.getTrueBitIndexes() bitLen 512	3.5660 us/op	3.7970 us/op	0.94
Buffer.concat 32 items	719.00 ns/op	630.00 ns/op	1.14
Uint8Array.set 32 items	953.00 ns/op	1.0340 us/op	0.92
Buffer.copy	2.5200 us/op	2.1270 us/op	1.18
Uint8Array.set - with subarray	1.5560 us/op	1.6350 us/op	0.95
Uint8Array.set - without subarray	1.0930 us/op	1.1890 us/op	0.92
getUint32 - dataview	199.00 ns/op	196.00 ns/op	1.02
getUint32 - manual	131.00 ns/op	123.00 ns/op	1.07
Set add up to 64 items then delete first	2.2118 us/op	2.7221 us/op	0.81
OrderedSet add up to 64 items then delete first	3.3046 us/op	3.7244 us/op	0.89
Set add up to 64 items then delete last	2.2767 us/op	3.0733 us/op	0.74
OrderedSet add up to 64 items then delete last	3.6547 us/op	4.4200 us/op	0.83
Set add up to 64 items then delete middle	2.4986 us/op	2.4804 us/op	1.01
OrderedSet add up to 64 items then delete middle	5.2453 us/op	5.9737 us/op	0.88
Set add up to 128 items then delete first	4.9718 us/op	5.4415 us/op	0.91
OrderedSet add up to 128 items then delete first	7.8621 us/op	8.6524 us/op	0.91
Set add up to 128 items then delete last	4.9589 us/op	5.5068 us/op	0.90
OrderedSet add up to 128 items then delete last	7.2505 us/op	9.0275 us/op	0.80
Set add up to 128 items then delete middle	4.7696 us/op	5.7053 us/op	0.84
OrderedSet add up to 128 items then delete middle	13.602 us/op	14.906 us/op	0.91
Set add up to 256 items then delete first	10.139 us/op	11.633 us/op	0.87
OrderedSet add up to 256 items then delete first	15.829 us/op	19.672 us/op	0.80
Set add up to 256 items then delete last	9.4887 us/op	13.298 us/op	0.71
OrderedSet add up to 256 items then delete last	14.484 us/op	18.324 us/op	0.79
Set add up to 256 items then delete middle	9.5219 us/op	12.376 us/op	0.77
OrderedSet add up to 256 items then delete middle	40.856 us/op	45.100 us/op	0.91
transfer serialized Status (84 B)	2.1750 us/op	2.1750 us/op	1.00
copy serialized Status (84 B)	1.2750 us/op	1.0920 us/op	1.17
transfer serialized SignedVoluntaryExit (112 B)	2.4270 us/op	2.1810 us/op	1.11
copy serialized SignedVoluntaryExit (112 B)	1.3070 us/op	1.1140 us/op	1.17
transfer serialized ProposerSlashing (416 B)	2.3600 us/op	2.2040 us/op	1.07
copy serialized ProposerSlashing (416 B)	1.3170 us/op	1.3640 us/op	0.97
transfer serialized Attestation (485 B)	2.3820 us/op	2.2180 us/op	1.07
copy serialized Attestation (485 B)	1.4610 us/op	1.3430 us/op	1.09
transfer serialized AttesterSlashing (33232 B)	2.7620 us/op	2.5010 us/op	1.10
copy serialized AttesterSlashing (33232 B)	3.7220 us/op	3.6210 us/op	1.03
transfer serialized Small SignedBeaconBlock (128000 B)	3.2540 us/op	2.8490 us/op	1.14
copy serialized Small SignedBeaconBlock (128000 B)	8.6830 us/op	8.5180 us/op	1.02
transfer serialized Avg SignedBeaconBlock (200000 B)	3.5590 us/op	3.3310 us/op	1.07
copy serialized Avg SignedBeaconBlock (200000 B)	12.708 us/op	13.328 us/op	0.95
transfer serialized BlobsSidecar (524380 B)	3.5660 us/op	3.5640 us/op	1.00
copy serialized BlobsSidecar (524380 B)	79.979 us/op	128.61 us/op	0.62
transfer serialized Big SignedBeaconBlock (1000000 B)	3.7290 us/op	3.5970 us/op	1.04
copy serialized Big SignedBeaconBlock (1000000 B)	108.32 us/op	112.76 us/op	0.96
pass gossip attestations to forkchoice per slot	2.7085 ms/op	2.7168 ms/op	1.00
forkChoice updateHead vc 100000 bc 64 eq 0	490.23 us/op	497.43 us/op	0.99
forkChoice updateHead vc 600000 bc 64 eq 0	3.0289 ms/op	3.0840 ms/op	0.98
forkChoice updateHead vc 1000000 bc 64 eq 0	5.1027 ms/op	5.2039 ms/op	0.98
forkChoice updateHead vc 600000 bc 320 eq 0	3.1123 ms/op	3.0742 ms/op	1.01
forkChoice updateHead vc 600000 bc 1200 eq 0	3.1236 ms/op	3.1097 ms/op	1.00
forkChoice updateHead vc 600000 bc 7200 eq 0	3.3823 ms/op	3.8398 ms/op	0.88
forkChoice updateHead vc 600000 bc 64 eq 1000	2.9599 ms/op	3.0164 ms/op	0.98
forkChoice updateHead vc 600000 bc 64 eq 10000	3.0889 ms/op	3.1193 ms/op	0.99
forkChoice updateHead vc 600000 bc 64 eq 300000	9.4145 ms/op	9.9964 ms/op	0.94
computeDeltas 1400000 validators 300 proto nodes	11.891 ms/op	12.388 ms/op	0.96
computeDeltas 1400000 validators 1200 proto nodes	11.850 ms/op	12.375 ms/op	0.96
computeDeltas 1400000 validators 7200 proto nodes	12.096 ms/op	12.235 ms/op	0.99
computeDeltas 2100000 validators 300 proto nodes	18.358 ms/op	18.419 ms/op	1.00
computeDeltas 2100000 validators 1200 proto nodes	18.515 ms/op	18.251 ms/op	1.01
computeDeltas 2100000 validators 7200 proto nodes	18.430 ms/op	18.230 ms/op	1.01
altair processAttestation - 250000 vs - 7PWei normalcase	2.0703 ms/op	2.0073 ms/op	1.03
altair processAttestation - 250000 vs - 7PWei worstcase	3.0183 ms/op	2.9988 ms/op	1.01
altair processAttestation - setStatus - 1/6 committees join	123.13 us/op	125.38 us/op	0.98
altair processAttestation - setStatus - 1/3 committees join	250.48 us/op	243.65 us/op	1.03
altair processAttestation - setStatus - 1/2 committees join	343.83 us/op	344.94 us/op	1.00
altair processAttestation - setStatus - 2/3 committees join	434.90 us/op	454.33 us/op	0.96
altair processAttestation - setStatus - 4/5 committees join	597.43 us/op	634.31 us/op	0.94
altair processAttestation - setStatus - 100% committees join	716.63 us/op	726.10 us/op	0.99
altair processBlock - 250000 vs - 7PWei normalcase	5.0973 ms/op	4.6197 ms/op	1.10
altair processBlock - 250000 vs - 7PWei normalcase hashState	47.291 ms/op	37.717 ms/op	1.25
altair processBlock - 250000 vs - 7PWei worstcase	40.579 ms/op	48.715 ms/op	0.83
altair processBlock - 250000 vs - 7PWei worstcase hashState	108.20 ms/op	98.183 ms/op	1.10
phase0 processBlock - 250000 vs - 7PWei normalcase	1.9123 ms/op	2.5675 ms/op	0.74
phase0 processBlock - 250000 vs - 7PWei worstcase	32.129 ms/op	20.813 ms/op	1.54
altair processEth1Data - 250000 vs - 7PWei normalcase	329.61 us/op	340.90 us/op	0.97
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	9.3670 us/op	9.7790 us/op	0.96
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	55.060 us/op	35.490 us/op	1.55
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	9.9140 us/op	16.248 us/op	0.61
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	6.3340 us/op	8.0320 us/op	0.79
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	129.60 us/op	243.90 us/op	0.53
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	1.6533 ms/op	1.9224 ms/op	0.86
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	2.1023 ms/op	2.3159 ms/op	0.91
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	2.0513 ms/op	2.3481 ms/op	0.87
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	4.2099 ms/op	4.4869 ms/op	0.94
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	2.1146 ms/op	2.4346 ms/op	0.87
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	4.3964 ms/op	4.6118 ms/op	0.95
Tree 40 250000 create	413.30 ms/op	409.45 ms/op	1.01
Tree 40 250000 get(125000)	137.56 ns/op	146.17 ns/op	0.94
Tree 40 250000 set(125000)	2.4232 us/op	1.5024 us/op	1.61
Tree 40 250000 toArray()	24.169 ms/op	16.257 ms/op	1.49
Tree 40 250000 iterate all - toArray() + loop	20.553 ms/op	15.518 ms/op	1.32
Tree 40 250000 iterate all - get(i)	53.780 ms/op	52.943 ms/op	1.02
Array 250000 create	3.1608 ms/op	2.8852 ms/op	1.10
Array 250000 clone - spread	822.48 us/op	801.03 us/op	1.03
Array 250000 get(125000)	0.41300 ns/op	0.41600 ns/op	0.99
Array 250000 set(125000)	0.43800 ns/op	0.43600 ns/op	1.00
Array 250000 iterate all - loop	81.985 us/op	82.235 us/op	1.00
phase0 afterProcessEpoch - 250000 vs - 7PWei	41.747 ms/op	41.371 ms/op	1.01
Array.fill - length 1000000	3.6711 ms/op	3.6485 ms/op	1.01
Array push - length 1000000	12.536 ms/op	12.656 ms/op	0.99
Array.get	0.26556 ns/op	0.26061 ns/op	1.02
Uint8Array.get	0.42077 ns/op	0.42063 ns/op	1.00
phase0 beforeProcessEpoch - 250000 vs - 7PWei	16.885 ms/op	16.638 ms/op	1.01
altair processEpoch - mainnet_e81889	251.99 ms/op	282.88 ms/op	0.89
mainnet_e81889 - altair beforeProcessEpoch	18.821 ms/op	19.398 ms/op	0.97
mainnet_e81889 - altair processJustificationAndFinalization	5.3470 us/op	5.7260 us/op	0.93
mainnet_e81889 - altair processInactivityUpdates	4.0828 ms/op	4.2603 ms/op	0.96
mainnet_e81889 - altair processRewardsAndPenalties	47.872 ms/op	39.482 ms/op	1.21
mainnet_e81889 - altair processRegistryUpdates	644.00 ns/op	675.00 ns/op	0.95
mainnet_e81889 - altair processSlashings	185.00 ns/op	176.00 ns/op	1.05
mainnet_e81889 - altair processEth1DataReset	169.00 ns/op	173.00 ns/op	0.98
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.1892 ms/op	1.1922 ms/op	1.00
mainnet_e81889 - altair processSlashingsReset	925.00 ns/op	937.00 ns/op	0.99
mainnet_e81889 - altair processRandaoMixesReset	1.1410 us/op	1.5390 us/op	0.74
mainnet_e81889 - altair processHistoricalRootsUpdate	179.00 ns/op	172.00 ns/op	1.04
mainnet_e81889 - altair processParticipationFlagUpdates	519.00 ns/op	518.00 ns/op	1.00
mainnet_e81889 - altair processSyncCommitteeUpdates	139.00 ns/op	139.00 ns/op	1.00
mainnet_e81889 - altair afterProcessEpoch	44.656 ms/op	42.938 ms/op	1.04
capella processEpoch - mainnet_e217614	1.0358 s/op	869.87 ms/op	1.19
mainnet_e217614 - capella beforeProcessEpoch	65.087 ms/op	62.367 ms/op	1.04
mainnet_e217614 - capella processJustificationAndFinalization	6.2010 us/op	5.2310 us/op	1.19
mainnet_e217614 - capella processInactivityUpdates	14.685 ms/op	13.422 ms/op	1.09
mainnet_e217614 - capella processRewardsAndPenalties	189.71 ms/op	213.05 ms/op	0.89
mainnet_e217614 - capella processRegistryUpdates	6.4430 us/op	7.7530 us/op	0.83
mainnet_e217614 - capella processSlashings	182.00 ns/op	170.00 ns/op	1.07
mainnet_e217614 - capella processEth1DataReset	183.00 ns/op	167.00 ns/op	1.10
mainnet_e217614 - capella processEffectiveBalanceUpdates	4.2181 ms/op	3.9836 ms/op	1.06
mainnet_e217614 - capella processSlashingsReset	1.0510 us/op	1.4470 us/op	0.73
mainnet_e217614 - capella processRandaoMixesReset	1.1410 us/op	1.2280 us/op	0.93
mainnet_e217614 - capella processHistoricalRootsUpdate	194.00 ns/op	170.00 ns/op	1.14
mainnet_e217614 - capella processParticipationFlagUpdates	521.00 ns/op	517.00 ns/op	1.01
mainnet_e217614 - capella afterProcessEpoch	115.89 ms/op	114.15 ms/op	1.02
phase0 processEpoch - mainnet_e58758	264.43 ms/op	300.75 ms/op	0.88
mainnet_e58758 - phase0 beforeProcessEpoch	69.042 ms/op	77.765 ms/op	0.89
mainnet_e58758 - phase0 processJustificationAndFinalization	7.0910 us/op	6.9120 us/op	1.03
mainnet_e58758 - phase0 processRewardsAndPenalties	44.831 ms/op	40.550 ms/op	1.11
mainnet_e58758 - phase0 processRegistryUpdates	3.0610 us/op	3.0010 us/op	1.02
mainnet_e58758 - phase0 processSlashings	175.00 ns/op	175.00 ns/op	1.00
mainnet_e58758 - phase0 processEth1DataReset	172.00 ns/op	185.00 ns/op	0.93
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.1659 ms/op	1.1455 ms/op	1.02
mainnet_e58758 - phase0 processSlashingsReset	1.0390 us/op	880.00 ns/op	1.18
mainnet_e58758 - phase0 processRandaoMixesReset	1.3270 us/op	1.5500 us/op	0.86
mainnet_e58758 - phase0 processHistoricalRootsUpdate	174.00 ns/op	171.00 ns/op	1.02
mainnet_e58758 - phase0 processParticipationRecordUpdates	889.00 ns/op	846.00 ns/op	1.05
mainnet_e58758 - phase0 afterProcessEpoch	44.972 ms/op	33.665 ms/op	1.34
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.8544 ms/op	1.6422 ms/op	1.13
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.9242 ms/op	2.7286 ms/op	0.71
altair processInactivityUpdates - 250000 normalcase	24.555 ms/op	18.747 ms/op	1.31
altair processInactivityUpdates - 250000 worstcase	22.726 ms/op	21.867 ms/op	1.04
phase0 processRegistryUpdates - 250000 normalcase	6.5280 us/op	8.0880 us/op	0.81
phase0 processRegistryUpdates - 250000 badcase_full_deposits	436.86 us/op	325.79 us/op	1.34
phase0 processRegistryUpdates - 250000 worstcase 0.5	115.82 ms/op	115.43 ms/op	1.00
altair processRewardsAndPenalties - 250000 normalcase	35.306 ms/op	33.401 ms/op	1.06
altair processRewardsAndPenalties - 250000 worstcase	36.545 ms/op	33.489 ms/op	1.09
phase0 getAttestationDeltas - 250000 normalcase	6.1574 ms/op	6.7726 ms/op	0.91
phase0 getAttestationDeltas - 250000 worstcase	6.7111 ms/op	6.5579 ms/op	1.02
phase0 processSlashings - 250000 worstcase	118.82 us/op	123.31 us/op	0.96
altair processSyncCommitteeUpdates - 250000	10.465 ms/op	10.954 ms/op	0.96
BeaconState.hashTreeRoot - No change	232.00 ns/op	221.00 ns/op	1.05
BeaconState.hashTreeRoot - 1 full validator	78.773 us/op	86.353 us/op	0.91
BeaconState.hashTreeRoot - 32 full validator	1.1789 ms/op	869.23 us/op	1.36
BeaconState.hashTreeRoot - 512 full validator	10.414 ms/op	9.8949 ms/op	1.05
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	130.12 us/op	108.59 us/op	1.20
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	2.6776 ms/op	1.3810 ms/op	1.94
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	24.038 ms/op	29.902 ms/op	0.80
BeaconState.hashTreeRoot - 1 balances	95.395 us/op	87.872 us/op	1.09
BeaconState.hashTreeRoot - 32 balances	840.39 us/op	1.2128 ms/op	0.69
BeaconState.hashTreeRoot - 512 balances	9.2667 ms/op	8.8105 ms/op	1.05
BeaconState.hashTreeRoot - 250000 balances	181.09 ms/op	199.27 ms/op	0.91
aggregationBits - 2048 els - zipIndexesInBitList	20.762 us/op	21.349 us/op	0.97
byteArrayEquals 32	51.979 ns/op	53.195 ns/op	0.98
Buffer.compare 32	16.734 ns/op	17.315 ns/op	0.97
byteArrayEquals 1024	1.5417 us/op	1.5921 us/op	0.97
Buffer.compare 1024	23.804 ns/op	24.695 ns/op	0.96
byteArrayEquals 16384	24.533 us/op	25.236 us/op	0.97
Buffer.compare 16384	187.29 ns/op	185.65 ns/op	1.01
byteArrayEquals 123687377	189.72 ms/op	187.74 ms/op	1.01
Buffer.compare 123687377	6.4976 ms/op	6.2904 ms/op	1.03
byteArrayEquals 32 - diff last byte	51.925 ns/op	52.420 ns/op	0.99
Buffer.compare 32 - diff last byte	16.855 ns/op	16.876 ns/op	1.00
byteArrayEquals 1024 - diff last byte	1.5653 us/op	1.5730 us/op	1.00
Buffer.compare 1024 - diff last byte	24.606 ns/op	24.497 ns/op	1.00
byteArrayEquals 16384 - diff last byte	24.963 us/op	24.928 us/op	1.00
Buffer.compare 16384 - diff last byte	191.96 ns/op	181.01 ns/op	1.06
byteArrayEquals 123687377 - diff last byte	184.71 ms/op	191.55 ms/op	0.96
Buffer.compare 123687377 - diff last byte	6.1549 ms/op	8.1001 ms/op	0.76
byteArrayEquals 32 - random bytes	4.9470 ns/op	5.1960 ns/op	0.95
Buffer.compare 32 - random bytes	16.551 ns/op	17.032 ns/op	0.97
byteArrayEquals 1024 - random bytes	4.9350 ns/op	5.1220 ns/op	0.96
Buffer.compare 1024 - random bytes	16.571 ns/op	17.060 ns/op	0.97
byteArrayEquals 16384 - random bytes	4.9520 ns/op	5.1370 ns/op	0.96
Buffer.compare 16384 - random bytes	16.455 ns/op	22.005 ns/op	0.75
byteArrayEquals 123687377 - random bytes	8.1200 ns/op	6.5400 ns/op	1.24
Buffer.compare 123687377 - random bytes	21.640 ns/op	18.540 ns/op	1.17
regular array get 100000 times	42.278 us/op	41.853 us/op	1.01
wrappedArray get 100000 times	31.743 us/op	33.773 us/op	0.94
arrayWithProxy get 100000 times	12.053 ms/op	12.670 ms/op	0.95
ssz.Root.equals	44.567 ns/op	46.439 ns/op	0.96
byteArrayEquals	43.627 ns/op	45.584 ns/op	0.96
Buffer.compare	9.9580 ns/op	10.448 ns/op	0.95
processSlot - 1 slots	10.348 us/op	10.273 us/op	1.01
processSlot - 32 slots	3.2098 ms/op	2.9084 ms/op	1.10
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	2.7924 ms/op	3.4569 ms/op	0.81
getCommitteeAssignments - req 1 vs - 250000 vc	2.1246 ms/op	2.1641 ms/op	0.98
getCommitteeAssignments - req 100 vs - 250000 vc	4.1303 ms/op	4.2461 ms/op	0.97
getCommitteeAssignments - req 1000 vs - 250000 vc	4.4028 ms/op	4.7369 ms/op	0.93
findModifiedValidators - 10000 modified validators	728.02 ms/op	777.12 ms/op	0.94
findModifiedValidators - 1000 modified validators	699.77 ms/op	745.37 ms/op	0.94
findModifiedValidators - 100 modified validators	271.10 ms/op	246.24 ms/op	1.10
findModifiedValidators - 10 modified validators	162.30 ms/op	141.27 ms/op	1.15
findModifiedValidators - 1 modified validators	169.47 ms/op	149.04 ms/op	1.14
findModifiedValidators - no difference	180.82 ms/op	270.35 ms/op	0.67
compare ViewDUs	6.2221 s/op	6.3802 s/op	0.98
compare each validator Uint8Array	2.0688 s/op	1.5102 s/op	1.37
compare ViewDU to Uint8Array	957.62 ms/op	1.1038 s/op	0.87
migrate state 1000000 validators, 24 modified, 0 new	796.55 ms/op	813.27 ms/op	0.98
migrate state 1000000 validators, 1700 modified, 1000 new	1.2086 s/op	1.2071 s/op	1.00
migrate state 1000000 validators, 3400 modified, 2000 new	1.3298 s/op	1.3348 s/op	1.00
migrate state 1500000 validators, 24 modified, 0 new	940.04 ms/op	945.58 ms/op	0.99
migrate state 1500000 validators, 1700 modified, 1000 new	1.1120 s/op	1.1611 s/op	0.96
migrate state 1500000 validators, 3400 modified, 2000 new	1.3054 s/op	1.2573 s/op	1.04
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	4.4100 ns/op	5.2600 ns/op	0.84
state getBlockRootAtSlot - 250000 vs - 7PWei	582.87 ns/op	415.75 ns/op	1.40
naive computeProposerIndex 100000 validators	53.107 ms/op	51.915 ms/op	1.02
computeProposerIndex 100000 validators	1.4837 ms/op	1.4913 ms/op	0.99
naiveGetNextSyncCommitteeIndices 1000 validators	7.4256 s/op	7.2387 s/op	1.03
getNextSyncCommitteeIndices 1000 validators	116.82 ms/op	106.87 ms/op	1.09
naiveGetNextSyncCommitteeIndices 10000 validators	8.0331 s/op	7.4287 s/op	1.08
getNextSyncCommitteeIndices 10000 validators	119.18 ms/op	103.81 ms/op	1.15
naiveGetNextSyncCommitteeIndices 100000 validators	9.2027 s/op	7.6577 s/op	1.20
getNextSyncCommitteeIndices 100000 validators	124.28 ms/op	108.77 ms/op	1.14
naive computeShuffledIndex 100000 validators	28.082 s/op	22.031 s/op	1.27
cached computeShuffledIndex 100000 validators	564.41 ms/op	572.61 ms/op	0.99
naive computeShuffledIndex 2000000 validators	513.62 s/op	503.57 s/op	1.02
cached computeShuffledIndex 2000000 validators	36.294 s/op	31.953 s/op	1.14
computeProposers - vc 250000	622.66 us/op	608.45 us/op	1.02
computeEpochShuffling - vc 250000	42.800 ms/op	40.559 ms/op	1.06
getNextSyncCommittee - vc 250000	10.822 ms/op	10.039 ms/op	1.08
computeSigningRoot for AttestationData	21.398 us/op	19.636 us/op	1.09
hash AttestationData serialized data then Buffer.toString(base64)	1.5593 us/op	1.5932 us/op	0.98
toHexString serialized data	1.1601 us/op	1.1514 us/op	1.01
Buffer.toString(base64)	158.41 ns/op	150.36 ns/op	1.05
nodejs block root to RootHex using toHex	150.28 ns/op	133.60 ns/op	1.12
nodejs block root to RootHex using toRootHex	85.641 ns/op	89.461 ns/op	0.96
nodejs fromhex(blob)	113.14 ms/op	112.35 ms/op	1.01
nodejs fromHexInto(blob)	100.79 ms/op	96.125 ms/op	1.05
browser block root to RootHex using the deprecated toHexString	214.48 ns/op	213.53 ns/op	1.00
browser block root to RootHex using toHex	173.82 ns/op	177.81 ns/op	0.98
browser block root to RootHex using toRootHex	163.03 ns/op	161.14 ns/op	1.01
browser fromHexInto(blob)	820.84 us/op	807.30 us/op	1.02
browser fromHex(blob)	778.82 ms/op	803.96 ms/op	0.97

by benchmarkbot/action

nflaig · 2025-10-13T14:18:30Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a significant feature: the ability to sync from an unfinalized checkpoint state. This is particularly useful for faster node startup and recovery, especially during periods of non-finality. The implementation is comprehensive, touching upon API definitions, CLI options, state initialization logic, and the core fork choice mechanism. The introduction of initializeForkChoiceFromUnfinalizedState with its proto-array scaffolding is a clever approach to handle this complex scenario. The code is well-structured, and the new "unsafe" options are appropriately hidden and documented with warnings. Overall, this is a solid implementation of a complex feature. I have a couple of minor suggestions for code quality and performance improvements.

packages/beacon-node/src/api/impl/lodestar/index.ts

packages/beacon-node/src/chain/stateCache/datastore/db.ts

nflaig · 2025-10-17T15:03:50Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a significant and well-implemented feature to allow syncing from an unfinalized checkpoint state. The changes are comprehensive, touching upon CLI options, API definitions, chain initialization, and the core fork choice logic. The implementation is robust, with a clear separation between finalized and unfinalized startup paths, and includes a clever mechanism to find the latest safe persisted checkpoint. The new complex logic, particularly in fork choice initialization, appears sound and is a great addition for faster node startup in certain scenarios. I have one minor suggestion regarding a misleading comment, but overall, this is an excellent contribution.

packages/beacon-node/src/chain/stateCache/datastore/db.ts

See #7509 (comment)

wemeetagain · 2025-10-20T13:38:09Z

packages/fork-choice/src/forkChoice/forkChoice.ts

    return block;
  }

+  getFinalizedBlockSlot(): Slot {


this is a little strange considering its just the finalized epoch start slot and not actually the finalized block slot.

maybe getFinalizedCheckpointSlot()?

I renamed it to getFinalizedCheckpointSlot, could also consider getAnchorBlockSlot

packages/beacon-node/src/sync/utils/remoteSyncType.ts

We need to pass validator count to fork choice since #8549 but that wasn't done in #8527 for `initializeForkChoiceFromUnfinalizedState` which breaks our code.

wemeetagain · 2025-11-04T18:53:40Z

🎉 This PR is included in v1.36.0 🎉

twoeths and others added 22 commits September 30, 2025 15:43

Fix type issues in tests

ed375a7

Merge branch 'unstable' into nflaig/sync-from-unfinalized

ddbd4f7

Lint

979dacb

Fix imports

85bd12b

Merge branch 'unstable' into nflaig/sync-from-unfinalized

701b950

Review changes

4cc8a20

Tweak comment

de08c89

Differentiate error cases when getting persisted cp state from api

020751c

Use 404 status code

7dd1690

Throw 400 api error if parsing of checkpoint fails

e09f7a0

Fix casing

ab75339

Make lastPersistedCheckpointState flag hidden

ad9a77f

Always return 404 api error

fab39a5

Tweak beacon state init logs

9ff88c7

Merge branch 'unstable' into nflaig/sync-from-unfinalized

9c48983

Merge branch 'unstable' into nflaig/sync-from-unfinalized

12a749a

Add --unsafeCheckpointState flag

651fd9c

Remove conflicts

0a7d848

refactor

6b1c96a

nflaig added 3 commits October 13, 2025 15:09

Throw error if last persisted checkoint state is not found

346198f

Check all checkpoint state args

db575f1

Merge branch 'unstable' into nflaig/sync-from-unfinalized

7404070

gemini-code-assist bot reviewed Oct 13, 2025

View reviewed changes

packages/beacon-node/src/api/impl/lodestar/index.ts Show resolved Hide resolved

packages/beacon-node/src/chain/stateCache/datastore/db.ts Show resolved Hide resolved

Check checkpointSyncUrl before unsafe options

81cdcb6

nflaig added 9 commits October 14, 2025 10:59

Merge branch 'unstable' into nflaig/sync-from-unfinalized

e00d519

Define regex outside of function

20420bd

Merge branch 'unstable' into nflaig/sync-from-unfinalized

4dce6d7

Post merge cleanup

ab68097

Merge branch 'unstable' into nflaig/sync-from-unfinalized

124fbd5

Update init beacon state

06d4bf0

More tweaks

91902e2

Minor update

def247c

Tweak db init log

cacc31e

gemini-code-assist bot reviewed Oct 17, 2025

View reviewed changes

packages/beacon-node/src/chain/stateCache/datastore/db.ts Show resolved Hide resolved

nflaig added 2 commits October 17, 2025 16:56

Wording

d5b80c9

Include lastProcessedSlot in log if synced from unfinalized

5ff5744

nflaig marked this pull request as ready for review October 17, 2025 16:34

nflaig requested a review from a team as a code owner October 17, 2025 16:34

Do not always start from local head in head sync

4087b35

See #7509 (comment)

wemeetagain reviewed Oct 20, 2025

View reviewed changes

nflaig added 2 commits October 20, 2025 16:08

Merge branch 'unstable' into nflaig/sync-from-unfinalized

caf63af

Merge branch 'unstable' into nflaig/sync-from-unfinalized

0a7b918

twoeths reviewed Oct 21, 2025

View reviewed changes

packages/beacon-node/src/sync/utils/remoteSyncType.ts Outdated Show resolved Hide resolved

nflaig added 3 commits October 21, 2025 10:15

Merge branch 'unstable' into nflaig/sync-from-unfinalized

6543bc6

Rename getFinalizedBlockSlot to getFinalizedCheckpointSlot

d9a0fc2

Update head sync comment

858013d

twoeths approved these changes Oct 22, 2025

View reviewed changes

wemeetagain approved these changes Oct 22, 2025

View reviewed changes

wemeetagain merged commit 7e3c184 into unstable Oct 22, 2025
25 of 26 checks passed

wemeetagain deleted the nflaig/sync-from-unfinalized branch October 22, 2025 19:22

nflaig mentioned this pull request Oct 22, 2025

chore: fix build after merge to unstable #8558

Merged

wemeetagain pushed a commit that referenced this pull request Oct 22, 2025

chore: fix build after merge to unstable (#8558)

9dbc67b

We need to pass validator count to fork choice since #8549 but that wasn't done in #8527 for `initializeForkChoiceFromUnfinalizedState` which breaks our code.

Uh oh!

feat: sync from unfinalized checkpoint state #8527

feat: sync from unfinalized checkpoint state #8527

Uh oh!

Conversation

nflaig commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Report

Uh oh!

nflaig commented Oct 13, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

nflaig commented Oct 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

wemeetagain Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

twoeths Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

nflaig Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wemeetagain commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nflaig commented Oct 13, 2025 •

edited

Loading

codecov bot commented Oct 13, 2025 •

edited

Loading

github-actions bot commented Oct 13, 2025 •

edited

Loading