这是indexloc提供的服务，不要输入任何密码
Skip to content

fix: min key for the archive data columns #8345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 31 commits into from
Sep 9, 2025
Merged

fix: min key for the archive data columns #8345

merged 31 commits into from
Sep 9, 2025

Conversation

@nazarhussain
Copy link
Contributor

@nazarhussain nazarhussain commented Sep 5, 2025 •
edited
Loading

Motivation

Fix bucket boundary for the min the key for data column sidecars.

Description

  • Fix the bucket boundary issue for data column side cars.

Steps to test or reproduce

Run all tests

Note

Level db have a builtin feature to define boundary for buckets called sublevel, I didn't realize earlier we are not using that, instead we have our own higher level logic for min/max key for the buckets. So didn't provided that second condition to limit to bucket boundary.

@nazarhussain nazarhussain requested a review from a team as a code owner September 5, 2025 18:59
gemini-code-assist[bot]
gemini-code-assist bot reviewed Sep 5, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @nazarhussain, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a potential issue with database key range queries by enforcing that both a start and end boundary are always specified. This change prevents ambiguous or unbounded queries, specifically fixing a bucket boundary problem related to archive data column sidecars by ensuring queries for these sidecars always have a defined minimum key.

Highlights

  • Database Query Range Enforcement: Implemented a new validation check in the PrefixedRepository to ensure that all key range queries (e.g., keys method) explicitly provide both a lower bound (gt or gte) and an upper bound (lt or lte). This prevents incomplete range specifications.
  • Archive Data Column Query Fix: Applied the new range enforcement by adding a gte (greater than or equal to) condition to the dataColumnSidecarArchive.keys query in archiveBlocks.ts, ensuring the query always starts from the minimum possible key (epoch 0).
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Sep 5, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request aims to fix an issue with data column sidecar pruning by adding a lower bound to the database query. The change in packages/beacon-node/src/chain/archiveStore/utils/archiveBlocks.ts correctly implements this fix. However, a related change in packages/db/src/abstractPrefixedRepository.ts introduces a new validation that is overly restrictive and will cause a regression in other parts of the system, specifically with blob sidecar pruning. My review focuses on this critical issue.

@codecov
Copy link

codecov bot commented Sep 5, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 63.38028% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.10%. Comparing base (09ef70b) to head (78ec79f).
⚠️ Report is 1 commits behind head on unstable.

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #8345      +/-   ##
============================================
+ Coverage     54.07%   54.10%   +0.02%     
============================================
  Files           849      850       +1     
  Lines         64148    64171      +23     
  Branches       4854     4853       -1     
============================================
+ Hits          34691    34719      +28     
+ Misses        29380    29375       -5     
  Partials         77       77
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nazarhussain
nazarhussain commented Sep 5, 2025
View reviewed changes
packages/beacon-node/src/db/repositories/dataColumnSidecar.ts Outdated

encodeKeyRaw(prefix: BlockRoot, id: ColumnIndex): Uint8Array {
return Buffer.concat([prefix, intToBytes(id, 4)]);
return Buffer.concat([prefix, intToBytes(id, 4, "be")]);
Copy link
Contributor Author

@nazarhussain nazarhussain Sep 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The leveldb keys are indexed in a way that makes gt, lt queries compatible with the values indexed as be integer values.

lodestar/packages/db/src/util.ts

Lines 16 to 18 in 083b7cc

} else if (typeof key === "number" || typeof key === "bigint") {
buf = Buffer.alloc(uintLen + prefixLength);
intToBytes(BigInt(key), uintLen, "be").copy(buf, prefixLength);

@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2025 •
edited
Loading

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: 5a04bf7 Previous: 09ef70b Ratio
arrayWithProxy get 100000 times 59.039 ms/op 13.776 ms/op 4.29
Full benchmark results
Benchmark suite Current: 5a04bf7 Previous: 09ef70b Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 832.44 us/op 1.7752 ms/op 0.47
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 33.941 us/op 54.024 us/op 0.63
BLS verify - blst 905.87 us/op 1.2643 ms/op 0.72
BLS verifyMultipleSignatures 3 - blst 1.3289 ms/op 1.9454 ms/op 0.68
BLS verifyMultipleSignatures 8 - blst 2.1319 ms/op 2.7102 ms/op 0.79
BLS verifyMultipleSignatures 32 - blst 4.4994 ms/op 7.7951 ms/op 0.58
BLS verifyMultipleSignatures 64 - blst 8.4387 ms/op 13.433 ms/op 0.63
BLS verifyMultipleSignatures 128 - blst 16.241 ms/op 24.070 ms/op 0.67
BLS deserializing 10000 signatures 638.86 ms/op 816.55 ms/op 0.78
BLS deserializing 100000 signatures 6.1658 s/op 9.7278 s/op 0.63
BLS verifyMultipleSignatures - same message - 3 - blst 919.57 us/op 1.4445 ms/op 0.64
BLS verifyMultipleSignatures - same message - 8 - blst 1.2348 ms/op 1.5099 ms/op 0.82
BLS verifyMultipleSignatures - same message - 32 - blst 1.6608 ms/op 2.5511 ms/op 0.65
BLS verifyMultipleSignatures - same message - 64 - blst 2.5326 ms/op 3.8005 ms/op 0.67
BLS verifyMultipleSignatures - same message - 128 - blst 4.2444 ms/op 7.0964 ms/op 0.60
BLS aggregatePubkeys 32 - blst 17.966 us/op 31.136 us/op 0.58
BLS aggregatePubkeys 128 - blst 63.536 us/op 115.33 us/op 0.55
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 51.952 ms/op 160.63 ms/op 0.32
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 39.909 ms/op 111.11 ms/op 0.36
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 37.601 ms/op 65.118 ms/op 0.58
getSlashingsAndExits - default max 51.552 us/op 318.85 us/op 0.16
getSlashingsAndExits - 2k 253.65 us/op 633.59 us/op 0.40
proposeBlockBody type=full, size=empty 4.8060 ms/op 10.792 ms/op 0.45
isKnown best case - 1 super set check 399.00 ns/op 384.00 ns/op 1.04
isKnown normal case - 2 super set checks 397.00 ns/op 341.00 ns/op 1.16
isKnown worse case - 16 super set checks 400.00 ns/op 337.00 ns/op 1.19
InMemoryCheckpointStateCache - add get delete 2.4480 us/op 4.0300 us/op 0.61
validate api signedAggregateAndProof - struct 1.4508 ms/op 2.3278 ms/op 0.62
validate gossip signedAggregateAndProof - struct 1.9880 ms/op 2.0686 ms/op 0.96
batch validate gossip attestation - vc 640000 - chunk 32 105.86 us/op 194.98 us/op 0.54
batch validate gossip attestation - vc 640000 - chunk 64 95.443 us/op 159.01 us/op 0.60
batch validate gossip attestation - vc 640000 - chunk 128 83.711 us/op 150.01 us/op 0.56
batch validate gossip attestation - vc 640000 - chunk 256 85.448 us/op 149.15 us/op 0.57
pickEth1Vote - no votes 722.13 us/op 1.2481 ms/op 0.58
pickEth1Vote - max votes 6.0521 ms/op 13.519 ms/op 0.45
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.8588 ms/op 18.783 ms/op 0.52
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.647 ms/op 29.061 ms/op 0.50
pickEth1Vote - Eth1Data fastSerialize value x2048 329.45 us/op 504.21 us/op 0.65
pickEth1Vote - Eth1Data fastSerialize tree x2048 2.4088 ms/op 6.1777 ms/op 0.39
bytes32 toHexString 515.00 ns/op 412.00 ns/op 1.25
bytes32 Buffer.toString(hex) 379.00 ns/op 276.00 ns/op 1.37
bytes32 Buffer.toString(hex) from Uint8Array 462.00 ns/op 371.00 ns/op 1.25
bytes32 Buffer.toString(hex) + 0x 389.00 ns/op 394.00 ns/op 0.99
Object access 1 prop 0.28500 ns/op 0.12500 ns/op 2.28
Map access 1 prop 0.30500 ns/op 0.14400 ns/op 2.12
Object get x1000 5.1260 ns/op 6.7160 ns/op 0.76
Map get x1000 5.6380 ns/op 7.0990 ns/op 0.79
Object set x1000 18.373 ns/op 36.049 ns/op 0.51
Map set x1000 15.795 ns/op 24.741 ns/op 0.64
Return object 10000 times 0.27440 ns/op 0.33280 ns/op 0.82
Throw Error 10000 times 3.2290 us/op 5.1906 us/op 0.62
toHex 88.961 ns/op 160.39 ns/op 0.55
Buffer.from 83.034 ns/op 136.17 ns/op 0.61
shared Buffer 59.653 ns/op 92.660 ns/op 0.64
fastMsgIdFn sha256 / 200 bytes 1.7930 us/op 2.4470 us/op 0.73
fastMsgIdFn h32 xxhash / 200 bytes 374.00 ns/op 391.00 ns/op 0.96
fastMsgIdFn h64 xxhash / 200 bytes 425.00 ns/op 314.00 ns/op 1.35
fastMsgIdFn sha256 / 1000 bytes 5.4830 us/op 8.2600 us/op 0.66
fastMsgIdFn h32 xxhash / 1000 bytes 484.00 ns/op 379.00 ns/op 1.28
fastMsgIdFn h64 xxhash / 1000 bytes 477.00 ns/op 373.00 ns/op 1.28
fastMsgIdFn sha256 / 10000 bytes 46.383 us/op 70.725 us/op 0.66
fastMsgIdFn h32 xxhash / 10000 bytes 1.8240 us/op 2.0080 us/op 0.91
fastMsgIdFn h64 xxhash / 10000 bytes 1.2790 us/op 1.4410 us/op 0.89
send data - 1000 256B messages 11.937 ms/op 22.417 ms/op 0.53
send data - 1000 512B messages 14.284 ms/op 25.806 ms/op 0.55
send data - 1000 1024B messages 19.951 ms/op 31.590 ms/op 0.63
send data - 1000 1200B messages 21.617 ms/op 34.335 ms/op 0.63
send data - 1000 2048B messages 20.910 ms/op 35.307 ms/op 0.59
send data - 1000 4096B messages 29.765 ms/op 35.695 ms/op 0.83
send data - 1000 16384B messages 34.975 ms/op 52.881 ms/op 0.66
send data - 1000 65536B messages 89.696 ms/op 140.03 ms/op 0.64
enrSubnets - fastDeserialize 64 bits 905.00 ns/op 1.0060 us/op 0.90
enrSubnets - ssz BitVector 64 bits 474.00 ns/op 358.00 ns/op 1.32
enrSubnets - fastDeserialize 4 bits 323.00 ns/op 144.00 ns/op 2.24
enrSubnets - ssz BitVector 4 bits 483.00 ns/op 367.00 ns/op 1.32
prioritizePeers score -10:0 att 32-0.1 sync 2-0 192.83 us/op 267.46 us/op 0.72
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 226.82 us/op 290.31 us/op 0.78
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 365.72 us/op 431.50 us/op 0.85
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 634.25 us/op 800.34 us/op 0.79
prioritizePeers score 0:0 att 64-1 sync 4-1 778.76 us/op 950.05 us/op 0.82
array of 16000 items push then shift 1.2000 us/op 1.7857 us/op 0.67
LinkedList of 16000 items push then shift 7.3780 ns/op 10.134 ns/op 0.73
array of 16000 items push then pop 62.541 ns/op 99.205 ns/op 0.63
LinkedList of 16000 items push then pop 6.0160 ns/op 8.5340 ns/op 0.70
array of 24000 items push then shift 1.7534 us/op 2.7318 us/op 0.64
LinkedList of 24000 items push then shift 6.8050 ns/op 8.9580 ns/op 0.76
array of 24000 items push then pop 92.638 ns/op 126.72 ns/op 0.73
LinkedList of 24000 items push then pop 6.0250 ns/op 7.7980 ns/op 0.77
intersect bitArray bitLen 8 5.1370 ns/op 7.4050 ns/op 0.69
intersect array and set length 8 32.141 ns/op 50.449 ns/op 0.64
intersect bitArray bitLen 128 25.205 ns/op 32.238 ns/op 0.78
intersect array and set length 128 518.85 ns/op 674.60 ns/op 0.77
bitArray.getTrueBitIndexes() bitLen 128 1.1510 us/op 1.1140 us/op 1.03
bitArray.getTrueBitIndexes() bitLen 248 1.8970 us/op 2.0570 us/op 0.92
bitArray.getTrueBitIndexes() bitLen 512 3.6640 us/op 4.3180 us/op 0.85
Buffer.concat 32 items 832.00 ns/op 737.00 ns/op 1.13
Uint8Array.set 32 items 1.7180 us/op 1.0840 us/op 1.58
Buffer.copy 2.5140 us/op 2.3180 us/op 1.08
Uint8Array.set - with subarray 2.3550 us/op 1.7220 us/op 1.37
Uint8Array.set - without subarray 1.8560 us/op 1.1860 us/op 1.56
getUint32 - dataview 372.00 ns/op 209.00 ns/op 1.78
getUint32 - manual 296.00 ns/op 137.00 ns/op 2.16
Set add up to 64 items then delete first 1.7078 us/op 3.4284 us/op 0.50
OrderedSet add up to 64 items then delete first 2.6460 us/op 5.2384 us/op 0.51
Set add up to 64 items then delete last 1.7707 us/op 2.5239 us/op 0.70
OrderedSet add up to 64 items then delete last 2.8878 us/op 5.7202 us/op 0.50
Set add up to 64 items then delete middle 1.8981 us/op 4.1781 us/op 0.45
OrderedSet add up to 64 items then delete middle 4.2160 us/op 13.399 us/op 0.31
Set add up to 128 items then delete first 3.7449 us/op 13.190 us/op 0.28
OrderedSet add up to 128 items then delete first 5.9952 us/op 19.019 us/op 0.32
Set add up to 128 items then delete last 3.6305 us/op 10.202 us/op 0.36
OrderedSet add up to 128 items then delete last 5.6496 us/op 18.984 us/op 0.30
Set add up to 128 items then delete middle 3.6368 us/op 11.032 us/op 0.33
OrderedSet add up to 128 items then delete middle 10.987 us/op 26.292 us/op 0.42
Set add up to 256 items then delete first 7.4995 us/op 21.659 us/op 0.35
OrderedSet add up to 256 items then delete first 12.080 us/op 34.890 us/op 0.35
Set add up to 256 items then delete last 7.2353 us/op 24.079 us/op 0.30
OrderedSet add up to 256 items then delete last 11.429 us/op 34.075 us/op 0.34
Set add up to 256 items then delete middle 7.2635 us/op 17.255 us/op 0.42
OrderedSet add up to 256 items then delete middle 33.885 us/op 48.507 us/op 0.70
transfer serialized Status (84 B) 2.4900 us/op 2.2570 us/op 1.10
copy serialized Status (84 B) 1.7330 us/op 1.1380 us/op 1.52
transfer serialized SignedVoluntaryExit (112 B) 2.4220 us/op 2.3380 us/op 1.04
copy serialized SignedVoluntaryExit (112 B) 1.7630 us/op 1.1940 us/op 1.48
transfer serialized ProposerSlashing (416 B) 3.1870 us/op 2.3530 us/op 1.35
copy serialized ProposerSlashing (416 B) 2.1900 us/op 1.4100 us/op 1.55
transfer serialized Attestation (485 B) 3.0780 us/op 2.4720 us/op 1.25
copy serialized Attestation (485 B) 2.2410 us/op 1.3040 us/op 1.72
transfer serialized AttesterSlashing (33232 B) 3.1270 us/op 2.4180 us/op 1.29
copy serialized AttesterSlashing (33232 B) 4.4150 us/op 3.9890 us/op 1.11
transfer serialized Small SignedBeaconBlock (128000 B) 3.1080 us/op 3.0210 us/op 1.03
copy serialized Small SignedBeaconBlock (128000 B) 9.1460 us/op 11.793 us/op 0.78
transfer serialized Avg SignedBeaconBlock (200000 B) 3.1620 us/op 3.5440 us/op 0.89
copy serialized Avg SignedBeaconBlock (200000 B) 13.773 us/op 17.458 us/op 0.79
transfer serialized BlobsSidecar (524380 B) 3.4720 us/op 3.5110 us/op 0.99
copy serialized BlobsSidecar (524380 B) 56.721 us/op 74.345 us/op 0.76
transfer serialized Big SignedBeaconBlock (1000000 B) 3.6990 us/op 3.8000 us/op 0.97
copy serialized Big SignedBeaconBlock (1000000 B) 105.02 us/op 131.62 us/op 0.80
pass gossip attestations to forkchoice per slot 2.2838 ms/op 3.0550 ms/op 0.75
forkChoice updateHead vc 100000 bc 64 eq 0 332.82 us/op 499.93 us/op 0.67
forkChoice updateHead vc 600000 bc 64 eq 0 2.2927 ms/op 3.4957 ms/op 0.66
forkChoice updateHead vc 1000000 bc 64 eq 0 3.8109 ms/op 6.4520 ms/op 0.59
forkChoice updateHead vc 600000 bc 320 eq 0 2.1076 ms/op 4.0560 ms/op 0.52
forkChoice updateHead vc 600000 bc 1200 eq 0 2.1523 ms/op 4.3226 ms/op 0.50
forkChoice updateHead vc 600000 bc 7200 eq 0 2.3756 ms/op 4.8020 ms/op 0.49
forkChoice updateHead vc 600000 bc 64 eq 1000 9.6013 ms/op 11.962 ms/op 0.80
forkChoice updateHead vc 600000 bc 64 eq 10000 9.4541 ms/op 11.725 ms/op 0.81
forkChoice updateHead vc 600000 bc 64 eq 300000 10.832 ms/op 38.716 ms/op 0.28
computeDeltas 500000 validators 300 proto nodes 3.1544 ms/op 4.6132 ms/op 0.68
computeDeltas 500000 validators 1200 proto nodes 3.1726 ms/op 4.5069 ms/op 0.70
computeDeltas 500000 validators 7200 proto nodes 3.1809 ms/op 4.4218 ms/op 0.72
computeDeltas 750000 validators 300 proto nodes 4.7031 ms/op 6.7260 ms/op 0.70
computeDeltas 750000 validators 1200 proto nodes 4.5887 ms/op 6.8424 ms/op 0.67
computeDeltas 750000 validators 7200 proto nodes 4.5549 ms/op 6.5487 ms/op 0.70
computeDeltas 1400000 validators 300 proto nodes 8.8848 ms/op 11.371 ms/op 0.78
computeDeltas 1400000 validators 1200 proto nodes 9.2420 ms/op 11.477 ms/op 0.81
computeDeltas 1400000 validators 7200 proto nodes 9.0397 ms/op 11.285 ms/op 0.80
computeDeltas 2100000 validators 300 proto nodes 13.743 ms/op 17.330 ms/op 0.79
computeDeltas 2100000 validators 1200 proto nodes 13.987 ms/op 17.647 ms/op 0.79
computeDeltas 2100000 validators 7200 proto nodes 13.755 ms/op 17.656 ms/op 0.78
altair processAttestation - 250000 vs - 7PWei normalcase 1.9689 ms/op 2.6667 ms/op 0.74
altair processAttestation - 250000 vs - 7PWei worstcase 2.6970 ms/op 3.3250 ms/op 0.81
altair processAttestation - setStatus - 1/6 committees join 105.01 us/op 143.16 us/op 0.73
altair processAttestation - setStatus - 1/3 committees join 203.82 us/op 269.07 us/op 0.76
altair processAttestation - setStatus - 1/2 committees join 300.32 us/op 369.04 us/op 0.81
altair processAttestation - setStatus - 2/3 committees join 368.38 us/op 494.46 us/op 0.75
altair processAttestation - setStatus - 4/5 committees join 523.59 us/op 637.02 us/op 0.82
altair processAttestation - setStatus - 100% committees join 633.65 us/op 786.14 us/op 0.81
altair processBlock - 250000 vs - 7PWei normalcase 9.8542 ms/op 6.1161 ms/op 1.61
altair processBlock - 250000 vs - 7PWei normalcase hashState 43.808 ms/op 30.116 ms/op 1.45
altair processBlock - 250000 vs - 7PWei worstcase 42.752 ms/op 44.403 ms/op 0.96
altair processBlock - 250000 vs - 7PWei worstcase hashState 89.810 ms/op 84.657 ms/op 1.06
phase0 processBlock - 250000 vs - 7PWei normalcase 1.5688 ms/op 2.2736 ms/op 0.69
phase0 processBlock - 250000 vs - 7PWei worstcase 33.108 ms/op 31.233 ms/op 1.06
altair processEth1Data - 250000 vs - 7PWei normalcase 261.98 us/op 380.67 us/op 0.69
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 6.3980 us/op 5.8650 us/op 1.09
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 40.950 us/op 60.090 us/op 0.68
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 11.512 us/op 13.114 us/op 0.88
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 8.0590 us/op 8.6850 us/op 0.93
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 158.29 us/op 207.55 us/op 0.76
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.4061 ms/op 1.9082 ms/op 0.74
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.8793 ms/op 2.5086 ms/op 0.75
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.8886 ms/op 2.5893 ms/op 0.73
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 3.8653 ms/op 4.9353 ms/op 0.78
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.8769 ms/op 2.8114 ms/op 0.67
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 3.8837 ms/op 5.3063 ms/op 0.73
Tree 40 250000 create 380.15 ms/op 500.17 ms/op 0.76
Tree 40 250000 get(125000) 109.22 ns/op 150.50 ns/op 0.73
Tree 40 250000 set(125000) 2.2666 us/op 2.0349 us/op 1.11
Tree 40 250000 toArray() 24.472 ms/op 21.356 ms/op 1.15
Tree 40 250000 iterate all - toArray() + loop 20.787 ms/op 21.263 ms/op 0.98
Tree 40 250000 iterate all - get(i) 57.673 ms/op 63.173 ms/op 0.91
Array 250000 create 2.1851 ms/op 3.0546 ms/op 0.72
Array 250000 clone - spread 646.03 us/op 1.6843 ms/op 0.38
Array 250000 get(125000) 0.55900 ns/op 0.46600 ns/op 1.20
Array 250000 set(125000) 0.57800 ns/op 0.46300 ns/op 1.25
Array 250000 iterate all - loop 73.976 us/op 110.88 us/op 0.67
phase0 afterProcessEpoch - 250000 vs - 7PWei 39.398 ms/op 44.260 ms/op 0.89
Array.fill - length 1000000 7.0235 ms/op 3.7072 ms/op 1.89
Array push - length 1000000 10.248 ms/op 16.736 ms/op 0.61
Array.get 0.26859 ns/op 0.29772 ns/op 0.90
Uint8Array.get 0.34982 ns/op 0.47688 ns/op 0.73
phase0 beforeProcessEpoch - 250000 vs - 7PWei 17.891 ms/op 22.551 ms/op 0.79
altair processEpoch - mainnet_e81889 281.67 ms/op 303.20 ms/op 0.93
mainnet_e81889 - altair beforeProcessEpoch 15.555 ms/op 17.403 ms/op 0.89
mainnet_e81889 - altair processJustificationAndFinalization 4.6490 us/op 7.3470 us/op 0.63
mainnet_e81889 - altair processInactivityUpdates 3.5562 ms/op 4.7084 ms/op 0.76
mainnet_e81889 - altair processRewardsAndPenalties 41.242 ms/op 38.773 ms/op 1.06
mainnet_e81889 - altair processRegistryUpdates 876.00 ns/op 793.00 ns/op 1.10
mainnet_e81889 - altair processSlashings 401.00 ns/op 193.00 ns/op 2.08
mainnet_e81889 - altair processEth1DataReset 396.00 ns/op 186.00 ns/op 2.13
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.4189 ms/op 1.2336 ms/op 1.15
mainnet_e81889 - altair processSlashingsReset 1.0050 us/op 968.00 ns/op 1.04
mainnet_e81889 - altair processRandaoMixesReset 1.2070 us/op 1.4030 us/op 0.86
mainnet_e81889 - altair processHistoricalRootsUpdate 388.00 ns/op 236.00 ns/op 1.64
mainnet_e81889 - altair processParticipationFlagUpdates 682.00 ns/op 578.00 ns/op 1.18
mainnet_e81889 - altair processSyncCommitteeUpdates 344.00 ns/op 153.00 ns/op 2.25
mainnet_e81889 - altair afterProcessEpoch 39.320 ms/op 45.767 ms/op 0.86
capella processEpoch - mainnet_e217614 952.07 ms/op 1.0271 s/op 0.93
mainnet_e217614 - capella beforeProcessEpoch 62.064 ms/op 73.786 ms/op 0.84
mainnet_e217614 - capella processJustificationAndFinalization 4.4530 us/op 5.6290 us/op 0.79
mainnet_e217614 - capella processInactivityUpdates 12.379 ms/op 16.617 ms/op 0.74
mainnet_e217614 - capella processRewardsAndPenalties 190.27 ms/op 201.18 ms/op 0.95
mainnet_e217614 - capella processRegistryUpdates 7.5390 us/op 6.6170 us/op 1.14
mainnet_e217614 - capella processSlashings 403.00 ns/op 182.00 ns/op 2.21
mainnet_e217614 - capella processEth1DataReset 394.00 ns/op 178.00 ns/op 2.21
mainnet_e217614 - capella processEffectiveBalanceUpdates 3.5232 ms/op 4.4005 ms/op 0.80
mainnet_e217614 - capella processSlashingsReset 1.0400 us/op 908.00 ns/op 1.15
mainnet_e217614 - capella processRandaoMixesReset 1.4270 us/op 1.2320 us/op 1.16
mainnet_e217614 - capella processHistoricalRootsUpdate 404.00 ns/op 180.00 ns/op 2.24
mainnet_e217614 - capella processParticipationFlagUpdates 769.00 ns/op 522.00 ns/op 1.47
mainnet_e217614 - capella afterProcessEpoch 110.74 ms/op 113.95 ms/op 0.97
phase0 processEpoch - mainnet_e58758 336.11 ms/op 302.32 ms/op 1.11
mainnet_e58758 - phase0 beforeProcessEpoch 82.729 ms/op 85.138 ms/op 0.97
mainnet_e58758 - phase0 processJustificationAndFinalization 6.3160 us/op 5.5370 us/op 1.14
mainnet_e58758 - phase0 processRewardsAndPenalties 32.625 ms/op 39.667 ms/op 0.82
mainnet_e58758 - phase0 processRegistryUpdates 2.7140 us/op 3.5320 us/op 0.77
mainnet_e58758 - phase0 processSlashings 387.00 ns/op 199.00 ns/op 1.94
mainnet_e58758 - phase0 processEth1DataReset 378.00 ns/op 176.00 ns/op 2.15
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 929.66 us/op 1.2663 ms/op 0.73
mainnet_e58758 - phase0 processSlashingsReset 1.1310 us/op 981.00 ns/op 1.15
mainnet_e58758 - phase0 processRandaoMixesReset 1.3460 us/op 1.4610 us/op 0.92
mainnet_e58758 - phase0 processHistoricalRootsUpdate 386.00 ns/op 187.00 ns/op 2.06
mainnet_e58758 - phase0 processParticipationRecordUpdates 1.2010 us/op 911.00 ns/op 1.32
mainnet_e58758 - phase0 afterProcessEpoch 32.539 ms/op 37.897 ms/op 0.86
phase0 processEffectiveBalanceUpdates - 250000 normalcase 993.95 us/op 1.9822 ms/op 0.50
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4281 ms/op 3.4376 ms/op 0.42
altair processInactivityUpdates - 250000 normalcase 16.819 ms/op 20.427 ms/op 0.82
altair processInactivityUpdates - 250000 worstcase 17.336 ms/op 23.665 ms/op 0.73
phase0 processRegistryUpdates - 250000 normalcase 5.7980 us/op 7.3730 us/op 0.79
phase0 processRegistryUpdates - 250000 badcase_full_deposits 301.27 us/op 283.05 us/op 1.06
phase0 processRegistryUpdates - 250000 worstcase 0.5 90.465 ms/op 138.32 ms/op 0.65
altair processRewardsAndPenalties - 250000 normalcase 31.376 ms/op 32.722 ms/op 0.96
altair processRewardsAndPenalties - 250000 worstcase 21.219 ms/op 31.508 ms/op 0.67
phase0 getAttestationDeltas - 250000 normalcase 5.1768 ms/op 7.0236 ms/op 0.74
phase0 getAttestationDeltas - 250000 worstcase 5.3764 ms/op 6.7722 ms/op 0.79
phase0 processSlashings - 250000 worstcase 90.864 us/op 104.72 us/op 0.87
altair processSyncCommitteeUpdates - 250000 10.160 ms/op 12.161 ms/op 0.84
BeaconState.hashTreeRoot - No change 436.00 ns/op 247.00 ns/op 1.77
BeaconState.hashTreeRoot - 1 full validator 75.161 us/op 99.259 us/op 0.76
BeaconState.hashTreeRoot - 32 full validator 908.54 us/op 1.3368 ms/op 0.68
BeaconState.hashTreeRoot - 512 full validator 13.358 ms/op 11.894 ms/op 1.12
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 108.08 us/op 117.16 us/op 0.92
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.7933 ms/op 1.9445 ms/op 0.92
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 23.679 ms/op 19.960 ms/op 1.19
BeaconState.hashTreeRoot - 1 balances 95.883 us/op 94.449 us/op 1.02
BeaconState.hashTreeRoot - 32 balances 741.71 us/op 690.87 us/op 1.07
BeaconState.hashTreeRoot - 512 balances 6.1691 ms/op 7.9184 ms/op 0.78
BeaconState.hashTreeRoot - 250000 balances 157.34 ms/op 156.95 ms/op 1.00
aggregationBits - 2048 els - zipIndexesInBitList 27.422 us/op 22.825 us/op 1.20
byteArrayEquals 32 44.737 ns/op 55.804 ns/op 0.80
Buffer.compare 32 14.865 ns/op 18.686 ns/op 0.80
byteArrayEquals 1024 1.2870 us/op 1.6494 us/op 0.78
Buffer.compare 1024 23.279 ns/op 26.047 ns/op 0.89
byteArrayEquals 16384 20.497 us/op 26.288 us/op 0.78
Buffer.compare 16384 202.06 ns/op 200.27 ns/op 1.01
byteArrayEquals 123687377 154.76 ms/op 193.93 ms/op 0.80
Buffer.compare 123687377 5.2332 ms/op 7.2234 ms/op 0.72
byteArrayEquals 32 - diff last byte 47.732 ns/op 55.324 ns/op 0.86
Buffer.compare 32 - diff last byte 16.487 ns/op 18.145 ns/op 0.91
byteArrayEquals 1024 - diff last byte 1.2775 us/op 1.6801 us/op 0.76
Buffer.compare 1024 - diff last byte 24.385 ns/op 27.658 ns/op 0.88
byteArrayEquals 16384 - diff last byte 20.426 us/op 26.588 us/op 0.77
Buffer.compare 16384 - diff last byte 200.09 ns/op 210.76 ns/op 0.95
byteArrayEquals 123687377 - diff last byte 152.87 ms/op 193.96 ms/op 0.79
Buffer.compare 123687377 - diff last byte 6.6432 ms/op 6.2286 ms/op 1.07
byteArrayEquals 32 - random bytes 4.9810 ns/op 5.0440 ns/op 0.99
Buffer.compare 32 - random bytes 15.892 ns/op 16.770 ns/op 0.95
byteArrayEquals 1024 - random bytes 5.0310 ns/op 5.0660 ns/op 0.99
Buffer.compare 1024 - random bytes 15.723 ns/op 16.885 ns/op 0.93
byteArrayEquals 16384 - random bytes 5.0500 ns/op 5.0500 ns/op 1.00
Buffer.compare 16384 - random bytes 15.814 ns/op 16.855 ns/op 0.94
byteArrayEquals 123687377 - random bytes 7.9300 ns/op 6.2200 ns/op 1.27
Buffer.compare 123687377 - random bytes 18.630 ns/op 17.960 ns/op 1.04
regular array get 100000 times 31.242 us/op 32.194 us/op 0.97
wrappedArray get 100000 times 31.254 us/op 42.906 us/op 0.73
arrayWithProxy get 100000 times 59.039 ms/op 13.776 ms/op 4.29
ssz.Root.equals 47.428 ns/op 45.482 ns/op 1.04
byteArrayEquals 38.836 ns/op 44.791 ns/op 0.87
Buffer.compare 8.7660 ns/op 10.107 ns/op 0.87
processSlot - 1 slots 9.5520 us/op 10.110 us/op 0.94
processSlot - 32 slots 3.4722 ms/op 3.6762 ms/op 0.94
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 4.0831 ms/op 2.9180 ms/op 1.40
getCommitteeAssignments - req 1 vs - 250000 vc 1.8150 ms/op 2.1534 ms/op 0.84
getCommitteeAssignments - req 100 vs - 250000 vc 3.6208 ms/op 4.1101 ms/op 0.88
getCommitteeAssignments - req 1000 vs - 250000 vc 3.8382 ms/op 4.3688 ms/op 0.88
findModifiedValidators - 10000 modified validators 821.59 ms/op 737.60 ms/op 1.11
findModifiedValidators - 1000 modified validators 741.93 ms/op 679.05 ms/op 1.09
findModifiedValidators - 100 modified validators 168.13 ms/op 264.59 ms/op 0.64
findModifiedValidators - 10 modified validators 178.34 ms/op 164.07 ms/op 1.09
findModifiedValidators - 1 modified validators 132.75 ms/op 135.21 ms/op 0.98
findModifiedValidators - no difference 143.94 ms/op 150.83 ms/op 0.95
compare ViewDUs 6.1051 s/op 6.4549 s/op 0.95
compare each validator Uint8Array 1.8473 s/op 1.4166 s/op 1.30
compare ViewDU to Uint8Array 799.94 ms/op 945.73 ms/op 0.85
migrate state 1000000 validators, 24 modified, 0 new 878.94 ms/op 925.03 ms/op 0.95
migrate state 1000000 validators, 1700 modified, 1000 new 1.1479 s/op 1.2452 s/op 0.92
migrate state 1000000 validators, 3400 modified, 2000 new 1.2419 s/op 1.4933 s/op 0.83
migrate state 1500000 validators, 24 modified, 0 new 912.22 ms/op 978.80 ms/op 0.93
migrate state 1500000 validators, 1700 modified, 1000 new 1.2277 s/op 1.2980 s/op 0.95
migrate state 1500000 validators, 3400 modified, 2000 new 1.3412 s/op 1.6462 s/op 0.81
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 6.1500 ns/op 4.6300 ns/op 1.33
state getBlockRootAtSlot - 250000 vs - 7PWei 500.64 ns/op 650.80 ns/op 0.77
naive computeProposerIndex 100000 validators 54.746 ms/op 70.904 ms/op 0.77
computeProposerIndex 100000 validators 1.2801 ms/op 1.7684 ms/op 0.72
naiveGetNextSyncCommitteeIndices 1000 validators 6.7993 s/op 10.192 s/op 0.67
getNextSyncCommitteeIndices 1000 validators 95.155 ms/op 163.43 ms/op 0.58
naiveGetNextSyncCommitteeIndices 10000 validators 6.4245 s/op 9.1834 s/op 0.70
getNextSyncCommitteeIndices 10000 validators 91.075 ms/op 133.38 ms/op 0.68
naiveGetNextSyncCommitteeIndices 100000 validators 6.0897 s/op 7.8684 s/op 0.77
getNextSyncCommitteeIndices 100000 validators 93.396 ms/op 125.35 ms/op 0.75
naive computeShuffledIndex 100000 validators 22.297 s/op 28.493 s/op 0.78
cached computeShuffledIndex 100000 validators 468.07 ms/op 599.77 ms/op 0.78
naive computeShuffledIndex 2000000 validators 414.67 s/op 616.06 s/op 0.67
cached computeShuffledIndex 2000000 validators 20.510 s/op 44.529 s/op 0.46
computeProposers - vc 250000 530.72 us/op 647.83 us/op 0.82
computeEpochShuffling - vc 250000 38.290 ms/op 43.835 ms/op 0.87
getNextSyncCommittee - vc 250000 9.3302 ms/op 11.162 ms/op 0.84
computeSigningRoot for AttestationData 16.143 us/op 22.310 us/op 0.72
hash AttestationData serialized data then Buffer.toString(base64) 1.1988 us/op 1.6716 us/op 0.72
toHexString serialized data 935.28 ns/op 1.2698 us/op 0.74
Buffer.toString(base64) 107.18 ns/op 157.50 ns/op 0.68
nodejs block root to RootHex using toHex 107.25 ns/op 148.09 ns/op 0.72
nodejs block root to RootHex using toRootHex 69.966 ns/op 93.407 ns/op 0.75
nodejs fromhex(blob) 96.481 ms/op 117.34 ms/op 0.82
nodejs fromHexInto(blob) 76.796 ms/op 98.268 ms/op 0.78
browser block root to RootHex using the deprecated toHexString 178.88 ns/op 228.75 ns/op 0.78
browser block root to RootHex using toHex 150.99 ns/op 186.07 ns/op 0.81
browser block root to RootHex using toRootHex 141.40 ns/op 171.27 ns/op 0.83
browser fromHexInto(blob) 661.04 us/op 859.46 us/op 0.77
browser fromHex(blob) 652.44 ms/op 848.90 ms/op 0.77

by benchmarkbot/action

nflaig
nflaig reviewed Sep 5, 2025
View reviewed changes
packages/db/src/abstractPrefixedRepository.ts

if(!hasUpper || !hasLower) {
throw new Error("Please provide both ranges to fetch the keys")
}
Copy link
Member

@nflaig nflaig Sep 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to have default keys defined here as we do in Repository?

the reason why keys() returns other values is because PrefixedRepository doesn't have default values

this part is missing

image

lodestar/packages/db/src/abstractRepository.ts

Line 255 in 083b7cc

protected dbFilterOptions(opts?: FilterOptions<I>): FilterOptions<Uint8Array> {

twoeths
twoeths reviewed Sep 6, 2025
View reviewed changes
Copy link
Contributor

@twoeths twoeths left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this issue should have been detected earlier, looks like we missed unit tests for these functions in order to reproduce/detect it?

@nazarhussain
Copy link
Contributor Author

nazarhussain commented Sep 8, 2025

this issue should have been detected earlier, looks like we missed unit tests for these functions in order to reproduce/detect it?

Yes I fully agree, this should have been detected earlier. Problem I feel is that key encoding to the be was such a vital information and had not enough comments/emphasis on it and neither it had any test coverage earlier.

Will be adding extensive tests right now.

@nazarhussain nazarhussain marked this pull request as draft September 8, 2025 08:42
@nazarhussain nazarhussain marked this pull request as ready for review September 8, 2025 13:04
@nazarhussain
Copy link
Contributor Author

nazarhussain commented Sep 8, 2025 •
edited
Loading

Tried the devnet-3 node locally and the metrics for db write for dataColumn seems correct. Trying to get more information on devnet-3 live node.

image

nflaig
nflaig reviewed Sep 8, 2025
View reviewed changes
packages/beacon-node/src/chain/archiveStore/utils/archiveBlocks.ts Outdated
const slotsToDelete = (
await db.dataColumnSidecarArchive.keys({
lt: db.dataColumnSidecarArchive.getMaxKeyRaw(computeStartSlotAtEpoch(dataColumnSidecarsMinEpoch)),
lt: {prefix: computeStartSlotAtEpoch(dataColumnSidecarsMinEpoch), id: 0},
Copy link
Member

@nflaig nflaig Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does id: 0 do? I suppose it's column indices related, but that's not really clear from reading the code

Copy link
Contributor Author

@nazarhussain nazarhussain Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it means column indices, it's the attribute name comes from generic abstract class. So far no way to support dynamic attribute names from generics, so we have to stick with it.

@nflaig nflaig mentioned this pull request Sep 9, 2025
Merged
wemeetagain
wemeetagain reviewed Sep 9, 2025
View reviewed changes
packages/beacon-node/src/db/repositories/dataColumnSidecarArchive.ts Outdated
return Buffer.concat([intToBytes(prefix, 4), intToBytes(NUMBER_OF_COLUMNS, 4)]);
return Buffer.concat([
encodeNumberForDbKey(prefix, slotByteSize),
encodeNumberForDbKey(NUMBER_OF_COLUMNS, columnIndexByteSize),
Copy link
Member

@wemeetagain wemeetagain Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is inclusive, then it it should be NUMBER_OF_COLUMNS - 1.
(mixing an index with a length)

Copy link
Member

@wemeetagain wemeetagain Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Else update the docstrings and don't use lte in *Stream functions below

Copy link
Contributor Author

@nazarhussain nazarhussain Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to have inclusive range on both min/max key to have consistency. Updated the keys range.

nflaig
nflaig reviewed Sep 9, 2025
View reviewed changes
@wemeetagain wemeetagain merged commit 1817ab1 into unstable Sep 9, 2025
17 of 20 checks passed
@wemeetagain wemeetagain deleted the nh/fix-data-column-query-delete branch September 9, 2025 16:04
This was referenced Sep 10, 2025
Closed
Closed
Closed
@wemeetagain
Copy link
Member

wemeetagain commented Oct 8, 2025

🎉 This PR is included in v1.35.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

@nflaig nflaig nflaig left review comments

@twoeths twoeths Awaiting requested review from twoeths

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nazarhussain @wemeetagain @twoeths @nflaig