ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

gargnitingoogle · 2025-07-17T09:48:19Z

Description

~~This is dependent on ci(read unfinalized objects): Add new e2e tests for reads of unfinalized objects while being appended #3473 .~~
This adds more test configurations i.e. metadata-cache on/off, file-cache on/off for the read while appending tests for unfinalized objects in rapid_appends test package (b/433142062).
- For metadata-cache enable cases, it reads before and after metadata ttl expiry to confirm that the reads should return pre-append and post-append sizes respectively (b/433143404)
Refactors the code in rapid_appends test package (b/433143407), primarily targeted at
- Moving management of secondary mount from setup_test.go to RapidAppendsSuite
- Splitting RapidAppendsSuite into single-mount and dual-mount suites
- configuration generation in TestMain to support more configuration

What is planned in next PR(s):

Move the primary mount, its flags/configs etc. also to RapidAppendsSuite .
Add support for sequential/random read configurations (b/433142769). This is WIP in ci(read unfinalized objects): Extend rapid_appends e2e tests - sequential and random read tests #3543 .

Link to the issue in case of a bug fix.

b/433142062
b/433143407
b/433143404

Testing details

Manual - NA
Unit tests - NA
Integration tests - Ran as part of presubmit: run1, run2, run3 .

Any backward incompatible change? If so, please explain.

gargnitingoogle · 2025-07-17T10:19:54Z

/gemini review

gemini-code-assist

Code Review

This pull request extends the e2e tests for rapid appends to cover more scenarios, including different combinations of metadata and file caching. The changes are well-structured, using a scenario-based approach in TestMain to run tests against multiple configurations. This makes the test suite more robust and easier to extend in the future.

tools/integration_tests/rapid_appends/setup_test.go

codecov · 2025-07-18T06:06:07Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (af30226) to head (21d6b1a).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #3539       +/-   ##
==========================================
- Coverage   79.68%       0   -79.69%     
==========================================
  Files         142       0      -142     
  Lines       18971       0    -18971     
==========================================
- Hits        15117       0    -15117     
+ Misses       3342       0     -3342     
+ Partials      512       0      -512

Flag	Coverage Δ
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This restricts primary mount to TestMain and secondary mount to RapidAppendsSuite class.

This reverts commit de1b76d.

hasSecondaryMount -> enableSecondaryMount

Split RapidAppendsSuite into single-mount and dual-mount.

gargnitingoogle · 2025-07-23T21:56:34Z

tools/integration_tests/rapid_appends/rapid_appends_test.go

-					assert.Truef(t.T(), strings.HasPrefix(t.fileContent, readContent), "With metadata-enabled, read content expected to be a prefix of the written content, but failed. Written content = %q, Read content = %q", t.fileContent, readContent)
+					assert.Equalf(t.T(), t.fileContent[:sizeBeforeAppend], readContent, "failed to match partial content in metadata-cache dual-mount after %v appends", i+1)
+
+					time.Sleep(time.Duration(metadataCacheTTLSecs) * time.Second) // Wait for metadata cache to be updated.


s/updated/expired

gargnitingoogle added the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

This comment was marked as outdated.

Sign in to view

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch 3 times, most recently from d6c295a to 64978d8 Compare July 17, 2025 10:01

gargnitingoogle removed the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

gargnitingoogle changed the base branch from meet2mky/append_files_functional_tests to master July 17, 2025 10:06

gargnitingoogle added the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from 64978d8 to c9bf214 Compare July 17, 2025 10:09

gargnitingoogle removed the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

gargnitingoogle changed the base branch from master to meet2mky/append_files_functional_tests July 17, 2025 10:16

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from c9bf214 to ce5273f Compare July 17, 2025 10:16

gargnitingoogle added the remind-reviewers Auto remind reviewers in attention set for review post 24hrs of inactivity on PR. label Jul 17, 2025

gargnitingoogle marked this pull request as ready for review July 17, 2025 10:19

gargnitingoogle requested review from Tulsishah and a team as code owners July 17, 2025 10:19

gargnitingoogle requested review from ashmeenkaur and removed request for a team July 17, 2025 10:19

gemini-code-assist bot reviewed Jul 17, 2025

View reviewed changes

tools/integration_tests/rapid_appends/setup_test.go Outdated Show resolved Hide resolved

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch 4 times, most recently from 0e72f62 to 3d71359 Compare July 17, 2025 11:25

gargnitingoogle mentioned this pull request Jul 18, 2025

ci(read unfinalized objects): Add new e2e tests for reads of unfinalized objects while being appended #3473

Merged

gargnitingoogle changed the base branch from meet2mky/append_files_functional_tests to master July 18, 2025 05:01

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from 3d71359 to 232b85c Compare July 18, 2025 05:48

gargnitingoogle changed the base branch from master to meet2mky/append_files_functional_tests July 18, 2025 05:49

gargnitingoogle added 28 commits July 23, 2025 20:53

change to prefix-check for metadata cache enabled test

c5bffeb

add more test scenarios

599fb8a

pure refactor

df5d8ab

pure refactor

7470d72

pure refactor

846ff86

pure refactor

4df1fcb

some cosmetic changes based on self-review

d32507b

fix rebase issue

d2393b6

restructure primaryMount and secondaryMount

05fd7df

remove unwanted code

7c45966

refactor

ffba22b

minor cleanup

dd059f8

minor cleanup

56d3461

minor fixes

c731694

address self-review comments

19cd829

Restrict secondary mount to RapidAppendsSuite

21a64ca

This restricts primary mount to TestMain and secondary mount to RapidAppendsSuite class.

minor code cleanup

9ef458b

split RapidAppendsSuite into SingleMount and DualMount

f06889d

cleanup

45e6c13

minor refactor

0092011

refactor code and clean-up

9980e8e

Move testdir creation to after mount for primary mount

5aafa56

move saving primary mount logfile to TestMain

4e3c72a

Revert "move saving primary mount logfile to TestMain"

56c1941

This reverts commit de1b76d.

address a review comment

acfd8d6

Address a self-review comment

41190f1

hasSecondaryMount -> enableSecondaryMount

Address another review comment

bbf97cd

Split RapidAppendsSuite into single-mount and dual-mount.

check read content size before and after metadata cache ttl expiry

0e04cba

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from ed09afe to 0e04cba Compare July 23, 2025 21:17

gargnitingoogle commented Jul 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

Uh oh!

gargnitingoogle commented Jul 17, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

gargnitingoogle commented Jul 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

codecov bot commented Jul 18, 2025 •

edited

Loading

Uh oh!

gargnitingoogle Jul 23, 2025

Uh oh!

Uh oh!

ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

Are you sure you want to change the base?

ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

Uh oh!

Conversation

gargnitingoogle commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What is planned in next PR(s):

Link to the issue in case of a bug fix.

Testing details

Any backward incompatible change? If so, please explain.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

gargnitingoogle commented Jul 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

codecov bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gargnitingoogle Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gargnitingoogle commented Jul 17, 2025 •

edited

Loading

codecov bot commented Jul 18, 2025 •

edited

Loading