ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

gargnitingoogle · 2025-07-17T09:48:19Z

Description

This pull request significantly enhances the e2e test package rapid_appends functionality in GCSFuse. The primary focus is on expanding test coverage to include various metadata and file cache configurations (b/433142062), ensuring correct read behavior for unfinalized objects, especially around metadata cache expiry (b/433143404). Concurrently, the test suite structure has undergone a substantial refactoring (b/433143407) to improve maintainability and provide more granular control over test environments.

Highlights

Expanded Test Coverage for Caching: New test configurations have been added to the rapid_appends test package, specifically enabling and disabling metadata and file caches. This ensures comprehensive validation of read operations on unfinalized objects under different caching scenarios.
Metadata Cache Expiry Validation: The tests now explicitly verify that reads return pre-append and post-append sizes correctly when metadata cache is enabled, by reading before and after the metadata TTL expiry.
Refactored Test Suite Architecture: The rapid_appends test package has been refactored into three distinct test suites: RapidAppendsSuite (base), SingleMountRapidAppendsSuite, and DualMountRapidAppendsSuite. This new structure allows each test to control its own mount flag sets, improving test isolation and management.

Changelog

tools/integration_tests/rapid_appends/appends_test.go
- Removed boilerplate RapidAppendsSuite and its helper methods, along with the TestAppendsAndRead function.
- Updated TestAppendSessionInvalidatedByAnotherClientUponTakeover and TestContentAppendedInNonAppendModeNotVisibleTillClose to run with various cache configurations, managing mount points within the test logic.
tools/integration_tests/rapid_appends/reads_after_appends_test.go
- New file: Introduced TestAppendsAndReads to specifically validate read consistency after appends across different metadata and file cache settings, including checks for metadata TTL expiry.
tools/integration_tests/rapid_appends/setup_test.go
- Removed global mount-related variables and simplified the TestMain function, delegating mount setup to the new test suites.
- Added new constants for metadata cache, file cache, and rapid appends flags.
tools/integration_tests/rapid_appends/suites_test.go
- New file: Introduced a new test suite architecture with RapidAppendsSuite (base), SingleMountRapidAppendsSuite, and DualMountRapidAppendsSuite to manage test environments and mount configurations more effectively.
- Added helper methods for mount point management, object creation/deletion, and file appending within the new suite structure.

What is planned in next PR(s):

Add support for sequential/random read configurations (b/433142769). This is WIP in ci(read unfinalized objects): Extend rapid_appends e2e tests - sequential and random read tests #3543 .

Link to the issue in case of a bug fix.

b/433142062
b/433143407
b/433143404

Testing details

Manual - NA
Unit tests - NA
Integration tests - Ran as part of presubmit: run1, run2, run3, run4, run5, run6 .

Any backward incompatible change? If so, please explain.

gargnitingoogle · 2025-07-17T10:19:54Z

/gemini review

gemini-code-assist

Code Review

This pull request extends the e2e tests for rapid appends to cover more scenarios, including different combinations of metadata and file caching. The changes are well-structured, using a scenario-based approach in TestMain to run tests against multiple configurations. This makes the test suite more robust and easier to extend in the future.

tools/integration_tests/rapid_appends/setup_test.go

codecov · 2025-07-18T06:06:07Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (af30226) to head (21d6b1a).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #3539       +/-   ##
==========================================
- Coverage   79.68%       0   -79.69%     
==========================================
  Files         142       0      -142     
  Lines       18971       0    -18971     
==========================================
- Hits        15117       0    -15117     
+ Misses       3342       0     -3342     
+ Partials      512       0      -512

Flag	Coverage Δ
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

gargnitingoogle · 2025-07-24T06:33:30Z

Converted to draft, as I am refactoring this code based on my discussion with @vadlakondaswetha . Will reopen for review soon.

github-actions · 2025-07-24T11:21:05Z

/gemini summary
/gemini review

tools/integration_tests/rapid_appends/appends_test.go

gargnitingoogle · 2025-07-24T11:49:41Z

/gemini review

gargnitingoogle · 2025-07-24T11:49:48Z

/gemini summary

gemini-code-assist

Code Review

This pull request refactors the rapid_appends integration tests, introducing a suite-based structure that improves organization and maintainability. The separation into RapidAppendsSuite, SingleMountRapidAppendsSuite, and DualMountRapidAppendsSuite handles different test scenarios effectively. The tests are extended to cover various metadata and file cache configurations. I have provided feedback on a potential bug in a test assertion, some redundant code, and minor style issues.

tools/integration_tests/rapid_appends/appends_test.go

tools/integration_tests/rapid_appends/reads_after_appends_test.go

tools/integration_tests/rapid_appends/setup_test.go

meet2mky

Added few comments, PTAL.

tools/integration_tests/rapid_appends/setup_test.go

tools/integration_tests/rapid_appends/suites_test.go

tools/integration_tests/rapid_appends/appends_test.go

tools/integration_tests/rapid_appends/setup_test.go

tools/integration_tests/rapid_appends/reads_after_appends_test.go

tools/integration_tests/rapid_appends/suites_test.go

tools/integration_tests/rapid_appends/reads_after_appends_test.go

- add test configurations for metadata cache enable/disable - split test suite between single and dual mounts add configurable options in test change to prefix-check for metadata cache enabled test add more test scenarios pure refactor pure refactor pure refactor pure refactor some cosmetic changes based on self-review fix rebase issue restructure primaryMount and secondaryMount remove unwanted code refactor minor cleanup minor cleanup minor fixes address self-review comments Restrict secondary mount to RapidAppendsSuite This restricts primary mount to TestMain and secondary mount to RapidAppendsSuite class. minor code cleanup split RapidAppendsSuite into SingleMount and DualMount cleanup minor refactor refactor code and clean-up Move testdir creation to after mount for primary mount move saving primary mount logfile to TestMain Revert "move saving primary mount logfile to TestMain" This reverts commit de1b76d. address a review comment Address a self-review comment hasSecondaryMount -> enableSecondaryMount Address another review comment Split RapidAppendsSuite into single-mount and dual-mount. check read content size before and after metadata cache ttl expiry address some review comments Refactor rapid_appends tests cleanup further refactor wip stable passing point clear up fmt printf calls minor refactor another minor refactor more refactor Switch one of the tests from DualMount to SingleMount address gemini comments

fix a rebase issue fix another rebase issue fix another rebase issue

gargnitingoogle · 2025-07-25T10:41:23Z

@vadlakondaswetha @meet2mky please review. I have addressed all your comments.

gargnitingoogle added the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

This comment was marked as outdated.

Sign in to view

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch 3 times, most recently from d6c295a to 64978d8 Compare July 17, 2025 10:01

gargnitingoogle removed the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

gargnitingoogle changed the base branch from meet2mky/append_files_functional_tests to master July 17, 2025 10:06

gargnitingoogle added the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from 64978d8 to c9bf214 Compare July 17, 2025 10:09

gargnitingoogle removed the execute-integration-tests-on-zb To run E2E tests on zonal bucket. label Jul 17, 2025

gargnitingoogle changed the base branch from master to meet2mky/append_files_functional_tests July 17, 2025 10:16

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from c9bf214 to ce5273f Compare July 17, 2025 10:16

gargnitingoogle added the remind-reviewers Auto remind reviewers in attention set for review post 24hrs of inactivity on PR. label Jul 17, 2025

gargnitingoogle marked this pull request as ready for review July 17, 2025 10:19

gargnitingoogle requested review from Tulsishah and a team as code owners July 17, 2025 10:19

gargnitingoogle requested review from ashmeenkaur and removed request for a team July 17, 2025 10:19

gemini-code-assist bot reviewed Jul 17, 2025

View reviewed changes

tools/integration_tests/rapid_appends/setup_test.go Outdated Show resolved Hide resolved

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch 4 times, most recently from 0e72f62 to 3d71359 Compare July 17, 2025 11:25

gargnitingoogle mentioned this pull request Jul 18, 2025

ci(read unfinalized objects): Add new e2e tests for reads of unfinalized objects while being appended #3473

Merged

gargnitingoogle changed the base branch from meet2mky/append_files_functional_tests to master July 18, 2025 05:01

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from 3d71359 to 232b85c Compare July 18, 2025 05:48

gargnitingoogle changed the base branch from master to meet2mky/append_files_functional_tests July 18, 2025 05:49

gargnitingoogle marked this pull request as draft July 24, 2025 06:32

auto-merge was automatically disabled July 24, 2025 06:32
Pull request was converted to draft

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from 0e59df9 to 764f166 Compare July 24, 2025 11:17

gargnitingoogle requested a review from meet2mky July 24, 2025 11:20

gargnitingoogle marked this pull request as ready for review July 24, 2025 11:20

kislaykishore requested a review from a team July 24, 2025 11:21

gargnitingoogle enabled auto-merge (squash) July 24, 2025 11:21

gargnitingoogle commented Jul 24, 2025

View reviewed changes

tools/integration_tests/rapid_appends/appends_test.go Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Jul 24, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch 2 times, most recently from 86cc6fd to 6cadf98 Compare July 24, 2025 12:56

meet2mky reviewed Jul 25, 2025

View reviewed changes

vadlakondaswetha reviewed Jul 25, 2025

View reviewed changes

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from 85d2290 to c36da0b Compare July 25, 2025 07:08

gargnitingoogle added 3 commits July 25, 2025 10:04

fix a linter error and move some code around

6f0be46

Address review comments

2415fe2

fix a rebase issue fix another rebase issue fix another rebase issue

gargnitingoogle force-pushed the gargnitin/extend-read-rapid-appends-e2e-tests/v1 branch from c36da0b to 2415fe2 Compare July 25, 2025 10:08

fix linter and a comment

67b9c93

gargnitingoogle requested review from meet2mky, anushka567 and vadlakondaswetha July 25, 2025 10:28

gargnitingoogle added 2 commits July 25, 2025 11:42

replace all gcsfuse flag variables/constants with literals

7cd3bad

fix format error

aed6ef1

ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

Are you sure you want to change the base?

ci(read unfinalized objects): Extends (add md-cache and file-cache configs) and refactors rapid_appends e2e tests #3539

Conversation

gargnitingoogle commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Highlights

What is planned in next PR(s):

Link to the issue in case of a bug fix.

Testing details

Any backward incompatible change? If so, please explain.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

gargnitingoogle commented Jul 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

codecov bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gargnitingoogle commented Jul 24, 2025

Uh oh!

github-actions bot commented Jul 24, 2025

Uh oh!

Uh oh!

gargnitingoogle commented Jul 24, 2025

Uh oh!

gargnitingoogle commented Jul 24, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

meet2mky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gargnitingoogle commented Jul 25, 2025

Uh oh!

Uh oh!

gargnitingoogle commented Jul 17, 2025 •

edited

Loading

codecov bot commented Jul 18, 2025 •

edited

Loading