feat(core): scanner v2 #6989

arendjr · 2025-07-23T18:34:25Z

Summary

Fixed #6965: Implemented smarter scanner for project rules.

Previously, if project rules were enabled Biome's scanner would scan all dependencies regardless of whether they were used by/reachable from source files or not. While this worked for a first version, it was far from optimal.

The new scanner first scans everything listed under the files.includes setting, and then descends into the dependencies that were discovered there, including transitive dependencies. This has three main advantages:

Dependencies that are not reachable from your source files don't get indexed.
Dependencies that have multiple type definitions, such as those with separate definitions for CommonJS and ESM imports, only have the relevant definitions indexed.
If vcs.useIgnoreFile is enabled, .gitignore gets respected as well. Assuming you have folders such as build/ or dist/ configured there, those will be automatically ignored by the scanner.

In addition, the following refactorings have been made:

The scanner has been moved out of the workspace module. The watcher has become a proper submodule of the scanner.
Both scanner and watcher have "bridge" traits now. They no longer take a reference to the workspace directly, and they don't share methods with the workspace anymore either. Instead, they take a reference to a bridge implementation which allows their communication with the workspace. The scanner bridge trait is still implemented directly on the WorkspaceServer (and the watcher bridge trait on top of that), so there's no/hardly any actual overhead, but it provides a much cleaner separation of responsibilities. As an added benefit, the traits allow for much easier unit testing of the scanner and watcher in isolation, which I've done for the watcher already.
is_opened_by_scanner has been renamed to is_indexed. The scanner and watcher also have their terminology changed so they no longer open/close a file, but they index/unload it. This is to highlight the differences between the scanner behaviour when they load a file and the behaviour when a client opens a file.

Test Plan

Tests added and expanded. It will take me a bit longer to get everything green already, but I thought it was ready for review at least.

Docs

biomejs/website#2853

changeset-bot · 2025-07-23T18:34:29Z

🦋 Changeset detected

Latest commit: a0bd572

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 14 packages

Name	Type
@biomejs/biome	Minor
@biomejs/wasm-bundler	Minor
@biomejs/wasm-nodejs	Minor
@biomejs/wasm-web	Minor
@biomejs/js-api	Major
@biomejs/cli-win32-x64	Minor
@biomejs/cli-win32-arm64	Minor
@biomejs/cli-darwin-x64	Minor
@biomejs/cli-darwin-arm64	Minor
@biomejs/cli-linux-x64	Minor
@biomejs/cli-linux-arm64	Minor
@biomejs/cli-linux-x64-musl	Minor
@biomejs/cli-linux-arm64-musl	Minor
@biomejs/backend-jsonrpc	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

codspeed-hq · 2025-07-23T18:48:00Z

CodSpeed Performance Report

Merging #6989 will not alter performance

_{Comparing arendjr:scanner-v2 (a0bd572) with next (d6e68b0)}

Summary

✅ 128 untouched benchmarks

.changeset/sane-scanner-smiles.md

ematipico · 2025-07-23T19:52:37Z

Can we run the ecosystem CI before merging and make sure we didn't introduce any regressions?

Conaclos · 2025-07-23T21:43:01Z

Nice! Have you performed some benchmarks/tests on big project to see the improvements?

crates/biome_lsp/src/session.rs

crates/biome_module_graph/src/module_graph.rs

ematipico · 2025-07-23T22:29:58Z

crates/biome_module_graph/src/module_graph.rs

-            imports.insert(path.to_path_buf(), visitor.collect_info());
+
+            let module_info = visitor.collect_info();
+            for import_path in module_info.all_import_paths() {


This is just amazing and mind-blowing!

crates/biome_service/src/scanner/workspace_bridges.rs

ematipico · 2025-07-23T22:38:46Z

crates/biome_service/src/scanner/workspace_bridges.rs

+where
+    T: WorkspaceScannerBridge,
+{
+    #[inline]


Could you test the implication of all these #[inline] whenever you can? i.e. check if there's a difference when removing them. Thank you

ematipico

I just tested this PR against a codebase, and it doesn't work anymore. Even without project rules, the CLI doesn't finish.

Without project rules, it's stuck. With debug logs, it only prints

2025-07-24T11:56:01.887489Z  INFO main crates/biome_cli/src/commands/mod.rs: Configuration file loaded: Some("**/biome.jsonc"), diagnostics detected 1
2025-07-24T11:56:01.888581Z DEBUG main insert_project: crates/biome_service/src/projects.rs: Insert workspace folder **

With a project rule enabled, it starts logging few index_file:open_file_internal, but then it stops, it doesn't log anything anymore, the memory of the process starts increasing slowly.

Blocking for now

arendjr · 2025-07-24T16:09:10Z

Yeah, sorry, the tests aren’t all green yet either, and it hang to cancel the runs yesterday because something was hanging indeed. Definitely needs some fixes still before it’s done.

@Conaclos Also still waiting for the tests to get green, then I’ll do some comparisons.

ematipico · 2025-07-31T09:37:37Z

Fixed the failures in 8a919a6 (#6989)

I think we can merge this PR to main because it is just an internal refactor that doesn't introduce any new APIs for the user. Considering that we are still receiving bug reports around leaks, we need this refactor more than ever, ASAP.

arendjr · 2025-08-03T19:04:04Z

Thanks, nice work!

I’ll have a closer look tomorrow, but just to temper expectations: I don’t think this PR will help much with the reports of leaks, because it does nothing except possibly reduce the amount of .d.ts files that are indexed. I haven’t even done benchmarks yet and before my holiday I wasn’t entirely confident with the current tests yet. I think it will improve performance for those using project rules, possibly significantly, but don’t expect any miracles 🙏

arendjr · 2025-08-04T07:55:21Z

Oh, I'm afraid the CI status was misleading. Tests are actually still failing here. I'm on it!

arendjr · 2025-08-04T15:25:24Z

How does this fix impact the LSP behaviour? I know we have a few tests in place, I wonder if a manual test would work.

I've done some manual testing indeed, and it appears the LSP behaviour is good. At least I've not gotten any CPU spikes in repositories where previously I could reproduce it, and I've tried running builds while the LSP is open too.

Regarding the entity of the change, from a user point of view prospective, it's still a fix. We essentially botched how we interpreted globs with folders and how we documented them. I don't have strong opinions, however this PR must have a docs PR, so we update our guides about how users can ignore folders and files (hopefully CLI and LSP behave the same)

Yes, I agree it's a fix. The reason I'm hesitant though is that we can anticipate that some users may require config changes, which makes it technically a breaking change. I don't think it's such a major change however, because it is still a fix. So if we're really strict in one direction, we would require releasing a v3, but if we don't care at all, it can go in a patch release. I think neither extreme is really helpful though, I think it deserves a small shoutout for users, so a minor it is :) We never strictly followed semver anyway...

arendjr · 2025-08-04T18:38:22Z

Bloody Windows 😅

@Conaclos I tried two open-source projects (unleash and aigne-framework), both with project rules enabled. One became twice as fast, the other almost :)

coderabbitai · 2025-08-05T12:59:09Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai generate unit tests to generate unit tests for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

# Conflicts: # crates/biome_lsp/src/server.rs # crates/biome_service/src/workspace/scanner.tests.rs # crates/biome_service/src/workspace/server.rs # crates/biome_service/src/workspace/server.tests.rs # crates/biome_service/src/workspace/watcher.tests.rs # crates/biome_service/tests/spec_tests.rs # packages/@biomejs/backend-jsonrpc/src/workspace.ts

siketyan

Awesome work!

crates/biome_service/src/scanner/watcher.rs

crates/biome_service/src/workspace/server.rs

siketyan · 2025-08-05T17:05:45Z

Just tested on my private large repository. No issues so far, and got faster ~1s.

v2.1.3

❯ time ./node_modules/.pnpm/@biomejs+cli-darwin-arm64@2.1.3/node_modules/@biomejs/cli-darwin-arm64/biome check
Checked 5251 files in 6s. No fixes applied.

________________________________________________________
Executed in    7.12 secs    fish           external
   usr time   43.87 secs    0.25 millis   43.87 secs
   sys time   13.83 secs    1.92 millis   13.83 secs

78f3460

❯ time ~/Projects/github.com/biomejs/biome/target/release-with-debug/biome check
Checked 5251 files in 5s. No fixes applied.

________________________________________________________
Executed in    5.99 secs    fish           external
   usr time   38.55 secs    0.20 millis   38.55 secs
   sys time   15.87 secs    1.47 millis   15.86 secs

arendjr · 2025-08-05T17:09:23Z

Thanks for confirming, @siketyan !

PR with website docs is ready too: biomejs/website#2853

Co-authored-by: Emanuele Stoppa <my.burning@gmail.com> Co-authored-by: Naoki Ikeguchi <me@s6n.jp>

flying-sheep · 2025-08-19T09:09:39Z

crates/biome_cli/tests/cases/included_files.rs

+    fs.insert(
+        file_path.into(),
+        r#"{
+  "files": { "includes": ["test.js", "!**/folder"] }
+}
+"#
+        .as_bytes(),
+    );
+
+    let test = Utf8Path::new("test.js");
+    fs.insert(test.into(), UNFORMATTED.as_bytes());
+
+    let test2 = Utf8Path::new("folder/test2.js");
+    fs.insert(test2.into(), UNFORMATTED.as_bytes());


this test doesn’t actually test what you want it to, see #7268 (comment)

so this whole feature was shipped completely broken and is breaking user code as we speak.

arendjr requested review from a team July 23, 2025 18:34

arendjr force-pushed the scanner-v2 branch from bcb3134 to d443a5b Compare July 23, 2025 18:34

github-actions bot added A-CLI Area: CLI A-Project Area: project A-LSP Area: language server protocol labels Jul 23, 2025

This comment was marked as resolved.

Sign in to view

arendjr changed the base branch from main to next July 23, 2025 18:43

arendjr force-pushed the scanner-v2 branch from d443a5b to ccaade0 Compare July 23, 2025 18:45

This comment was marked as resolved.

Sign in to view

ematipico reviewed Jul 23, 2025

View reviewed changes

.changeset/sane-scanner-smiles.md Outdated Show resolved Hide resolved

.changeset/sane-scanner-smiles.md Show resolved Hide resolved

ematipico reviewed Jul 23, 2025

View reviewed changes

ematipico previously requested changes Jul 24, 2025

View reviewed changes

arendjr force-pushed the scanner-v2 branch from e5407e4 to dba87d7 Compare July 24, 2025 18:55

arendjr and others added 7 commits August 4, 2025 10:30

feat(core): scanner v2

7091d11

don't store indexed files among documents

55ab45a

optimise symlink handling

7e4a215

better named bridge

5c69911

fix: scanner of manifests and configs

f07bc1e

avoid pulling manifests from documents

53d7e95

test fixes

8ec4d30

arendjr force-pushed the scanner-v2 branch from 811239d to 8ec4d30 Compare August 4, 2025 09:18

arendjr added 4 commits August 4, 2025 15:20

fixes

a4df40c

update changeset

8c4ecd5

test fixes

1f851e4

stabilise test

66e1c2c

arendjr mentioned this pull request Aug 5, 2025

🐛 Biome taking up all the CPU #7020

Closed

1 task

properly unload index on folder deletion

f4e7ab6

siketyan added 2 commits August 5, 2025 22:27

fix: correctly unloads the path on Windows

af53d15

test: fix Windows paths

ad349b5

siketyan force-pushed the scanner-v2 branch from 654b478 to d512381 Compare August 5, 2025 14:55

siketyan force-pushed the scanner-v2 branch from d512381 to 78f3460 Compare August 5, 2025 14:58

siketyan approved these changes Aug 5, 2025

View reviewed changes

crates/biome_service/src/scanner/watcher.rs Outdated Show resolved Hide resolved

siketyan reviewed Aug 5, 2025

View reviewed changes

crates/biome_service/src/workspace/server.rs Outdated Show resolved Hide resolved

feedback

a0bd572

arendjr force-pushed the scanner-v2 branch from 9227f29 to a0bd572 Compare August 5, 2025 16:49

arendjr mentioned this pull request Aug 5, 2025

docs: scanner updates biomejs/website#2853

Merged

arendjr merged commit 85b1128 into biomejs:next Aug 5, 2025
32 checks passed

arendjr mentioned this pull request Aug 5, 2025

💅 noFloatingPromise fails to find some floating promises #7111

Closed

1 task

siketyan mentioned this pull request Aug 6, 2025

fix: revive wasm build #7136

Merged

ematipico added a commit that referenced this pull request Aug 12, 2025

feat(core): scanner v2 (#6989)

f88c5e5

Co-authored-by: Emanuele Stoppa <my.burning@gmail.com> Co-authored-by: Naoki Ikeguchi <me@s6n.jp>

github-actions bot mentioned this pull request Aug 13, 2025

ci: release #7157

Merged

BenjaminBrienen mentioned this pull request Aug 13, 2025

📎 Implement smarter scanner #6965

Closed

flying-sheep reviewed Aug 19, 2025

View reviewed changes

flying-sheep mentioned this pull request Aug 19, 2025

🐛 v2.2.0 files.includes patterns to not work with exact file paths. #7268

Closed

1 task

flying-sheep mentioned this pull request Sep 4, 2025

enh: add simple from_sparse method for Graph holoviz/holoviews#6673

Merged

Uh oh!

feat(core): scanner v2 #6989

feat(core): scanner v2 #6989

Uh oh!

Conversation

arendjr commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Docs

Uh oh!

changeset-bot bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

codspeed-hq bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #6989 will not alter performance

Summary

Uh oh!

Uh oh!

Uh oh!

ematipico commented Jul 23, 2025

Uh oh!

Conaclos commented Jul 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ematipico Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ematipico Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

ematipico left a comment

Choose a reason for hiding this comment

Uh oh!

arendjr commented Jul 24, 2025

Uh oh!

ematipico commented Jul 31, 2025

Uh oh!

arendjr commented Aug 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arendjr commented Aug 4, 2025

Uh oh!

arendjr commented Aug 4, 2025

Uh oh!

arendjr commented Aug 4, 2025

Uh oh!

coderabbitai bot commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

Uh oh!

siketyan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

siketyan commented Aug 5, 2025

Uh oh!

arendjr commented Aug 5, 2025

Uh oh!

Uh oh!

flying-sheep Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

arendjr commented Jul 23, 2025 •

edited

Loading

changeset-bot bot commented Jul 23, 2025 •

edited

Loading

codspeed-hq bot commented Jul 23, 2025 •

edited

Loading

arendjr commented Aug 3, 2025 •

edited

Loading

coderabbitai bot commented Aug 5, 2025 •

edited

Loading