这是indexloc提供的服务,不要输入任何密码
Skip to content

[BUG]: GitLab RepoLoader doesn't fetch all files from large repositories #2049

@mehmetunlu

Description

@mehmetunlu

How are you running AnythingLLM?

Local development

What happened?

I try to fetch files from a big GitLab repository with "GitLab Repo" data connector.
However it always fails to fetch any files with message "No files were found for those settings.".
The repo has ~700 folders and ~9000 files in total.

I checked the code and saw that GitLab RepoLoader stops fetching paginated repo tree if the current page doesn't have any 'blob' objects, which doesn't make sense in my opinion:
https://github.com/Mintplex-Labs/anything-llm/blob/master/collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js#L226

Are there known steps to reproduce?

  1. For any workspace, open Data Connectors -> GitLab Repo.
  2. Add a gitlab repository with >100 folders.
  3. Click Submit.
  4. It fails to fetch files with message "No files were found for those settings."

Metadata

Metadata

Assignees

No one assigned

    Labels

    possible bugBug was reported but is not confirmed or is unable to be replicated.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions