θΏ™ζ˜―indexlocζδΎ›ηš„ζœεŠ‘οΌŒδΈθ¦θΎ“ε…₯任何密码
Skip to content

Conversation

@DipFlip
Copy link
Contributor

@DipFlip DipFlip commented Jul 22, 2024

Pull Request Type

  • ✨ feat
  • πŸ› fix
  • ♻️ refactor
  • πŸ’„ style
  • πŸ”¨ chore
  • πŸ“ docs

Relevant Issues

resolves #812

What is in this change?

This copies the github data connector and ajusts it where necessary to work with gitlab.

Additional Information

I've tested this against public and private gitlab.com repos with access tokens. I have not tested against self-hosted gitlab server.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

@timothycarambat
Copy link
Member

Does this work? Do GitHub and GitLab have the same API's?

@DipFlip
Copy link
Contributor Author

DipFlip commented Jul 22, 2024

@timothycarambat yes this is working. You can use the new gitlab data connector to load gitlab repos just like the github connector worked before. The github and gitlab APIs are similar but not identical. Compare for instance the fetchSingleFile function for github and gitlab:

Github:

  async fetchSingleFile(sourceFilePath) {
    try {
      return fetch(
        `https://api.github.com/repos/${this.author}/${this.project}/contents/${sourceFilePath}?ref=${this.branch}`,
        {
          method: "GET",
          headers: {
            Accept: "application/vnd.github+json",
            "X-GitHub-Api-Version": "2022-11-28",
            ...(!!this.accessToken
              ? { Authorization: `Bearer ${this.accessToken}` }
              : {}),
          },
        }
      )
        .then((res) => {
          if (res.ok) return res.json();
          throw new Error(`Failed to fetch from Github API: ${res.statusText}`);
        })
        .then((json) => {
          if (json.hasOwnProperty("status") || !json.hasOwnProperty("content"))
            throw new Error(json?.message || "missing content");
          return atob(json.content);
        });
    } catch (e) {
      console.error(`RepoLoader.fetchSingleFile`, e);
      return null;
    }
  }
}

Gitlab:

  async fetchSingleFile(sourceFilePath) {
    try {
      const data = await this.#makeRequest(
        `https://gitlab.com/api/v4/projects/${
          this.projectId
        }/repository/files/${encodeURIComponent(sourceFilePath)}/raw?ref=${
          this.branch
        }`
      );
      return data;
    } catch (e) {
      console.error(`RepoLoader.fetchSingleFile`, e);
      return null;
    }
  }

  #makeRequest(url) {
    return new Promise((resolve, reject) => {
      const options = {
        headers: this.accessToken ? { "PRIVATE-TOKEN": this.accessToken } : {},
      };

      https
        .get(url, options, (res) => {
          let data = "";
          res.on("data", (chunk) => (data += chunk));
          res.on("end", () => {
            if (res.statusCode >= 200 && res.statusCode < 300) {
              resolve(data);
            } else {
              reject(
                new Error(`Request failed with status code ${res.statusCode}`)
              );
            }
          });
        })
        .on("error", reject);
    });
  }
}

@timothycarambat timothycarambat self-assigned this Jul 22, 2024
@timothycarambat timothycarambat added the PR:needs review Needs review by core team label Jul 22, 2024
@DipFlip
Copy link
Contributor Author

DipFlip commented Jul 23, 2024

snuck in a little addition. As mentioned in #1922 , the input field for repo addresses now takes multiple repos which will be collected one by one when Submit is clicked.

@timothycarambat
Copy link
Member

timothycarambat commented Jul 23, 2024

Closing this as I needed to make a bit of changes so moving PR to #1932 - some issues/quirks I came across during review was:

  • Duplication of endpoints (same endpoint for /branches and /repo were duplicated for GitLab)
  • Multi-repo loading (cool, but that should be in a new PR, should not be in same PR as its changes the complexity and scope in one PR)
  • File ignoring did not function as expected (looks like it was explicitly matching of filenames, not the blob-type like Github is)
  • Would not function for local/self-hosted GitLab, which is part of issue request
  • Some general refactor opportunities to keep things less complex or deduplicated

TBH, this is a massive effort and what you did write is a very solid point to start! Youll still get credit for the merge and such, its just on another PR because that is just how Github works when i edit a PR in the gh cli.

πŸš€

@DipFlip
Copy link
Contributor Author

DipFlip commented Jul 23, 2024

Thanks for checking it and the refactored implementation is nice!

@timothycarambat
Copy link
Member

Thank you for your work, it made doing this about 100x simpler for me to get going - I hadn't used GitLab prior so it would've been a steeper learning curve and taken longer to merge. Thanks again πŸ‘‹

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR:needs review Needs review by core team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAT]: Self-Hosted Gitlab Data Connector

2 participants