feat: Autogenerate contributor pool report #593
Conversation
| body: JSON.stringify({ | ||
| model: AI_MODEL, | ||
| messages: [ | ||
| { role: "system", content: PROMPT }, | ||
| { role: "user", content: JSON.stringify(grouped) } | ||
| ], | ||
| temperature: 0.7 | ||
| }) |
There was a problem hiding this comment.
I tried to start the new workflow in a test repo today, and it run into an error:
> tsc-meetings-test@1.0.0 generate:contributor-pool
> node ./scripts/generate-contributor-pool.js
Fetching PRs from GitHub API with query:
org:eslint type:pr label:"contributor pool" merged:2025-06-01..2025-06-30
Fetching page 1...
Retrieved 22 of 22 total results
Fetched 22 PRs total
Found 21 contributor PRs merged.
{"error":{"code":"tokens_limit_reached","message":"Request body too large for gpt-4o model. Max size: 8000 tokens.","details":"Request body too large for gpt-4o model. Max size: 8000 tokens."}}
/home/runner/work/Test/Test/scripts/generate-contributor-pool.js:176
throw new Error(`AI model request failed with status ${response.status}`);
^
Error: AI model request failed with status 413
at fetchModelResponse (/home/runner/work/Test/Test/scripts/generate-contributor-pool.js:176:15)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async /home/runner/work/Test/Test/scripts/generate-contributor-pool.js:211:23
Node.js v22.16.0
Error: Process completed with exit code 1.
The error message states that the maximum body size is 8000 tokens, but I'm not sure how tokens are counted in this context. I added a log line and it showed that the body length was 45026 characters.
There was a problem hiding this comment.
That's what I was afraid of...the token limit on GitHub is really small. I already crunched the data as small as I could. Apparently my personal account has a higher token limit (16,000) so when I was testing with a token I generated, it worked.
The only option to make this work is to move into the paid service. These types of requests are fairly cheap, typically well under $1 to run. I think it could be worthwhile to do that to help automate more tasks for us.
There was a problem hiding this comment.
Apparently my personal account has a higher token limit (16,000) so when I was testing with a token I generated, it worked.
Now that you mention it, I was running the test in a private repo of my own GitHub user, and that's where the 8000 token limit came from. It's possible that the token limit for the eslint organization is higher than that. I guess we'll have to run the workflow to know for sure.
The only option to make this work is to move into the paid service. These types of requests are fairly cheap, typically well under $1 to run. I think it could be worthwhile to do that to help automate more tasks for us.
I was thinking about splitting the query into multiple requests to stay under the limit, but if we can get the whole response in one go for just a few cents, that would be even better.
There was a problem hiding this comment.
It's possible that the token limit for the eslint organization is higher than that. I guess we'll have to run the workflow to know for sure.
That's a good point. I'll merge this and run it to see what happens. If it can't run, I'll follow up with another PR that uses a separate token from Open AI.
This pull request introduces a new workflow for generating a monthly Contributor Pool Report, along with the necessary script, configurations, and documentation updates. On the first of every month, a report will be generated and a pull request will be created to review the report.
Workflow Automation:
.github/workflows/generate-contributor-pool.yml: Added a new GitHub Actions workflow to generate a Contributor Pool Report on the 1st of every month. It installs dependencies, runs the report generation script, and creates a pull request with the report.New Script:
scripts/generate-contributor-pool.js: Added a script to fetch pull request data from the GitHub API, process it using an AI model for summarization, and generate a Markdown report. The script includes logic for grouping PRs by contributor and estimating effort levels.Configuration Updates:
package.json: Added a newgenerate:contributor-poolscript to thescriptssection for running the contributor pool report generation script.Documentation:
notes/2025/2025-06-01-contributor-pool.md: Added an example Contributor Pool Report for May 2025, showcasing the format and content generated by the new script.