Initial PR for Synthetic data generation #767
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced `quick-synthetic.py` example demonstrating the use of `SimpleSyntheticStage` for generating synthetic text data using an LLM. - Added `LLMClient` interface and `OpenAIClient` implementation for querying OpenAI models. - Created `SimpleSyntheticStage` for processing empty tasks and generating document batches from prompts. - Implemented necessary imports and setup for the new classes and example. This commit enhances the functionality of the Ray Curator by providing a practical example and a structured way to interact with LLMs for synthetic data generation. Signed-off-by: Abhinav Garg <abhinavg@stanford.edu>
- Changed the hardcoded API key in `quick-synthetic.py` to a placeholder "<your-nvidia-api-key>" for better security practices. - Added a comment to guide users on where to obtain their API key. - The key has been rotated This update enhances the usability and security of the synthetic data generation example. Signed-off-by: Abhinav Garg <abhinavg@stanford.edu>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Intial PR for synth data to Ray
Usage
python ray_curator/examples/quick-synthetic.pyChecklist