Summary: Standardized Generation Parameters for vLLM and Transformers Wrappers

Overview

This PR introduces standardized generation parameters across vLLM and Transformers wrappers, enabling cross-backend compatibility while maintaining full backward compatibility.

Key Changes

1. Standardized Parameter Names

• Introduced common parameter names that work across both vLLM and Transformers wrappers
• Core parameters: max_new_tokens, num_return_sequences, temperature, top_p, top_k, repetition_penalty, do_sample, num_beams, length_penalty, early_stopping, stop_sequences, skip_special_tokens, logprobs

2. Legacy Parameter Support

• Automatic conversion of legacy parameter names:
  • max_tokens → max_new_tokens
  • n → num_return_sequences
• Ensures existing code continues to work without modification

3. Parameter Conflict Resolution

• Silent override behavior: When both legacy and standardized names are provided, legacy names prevail
• Example: max_tokens=20 + max_new_tokens=10 → max_tokens=20 wins
• This ensures backward compatibility with existing code

4. Backend-Specific Mappings

• vLLM: Maps standardized names to vLLM's SamplingParams format
  • max_new_tokens → max_tokens
  • num_return_sequences → n
  • num_beams → best_of
  • do_sample=False → temperature=0.0 (greedy decoding)
• Transformers: Maps to Hugging Face's generation arguments
  • logprobs → output_scores
  • Filters unsupported parameters like length_penalty, early_stopping

5. Comprehensive Documentation

• Added detailed docstrings to all three classes explaining:
  • Standardized parameter list
  • Legacy parameter support
  • Parameter conflict resolution behavior
  • Backend-specific mappings
• Created comprehensive documentation file with examples and best practices

6. Test Coverage

• Added comprehensive test suite covering:
  • Standardized parameter usage
  • Legacy parameter conversion
  • Parameter conflict resolution
  • Cross-backend compatibility

Files Modified

Core Implementation

• torchrl/modules/llm/policies/common.py: Added standardization logic and common parameter definitions
• torchrl/modules/llm/policies/vllm_wrapper.py: Integrated standardization with vLLM-specific mappings
• torchrl/modules/llm/policies/transformers_wrapper.py: Integrated standardization with Transformers-specific mappings

Documentation

• docs/standardized_generation_parameters.md: Comprehensive guide with examples and best practices

Tests

• test/llm/test_wrapper.py: Added test cases for all new functionality

Benefits

1. Cross-Backend Compatibility: Same parameter names work with both vLLM and Transformers
2. Backward Compatibility: Existing code continues to work without changes
3. Clear Documentation: Users understand exactly how parameters are handled
4. Future-Proof: Easy to add new standardized parameters
5. Consistent API: Unified interface across different LLM backends

Usage Examples

# ✅ Standardized parameters (recommended)
generate_kwargs = {
    "max_new_tokens": 50,
    "temperature": 0.7,
    "do_sample": True,
}

# ✅ Legacy parameters (backward compatible)
generate_kwargs = {
    "max_tokens": 50,
    "n": 1,
    "temperature": 0.7,
}

# ⚠️ Mixed usage (legacy wins)
generate_kwargs = {
    "max_tokens": 20,      # This wins
    "max_new_tokens": 10,  # This is ignored
}

Breaking Changes

None - This is a purely additive change that maintains full backward compatibility.

Testing

All existing tests pass, and new tests verify the standardization functionality works correctly across both backends.