这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@fengtality
Copy link
Contributor

@fengtality fengtality commented Nov 5, 2025

Before submitting this PR, please make sure:

  • Your code builds clean without any errors or warnings
  • You are using approved title ("feat/", "fix/", "docs/", "refactor/")

Overview

This PR implements a comprehensive caching and monitoring system for Solana CLMM connectors (PancakeSwap-Sol, Raydium, Meteora) with WebSocket support, token/pool discovery, and automatic cache integration.

Key Features

1. Helius and Infura WebSocket Monitoring

  • Real-time transaction monitoring for Solana and Ethereum

2. Error Handling for Rate Limits and other common errors

  • Real-time transaction monitoring for Solana and Ethereum
  • Position not found, insufficient balance for CLMM operations

3. Token & Pool Discovery (CoinGecko Integration)

  • API Key Configuration: Add CoinGecko API key in conf/coingecko.yml
  • Find Tokens: POST /tokens/find?chainNetwork=solana-mainnet-beta&tokenAddress=<ADDRESS>
  • Save Tokens: POST /tokens/save/<ADDRESS>?chainNetwork=solana-mainnet-beta
  • Find Pools: GET /pools/find?chainNetwork=solana-mainnet-beta&tokenA=SOL&tokenB=USDC
  • Save Pools: POST /pools/save?chainNetwork=solana-mainnet-beta&connector=raydium&type=clmm

4. Connector Updates

  • Removes pool-address from all Positions-Owned routes for CLMM connectors
  • Updated Meteora SDK to 1.7.5
  • Fixed Pancakeswap Solana connector

QA Testing Guide

Prerequisites

  1. Setup CoinGecko API Key:
# Create conf/coingecko.yml
cat > conf/coingecko.yml << 'YAML'
apiKey: your_coingecko_api_key_here
YAML
  1. Setup Helius RPC (optional for WebSocket monitoring):
# Update conf/rpc/helius.yml
apiKey: your_helius_api_key
region: slc  # or ewr, lon, fra, ams, sg, tyo
  1. Start Gateway:
GATEWAY_PASSPHRASE=a pnpm start --dev

Test 1: Token Discovery

Find Token via CoinGecko

# Find token by address
curl -X POST "http://localhost:15888/tokens/find?chainNetwork=solana-mainnet-beta&tokenAddress=2zMMhcVQEXDtdE6vsFS7S7D5oUodfJHE8vd1gnBouauv"

Expected: Returns token info (name, symbol, decimals, imageUrl, etc.) from CoinGecko

Save Token

# Save token (triggers token list reload + balance refresh)
curl -X POST "http://localhost:15888/tokens/save/2zMMhcVQEXDtdE6vsFS7S7D5oUodfJHE8vd1gnBouauv?chainNetwork=solana-mainnet-beta"

# Verify token was added
curl "http://localhost:15888/chains/solana/tokens?network=mainnet-beta" | grep PENGU

# Check that balances include the new token
curl "http://localhost:15888/chains/solana/balances?address=<WALLET>&network=mainnet-beta" | grep PENGU

Expected Logs:

Reloaded token list for solana/mainnet-beta with new token PENGU
Refreshing balances for X wallet(s) to include new token

Test 2: Pool Discovery

Find Pools via CoinGecko

# Find SOL/USDC pools
curl "http://localhost:15888/pools/find?chainNetwork=solana-mainnet-beta&tokenA=SOL&tokenB=USDC"

# Find top pools by network
curl "http://localhost:15888/pools/find?chainNetwork=solana-mainnet-beta&limit=5"

Expected: Returns pool info from CoinGecko (poolAddress, dex, type, liquidity, volume, etc.)

Save Pool

# Save pool (triggers pool-info fetch + cache population)
curl -X POST "http://localhost:15888/pools/save?chainNetwork=solana-mainnet-beta&connector=pancakeswap-sol&type=clmm&saveLimit=1"

# Verify pool was added to template
cat src/templates/pools/pancakeswap-sol.json

# Verify pool is in cache (should be fast)
time curl "http://localhost:15888/connectors/pancakeswap-sol/clmm/pool-info?network=mainnet-beta&poolAddress=<SAVED_POOL_ADDRESS>"

Expected:

  • Pool added to src/templates/pools/<connector>.json
  • Second pool-info request <50ms (cache HIT)

A. Raydium Position Opening

# Open position
curl -X POST "http://localhost:15888/connectors/raydium/clmm/open-position" \
  -H "Content-Type: application/json" \
  -d '{
    "network": "mainnet-beta",
    "walletAddress": "<WALLET>",
    "poolAddress": "58oQChx4yWmvKdwLLZzBi4ChoCc2fqCUWBkwMihLYQo2",
    "lowerPrice": 95,
    "upperPrice": 105,
    "baseTokenAmount": 0.1,
    "slippagePct": 1
  }'

B. Pancakeswap Solana Position Opening

C. Meteora Position Opening

# Open position
curl -X POST "http://localhost:15888/connectors/meteora/clmm/open-position" \
  -H "Content-Type: application/json" \
  -d '{
    "network": "mainnet-beta",
    "walletAddress": "<WALLET>",
    "poolAddress": "<METEORA_POOL>",
    "lowerPrice": 95,
    "upperPrice": 105,
    "baseTokenAmount": 0.1,
    "slippagePct": 1
  }'
**Expected**: Same cache refresh behavior

#### D. Position Closing (Auto-Removal)
```bash
# Close position
curl -X POST "http://localhost:15888/connectors/pancakeswap-sol/clmm/close-position" \
  -H "Content-Type: application/json" \
  -d '{
    "network": "mainnet-beta",
    "walletAddress": "<WALLET>",
    "positionAddress": "<POSITION_ADDRESS>"
  }'

**Expected**: Closed position no longer appears in positions-owned

---

### Test 4: Helius WebSocket Monitoring

#### Enable WebSocket Monitoring
```bash
# Update conf/rpc/helius.yml
websocket:
  enabled: true
  reconnectInterval: 5000
  pingInterval: 30000

Test Transaction Monitoring

# Start gateway and watch logs
GATEWAY_PASSPHRASE=a pnpm start --dev | grep -i websocket

# Expected logs:
# [Helius WebSocket] Connected to wss://mainnet.helius-rpc.com/?api-key=...
# [Helius WebSocket] Subscribed to account: <wallet>
# [Helius WebSocket] Warming sender connections...

Regression Testing

Verify existing functionality still works:

  • ✅ All swap operations (Jupiter, 0x, Uniswap, PancakeSwap, Raydium)
  • ✅ Pool operations (add/remove liquidity)
  • ✅ Position operations (open/close/collect fees)
  • ✅ Balance queries
  • ✅ Transaction submission

Edge Cases to Test

  1. Invalid addresses: Proper 400/404 errors
  2. Cache disabled: Fallback to RPC works
  3. Concurrent requests: Cache handles simultaneous access
  4. Case-insensitive tokens: SOL and sol both work
  5. Non-existent pools/positions: Proper error handling

fengtality and others added 6 commits November 2, 2025 01:27
… via WebSocket

Implement comprehensive WebSocket account subscription support enabling real-time
monitoring of Solana wallet balances (SOL + SPL tokens).

## HeliusService Enhancements (helius-service.ts):
- Add AccountSubscription interfaces and callbacks
- Implement subscribeToAccount() method for account monitoring
- Implement unsubscribeFromAccount() for cleanup
- Add accountNotification message handling
- Implement automatic subscription restoration after WebSocket reconnection
- Track account subscriptions separately from signature subscriptions

## Solana Class Enhancements (solana.ts):
- Add subscribeToWalletBalance() method for monitoring wallet balances
- Implement parseWalletBalances() to extract SOL + token balances
- Add unsubscribeFromWalletBalance() for cleanup
- Integrate with token list from default network configuration
- Real-time updates trigger callbacks with balance changes

## API Routes (routes/balances.ts):
- POST /chains/solana/subscribe-balances - Subscribe to wallet balance updates
  - Returns subscription ID and initial balances
  - Monitors SOL and all SPL tokens from configured token list
- DELETE /chains/solana/unsubscribe-balances - Unsubscribe from updates
  - Cleanup subscription and free resources

## Features:
✅ Real-time balance updates via WebSocket (2-3s latency)
✅ Automatic subscription restoration on reconnection
✅ SOL + SPL token monitoring from default network token list
✅ Proper cleanup on disconnect
✅ Clear error messages when WebSocket unavailable

## Configuration Required:
Users must enable in conf/rpc/helius.yml:
```yaml
apiKey: 'YOUR_HELIUS_API_KEY'
useWebSocketRPC: true
```

Part of Phase 2 implementation from docs/helius-websocket-implementation-plan.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…ocket

Implement Phase 3 of Helius WebSocket implementation plan, enabling real-time
monitoring of Meteora DLMM pool state changes.

Key Changes:
- Add subscribeToPoolUpdates() method to Meteora connector
- Add unsubscribeFromPool() method
- Add Server-Sent Events streaming endpoint at /pool-info-stream
- Parse pool state updates: activeBinId, reserves, fees, and price
- Automatic cleanup on client disconnect
- Keepalive support for long-lived connections

Implementation Details:
- Uses HeliusService.subscribeToAccount() for WebSocket subscriptions
- Refetches pool state on each notification to ensure latest data
- Adjusts prices for decimal differences between token pairs
- Parses baseFactor from pool parameters for fee calculation
- Sends real-time pool updates as SSE events to connected clients

Technical Notes:
- Pool info streamed via text/event-stream content type
- Subscription automatically cleaned up when client disconnects
- Requires Helius WebSocket enabled in conf/rpc/helius.yml
- Falls back to standard RPC when WebSocket unavailable

Endpoint:
GET /connectors/meteora/clmm/pool-info-stream?network={network}&poolAddress={poolAddress}

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Implement automatic WebSocket subscription to all Solana wallets in conf/wallets/solana
when Gateway initializes with Helius WebSocket enabled.

Key Changes:
- Add autoSubscribeToWallets() method to Solana class
- Scan conf/wallets/solana for wallet files on initialization
- Fetch initial balance for each wallet (SOL + tokens)
- Subscribe to real-time balance updates via WebSocket
- Log initial balances and real-time updates to console

Implementation Details:
- Called during Solana.init() after HeliusService initialization
- Uses getWalletAddresses() to scan wallet directory
- Fetches initial balance via getBalances() before subscribing
- Subscribes via subscribeToWalletBalance() with logging callback
- Logs truncated address (first 8 chars) for readability
- Shows up to 3 tokens per update with overflow indicator

User Experience:
- Zero configuration required - just add wallets to conf/wallets/solana
- Initial balances logged on startup
- Real-time updates logged when transactions occur
- Automatic subscription restoration after WebSocket reconnection
- Graceful error handling per wallet (doesn't fail entire batch)

Startup Logs:
  Auto-subscribing to N Solana wallet(s)...
  [ADDRESS...] Initial balance: X.XXXX SOL, Y token(s)
  Subscribed to wallet balance updates for ADDRESS...
  ✅ Auto-subscribed to N/N Solana wallet(s)

Real-time Update Logs:
  [ADDRESS...] Balance update at slot XXXXXX:
    SOL: X.XXXX, Tokens: Y
    - SYMBOL: X.XXXX
    ... and N more

Testing:
- Created test-auto-subscription.sh for manual testing
- Created test-websocket-monitoring.ts for programmatic testing
- Created websocket-monitoring-guide.md with comprehensive documentation

Technical Notes:
- Only activated when Helius WebSocket is connected
- Skips auto-subscription if no wallets found
- Filters out hardware-wallets.json from subscription
- Validates Solana address format (length 32-44)
- Uses fse.readdir() to list wallet files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Fix issue where auto-subscription to Solana wallets wasn't triggering during
Gateway startup. The startup banner was creating a standalone Connection object
instead of initializing the Solana instance, so autoSubscribeToWallets() never ran.

Fix:
- Update startup-banner.ts to call Solana.getInstance() instead of new Connection()
- This triggers full Solana initialization including HeliusService and auto-subscription
- Auto-subscription now runs immediately on Gateway startup

Result:
- Wallets in conf/wallets/solana are now automatically monitored on startup
- Initial balances logged during startup sequence
- Real-time balance updates logged when transactions occur

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Fix error in auto-subscription where code was accessing initialBalances.balances['SOL']
but getBalances() directly returns Record<string, number>, not wrapped in a balances property.

Error:
  Cannot read properties of undefined (reading 'SOL')

Fix:
- Change initialBalances.balances['SOL'] to balances['SOL']
- getBalances() returns the balances object directly
- Only the API route wraps it with { balances }

Result:
- Auto-subscription now correctly fetches and logs initial balances
- Wallet monitoring works as expected on startup

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@fengtality fengtality changed the base branch from main to development November 5, 2025 21:41
fengtality and others added 17 commits November 5, 2025 13:42
…ket updates

Add cache-first balance fetching strategy that reduces RPC calls:
- Generic CacheManager utility for balances, positions, and pools
- Unified cache config in RPC provider files (helius.yml, infura.yml)
- Cache-first getBalances() with stale detection and background refresh
- WebSocket updates automatically populate cache in real-time
- Periodic refresh every 5s for all cached entries
- TTL-based cleanup (60s) for unused cache entries
- Provider-agnostic design works with Helius, Infura, future providers

Cache configuration (refreshInterval: 5s, maxAge: 10s, ttl: 60s):
- refreshInterval: Full cache refresh every 5 seconds
- maxAge: Data considered stale after 10 seconds
- ttl: Unused entries removed after 60 seconds

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Extend cache system to support positions and pools:
- Added PositionData and PoolData interfaces for CLMM position and pool caching
- Added trackBalances, trackPositions, trackPools flags to RPC cache config
- Initialize separate caches based on flags: balance-cache, position-cache, pool-cache
- Updated cache naming to include chain: solana-{network}-{type}-cache
- Export cache data interfaces and accessor methods for connector use
- Updated default cache settings: refreshInterval=10s, maxAge=20s, ttl=60s
- Clean up all caches in disconnect()

Cache architecture:
- Balances: Track wallet SOL and token balances via WebSocket
- Positions: Track CLMM positions owned by wallets
- Pools: Track pool info for pools in conf/pools/{connector}

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Populate pool and position caches during initialization:
- trackWalletPositions(): Initialize position tracking for all wallets (TODO: fetch actual positions from CLMM connectors)
- trackPools(): Load pools from conf/pools/*.json and cache them by connector:address key
- Logs show pools loaded and positions tracked at startup

Pool cache key format: {connector}:{poolAddress}
Position cache key format: {walletAddress}

Next step: Implement position fetching from CLMM connectors (meteora, raydium, pancakeswap-sol)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Use different emojis for each cache tracking type:
- 💰 Balance tracking
- 📍 Position tracking
- 🏊 Pool tracking

Makes logs easier to scan and distinguish between different tracking types.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Implement cache-first strategy for CLMM pool-info:
- Check pool cache before fetching from RPC
- Cache key format: pancakeswap-sol:{poolAddress}
- Background refresh when data is stale (>20s)
- Log cache hits/misses/sets for debugging

First request: cache MISS, fetch from RPC, populate cache
Subsequent requests: cache HIT, return cached data instantly
Stale data: return cached data, trigger non-blocking refresh

Example for other connectors (meteora, raydium) to implement cache-first pool-info.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Implement cache-first strategy for CLMM position-info:
- Check position cache before fetching from RPC
- Cache key format: pancakeswap-sol:{positionAddress}
- Background refresh when data is stale (>20s)
- Log cache hits/misses/sets for debugging

Position cache stores PositionData with connector metadata:
- connector, positionId, poolAddress, baseToken, quoteToken
- liquidity = baseTokenAmount + quoteTokenAmount
- Spread operator includes all PositionInfo fields

First request: cache MISS, fetch from RPC, populate cache
Subsequent requests: cache HIT, return cached data instantly
Stale data: return cached data, trigger non-blocking refresh

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Don't pre-load Pool metadata from conf/pools/*.json into pool cache
because Pool interface only contains basic metadata, not full PoolInfo.

Pool cache is now populated on-demand when pool-info endpoint fetches
complete PoolInfo data from RPC. This prevents validation errors when
returning cached data that's missing required fields like price,
baseTokenAmount, quoteTokenAmount, activeBinId, etc.

Changes:
- Made binStep optional in PoolInfoSchema (Meteora-specific field)
- trackPools() now only counts available pools instead of pre-loading
- Added explanatory comment about on-demand population
- Log message updated to indicate cache is ready for first request

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Created general-purpose RateLimiter utility and integrated it into
pool tracking to prevent overwhelming RPC endpoints with concurrent
requests at startup.

Changes:
- Created rate-limiter.ts service with token bucket algorithm
- Configurable maxConcurrent and minDelay parameters
- Integrated into trackPools() with 2 concurrent, 500ms delay
- Fetches full PoolInfo from WebSocket RPC for each saved pool
- Prevents 429 rate limit errors during startup

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Pool fetching at startup was causing infinite recursive loop because
each connector (Meteora, Raydium, PancakeSwap) calls Solana.getInstance()
which re-triggers initialization and trackPools() again.

Changes:
- Disabled pool fetching in trackPools() method
- Only counts available pools without fetching PoolInfo
- Pool cache populated on-demand when pool-info endpoints are called
- Added TODO to fix circular dependency before re-enabling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Added isTrackingPools flag to prevent infinite recursion when
connectors call Solana.getInstance() during pool fetching.

How it works:
1. trackPools() sets isTrackingPools=true before fetching
2. Connectors (Meteora/Raydium/PancakeSwap) call Solana.getInstance()
3. Solana.getInstance() returns existing instance (already initialized)
4. No recursive trackPools() call because flag is true
5. Flag reset in finally block after all pools fetched

Changes:
- Added isTrackingPools boolean flag to Solana class
- Re-enabled pool fetching with rate limiting (2 concurrent, 500ms delay)
- Fetches full PoolInfo from WebSocket RPC at startup
- Populates pool cache for immediate availability

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
The root cause was that instance was added to _instances AFTER init()
completed, but init() calls trackPools() which triggers connector
getInstance() calls that check _instances[network]. Since it was still
undefined, new instances were created.

Fix: Add instance to _instances BEFORE calling init(). Now when
trackPools() → connector.getInstance() → Solana.getInstance() is
called during initialization, it finds and returns the existing
instance instead of creating a new one.

Changes:
- Moved `Solana._instances[network] = instance` before `await instance.init()`
- Added comment explaining why this order is critical
- isTrackingPools flag now works as intended (no recursive calls)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…oint

Positions are already loaded at startup into the position cache, but
the positions-owned endpoint was still fetching from RPC. Now it uses
cache-first strategy for instant responses.

Changes:
- Check position cache first before fetching from RPC
- Filter cached positions by connector (pancakeswap-sol only)
- Trigger background refresh when cache is stale (>20s)
- Populate cache on miss for future requests
- Split RPC fetching logic into separate function
- Add background refresh helper function

Benefits:
- Near-instant response for cached positions (< 10ms vs 635ms)
- Reduced RPC load
- Automatic background updates keep data fresh

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Position and pool caches were created but never had periodic refresh
enabled, unlike balance cache. Now all three caches refresh at the
configured intervals (default: every 10s).

Changes:
- Added startPeriodicRefresh() calls for position and pool caches
- Implemented refreshPositionsInBackground() helper (placeholder for now)
- Implemented refreshPoolInBackground() helper with full logic
- Pool refresh supports meteora, raydium, and pancakeswap-sol connectors
- Raydium pool refresh detects AMM vs CLMM from cached poolType

Refresh intervals (from conf/rpc/*.yml):
- refreshInterval: 10s (refresh all cached entries)
- maxAge: 20s (trigger background refresh when stale)
- ttl: 60s (remove unused entries)

Benefits:
- Pools automatically stay fresh without manual endpoint calls
- Positions will auto-refresh once fetching logic is implemented
- Consistent behavior across all cache types

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Store the original pool type (amm/clmm) from conf/pools/*.json alongside
PoolInfo in cache, eliminating need to inspect cached poolInfo structure
during refresh.

Changes:
- Added poolType field to PoolData interface
- Store poolType during initial pool load at startup
- Use stored poolType in refreshPoolInBackground() for Raydium
- Preserve poolType when updating cache during refresh
- Removed complex logic that tried to detect pool type from cached data

Benefits:
- Simpler, more maintainable code
- No need to check for 'cpmm' (we only use 'amm' in raydium.json)
- Direct mapping: 'amm' → getAmmPoolInfo(), 'clmm' → getClmmPoolInfo()
- poolType is the source of truth from config files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…MM connectors

## Implementation Changes

### Unified Cache Keys (Simplified)
- Pool cache: Use `poolAddress` as key (removed connector prefix)
- Position cache: Use `positionAddress` as key (removed connector prefix)
- Balance cache: Filter by requested tokens, fetch unknowns from RPC

### Pool Info Routes (Cache-First)
- pancakeswap-sol/clmm-routes/poolInfo.ts: Cache HIT → return, STALE → background refresh, MISS → fetch + cache
- raydium/clmm-routes/poolInfo.ts: Same pattern
- meteora/clmm-routes/poolInfo.ts: Same pattern

### Position Info Routes (Cache-First)
- pancakeswap-sol/clmm-routes/positionInfo.ts: Cache HIT → return, STALE → background refresh, MISS → fetch + cache
- raydium/clmm-routes/positionInfo.ts: Same pattern
- meteora/clmm-routes/positionInfo.ts: Same pattern

### Positions Owned Routes (Individual Position Caching)
- pancakeswap-sol/clmm-routes/positionsOwned.ts: Fetch from RPC, populate individual position caches by address
- raydium/clmm-routes/positionsOwned.ts: Same pattern
- meteora/clmm-routes/positionsOwned.ts: Same pattern

Note: positions-owned always fetches fresh data from RPC because it's a wallet-level query that needs to discover all positions. However, it populates the cache for each individual position so subsequent position-info requests can be served from cache.

### Balance Cache Token Filtering
- solana.ts: Filter cached balances by requested tokens
- Fetch unknown tokens from RPC (treat as token addresses)
- Case-insensitive symbol matching

### Services
- PositionsService: New service for position tracking and refresh
- PoolService: Updated to use simplified pool address keys

## Test Coverage (28 Tests)

### Pool Cache Tests (9 tests)
- Cache HIT/MISS/STALE scenarios for all 3 connectors
- Verify simplified cache key format (no connector prefix)

### Position Cache Tests (13 tests)
- position-info: Cache HIT/MISS/STALE for all 3 connectors
- positions-owned: Verify individual position cache population
- Verify simplified cache key format

### Balance Cache Tests (16 tests)
- Cache HIT/MISS/STALE scenarios
- Token filtering with cached and non-cached tokens
- Case-insensitive lookup
- Edge cases (empty list, all unknown tokens, mixed tokens)

## Unified Behavior

All Solana CLMM connectors (PancakeSwap-Sol, Raydium, Meteora) now:
- Use identical cache-first pattern
- Use simplified cache keys (address only)
- Support background refresh on stale data
- Populate individual position caches via positions-owned
- Work seamlessly with unified /trading/clmm/* routes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add integration with GeckoTerminal API to fetch top trading pools for any token across supported networks.

Changes:
- Add CoinGeckoService with network mapping and API client
- Add /tokens/top-pools/:symbolOrAddress endpoint
- Support both token symbols and addresses
- Add coingeckoAPIKey to server.yml config
- Map Gateway chain-network format to GeckoTerminal network IDs
- Return pool metrics (price, volume, liquidity, txns)

Supported networks: Ethereum, Solana, BSC, Polygon, Arbitrum, Optimism, Base, Avalanche, Celo

Example usage:
GET /tokens/top-pools/SOL?chainNetwork=solana-mainnet-beta&limit=5

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
fengtality and others added 5 commits November 5, 2025 21:23
…ttern

Change error-level logging to debug-level when connectors fail to decode pools during background refresh. This is expected behavior since the refresh logic tries each connector (Meteora, Raydium, PancakeSwap) in sequence until one succeeds.

Changes:
- Meteora: error → debug when unable to decode pool
- Raydium CLMM: error → debug when unable to decode pool
- Raydium AMM: error → debug when unable to decode pool
- PancakeSwap-Sol: error → debug when unable to decode pool

Fixes excessive error logs like:
  Error getting pool info for <address>: Invalid account discriminator

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Fix pool refresh failures by ensuring all connector methods return results properly.

Changes:
1. PancakeSwap-Sol: Return null instead of throwing on decode failure (consistent with other connectors)
2. Solana pool refresh: Check result from each connector before returning
3. Raydium fallback: Try both AMM and CLMM when poolType hint fails
4. Add explicit null checks to ensure we only return valid pool info

This fixes the issue where 6/8 pools were failing to refresh during periodic cache updates.

Before: Only Meteora pools refreshed successfully
After: All pools refresh using their correct connector

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…tors

Implement periodic position refresh that fetches positions from all Solana CLMM connectors (Meteora, Raydium, PancakeSwap) and updates the cache.

Changes:
1. PositionsService.refreshPositions(): Fetch and cache positions from all connectors
2. Solana.refreshPositionsInBackground(): Aggregate positions from:
   - Meteora: via getAllPositionsForWallet()
   - Raydium: via raydiumSDK.clmm.getOwnerPositionInfo()
   - PancakeSwap: via NFT token account scanning
3. Each position cached individually by position address with connector metadata

This enables:
- Automatic position updates every 60 seconds (or configured interval)
- Fresh position data without manual endpoint calls
- Cache-first reads with background refresh

Replaces placeholder "Position refresh not yet implemented" log message.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Remove duplicate refreshPositions method and use trackPositions for both initial loading and periodic refresh.

Changes:
- Remove PositionsService.refreshPositions() (duplicate of trackPositions)
- Update trackPositions to cache by position address (not wallet address)
- Use trackPositions([address], ...) for single wallet refresh
- Consolidate caching logic in one place

Benefits:
- Single source of truth for position tracking
- Consistent caching behavior (by position address)
- Simpler API with one method for all use cases
- Works for both single wallet and multiple wallets

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Change position cache keys from just "address" to "connector:clmm:address" format to support future AMM positions and make it easier to reference specific connector positions.

Changes:
- PositionsService: Cache as "connector:clmm:address"
- Meteora positionsOwned: Cache as "meteora:clmm:address"
- Raydium positionsOwned: Cache as "raydium:clmm:address"
- PancakeSwap positionsOwned: Cache as "pancakeswap-sol:clmm:address"

Benefits:
- Future-proof for AMM positions (can use "connector:amm:address")
- Easier to identify which connector owns a position
- Avoids fetching from connectors we know don't have positions
- Consistent with pool cache format (connector:poolAddress)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
fengtality and others added 3 commits November 15, 2025 16:39
- Delete src/services/chain-config.ts (redundant with ConfigManagerV2)
- Add helper methods to ConfigManagerV2: getChainId(), getGeckoTerminalId(), parseChainNetwork(), getSupportedChainNetworks()
- Update all imports to use ConfigManagerV2 instead of chain-config
- Remove cache-related test files: token-find-save-cache.test.ts, token-gecko-data.test.ts, openPosition-cache.test.ts (raydium and pancakeswap-sol)

ConfigManagerV2 now serves as the single source of truth for chain configuration,
reading chainID and geckoId directly from network config files. This eliminates
the redundant chain-config module.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…2 calls

Replace ConfigManagerV2.getInstance().getSupportedChainNetworks() with
hardcoded examples to avoid module initialization ordering issues.

The schemas are evaluated at module load time before ConfigManagerV2
is fully initialized, causing runtime errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…etwork

Updated test mocks in both Solana and Ethereum status tests to use the
correct method name after RPC provider refactoring.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@fengtality fengtality changed the base branch from development to fix/meteora-struct-error-and-position-width November 16, 2025 00:59
- Removed console.log statements that display URLs containing API keys
- Added comments to warn about sensitive data in URL variables
- Updated test-infura-websocket-live.js to not log HTTP/WebSocket URLs
- Updated test-helius-live.js with comments about sensitive URLs

This prevents accidental exposure of API keys in logs or console output.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@fengtality fengtality changed the base branch from fix/meteora-struct-error-and-position-width to development November 16, 2025 01:03
Removed test scripts for features that were removed during cache cleanup:
- test-auto-subscription.sh: Auto-subscription feature removed
- test-autosubscribe.ts: Auto-subscription with Helius removed
- test-sse-stream.sh: SSE streaming feature removed
- test-token-accounts.ts: One-off debug script no longer needed
- test-websocket-monitoring.ts: WebSocket monitoring caching removed

Keeping only test-helius-live.js and test-infura-live.js for RPC provider testing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@fengtality fengtality force-pushed the feat/helius-websocket-monitoring branch from 702f1c1 to dc6903d Compare November 16, 2025 01:06
fengtality and others added 14 commits November 15, 2025 17:09
Simplified fetchPositionsForWallet by using the existing getPositionsOwned
helpers from connector route files instead of duplicating connector logic.

This makes the code more maintainable and ensures consistency with the API
endpoints.

Changes:
- Import getPositionsOwned from meteora/clmm-routes/positionsOwned
- Import getPositionsOwned from raydium/clmm-routes/positionsOwned
- Import getPositionsOwned from pancakeswap-sol/clmm-routes/positionsOwned
- Removed duplicate connector-specific position fetching logic
- Reduced code from ~125 lines to ~80 lines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Created token-lookup-helper.ts to share token fetching logic between findToken and save routes
- Created token-error-handler.ts for consistent error handling across all token routes
- Created pool-error-handler.ts for consistent error handling across pool routes
- Renamed tokens/routes/findSave.ts to save.ts for concise naming
- Refactored all token routes (findToken, save, addToken, removeToken, getToken) to use shared helpers
- Refactored pool routes (findPools, save) to use shared error handler
- Removed unused Solana imports from meteora connector files
- Removed unused subscription methods (subscribeToPoolUpdates, unsubscribeFromPool) from meteora.ts

Benefits:
- Eliminated ~200+ lines of duplicated error handling and data fetching code
- Centralized token data fetching logic in one reusable helper
- Consistent error responses across all token and pool endpoints
- Improved maintainability - changes to error handling or data fetching now happen in one place
- Net code reduction of ~70-100 lines while adding 3 reusable helper files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Added redactUrl() helper to mask API keys and credentials in URLs
- Applied redaction to both logger.error() calls and error messages
- Prevents sensitive API keys from being exposed in logs
- Fixes CodeQL security warning about clear-text logging of sensitive information

The redactUrl() function masks:
- Query parameter API keys (?api-key=xxx or &api_key=xxx)
- Basic auth credentials (//user:pass@host)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…nd Ethereum

- Created src/services/rpc-connection-interceptor.ts with chain-agnostic rate limit detection
- Supports both Solana Connection and Ethereum Provider (ethers.js)
- Replaced Solana-specific solana-connection-interceptor.ts with generic solution
- Applied rate limit detection to ALL Ethereum providers:
  - Standard RPC (StaticJsonRpcProvider)
  - Infura HTTP and WebSocket providers
  - All fallback scenarios
- Enhanced 429 error detection patterns (added "rate limit" keyword)
- Network-specific error messages for better UX

Benefits:
- Single source of truth for rate limit handling across both chains
- Ethereum now has same rate limit protection as Solana
- Consistent error messages and handling
- Reduced code duplication

Breaking changes: None (backward compatible via legacy export)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Removed all console.log statements to prevent potential sensitive data exposure
- Simplified error handling to use throw/exit codes instead of logging
- Maintains test functionality while avoiding CodeQL security warnings
- Test still validates WebSocket connections and disconnections via exit codes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Remove optional poolAddress parameter from CLMM positionsOwned endpoints to always return all positions owned by a wallet instead of filtering by pool.

Changes:
- Remove poolAddress parameter from Raydium and Meteora positionsOwned routes
- Remove poolAddress from request schemas
- Update test imports to use new rpc-connection-interceptor location
- Pancakeswap and Uniswap already correct (no poolAddress parameter)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Remove account subscription features from HeliusService to focus solely on transaction monitoring, matching InfuraService's simpler approach.

Removed:
- subscribeToAccount() and unsubscribeFromAccount() public methods
- accountSubscriptions Map and related interfaces
- Account notification handling in WebSocket message handler
- Account subscription restoration after reconnection

The service now only provides WebSocket transaction monitoring via monitorTransaction(), which is the only feature currently being used.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Consolidate redundant error handling in wrap and unwrap routes by creating a shared handleSolanaTransactionError helper function.

Changes:
- Add handleSolanaTransactionError() helper to both files
- Reduce error handling from ~20 lines to 2 lines per route
- Simplify route handlers by removing extra blank lines
- Keep wrap/unwrap logic compartmentalized in their respective files
- Maintain proper error propagation for custom errors (statusCode check)

The error handler now handles all common Solana transaction errors:
- Insufficient funds
- Transaction timeout
- Ledger device errors (rejected, locked, wrong app)
- Generic transaction failures

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Remove redundant positionAddress variable extraction that created duplicate property in return object.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Consolidate the fee fetching logic for both CLMM and AMM pools into a single code block.

Changes:
- Combine CLMM and AMM fee fetching into unified if block
- V3 (CLMM) pools: Fetch fee from pool contract's fee() method
- V2 (AMM) pools: Use standard fees per DEX (0.3% Uniswap, 0.25% PancakeSwap)
- Add comment explaining V2 fees aren't exposed on-chain

Note: V2/AMM pool fees are still hardcoded per connector because Uniswap V2
and PancakeSwap V2 don't expose the fee percentage on-chain. The fee is
hardcoded in the factory contract logic, not stored as a variable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Remove redundant TokenService fallback and blockchain fetching. Use only local token list for symbol resolution.

Changes:
- Remove 40+ lines of TokenService fallback logic
- Use chain.getToken() for both Solana and Ethereum
- No blockchain fetching - local token list only
- Simpler, faster, and more predictable behavior

If tokens aren't in the local list, symbols will be undefined (which is fine - they're optional).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Changed ethereum.getToken() to async and removed blockchain fetching
- All token lookups now use local token list only (no getOrFetchToken)
- Simplified connector token methods (getTokenBySymbol/getTokenByAddress)
  to call ethereum.getToken() and convert to SDK Token type
- Updated all route files to await async getToken calls
- Cleaned up duplicate JSDoc comments
- Removed TokenService fallback from pool-info-helpers

This keeps the codebase simple by only using local token lists instead
of fetching from blockchain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…gle getToken method

- Renamed connector methods from getTokenBySymbol/getTokenByAddress to getToken
- Both methods were redundant since ethereum.getToken() handles both symbols and addresses
- Updated all route files to use the simplified getToken() method
- Maintains SDK Token conversion (TokenInfo -> Pancakeswap/Uniswap SDK Token)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Updated test mocks for uniswap and pancakeswap AMM quote-swap tests
- Removed getOrFetchTokenBySymbol mock (method no longer exists)
- Changed getTokenBySymbol to getToken in connector mocks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@rapcmia rapcmia dismissed their stale review November 17, 2025 08:04

pending for retest

@rapcmia
Copy link
Contributor

rapcmia commented Nov 17, 2025

Commit c66321e

  • Removed and previous installation then do clean install ✅
  • Setup this PR with hummingbot dev and secure mode ✅
  • On Helius.yml, websocket is not showing any of the parameters anymore from previous tests
    • pnpm build and run setup same results
  • Balance returns expected amount compared to wallet
  • For testing cached, each endpoint is executed consecutively without delay

Token Discovery & Cache Integration

- Added  ./conf/coingecko.yml manually and initialised with gateway successfully
- GET token/find/ endpoint works only when using token address
- POST /token/save/<token address>  endpoint saved on conf/token/solana/mainnet-beta
```
  {
    "chainId": 101,
    "name": "Gundam",
    "symbol": "Gundam",
    "address": "7c21xTnFy2CH4Td13JbDvPtFHnKqpbsM4Pw5Z3pCbonk",
    "decimals": 6,
    "geckoData": {
      "coingeckoCoinId": null,
      "imageUrl": "https://assets.geckoterminal.com/wflylvnpcv70233tokgp23vvebxw",
      "priceUsd": null,
      "volumeUsd24h": "0.0",
      "marketCapUsd": null,
      "fdvUsd": null,
      "totalSupply": "999428243.458183",
      "topPools": [],
      "timestamp": 1763379747164
    }
  }
]
```
  - Compared to exiting tokens, observed added entry geckoData above ✅

Pool Discovery & Cache Integration

  • Deleted all JUP-USDC pools on {meteora, pancakeswap-sol}.yml files
  • Run find and save endpoint using JUP-USDC, successfully save to conf/pools/{meteora, pancake-swap and raydium}.json successfully including geckoData
  • Run pool-info endpoint twice to time the cache
    • Meteora cached around 0.55s
      # 1st uncache
      /usr/bin/time -f 'time_total: %e s' curl -s "http://localhost:15888/trading/clmm/pool-info?connector=meteora&chainNetwork=solana-mainnet-beta&poolAddress=BhQEFZCRnWKQ21LEt4DUby7fKynfmLVJcNjfHNqjEF61" | jq
      
      time_total: 0.97 s
      {
        "address": "BhQEFZCRnWKQ21LEt4DUby7fKynfmLVJcNjfHNqjEF61",
        "baseTokenAddress": "JUPyiwrYJFskUPiHa7hkeR8VUtAeFoSYbKedZNsDvCN",
        "quoteTokenAddress": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
        "feePct": 0.1,
        "price": 0.27988865255566,
        "baseTokenAmount": 93659.546438,
        "quoteTokenAmount": 9327.188198,
        "activeBinId": -1274,
        "binStep": 10
      }
      
      
      # 2nd cached
      /usr/bin/time -f 'time_total: %e s' curl -s "http://localhost:15888/trading/clmm/pool-info?connector=meteora&chainNetwork=solana-mainnet-beta&poolAddress=BhQEFZCRnWKQ21LEt4DUby7fKynfmLVJcNjfHNqjEF61" | jq
      time_total: 0.55 s
      {
        "address": "BhQEFZCRnWKQ21LEt4DUby7fKynfmLVJcNjfHNqjEF61",
        "baseTokenAddress": "JUPyiwrYJFskUPiHa7hkeR8VUtAeFoSYbKedZNsDvCN",
        "quoteTokenAddress": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
        "feePct": 0.1,
        "price": 0.27988865255566,
        "baseTokenAmount": 93659.546438,
        "quoteTokenAmount": 9327.188198,
        "activeBinId": -1274,
        "binStep": 10
      }
      
    • Pancakeswap-sol around 0.71s
      # 1st uncache
      /usr/bin/time -f 'time_total: %e s' curl -s "http://localhost:15888/trading/clmm/pool-info?connector=pancakeswap-sol&chainNetwork=solana-mainnet-beta&poolAddress=FG1EitLebiWFhJyx8XHvryds2fnrD59xdge9kMVkSQnY" | jq
      time_total: 5.87 s
      {
        "address": "FG1EitLebiWFhJyx8XHvryds2fnrD59xdge9kMVkSQnY",
        "baseTokenAddress": "JUPyiwrYJFskUPiHa7hkeR8VUtAeFoSYbKedZNsDvCN",
        "quoteTokenAddress": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
        "feePct": 0.25,
        "price": 0.2994105100164565,
        "baseTokenAmount": 278973.580721,
        "quoteTokenAmount": 326.049576,
        "activeBinId": -12060,
        "binStep": 60
      }
      
      # 2nd cached
      /usr/bin/time -f 'time_total: %e s' curl -s "http://localhost:15888/trading/clmm/pool-info?connector=pancakeswap-sol&chainNetwork=solana-mainnet-beta&poolAddress=FG1EitLebiWFhJyx8XHvryds2fnrD59xdge9kMVkSQnY" | jq
      time_total: 0.71 s
      {
        "address": "FG1EitLebiWFhJyx8XHvryds2fnrD59xdge9kMVkSQnY",
        "baseTokenAddress": "JUPyiwrYJFskUPiHa7hkeR8VUtAeFoSYbKedZNsDvCN",
        "quoteTokenAddress": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
        "feePct": 0.25,
        "price": 0.2994105100164565,
        "baseTokenAmount": 278973.580721,
        "quoteTokenAmount": 326.049576,
        "activeBinId": -12060,
        "binStep": 60
      }
      
    • Raydium around 0.74s
      # 1st uncache
      /usr/bin/time -f 'time_total: %e s' curl -s "http://localhost:15888/trading/clmm/pool-info?connector=raydium&chainNetwork=solana-mainnet-beta&poolAddress=Fu5uXjR17rGbUur294Ke3UxCH8JF5nWYYCe9wqNjDuFr"
      | jq
      time_total: 1.16 s
      {
        "address": "Fu5uXjR17rGbUur294Ke3UxCH8JF5nWYYCe9wqNjDuFr",
        "baseTokenAddress": "JUPyiwrYJFskUPiHa7hkeR8VUtAeFoSYbKedZNsDvCN",
        "quoteTokenAddress": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
        "feePct": 0.25,
        "price": 0.2796311252906035,
        "baseTokenAmount": 24147.943849,
        "quoteTokenAmount": 555.920749,
        "activeBinId": -12744,
        "binStep": 60
      }
      
      # 2nd cached
      /usr/bin/time -f 'time_total: %e s' curl -s "http://localhost:15888/trading/clmm/pool-info?connector=raydium&chainNetwork=solana-mainnet-beta&poolAddress=Fu5uXjR17rGbUur294Ke3UxCH8JF5nWYYCe9wqNjDuFr" | jq
      time_total: 0.74 s
      {
        "address": "Fu5uXjR17rGbUur294Ke3UxCH8JF5nWYYCe9wqNjDuFr",
        "baseTokenAddress": "JUPyiwrYJFskUPiHa7hkeR8VUtAeFoSYbKedZNsDvCN",
        "quoteTokenAddress": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
        "feePct": 0.25,
        "price": 0.2796311252906035,
        "baseTokenAmount": 24147.943849,
        "quoteTokenAmount": 555.920749,
        "activeBinId": -12744,
        "binStep": 60
      }
      

Position Cache Integration (All Connectors)

  • Successfully opened positions and closed
  • Raydium: position-info cached response around 0.81s
  • Meteora: position-info cached response around 0.51s
  • Pancakeswap-sol: position-info cached response around 0.85s

Added endpoints used on the test however when testing helius, it overwrite the existing gateway logs (can't undo, can't recover the data):
test-endpoint.log

@fengtality
Copy link
Contributor Author

fengtality commented Nov 17, 2025 via email

@fengtality fengtality merged commit 75155e1 into development Nov 17, 2025
5 checks passed
@fengtality fengtality deleted the feat/helius-websocket-monitoring branch November 17, 2025 15:56
@rapcmia rapcmia moved this from Under Review to Development v2.11.0 in Pull Request Board Nov 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development v2.11.0

Development

Successfully merging this pull request may close these issues.

4 participants