(feat) move ethereum token lists to `conf` #126

fengtality · 2023-06-02T22:12:59Z

A description of the changes proposed in the pull request:

Since default token lists are stored in src, Docker users have been unable to easily add new tokens
Changes in this PR (for Ethereum only)
- Moves the default list location to conf/[list].json
- Moves the token lists for Ethereum into src/templates
- Modifies gateway-setup.sh so that it creates all files from templates directory, including the new lists

Tests performed by the developer:

I tested this with a random Arbitrum new token: https://arbiscan.io/token/0xBfbCFe8873fE28Dfa25f1099282b088D52bbAD9C

Added token info to Arbitrum list in conf/lists

Postman returns a 0 balance for EQR - if no token is found, it would not return a value

Hummingbot client (after restarting) also shows EQR balance

Tips for QA testing:

Please test Docker build
There may be permissions issues with new lists folder

rapcmia · 2023-06-06T04:33:30Z

PR update:

Overview of PR, changes are applied only for Ethereum
Setup this gateway PR and latest client development
- Confirmed no tokenlist.json file is located on previous folder for Ethereum (others are still on default src/chain folders
- gateway-setup.sh now copies src/template/list to conf/list
- Check gateway config ethereum
  - confirmed new location is available 💯
Build docker image successful for this PR
Confirm gateway config ethereum changes worked, location is the same like the screenshot above however:

Due to location of tokenListSource it is now returning an error when setting up wallet on ethereum connectors:

2023-06-06 03:52:33,214 - 1 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/wallet/add failed. See logs for more details.
2023-06-06 03:52:33,214 - 1 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Unknown error.', 'httpErrorCode': 503, 'errorCode': 1099, 'stack': "Error: ENOENT: no such file or directory, open 'conf/lists/arbitrum_one_tokens.json'"}
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
    return await c
  File "/home/hummingbot/hummingbot/client/command/gateway_command.py", line 256, in _gateway_connect
    wallet_address, additional_prompt_values = await self._prompt_for_wallet_address(
  File "/home/hummingbot/hummingbot/client/command/gateway_command.py", line 382, in _prompt_for_wallet_address
    response: Dict[str, Any] = await self._get_gateway_instance().add_wallet(
  File "/home/hummingbot/hummingbot/core/gateway/gateway_http_client.py", line 265, in add_wallet
    return await self.api_request(method="post", path_url="wallet/add", params=request)
  File "/home/hummingbot/hummingbot/core/gateway/gateway_http_client.py", line 218, in api_request
    raise e
  File "/home/hummingbot/hummingbot/core/gateway/gateway_http_client.py", line 206, in api_request
    raise ValueError(f"Error on {method.upper()} {url} Error: {parsed_response}")
ValueError: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Unknown error.', 'httpErrorCode': 503, 'errorCode': 1099, 'stack': "Error: ENOENT: no such file or directory, open 'conf/lists/arbitrum_one_tokens.json'"}

the gateway_container_file/conf/ does not have list folder
This is most likely due to the helper or compose script when creating image, does not copy the src/templates/list to the gateway_container_file/conf/list
Tried to modify the script and was able to create the conf/list folder but there are no default tokenlist that can be modified like on source build.
Change permission sudo chmod -R a+rw has been applied too the test but no go

Steps to reproduce:

Build docker image for this PR docker build -t hummingbot/gateway:126 .
Setup latest docker image for client development and this PR126
Run gateway config ethereum and observe the tokenListSource dir
Added any wallet from supported uniswap connectors
Observe error messages after adding the connector

logs_hummingbot.log

…eat/move-list

fengtality · 2023-06-08T01:42:21Z

@cardosofede for this to work when users install Gateway using Docker, I think the lists stored in /src/templates/lists to /conf/lists locally.

I tried to do this in the last commit, but the JSON files are still not being copied over.

fengtality · 2023-06-09T03:39:52Z

I changed the Dockerfile and created this PR in the deploy-examples repo: hummingbot/deploy#19

Tips for QA

Test this PR by installing Gateway from 1) source using gateway-setup.sh and 2) Docker using the Hummingbot+Gateway example from this PR Fix/add copy scripts deploy#19
Follow the instructionsin docs and README as if you are a new user
In both cases, you should be able to add a DEX connector with gateway connect uniswap
You should also be able to see WETH balances after gateway connector-tokens [exchange] WETH
Add a token entry to the list in the local directory
Afterwards you should be able to query for that token's balance from Hummingbot

rapcmia · 2023-06-09T06:38:00Z

PR update:

Test commit 143fa04b08ede19e84f7f11f8cdc3785e5e5aaff
Test this PR and latest client development version using source

Test macbook pro (m1)

Balance command works ✅
Ran gateway approve-token command and able to approve ✅
Setup simple AMM-ARB strategy and observed expected behavior ✅

Test WSL ubuntu 20.04LTS

Getting an error of sed: can't read s|/home/gateway/conf/lists/|conf/lists/|g: No such file or directory which seems to be on line 83 under replace_list_source()

While this happened client and gateway successfully connected ✅

tokenListSource is changed which is correct base on the src/templates/ folder for both WSL ubuntu 20.04 and MBP(m1) but we are assuming that the error from line 83 of gateway_setup.sh could be the reason.

Due to the changes on the default tokenListSource we are now getting the following error:

Call to https://localhost:15888/wallet/add failed. See logs for more details.
Error: ENOENT: no such file or directory, open '/home/gateway/conf/lists/arbitrum_one_tokens.json’

 2023-06-09 12:08:14,017 - 6264 - hummingbot.core.gateway.gateway_status_monitor - INFO - Gateway Service is ONLINE.
 2023-06-09 12:15:22,348 - 6264 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/wallet/add failed. See logs for more details.
 2023-06-09 12:15:22,348 - 6264 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Unknown error.', 'httpErrorCode': 503, 'errorCode': 1099, 'stack': "Error: ENOENT: no such file or directory, open '/home/gateway/conf/lists/arbitrum_one_tokens.json'"}
 Traceback (most recent call last):
   File "/home/rapcmia/github/hummingbot/dev/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
     return await c
   File "/home/rapcmia/github/hummingbot/dev/hummingbot/client/command/gateway_command.py", line 256, in _gateway_connect
     wallet_address, additional_prompt_values = await self._prompt_for_wallet_address(
   File "/home/rapcmia/github/hummingbot/dev/hummingbot/client/command/gateway_command.py", line 382, in _prompt_for_wallet_address
     response: Dict[str, Any] = await self._get_gateway_instance().add_wallet(
   File "/home/rapcmia/github/hummingbot/dev/hummingbot/core/gateway/gateway_http_client.py", line 265, in add_wallet
     return await self.api_request(method="post", path_url="wallet/add", params=request)
   File "/home/rapcmia/github/hummingbot/dev/hummingbot/core/gateway/gateway_http_client.py", line 218, in api_request
     raise e
   File "/home/rapcmia/github/hummingbot/dev/hummingbot/core/gateway/gateway_http_client.py", line 206, in api_request
     raise ValueError(f"Error on {method.upper()} {url} Error: {parsed_response}")
 ValueError: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Unknown error.', 'httpErrorCode': 503, 'errorCode': 1099, 'stack': "Error: ENOENT: no such file or directory, open '/home/gateway/conf/lists/arbitrum_one_tokens.json'"}

Able to have connect wallet successfully by changing the tokenListSource using the full directory where the .json files are located

Pending:

Docker using Hummingbot + gateway from PR19

nikspz · 2023-06-09T11:11:42Z

Docker using the Hummingbot+Gateway example from this PR hummingbot/deploy#19

Steps:

manually created docker image from gwPR126 and development branch
cloned Fix/add copy scripts deploy#19 and used hummingbot_gateway_compose/docker-compose.yml filed
used installation instructions from hummingbot_gateway_compose example
reviewed hummingbot client run and gateway ONLINE
gateway connect uniswap arbitrum

Actual:
Got an issue: Failed to add wallet. Also review lists folder not created in /conf/lists

Exit the Client and tried to connect wallet again - successfully connected, however still no lists folder created, Failed to ran amm_arb / approve tokens (on development branch same command approve tokens successfully/ amm_arb showing profitability on dev)

fengtality · 2023-06-09T13:37:01Z

@nikspz Did you see this line in the new instructions in the deploy-examples README?

Populate Gateway lists folder from token lists from the Gateway image:

docker cp gateway:/home/gateway/src/templates/lists ./gateway_files/conf/lists

david-hummingbot · 2023-06-09T17:17:24Z

Was able to replicate Ralph's issue on Source (amd64)

Got the sed -i '' 's|/home/gateway/conf/lists/|conf/lists/|g' $file error after running the gateway_setup.sh script.

Got an error in Gateway saying ENOENT: no such file or directory, open '/home/gateway/conf/lists/arbitrum_one_tokens.json' when trying to connect to uniswap arbitrum_one but running gateway config ethereum shows that there is an arbitrum_one_tokens.json file in the correct directory.

fengtality · 2023-06-10T16:16:04Z

@david-hummingbot I changed the sed command to perl in the setup script since ChatGPT told me it should work on all systems =)

If you see that the ethereum.yml file has been modified afterwards so that tokenListSource is conf/lists/...json, then it worked.

When the tokenListSource is /home/gateway/, it should work without modification for Docker users, since that is the location in the Docker volume that is mapped to the user's local gateway_files/conf/lists folder.

When the user is running from source, the perl command in gateway-setup.sh should replace the /home/gateway/ references with the local conf/lists folder location.

nikspz · 2023-06-13T09:18:05Z

https://github.com/hummingbot/gateway/pull/126/files#diff-046085b9f807c9537072f01918c95c25b997d185a4acbf552b26f691a7f67aefR107

https://github.com/hummingbot/gateway/pull/126/files#diff-046085b9f807c9537072f01918c95c25b997d185a4acbf552b26f691a7f67aefR17

We should add Yes and True as an option as well (as we do in hummingbot PRs before)

Steps:
Clone and install PR6373 and #126
Open Client generate certs
in the terminal cd gateway folder and yarn + yarn build
on ./gateway-setup.sh
Do you want to copy over client certificates (Y/N) >>> Yes

Actual:
Skipping copying client certificates

Expected:
Yes and True promt supported

david-hummingbot

Docker

initial test using hummingbot/hummingbot:scripts-copy image - ok
test using deploy-examples PR#19 - ok
Docker image built locally - ok
docker compose up -d
sudo chmod -R a+rw ./hummingbot_files
docker cp hummingbot:/home/hummingbot/scripts-copy/. ./hummingbot_files/scripts/
verified scripts files are present in the /scripts folder
docker attach hummingbot
run start --script and made sure scripts are showing - ok
run script - ok

Source install

install gateway ok
run ./gateway-setup.sh, no errors
checked ethereum.yml file and verified tokenlistsource shows conf/lists
checked conf/lists folder and .json files are present
start up gateway and client shows online
gateway connect uniswap arbitrum
run gateway config ethereum and confirm tokenListSource directory

fengtality · 2023-06-13T18:03:40Z

We should add Yes and True as an option as well (as we do in hummingbot PRs before)

@nikspz For bash scripts, we previously only supported "Y" or "y" - see https://github.com/hummingbot/gateway/tree/main/docker. I think we only try to support Yes or True in the client.

(feat) move eth lists and modify gateway setup

e2ebba3

fengtality added the enhancement New feature or request label Jun 2, 2023

fengtality requested a review from rapcmia June 2, 2023 22:13

rapcmia requested review from david-hummingbot and nikspz June 6, 2023 01:10

Merge branch 'development' into feat/move-list

784c65e

nikspz and others added 3 commits June 6, 2023 23:19

Merge branch 'development' into feat/move-list

3239f5e

(wip) add lists to docker image

6339128

Merge branch 'feat/move-list' of github.com:hummingbot/gateway into f…

8956316

…eat/move-list

fengtality added 2 commits June 8, 2023 06:15

changes

563f798

redo docker setup

143fa04

fengtality mentioned this pull request Jun 9, 2023

Fix/add copy scripts hummingbot/deploy#19

Merged

(fix) change sed to perl and update lists

dfce457

fengtality mentioned this pull request Jun 10, 2023

Uniswap on arbitrum_one does not support ARB token in tokenlist hummingbot/hummingbot#6340

Closed

david-hummingbot approved these changes Jun 13, 2023

View reviewed changes

fengtality merged commit 77ac9f3 into development Jun 13, 2023

fengtality deleted the feat/move-list branch June 13, 2023 18:03

fengtality mentioned this pull request Jun 14, 2023

Feat/move all lists #132

Merged

2 tasks

This was referenced Jun 19, 2023

sync gateway / gateway development -> staging #134

Merged

sync gateway / gateway staging -> main for Hummingbot gateway version 1.17.0 #137

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

(feat) move ethereum token lists to `conf` #126

(feat) move ethereum token lists to `conf` #126

Uh oh!

fengtality commented Jun 2, 2023 •

edited

Loading

Uh oh!

rapcmia commented Jun 6, 2023

Uh oh!

fengtality commented Jun 8, 2023

Uh oh!

fengtality commented Jun 9, 2023

Uh oh!

rapcmia commented Jun 9, 2023

Uh oh!

nikspz commented Jun 9, 2023 •

edited

Loading

Uh oh!

fengtality commented Jun 9, 2023

Uh oh!

david-hummingbot commented Jun 9, 2023

Uh oh!

fengtality commented Jun 10, 2023 •

edited

Loading

Uh oh!

nikspz commented Jun 13, 2023 •

edited

Loading

Uh oh!

david-hummingbot left a comment

Uh oh!

fengtality commented Jun 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

(feat) move ethereum token lists to conf #126

(feat) move ethereum token lists to conf #126

Uh oh!

Conversation

fengtality commented Jun 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rapcmia commented Jun 6, 2023

Uh oh!

fengtality commented Jun 8, 2023

Uh oh!

fengtality commented Jun 9, 2023

Uh oh!

rapcmia commented Jun 9, 2023

Uh oh!

nikspz commented Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fengtality commented Jun 9, 2023

Uh oh!

david-hummingbot commented Jun 9, 2023

Uh oh!

fengtality commented Jun 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nikspz commented Jun 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

david-hummingbot left a comment

Choose a reason for hiding this comment

Uh oh!

fengtality commented Jun 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

(feat) move ethereum token lists to `conf` #126

(feat) move ethereum token lists to `conf` #126

fengtality commented Jun 2, 2023 •

edited

Loading

nikspz commented Jun 9, 2023 •

edited

Loading

fengtality commented Jun 10, 2023 •

edited

Loading

nikspz commented Jun 13, 2023 •

edited

Loading