Integration test for Sycamore Query demo. #985
Conversation
| "--exec-mode", type=str, choices=["ray", "local"], default="ray", help="Configure Sycamore execution mode." | ||
| ) | ||
| argparser.add_argument("--chat", action="store_true", help="Only show the chat demo pane.") | ||
| argparser.add_argument("--test", action="store_true", help="Only run the integration test.") |
There was a problem hiding this comment.
Do you need both chat and test if both are never used together? Why not a mode argument that is either chat or test?
There was a problem hiding this comment.
Well, for now --test is kind of a whole different mode of execution of the entire script, which will subsume a bunch of the other arguments anyway. Maybe the specific test for not using --chat does not make sense in that light.
There was a problem hiding this comment.
Yes, it would help clarify this architecture better as well. I didn't realize that they encompass overlapping things while reading this script.
| "--exec-mode", type=str, choices=["ray", "local"], default="ray", help="Configure Sycamore execution mode." | ||
| ) | ||
| argparser.add_argument("--chat", action="store_true", help="Only show the chat demo pane.") | ||
| argparser.add_argument("--test", action="store_true", help="Only run the integration test.") |
There was a problem hiding this comment.
Yes, it would help clarify this architecture better as well. I didn't realize that they encompass overlapping things while reading this script.
This PR uses the Streamlit app testing library (which is incredibly handy) to provide an integration test for the Luna demo.
This is needed because we often find that small changes to Luna or the UI code can break the demo. I need a way to run an integration test when the demo is deployed, and this is it.
The intention is not to run this test as part of CI, since it needs a specifically-configured OpenSearch index. Instead, it will be run as part of the demo deployment process.
The Streamlit AppTest class lets us run a "headless" Streamlit app, interact with it programmatically, and access its state, including all of the UI elements. To make writing test cases easier, I've implemented a few helper functions to walk the UI tree and test specific conditions on it.
I realize that understanding this test code is going to be difficult if you don't know the ins and outs of the Luna demo code itself, but I'm not expecting reviewers to have that background. I'd appreciate your looking for any glaring mistakes or stylistic suggestions. Thanks!