[dataquery] Add ability to retrieve results of previous run from API #9301

driusan · 2024-06-20T14:01:02Z

This implements the endpoint that is listed as "aspirational" in the
dataquery api schema to load the results of a previously run query.

It was originally left out for fear of performance issues but with
this implementation performance seems to be comparable to without
it.

This does not implement a frontend to reload query results, but
the API functionality is useful for i.e. api callbacks that want
to process the results of a query without the need to re-run
it (and the possibility to get different results.)

driusan · 2024-06-20T14:01:49Z

modules/dataquery/php/endpoints/queries/query/run.class.inc

+                function (
+                    $candid,
+                    $rowsArray,
+                    /* @phan-suppress-next-line PhanUnusedClosureParameter */


Closure signature needs to match the callback of DataIteratorBinaryStream, so the unused parameter can't be removed.

driusan · 2024-06-20T14:02:22Z

modules/dataquery/php/endpoints/queries/query/run.class.inc

-     *
-     * @return ResponseInterface
-     */
-    public function countResults(\User $user) : ResponseInterface


This function was unused in this class, so I removed it while I was cleaning up the code.

modules/dataquery/php/endpoints/queries/query/run/run.class.inc

CamilleBeau · 2024-10-29T15:08:37Z

@driusan needs to have the conflict resolved but otherwise this is working well!

This implements the endpoint that is listed as "aspirational" in the dataquery api schema to load the results of a previously run query. It was originally left out for fear of performance issues but with this implementation performance seems to be comparable to without it. This does not implement a frontend to reload query results, but the API functionality is useful for eg api callbacks that want to process the results of a query without the need to re-run it (and the possibility to get different results.)

CamilleBeau

Running a query from this PR isn't working. Getting the following error:

[Wed Jan 22 11:00:02.344866 2025] [php:error] [pid 7561] [client ::1:42036] PHP Fatal error: Uncaught TypeError: LORIS\dataquery\QueryRun::getCandidates(): Return value must be of type array, Generator returned in /var/www/loris/modules/dataquery/php/queryrun.class.inc:103\nStack trace:\n#0 /var/www/loris/modules/dataquery/php/querydataprovisioner.class.inc(51): LORIS\dataquery\QueryRun->getCandidates()\n#1 /var/www/loris/src/Data/ProvisionerInstance.php(116): LORIS\dataquery\QueryDataProvisioner->getAllInstances()\n#2 /var/www/loris/modules/dataquery/php/querydataprovisioner.class.inc(41): LORIS\Data\ProvisionerInstance->execute()\n#3 /var/www/loris/modules/dataquery/php/endpoints/queries/query/run.class.inc(70): LORIS\dataquery\QueryDataProvisioner->execute()\n#4 /var/www/loris/modules/dataquery/php/endpoints/queries/query/run.class.inc(47): LORIS\dataquery\endpoints\queries\query\Run->runQuery()\n#5 /var/www/loris/src/Middleware/Compression.php(36): LORIS\dataquery\endpoints\queries\query\Run->handle()\n#6 /var/www/loris/modules/dataquery/php/queries.class.inc(83): LORIS\Middleware\Compression->process()\n#7 /var/www/loris/src/Middleware/UserPageDecorationMiddleware.php(248): LORIS\dataquery\Queries->handle()\n#8 /var/www/loris/src/Middleware/PageDecorationMiddleware.php(59): LORIS\Middleware\UserPageDecorationMiddleware->process()\n#9 /var/www/loris/php/libraries/NDB_Page.class.inc(726): LORIS\Middleware\PageDecorationMiddleware->process()\n#10 /var/www/loris/php/libraries/Module.class.inc(322): NDB_Page->process()\n#11 /var/www/loris/src/Middleware/ResponseGenerator.php(51): Module->handle()\n#12 /var/www/loris/src/Middleware/AuthMiddleware.php(64): LORIS\Middleware\ResponseGenerator->process()\n#13 /var/www/loris/src/Router/ModuleRouter.php(75): LORIS\Middleware\AuthMiddleware->process()\n#14 /var/www/loris/src/Middleware/ExceptionHandlingMiddleware.php(55): LORIS\Router\ModuleRouter->handle()\n#15 /var/www/loris/src/Router/BaseRouter.php(138): LORIS\Middleware\ExceptionHandlingMiddleware->process()\n#16 /var/www/loris/src/Middleware/ResponseGenerator.php(51): LORIS\Router\BaseRouter->handle()\n#17 /var/www/loris/src/Middleware/ContentLength.php(53): LORIS\Middleware\ResponseGenerator->process()\n#18 /var/www/loris/htdocs/index.php(74): LORIS\Middleware\ContentLength->process()\n#19 {main}\n thrown in /var/www/loris/modules/dataquery/php/queryrun.class.inc on line 103, referer: http://localhost:8080/dataquery/

driusan · 2025-01-22T18:13:07Z

@CamilleBeau I fixed the error (and a couple other introduced by the rebase). Should be ready now.

CamilleBeau

Running a query, saving a query, filters, and accessing previous queries from API are all working now.

Just a couple of notes on the code. There are tests failing as well

CamilleBeau · 2025-01-22T19:00:25Z

modules/dataquery/php/endpoints/queries/query/run.class.inc


-        }
-    }
+        set_time_limit(300);


Should this be configurable?

Maybe?

I think it can just be removed. Our documentation says to set max_execution_time to 10800 to support large files which is the php config setting that php would fall back on.

https://acesloris.readthedocs.io/en/latest/docs/wiki/00_SERVER_INSTALL_AND_CONFIGURATION/02_Website_Configuration/Handling_Large_File_Uploads.html

modules/dataquery/php/endpoints/queries/query/run.class.inc

CamilleBeau · 2025-01-22T19:03:59Z

modules/dataquery/php/endpoints/queries/query/run/run.class.inc

+    public function queryResults(\User $user) : ResponseInterface
+    {
+            $db = $this->loris->getDatabaseConnection();
+        // FIXME: Check permissions. Check if count(*) > 0.


Permission check needed still?

Oops.. you're right. That's still needed.

modules/dataquery/php/endpoints/queries/query/run/run.class.inc

CamilleBeau

All the tests I ran are passing now for the saved queries accessed from API, running queries, saving queries, and accessing queries / data from a user with limited access. One thing to note is that you can have access to another user's private query using the API URL as long as you have the access to all of the fields in the query. If this is intended then all looks good!

driusan commented Jun 20, 2024

View reviewed changes

modules/dataquery/php/endpoints/queries/query/run/run.class.inc Outdated Show resolved Hide resolved

driusan added Release: Add to release notes PR whose changes should be highlighted in the release notes Category: Feature PR or issue that aims to introduce a new feature labels Jun 20, 2024

cmadjar assigned CamilleBeau Oct 8, 2024

CamilleBeau assigned driusan Nov 11, 2024

regisoc and others added 2 commits January 22, 2025 10:14

Fix documentation and add missing files

ef357da

driusan force-pushed the DQTSaveResults branch from 2e51649 to ef357da Compare January 22, 2025 15:14

driusan marked this pull request as ready for review January 22, 2025 15:15

CamilleBeau requested changes Jan 22, 2025

View reviewed changes

CamilleBeau added the State: Needs work PR awaiting additional work by the author to proceed label Jan 22, 2025

Fix errors introduced after rebase

730b943

driusan removed the State: Needs work PR awaiting additional work by the author to proceed label Jan 22, 2025

Remove debug changes

fc6ac90

CamilleBeau reviewed Jan 22, 2025

View reviewed changes

Remove set_time_limit, unnecessary commented out lines

85f8e12

driusan added the State: Needs work PR awaiting additional work by the author to proceed label Jan 22, 2025

Add checking for permissions to retrieving of QueryRun results.

97f10ad

CamilleBeau approved these changes Jan 27, 2025

View reviewed changes

CamilleBeau removed the State: Needs work PR awaiting additional work by the author to proceed label Jan 27, 2025

driusan merged commit 7ee7478 into aces:main Feb 4, 2025
19 checks passed

[dataquery] Add ability to retrieve results of previous run from API #9301

[dataquery] Add ability to retrieve results of previous run from API #9301

Uh oh!

Conversation

driusan commented Jun 20, 2024

Uh oh!

driusan Jun 20, 2024

Choose a reason for hiding this comment

Uh oh!

driusan Jun 20, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CamilleBeau commented Oct 29, 2024

Uh oh!

CamilleBeau left a comment

Choose a reason for hiding this comment

Uh oh!

driusan commented Jan 22, 2025

Uh oh!

CamilleBeau left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CamilleBeau Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

driusan Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

CamilleBeau Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

driusan Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

driusan Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CamilleBeau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CamilleBeau left a comment •

edited

Loading