COCOS-165 - Add Docker support #180

danko-miladinovic · 2024-07-22T17:07:53Z

What type of PR is this?

This is a feature because it enables the Agent to run Docker images inside the VM.

What does this do?

This PR introduces a new way of executing algorithms by using Docker.

Which issue(s) does this PR fix/relate to?

Resolves #165

Have you included tests for your changes?

The agent-config/main.go has been changed to create an option for the Agent to run docker.

Did you document any new/modified feature?

The official documentation will be updated in a separate PR. The manual testing documentation has been updated.

Notes

SammyOina · 2024-07-31T15:10:06Z

test/manual/README.md

 # 2023/09/21 10:43:53 Uploading algorithm binary: test/manual/algo/lin_reg.bin

+# In order to run the Docker image, run the CLI program with the algorithm docker option
+go run ./cmd/cli/main.go algo -a docker -d "python3 /cocos/lin_reg.py" <path_to_docker_image.tar> <private_key_file_path>


please add the docker file to test/algo for the example used, also add docs there on how to build and output the docker image

I have added an README file for the docker.

SammyOina

results fails

go run ./cmd/cli/main.go result private.pem                                                                                                                                                            ✔  10s  
2024/08/02 16:57:50 Retrieving computation result file
2024/08/02 16:58:00 Error retrieving computation result: rpc error: code = Unknown desc = could not copy from the Docker container: failed to copy from container: Error response from daemon: Could not find the file /result/result.bin in container agent_container
exit status 1

danko-miladinovic · 2024-08-02T14:35:38Z

results fails

go run ./cmd/cli/main.go result private.pem                                                                                                                                                            ✔  10s  
2024/08/02 16:57:50 Retrieving computation result file
2024/08/02 16:58:00 Error retrieving computation result: rpc error: code = Unknown desc = could not copy from the Docker container: failed to copy from container: Error response from daemon: Could not find the file /result/result.bin in container agent_container
exit status 1

Fixed, the problem was in the lin_reg.py file. My mistake was that I left the dataset path be /datasets/iris.csv but the datasets are renamed when the Agent saves them, so it did not work.

SammyOina · 2024-08-06T17:11:46Z

agent/service.go

+			return
+		}
+	} else if err != nil {
+		fmt.Printf("Could not create result dir\n")


Suggested change

fmt.Printf("Could not create result dir\n")

SammyOina · 2024-08-06T17:14:31Z

agent/algorithm/docker/docker.go

+func (d *docker) AddDataset(dataset string) {
+	d.logger.Debug("dataset method called")
+}


SammyOina · 2024-08-06T17:15:51Z

agent/algorithm/docker/docker.go

+	case <-statusCh:
+	}
+
+	agentResultOutput := path.Join(algorithm.AlgorithmResultOutputDir, algorithm.AlgorithmResultOutputFile)


there could be multiple files. results dir is zipped by agent

The code is now changed to reflect this change.

SammyOina · 2024-08-07T13:06:55Z

test/manual/algo/lin_reg.py

+DATA_DIR = "/datasets"
+RESULTS_DIR = "/results"


this absolute path will work on docker but not on python or python or bin. on docker can you have the algo mount be in a relative path to datasets and results mounts

drasko · 2024-08-07T22:39:34Z

agent/algorithm/docker/docker.go

+
+const (
+	containerName       = "agent_container"
+	DockerRunCommand    = "python3 /cocos/algorithm.py"


This is not clear to me, what is this const? Does that mean that we can run only Python algorithms? Should we just run a docker, whatever the binary is inside, and it is a docker creator's obligation to ensure that the container will boot properly, that the binary (or script, or whatever) will run inside and that it will find datasets mapped on the volume

it is just the default, but the user can supply their own command to run in docker

I have changed the code so that the docker start command must be a part of the docker image. The datasets and results are left as options in the CLI. They are there because the python, wasm and bin will not work if the datasets and results dirs are /datasets and /results. They need to be relative for those algorithms.

dborovcanin · 2024-08-14T14:53:33Z

@drasko, @SammyOina Please re-review.

danko-miladinovic · 2024-08-19T14:59:44Z

Now, the /cocos directory is the default working directory for the Agent. The Docker image must be built so it has the /cocos directory. In this directory the datasets and results will be mounted.

SammyOina · 2024-08-20T09:44:59Z

test/manual/docker/README.md

move to algo readme and combine with existing readme

SammyOina · 2024-08-20T11:42:19Z

test/manual/README.md


 ```sh
-python3 test/manual/algo/lin_reg_test.py test/manual/data/iris.csv result.bin
+python3 test/manual/algo/lin_reg.py predict result.zip /test/manual/data


Sorry, mistake happened during copy/paste.

danko-miladinovic marked this pull request as ready for review July 22, 2024 22:36

danko-miladinovic force-pushed the docker branch 2 times, most recently from 656d428 to 78d7204 Compare July 31, 2024 11:07

SammyOina requested changes Jul 31, 2024

View reviewed changes

SammyOina requested changes Aug 2, 2024

View reviewed changes

danko-miladinovic requested a review from SammyOina August 2, 2024 14:41

SammyOina approved these changes Aug 5, 2024

View reviewed changes

SammyOina requested changes Aug 6, 2024

View reviewed changes

danko-miladinovic force-pushed the docker branch from 90b3f44 to 7e45b82 Compare August 7, 2024 12:35

SammyOina requested changes Aug 7, 2024

View reviewed changes

danko-miladinovic force-pushed the docker branch from 77dbc27 to 62fb08e Compare August 7, 2024 15:18

SammyOina approved these changes Aug 7, 2024

View reviewed changes

drasko requested changes Aug 7, 2024

View reviewed changes

danko-miladinovic force-pushed the docker branch from 9499f1a to c321b3b Compare August 19, 2024 13:57

SammyOina requested changes Aug 20, 2024

View reviewed changes

test/manual/docker/README.md Outdated

Copy link

Contributor

SammyOina Aug 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to algo readme and combine with existing readme

SammyOina requested changes Aug 20, 2024

View reviewed changes

SammyOina approved these changes Aug 20, 2024

View reviewed changes

danko-miladinovic force-pushed the docker branch from 5a5a568 to 7f39ff4 Compare August 21, 2024 09:11

danko-miladinovic added 8 commits August 21, 2024 12:32

add docker support

db08d2f

add copyright clause

2dad1f7

rebase docker support

a064f8c

address blank lines

969708b

update manual tests to include docker

4f8a65c

fix algo test

67495fe

fix docker command

0142529

add docker doc

d812c92

danko-miladinovic added 15 commits August 21, 2024 12:32

fix AddDataset method

6fcdbb3

fixed lin_reg.py

343fd28

rebsed docker implementation

089d34b

fix NewAlgorithm error

b9729cb

change docker README.md based on rebase

9428ed9

fix docker README

5f62dfb

fix docker.go gofumpt

3e1d0f6

add option for datasets and results mount

8e61588

edit README for docker

8a86f56

make docker container run command a part of docker image

955d07e

remove unused code

0d98f3f

make /cocos the default directory

b31a1d1

updated documentation

d3ce1d9

removed docker dir

89e859a

rebased docker

67a723c

danko-miladinovic force-pushed the docker branch from 7f39ff4 to 67a723c Compare August 21, 2024 10:32

drasko merged commit ee83704 into ultravioletrs:main Aug 21, 2024

danko-miladinovic deleted the docker branch August 8, 2025 15:31

		DATA_DIR = "/datasets"
		RESULTS_DIR = "/results"

COCOS-165 - Add Docker support #180

COCOS-165 - Add Docker support #180

Uh oh!

Conversation

danko-miladinovic commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What does this do?

Which issue(s) does this PR fix/relate to?

Have you included tests for your changes?

Did you document any new/modified feature?

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SammyOina left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danko-miladinovic commented Aug 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dborovcanin commented Aug 14, 2024

Uh oh!

danko-miladinovic commented Aug 19, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

danko-miladinovic commented Jul 22, 2024 •

edited

Loading

SammyOina left a comment •

edited

Loading