-
Notifications
You must be signed in to change notification settings - Fork 12
COCOS-346 - Explore cloud init for Cloud setup #357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
dborovcanin
merged 10 commits into
ultravioletrs:main
from
smithjilks:cocos-346-explore-cloud-init
Jan 31, 2025
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
24ebef1
Add qemu cloud init
smithjilks 5091671
Update qemu cloud init
smithjilks 134321d
Add qemu cloud init
smithjilks 1a1e90b
Update qemu cloud init
smithjilks dff9f1c
Update qemu cloud config
smithjilks 2789c85
Merge remote-tracking branch 'origin/cocos-346-explore-cloud-init' in…
smithjilks 4169d8d
Update cloud init
smithjilks b68819c
Update cloud init
smithjilks 2327acb
Add cloud init README.md
smithjilks b348602
Add cocos release workflow
smithjilks File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,9 @@ | ||
name: Build and Release | ||
name: Build and Release Hal | ||
|
||
on: | ||
push: | ||
tags: | ||
- '*' | ||
- "*" | ||
|
||
jobs: | ||
build: | ||
|
@@ -32,8 +32,8 @@ jobs: | |
with: | ||
root-reserve-mb: 35000 | ||
swap-size-mb: 1024 | ||
remove-dotnet: 'true' | ||
remove-android: 'true' | ||
remove-dotnet: "true" | ||
remove-android: "true" | ||
- name: Check free space | ||
run: | | ||
echo "Free space:" | ||
|
@@ -48,26 +48,33 @@ jobs: | |
- name: Checkout cocos | ||
uses: actions/checkout@v4 | ||
with: | ||
repository: 'ultravioletrs/cocos' | ||
repository: "ultravioletrs/cocos" | ||
path: cocos | ||
|
||
- name: Checkout buildroot | ||
uses: actions/checkout@v4 | ||
with: | ||
repository: 'buildroot/buildroot' | ||
repository: "buildroot/buildroot" | ||
path: buildroot | ||
ref: 2024.11-rc2 | ||
|
||
- name: Build | ||
- name: Build hal | ||
run: | | ||
cd buildroot | ||
make BR2_EXTERNAL=../cocos/hal/linux cocos_defconfig | ||
make | ||
|
||
- name: Build cocos | ||
run: | | ||
cd cocos | ||
make | ||
|
||
- name: Release | ||
uses: softprops/action-gh-release@v2 | ||
with: | ||
files: | | ||
buildroot/output/images/bzImage | ||
buildroot/output/images/rootfs.cpio.gz | ||
|
||
cocos/build/cocos-agent | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is there a step to build these binaries? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. There is. There is a make run. |
||
cocos/build/cocos-cli | ||
cocos/build/cocos-manager |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,89 @@ | ||
#### memory config | ||
MEMORY_SIZE=2048M | ||
MEMORY_SLOTS=5 | ||
MAX_MEMORY=30G | ||
|
||
|
||
#### ovmf code config | ||
OVMF_CODE_IF=pflash | ||
OVMF_CODE_FORMAT=raw | ||
OVMF_CODE_UNIT=0 | ||
OVMF_CODE_FILE=/usr/share/OVMF/OVMF_CODE.fd | ||
OVMF_CODE_READONLY=on | ||
OVMF_VERSION= | ||
|
||
#### ovmf vars config | ||
OVMF_VARS_IF=pflash | ||
OVMF_VARS_FORMAT=raw | ||
OVMF_VARS_UNIT=1 | ||
OVMF_VARS_FILE=/usr/share/OVMF/OVMF_VARS.fd | ||
|
||
#### net dev config | ||
NET_DEV_ID=vmnic | ||
NET_DEV_HOST_FWD_AGENT=7020 | ||
NET_DEV_GUEST_FWD_AGENT=7002 | ||
|
||
#### Virtio Net Pci Config | ||
VIRTIO_NET_PCI_DISABLE_LEGACY=on | ||
VIRTIO_NET_PCI_IOMMU_PLATFORM=true | ||
VIRTIO_NET_PCI_ADDR=0x2 | ||
VIRTIO_NET_PCI_ROMFILE= | ||
|
||
#### Disk image config | ||
DISK_IMG_KERNEL_FILE= | ||
DISK_IMG_ROOTFS_FILE= | ||
|
||
KERNEL_COMMAND_LINE="quiet console=null" | ||
|
||
#### Sev Config | ||
SEV_ID=sev0 | ||
SEV_CBIT_POS=51 | ||
SEV_REDUCED_PHYS_BITS=1 | ||
SEV_HOST_DATA= | ||
|
||
#### VSock Config | ||
VSOCK_ID=vhost-vsock-pci0 | ||
VSOCK_GUEST_CID=3 | ||
|
||
BIN_PATH=qemu-system-x86_64 | ||
USE_SUDO=false | ||
ENABLE_SEV=false | ||
ENABLE_SEV_SNP=false | ||
|
||
ENABLE_KVM=true | ||
|
||
MACHINE=q35 | ||
CPU=EPYC | ||
SMP_COUNT=8 | ||
SMP_MAXCPUS=64 | ||
MEM_ID=ram1 | ||
|
||
KERNEL_HASH=false | ||
|
||
NO_GRAPHIC=true | ||
MONITOR=pty | ||
HOST_FWD_RANGE=6100-6200 | ||
CERTS_MOUNT=/etc/cocos/certs | ||
ENV_MOUNT=/etc/cocos/environment | ||
|
||
COCOS_AGENT_VERSION=v0.3.1 | ||
|
||
#### Base image URL and names | ||
BASE_IMAGE_URL=https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img | ||
BASE_IMAGE=ubuntu-base.qcow2 | ||
CUSTOM_IMAGE=ubuntu-custom.qcow2 | ||
|
||
#### Paths for OVMF firmware | ||
OVMF_CODE=/usr/share/ovmf/x64/OVMF_CODE.4m.fd | ||
OVMF_VARS=/usr/share/ovmf/x64/OVMF_VARS.4m.fd | ||
|
||
#### VM parameters | ||
VM_NAME=cocos-vm | ||
RAM=16G | ||
DISK_SIZE=10G # Size for root filesystem | ||
QEMU_BINARY=qemu-system-x86_64 | ||
|
||
AGENT_GRPC_SERVER_CERT=/etc/cocos/certs/server.pem | ||
AGENT_GRPC_SERVER_KEY=/etc/cocos/certs/key.pem | ||
AGENT_GRPC_SERVER_CA_CERTS=/etc/cocos/ca.pem | ||
AGENT_GRPC_CLIENT_CA_CERTS=/etc/cocos/ca.pem |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
# Agent Cloud Init Setup | ||
|
||
## Overview | ||
|
||
The `hal/cloud` directory contains essential files required for setting up a virtual machine (VM) with cloud-init. This setup ensures the automated installation of dependencies, configuration of the environment, and deployment of the Cocos agent as a systemd service. | ||
|
||
### Directory Contents | ||
|
||
- **`config.yaml`**: This YAML file provides configuration instructions for the cloud image. | ||
- **`meta-data`**: Contains VM metadata, such as instance-specific details and identifiers. | ||
- **`qemu.sh`**: A Bash script for downloading and configuring a cloud image, running QEMU to simulate a VM with the cloud-init configuration. | ||
- **`.env`**: Contains environment variables for starting the VM in different modes, configuring disk space, memory allocation, and other parameters. | ||
|
||
## Configuration | ||
|
||
### Preparing the Cloud-Config File | ||
|
||
The `config.yaml` file defines system configurations, including user creation, package installations, file management, and command execution. | ||
Ensure that the cloud-config file is set up with the following configurations: | ||
|
||
- **User Credentials**: Specify the default username and password. | ||
- **Certificates and Keys**: Certificate files for agent for secure communication. | ||
- **Environment Variables**: Configuration parameters required by the system. | ||
|
||
The `config.yaml` file is divided into multiple sections, each addressing a specific aspect of the setup process. | ||
|
||
### 1. User Configuration | ||
|
||
This section creates a default user with specific permissions and configurations: | ||
|
||
- Creates a user named **`cocos_user`**. | ||
- Adds `cocos_user` to the `sudo` and `docker` groups. | ||
- Sets a default password (should be changed for production use). | ||
- Configures the user’s shell as `/bin/bash`. | ||
|
||
### 2. Package Installation | ||
|
||
Installs essential system packages required for various operations: | ||
|
||
- **`curl`**: For downloading files from the web. | ||
- **`make`**: A utility for building software. | ||
- **`git`**: Version control system for managing code repositories. | ||
- **`python3` and `python3-dev`**: Required for running Python-based tools. | ||
- **`net-tools`**: Provides networking utilities such as `ifconfig` and `route`. | ||
|
||
### 3. File Management (write_files) | ||
|
||
Creates and configures critical files required for the setup: | ||
|
||
- **Certificates**: Cert files (`cert.pem`, `ca.pem`, `key.pem`) located at `/etc/cocos/certs/`. | ||
- **Environment Variables**: An env file stored at `/etc/cocos/environment`. | ||
- **Systemd Service File**: Cocos agent service configuration file at `/etc/systemd/system/cocos-agent.service` for managing the Cocos agent. | ||
- **Agent Scripts**: | ||
- `agent_setup.sh`: Configures network interfaces and resizes the root filesystem. | ||
- `agent_start_script.sh`: Sets up Docker and starts the Cocos agent. | ||
|
||
### 4. Execution of Commands (runcmd) | ||
|
||
A sequence of commands is executed to finalize the setup: | ||
|
||
- Creates necessary directories: `/cocos`, `/cocos_init`, `/var/log/cocos`, `/etc/cocos`. | ||
- Downloads and installs the Cocos agent binary. | ||
- Installs **Wasmtime** and configures its environment variables. | ||
- Installs **Docker** and adds `cocos_user` to the Docker group. | ||
- Reloads systemd and enables the Cocos agent service. | ||
|
||
## Running the Agent | ||
|
||
To test the cloud-init configuration, execute the `qemu.sh` script to bring up a VM using QEMU: | ||
|
||
```bash | ||
sudo ./qemu.sh | ||
``` | ||
|
||
**Important:** The script must be executed as root. | ||
|
||
Once the QEMU boots the VM, the Cocos agent will run as a systemd service. The service is configured to start automatically on boot and restart in case of failure. | ||
|
||
## Debugging and Monitoring | ||
|
||
For troubleshooting and monitoring the Cocos agent service, use the following commands within the VM: | ||
|
||
### Manually Start the Service | ||
|
||
To manually start the agent service, execute: | ||
|
||
```bash | ||
sudo systemctl start cocos-agent.service | ||
``` | ||
|
||
### Verify Service Status | ||
|
||
To check if the service is running properly, use: | ||
|
||
```bash | ||
sudo systemctl status cocos-agent.service | ||
``` | ||
|
||
### View Service Logs | ||
|
||
To inspect logs generated by the agent service, execute: | ||
|
||
```bash | ||
journalctl -u cocos-agent.service | ||
``` | ||
|
||
### Check Standard Output and Error Logs | ||
|
||
To check logs stored in the system, use: | ||
|
||
```bash | ||
cat /var/log/cocos/agent.stdout.log | ||
cat /var/log/cocos/agent.stderr.log | ||
``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is the run.