Cloud Deployment (GKE)

Deploy agents to Google Kubernetes Engine for production-grade scalability and reliability

Overview

Cloud deployment uses Google Kubernetes Engine (GKE) to run agents in a managed Kubernetes environment. This provides high availability, automatic scaling, and seamless integration with Google Cloud services.

Architecture

Prerequisites

Google Cloud Account

Active GCP project with billing enabled

GKE Cluster

Running Kubernetes cluster in your GCP project

kubectl

Kubernetes command-line tool configured for your cluster

Docker Image

Built and pushed to container registry (GHCR)

Deployment Architecture

Kubernetes Resources

The application runs in the agents namespace with multiple deployments:

pma-agent · Deployment

Main prediction market agent deployment running configured agents

pma-agent-monitoring · Deployment

Monitoring and observability services for agent performance tracking

autonomous-trader-agent · Deployment

Autonomous trading agent with microchain framework (deployed here)

deployed-general-agent-viewer · Deployment

Web interface for viewing deployed agent status and performance

treasury · Deployment

Treasury management service for fund allocation and tracking

Setup Instructions

1. Configure kubectl

   Connect kubectl to your GKE cluster:

   gcloud container clusters get-credentials <cluster-name> \
     --region <region> \
     --project <project-id>
   

   Verify connection:

   kubectl cluster-info
   kubectl get nodes
   
2. Create Namespace

   Create the agents namespace if it doesn’t exist:

   kubectl create namespace agents
   

   Set as default namespace:

   kubectl config set-context --current --namespace=agents
   
3. Configure Secrets

   Create Kubernetes secrets for sensitive environment variables:

   kubectl create secret generic agent-secrets \
     --from-literal=BET_FROM_PRIVATE_KEY="your_private_key" \
     --from-literal=OPENAI_API_KEY="your_openai_key" \
     --from-literal=SERPER_API_KEY="your_serper_key" \
     --from-literal=TAVILY_API_KEY="your_tavily_key" \
     --namespace=agents
   

   Never commit secrets to version control. Use secret management tools like Google Secret Manager.

4. Create Deployment

   Create a deployment manifest:

   deployment.yaml

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: pma-agent
     namespace: agents
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: pma-agent
     template:
       metadata:
         labels:
           app: pma-agent
       spec:
         containers:
         - name: agent
           image: ghcr.io/gnosis/prediction-market-agent:main
           env:
           - name: runnable_agent_name
             value: "prophet_gpt4o"
           - name: market_type
             value: "omen"
           - name: BET_FROM_PRIVATE_KEY
             valueFrom:
               secretKeyRef:
                 name: agent-secrets
                 key: BET_FROM_PRIVATE_KEY
           - name: OPENAI_API_KEY
             valueFrom:
               secretKeyRef:
                 name: agent-secrets
                 key: OPENAI_API_KEY
           resources:
             requests:
               memory: "2Gi"
               cpu: "1000m"
             limits:
               memory: "4Gi"
               cpu: "2000m"
           imagePullPolicy: Always
         imagePullSecrets:
         - name: ghcr-secret
   

   Apply the deployment:

   kubectl apply -f deployment.yaml
   
5. Verify Deployment

   Check deployment status:

   kubectl get deployments -n agents
   kubectl get pods -n agents
   kubectl logs -f <pod-name> -n agents
   

Deployment Configuration

Environment Variables

Configure agents via environment variables in your deployment:

env:
- name: runnable_agent_name
  value: "prophet_gpt4o"  # Agent to run
- name: market_type
  value: "omen"  # Target market
- name: BET_FROM_PRIVATE_KEY
  valueFrom:
    secretKeyRef:
      name: agent-secrets
      key: BET_FROM_PRIVATE_KEY

Resource Limits

Set appropriate resource requests and limits:

Light Agent

resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "2Gi"
    cpu: "1000m"

Image Pull Policy

For production, use specific tags instead of latest:

containers:
- name: agent
  image: ghcr.io/gnosis/prediction-market-agent:v1.2.3
  imagePullPolicy: IfNotPresent  # Only pull if not cached

For development, always pull latest:

containers:
- name: agent
  image: ghcr.io/gnosis/prediction-market-agent:main
  imagePullPolicy: Always  # Always pull latest

Managing Deployments

Restarting Deployments

The restart_app_deployments.sh script restarts all agent deployments to pick up new container images:

restart_app_deployments.sh

#!/bin/bash
kubectl rollout restart deploy pma-agent -n agents
kubectl rollout restart deploy pma-agent-monitoring  -n agents
kubectl rollout restart deploy autonomous-trader-agent -n agents
kubectl rollout restart deploy deployed-general-agent-viewer -n agents
kubectl rollout restart deploy treasury -n agents

Run the script:

bash restart_app_deployments.sh

This script is currently used until CI/CD pipeline integration with GKE is complete. DevOps is working on securing the connection between GitHub and GKE for automated deployments.

Rolling Updates

Kubernetes performs rolling updates automatically when you change the deployment:

# Update image
kubectl set image deployment/pma-agent \
  agent=ghcr.io/gnosis/prediction-market-agent:v1.2.3 \
  -n agents

# Check rollout status
kubectl rollout status deployment/pma-agent -n agents

Scaling

Scale deployments up or down:

# Scale to 3 replicas
kubectl scale deployment/pma-agent --replicas=3 -n agents

# Auto-scale based on CPU
kubectl autoscale deployment/pma-agent \
  --min=1 --max=5 --cpu-percent=80 \
  -n agents

Rollback

Rollback to a previous deployment version:

# View rollout history
kubectl rollout history deployment/pma-agent -n agents

# Rollback to previous version
kubectl rollout undo deployment/pma-agent -n agents

# Rollback to specific revision
kubectl rollout undo deployment/pma-agent --to-revision=3 -n agents

Multiple Agent Deployment

Run multiple agents simultaneously with different configurations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pma-agent-prophet-gpt4o
  namespace: agents
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pma-agent
      agent-type: prophet-gpt4o
  template:
    metadata:
      labels:
        app: pma-agent
        agent-type: prophet-gpt4o
    spec:
      containers:
      - name: agent
        image: ghcr.io/gnosis/prediction-market-agent:main
        env:
        - name: runnable_agent_name
          value: "prophet_gpt4o"
        - name: market_type
          value: "omen"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pma-agent-microchain
  namespace: agents
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pma-agent
      agent-type: microchain
  template:
    metadata:
      labels:
        app: pma-agent
        agent-type: microchain
    spec:
      containers:
      - name: agent
        image: ghcr.io/gnosis/prediction-market-agent:main
        env:
        - name: runnable_agent_name
          value: "microchain"
        - name: market_type
          value: "omen"

Monitoring and Observability

Viewing Logs

# Tail logs from a specific pod
kubectl logs -f <pod-name> -n agents

# Logs from all pods in a deployment
kubectl logs -f deployment/pma-agent -n agents

# Previous container logs (if crashed)
kubectl logs <pod-name> -n agents --previous

# Last 100 lines
kubectl logs <pod-name> -n agents --tail=100

Pod Status

# Get pod status
kubectl get pods -n agents

# Detailed pod information
kubectl describe pod <pod-name> -n agents

# Watch pod status in real-time
kubectl get pods -n agents -w

Resource Usage

# CPU and memory usage
kubectl top pods -n agents

# Node resource usage
kubectl top nodes

Deployment Dashboard

View agent activity on the Dune dashboard:

AI Agents Overview

Track on-chain activity of deployed agents from this repo

Configuration Management

ConfigMaps

Use ConfigMaps for non-sensitive configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-config
  namespace: agents
data:
  market_type: "omen"
  log_level: "INFO"
  free_for_everyone: "1"

Reference in deployment:

envFrom:
- configMapRef:
    name: agent-config

Secrets Management

Integrate with Google Secret Manager:

# Enable Secret Manager API
gcloud services enable secretmanager.googleapis.com

# Create secret
echo -n "your-api-key" | \
  gcloud secrets create openai-api-key --data-file=-

# Grant access to GKE service account
gcloud secrets add-iam-policy-binding openai-api-key \
  --member="serviceAccount:<gke-service-account>" \
  --role="roles/secretmanager.secretAccessor"

Use with Workload Identity or External Secrets Operator for automatic secret injection.

CI/CD Pipeline

Current State

Images are automatically built and pushed to GitHub Container Registry on:

• Pushes to main branch
• Pull requests with “build please” in description

Deployment Process

1. Code Push

   Developer pushes code to GitHub

2. CI Build

   GitHub Actions builds Docker image and pushes to GHCR

3. Manual Deployment

   Currently, deployments are manually triggered using restart_app_deployments.sh

4. Future: Automated CD

   DevOps is working on automating deployment after successful builds

The deployment process will be fully automated once the connection between GitHub Actions and GKE is secured.

High Availability

Multiple Replicas

Run multiple replicas for redundancy:

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

Pod Disruption Budget

Ensure minimum availability during updates:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pma-agent-pdb
  namespace: agents
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: pma-agent

Health Checks

Add liveness and readiness probes:

livenessProbe:
  exec:
    command:
    - python
    - -c
    - "import sys; sys.exit(0)"
  initialDelaySeconds: 30
  periodSeconds: 10
  
readinessProbe:
  exec:
    command:
    - python
    - -c
    - "import sys; sys.exit(0)"
  initialDelaySeconds: 10
  periodSeconds: 5

Cost Optimization

Right-size Resources

Start with smaller resource requests and scale up based on actual usage:

kubectl top pods -n agents

Adjust resource limits based on observed consumption.

Use Preemptible Nodes

For non-critical workloads, use GKE’s preemptible nodes for ~80% cost savings:

nodeSelector:
  cloud.google.com/gke-preemptible: "true"

Horizontal Pod Autoscaling

Scale pods based on actual demand:

kubectl autoscale deployment/pma-agent \
  --min=1 --max=10 --cpu-percent=70 \
  -n agents

Cluster Autoscaling

Enable GKE cluster autoscaling to add/remove nodes based on pod requirements.

Troubleshooting

ImagePullBackOff

Pod can’t pull the Docker image:

# Check image exists
docker pull ghcr.io/gnosis/prediction-market-agent:main

# Verify image pull secret
kubectl get secret ghcr-secret -n agents -o yaml

# Recreate secret if needed
kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=<username> \
  --docker-password=<token> \
  -n agents

CrashLoopBackOff

Pod keeps crashing:

# Check logs
kubectl logs <pod-name> -n agents --previous

# Describe pod for events
kubectl describe pod <pod-name> -n agents

# Common causes:
# - Missing environment variables
# - Invalid configuration
# - Resource constraints

Pending Pods

Pods stuck in Pending state:

# Check pod events
kubectl describe pod <pod-name> -n agents

# Common causes:
# - Insufficient cluster resources
# - Node selector/affinity issues
# - Persistent volume claims not bound

Resource Exhaustion

Out of memory or CPU:

# Check resource usage
kubectl top pods -n agents

# Increase resource limits in deployment
# or scale horizontally

Next Steps

Environment Config

Complete environment variable reference for all agents

Docker Deployment

Learn about the Docker build and container setup

Local Development

Test agents locally before deploying to cloud

Monitoring Dashboard

View live agent performance and activity