Figure 1: The 4 Architectural Pillars of the Gemma Enterprise Agent Platform on Google Cloud
Figure 1: The four foundational pillars I used to take an open-weights Gemma agent from a local prototype to governed enterprise production on Google Cloud.

Most AI tutorials stop at a basic "Hello World" chatbot running on a laptop. However, when engineering teams try to move open-weights models like Google’s Gemma 3 (gemma-3-4b-it, gemma-3-12b-it, gemma-3-27b-it) into real enterprise production, they immediately hit four hard engineering questions:

  1. Grounding: How do I connect Gemma to live enterprise data (like Cloud Firestore) so it doesn't hallucinate?
  2. Serverless GPU Scaling: How do I serve Gemma on GPUs (Cloud Run + vLLM + Cloud Storage FUSE) so it scales to zero when idle instead of burning thousands of dollars a month?
  3. Reliability & Governance: How do I guarantee the agent cannot pass malformed tool parameters or accidentally take down a production service?
  4. Testing & Optimization: How do I unit-test an AI agent's tool calls in CI/CD, and how do I fine-tune a small, fast Gemma 4B model with LoRA on Vertex AI so it performs like a 27B model?

I wrote this article to serve as a complete, end-to-end Enterprise Reference Blueprint that answers all four questions using a single, working project (CloudGuard) verified in Google Cloud Shell (all working code is available in my cloudguard-gemma-agent GitHub repository).

Where This Blueprint Is Useful in Production

  • DevOps, SRE & Automated Incident Response: On-call engineers often waste 20-30 minutes during an outage manually querying logs, searching old postmortems, and finding remediation runbooks. CloudGuard automatically pulls live incident telemetry from Cloud Firestore (get_service_incident), matches symptoms against historical postmortems in its Long-Term Memory Bank (MEM-2026-014), and executes approved mitigations in staging (mitigate_runaway_workload).
  • Cloud FinOps & GPU Quota Governance: A single misconfigured ML batch job or unthrottled retry loop can spin up dozens of GPUs overnight. CloudGuard checks real-time regional GPU utilization (NVIDIA L4 and RTX PRO 6000) and monthly budget burn (check_gpu_quota_and_cost), immediately scaling down runaway staging workloads and calculating hourly dollar savings ($36.12/hr).
  • Regulated Enterprises Requiring Data Sovereignty (Banking, Healthcare, Government, FinOps): Organizations prohibited from sending application telemetry logs, credentials, or infrastructure schemas to external third-party APIs can deploy open-weights Gemma inside their own Google Cloud project behind --no-allow-unauthenticated service and Model Armor guardrails (cloudguard_agent/guardrails.py), where accidentally pasted GCP API keys (AIza...) are redacted before reaching the model.
  • AI Engineering & MLOps CI/CD Pipelines: Instead of relying on manual "vibe checks" when prompts or tools change, my ADK Eval harness (scripts/run_eval.py) scores Tool Trajectory Accuracy (1.00) and Response Factuality (1.00) on every build, while my Vertex AI QLoRA pipeline (scripts/prepare_and_verify_lora.py) lets teams replace a 54 GB VRAM model (Gemma 27B) with an 8.9 GB VRAM model (Gemma 4B + LoRA) running in ~51ms on a single NVIDIA L4 GPU.

Here are the six stages of the production lifecycle I will walk through:

  1. Build: Scaffolding a model-agnostic Google ADK agent in Python that runs against local Gemma weights (via Ollama) during development and seamlessly switches to Cloud Run GPUs or Vertex AI Model Garden in production.
  2. Connect: Authoring native Python function-calling tools backed by Cloud Firestore to inspect live production incidents and regional GPU FinOps quotas.
  3. Scale: Deploying vLLM on Cloud Run Serverless GPUs (NVIDIA L4 / RTX PRO 6000) with Cloud Storage FUSE for zero-download weight streaming, paired with Short-Term Session State and a Long-Term Memory Bank.
  4. Govern: Enforcing deterministic parameter validation with private Cloud Run service configurations, Model Armor parameter formatting (before_model_callback), and deterministic RBAC tool policies (before_tool_callback).
  5. Optimize: Automating CI/CD release gates with ADK Eval (tool_trajectory_avg_score and response_match_score) and OpenTelemetry tracing for Google Cloud Trace.
  6. Specialize: Fine-tuning a compact google/gemma-3-4b-it model using QLoRA on Vertex AI so it matches gemma-3-27b-it tool-calling accuracy (1.00) at 2.8x faster latency and 6x lower GPU memory.

Figure 2: CloudGuard End-to-End Request Flow Architecture
Figure 2: End-to-end request lifecycle for my CloudGuard Incident & FinOps Agent across Cloud Run, vLLM, Cloud Firestore, Model Armor guardrails, and Cloud Trace.

Part 1: Platform Foundations & Scaffolding the First Gemma Agent

Prerequisites & Google Cloud Setup

To follow along with my setup, you will need:

  • A Google Cloud Project with billing enabled.
  • The Google Cloud SDK (gcloud) installed and authenticated.
  • Python 3.10+ and uv (the fast Python package and virtual environment manager I use across this project).
  • Ollama installed locally (ollama pull gemma3:4b or ollama pull gemma3:27b) for zero-cost local prototyping before deploying to Google Cloud GPUs.

First, I set my project and region environment variables and enabled the required Google Cloud APIs:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Next, I configured Application Default Credentials (ADC) so my local Python environment can authenticate with Google Cloud services:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Scaffolding the Project (Local or 1-Click in Google Cloud Shell)

You can either clone and run this project directly inside Google Cloud Shell (which comes pre-authenticated with gcloud and Python in your browser) by clicking Open in Google Cloud Shell, or initialize the workspace locally with uv:

Cloning the CloudGuard Gemma Agent repository directly into Google Cloud Shell
Step 1: Launching the RobinaMirbahar/cloudguard-gemma-agent repository directly inside Google Cloud Shell.

Authorizing Google Cloud Shell to access Google Cloud API credentials
Step 2: Authorizing Google Cloud Shell credentials while the repository clones into the Cloud Shell workspace.

Running the CloudGuard setup and verification commands inside Google Cloud Shell Editor
Step 3: Executing the CloudGuard environment setup, database seeding, and ADK evaluation commands inside the Google Cloud Shell terminal.

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

I structured the repository so that agent orchestration, native tools, memory management, validation callbacks, and evaluation suites are cleanly separated (and runnable offline via a deterministic local simulation harness before touching cloud GPUs):

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Repository structure of cloudguard-gemma-agent on GitHub
Repository structure of RobinaMirbahar/cloudguard-gemma-agent showing the modular separation across agent orchestration, Firestore tools, memory, guardrails, deployment manifests, and evaluation scripts.

In pyproject.toml, I declared google-adk, litellm, google-cloud-firestore, and opentelemetry-sdk:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Then I locked and installed the dependencies:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Decoupling Local vs. Cloud GPU Inference (.env.example)

One of the strongest architectural benefits of Google ADK is its model-agnostic LiteLlm integration. By defining a .env configuration file, I can point my agent to a local Ollama server on localhost:11434 during development and switch to a self-hosted vLLM endpoint on Cloud Run GPUs (or Vertex AI Model Garden) in production without changing a single line of agent orchestration code:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Defining the CloudGuard Agent Persona (cloudguard_agent/agent.py)

When building agents with open-weights models like Gemma, explicit, numbered system instructions are essential. Here is how I defined the agent's identity, tool bindings, and strict operational constraints in cloudguard_agent/agent.py:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Part 2: Native Python Tools & Connecting to Cloud Firestore

Figure 3: Part 2 - ADK Native Function Calling & Cloud Firestore Integration
Figure 3: How Google ADK introspects my Python type hints and docstrings to drive structured Gemma tool calls against Cloud Firestore.

An isolated LLM knows nothing about active Cloud Run services, operational SRE runbooks, or current monthly GPU FinOps burn rates. To bridge that gap, Google ADK automatically converts standard Python functions into JSON Schema tool definitions for Gemma.

My Engineering Rule for Open-Weights Gemma: Compact open-weights models (Gemma 4B, 12B, 27B) rely heavily on explicit Args: descriptions and concrete format examples (such as 'INC-YYYY-NNNN') in Python docstrings to construct valid JSON tool arguments. I treat Python docstrings as first-class prompt engineering.

Implementing the Operational Tools (cloudguard_agent/tools.py)

I built four native Python tools backed by Cloud Firestore (plus a zero-config local JSON schema mirror (USE_LOCAL_MOCK_DB="true") so the entire test suite can be verified locally before provisioning cloud resources):

  • get_service_incident(incident_id: str): Retrieves severity, symptom, GPU type, hourly USD burn rate, and remediation runbook for a specific incident ticket.
  • list_active_alerts(environment: str = "all"): Lists open incidents filtered by production, staging, or all.
  • check_gpu_quota_and_cost(region: str): Checks regional NVIDIA L4 and RTX PRO 6000 quota limits, active usage, and monthly FinOps budget percentage.
  • mitigate_runaway_workload(...): Scales down a runaway Cloud Run service or batch job and records the hourly cost savings. 
export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Seeding the Database (scripts/seed_firestore.py)

When connecting to a live Cloud Firestore instance in Google Cloud, you can provision the database in Native mode:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

I then ran scripts/seed_firestore.py to populate the incidents and quotas collections:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Here is the terminal output from running the database seeder:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Live Google Cloud Shell output showing the seeded incidents and regional GPU FinOps quota collections
Running scripts/seed_firestore.py inside Google Cloud Shell to populate the incidents and quotas collections.

Part 3: Scaling with Cloud Run Serverless GPUs, Vertex AI Model Garden & Dual Memory

Figure 4: Part 3 - Serverless GPU Scaling on Cloud Run & Dual Memory Architecture
Figure 4: Streaming Gemma weights from Cloud Storage FUSE into Cloud Run Serverless GPUs with vLLM, backed by Short-Term Session State and a Long-Term Enterprise Memory Bank.

Moving an open-weights Gemma model from a developer laptop into enterprise production on Google Cloud offers two primary deployment patterns: Cloud Run Serverless GPUs with vLLM (for scale-to-zero serverless containers) or Vertex AI Model Garden (for one-call managed GPU endpoints).

Option A: Serverless GPU Serving on Cloud Run with vLLM & Cloud Storage FUSE

Instead of baking multi-gigabyte model weights into a container image - or downloading them from Hugging Face on every cold start - I stage the Gemma weights in a regional Google Cloud Storage (GCS) bucket and mount that bucket at /mnt/models using Cloud Storage FUSE.

VRAM Sizing Note on Google Cloud Run GPUs: A single NVIDIA L4 GPU on Cloud Run provides 24 GB of VRAM, which easily fits google/gemma-3-4b-it (~8.4 GB in bfloat16), google/gemma-3-12b-it, or a 4-bit AWQ quantized google/gemma-3-27b-it. To run unquantized bfloat16 google/gemma-3-27b-it (~54 GB VRAM) on a single serverless instance, use --gpu-type=nvidia-rtx-pro-6000 on Cloud Run or deploy on Vertex AI Model Garden.

Here is my deploy/Dockerfile.vllm configuration:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

And here is the gcloud run deploy command from deploy/deploy_cloud_run_gpu.sh mounting the GCS FUSE volume:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Option B: Managed Deployment via Vertex AI Model Garden

Alternatively, if you prefer a managed endpoint provisioned directly from Vertex AI Model Garden, you can deploy google/gemma3@gemma-3-4b-it or google/gemma3@gemma-3-27b-it programmatically using the Vertex AI Python SDK:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Dual-Tier Memory: Short-Term Session State vs. Long-Term Memory Bank

In cloudguard_agent/memory.py, I equipped the agent with two distinct memory layers:

  1. Short-Term Conversational Session (AgentSessionState): Tracks the active incident ID (active_incident_id) within the current troubleshooting thread. When an engineer inspects INC-2026-0891 in Turn 2 and says "Go ahead and mitigate that runaway staging workload" in Turn 4, the agent resolves the reference automatically.
  2. Long-Term Organizational Memory (EnterpriseMemoryBank): Stores verified postmortem learnings across all sessions. When INC-2026-0891 triggers on ml-batch-indexer-dev, the agent automatically searches historical postmortems and retrieves MEM-2026-014.

Here is the execution output from running my multi-turn simulation harness (uv run python scripts/simulate_parts.py):

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Live Google Cloud Shell execution of the 4-turn incident triage, FinOps GPU quota check, and session-aware mitigation workflow
Live execution of scripts/simulate_parts.py in Google Cloud Shell showing multi-turn tool calling, Long-Term Memory Bank matching (MEM-2026-014), and Short-Term Session State mitigation.

Part 4: Deterministic Input Formatting & Tool Parameter Validation

Figure 5: Part 4 - Deterministic Input & Parameter Validation Callbacks
Figure 5: Three sequential validation gates I implemented to protect the CloudGuard agent: Cloud Run service routing, Before-Model parameter formatting & injection blocking, and Before-Tool RBAC enforcement.

Giving an AI agent write-access to infrastructure requires deterministic validation boundaries. Telling an LLM "Never scale down production" inside a system prompt is a probabilistic guideline - not an enterprise reliability control.

I implemented defense-in-depth across three sequential gates:

  • Gate 1: Service-to-Service Cloud Run Routing: The gemma-vllm-service is configured with --no-allow-unauthenticated. Only the ADK orchestrator's service account (cloudguard-agent-sa) holds roles/run.invoker permission.
  • Gate 2: Before-Model Input Sanitization (inspect_and_sanitize_prompt): Modeled on Google Cloud Model Armor, this interceptor scans every incoming prompt for adversarial jailbreak signatures and automatically redacts live credentials (such as AIza... GCP API keys or ya29... bearer tokens) before they reach the Gemma model or telemetry logs.
  • Gate 3: Before-Tool Deterministic RBAC (validate_tool_invocation): Intercepts every mitigate_runaway_workload call before Python executes it, blocking any production mutation unless the caller holds sre-incident-commander privileges (POL-PROD-MUTATION-001).

Guardrail Implementation (cloudguard_agent/guardrails.py) 

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Validation Verification Output

Here is the audit trace from testing invalid parameter formatting, unauthorized production scaling, and prompt injection against my guardrails:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Live Google Cloud Shell execution of the Part 4 Model Armor input sanitization and RBAC tool policy guardrails
Live execution of scripts/simulate_parts.py in Google Cloud Shell verifying live API key redaction ([REDACTED_GCP_API_KEY]), RBAC production scaling denial (POL-PROD-MUTATION-001), and Model Armor prompt injection blocking.

Part 5: Observability & Deterministic Evaluation (ADK Eval)

Figure 6: Part 5 - Deterministic CI/CD Evaluation (ADK Eval) & OpenTelemetry Tracing
Figure 6: Automated CI/CD evaluation pipeline scoring both Tool Trajectory Accuracy and Response Factuality while exporting OpenTelemetry spans to Google Cloud Trace.

Manual "vibe checks" in a chat UI cannot prevent regressions when prompts, tools, or model weights change. Following ADK Eval best practices, I built scripts/run_eval.py to evaluate the agent against a golden dataset (tests/eval_dataset.json) across two quantitative metrics:

  • Tool Trajectory Score (tool_trajectory_avg_score): Verifies that the agent invoked the exact expected sequence of tools with the exact expected JSON arguments (1.00 = 100% match).
  • Response Factuality Score (response_match_score): Verifies that required domain identifiers (Incident IDs, runbook codes, dollar savings, or RBAC policy codes) appear in the synthesized response.

Running uv run python scripts/run_eval.py produces the following evaluation table and Google Cloud Trace OpenTelemetry span:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Live Google Cloud Shell execution of the ADK Deterministic Evaluation Suite (5/5 Passed) and OpenTelemetry Cloud Trace span export
Executing scripts/run_eval.py in Google Cloud Shell: 5/5 golden test cases passing with a 1.00 Tool Trajectory score and exported OpenTelemetry `gen_ai.` span.*

Part 6: Fine-Tuning Gemma with QLoRA on Vertex AI for Domain Tool Calling

Figure 7: Part 6 - Fine-Tuning Gemma with QLoRA on Vertex AI for Domain Tool Calling
Figure 7: End-to-end domain specialization lifecycle - converting verified production traces into a Gemma SFT dataset, training a QLoRA adapter on Vertex AI, and hot-loading it into Cloud Run vLLM.

While google/gemma-3-27b-it achieves a 1.00 tool-trajectory score out of the box, unquantized bfloat16 weights require ~54 GB of VRAM. To run cleanly on a single 24 GB NVIDIA L4 GPU on Cloud Run with plenty of room for concurrent KV-cache requests, I wanted to specialize the compact google/gemma-3-4b-it model (which uses only 8.9 GB of VRAM and responds in ~51ms).

Using cloudguard_agent/finetuning.py (uv run python scripts/prepare_and_verify_lora.py), I exported verified multi-turn tool trajectories into Gemma's official <start_of_turn> SFT format (data/cloudguard_gemma_sft_train.jsonl) and generated the Vertex AI Custom Training (NVIDIA_L4 QLoRA r=16, lora_alpha=32) specification:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Live Google Cloud Shell execution showing the exported Gemma SFT tool-calling training example
Live execution in Google Cloud Shell exporting golden SFT trajectories formatted with Gemma's <start_of_turn> and tool_code tokens.

Once Vertex AI writes the LoRA adapter weights to gs://${PROJECT_ID}-gemma-weights/adapters/cloudguard-sre-lora-v1, I hot-load the adapter into vLLM on Cloud Run (--enable-lora --lora-modules cloudguard-sre-lora-v1=/mnt/models/adapters/cloudguard-sre-lora-v1).

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Live Google Cloud Shell execution of the Vertex AI QLoRA CustomJob specification and post-tuning ADK Eval benchmark
Live execution inside Google Cloud Shell generating the Vertex AI NVIDIA_L4 QLoRA job specification and verifying the post-tuning ADK Eval benchmark.

With domain QLoRA fine-tuning, the compact Gemma 4B model matches the 1.00 Tool Trajectory score of the 27B model while cutting latency from 144.1ms down to 51.4ms and fitting effortlessly on a single Cloud Run NVIDIA L4 GPU.

Cleaning Up Cloud Resources (FinOps Hygiene)

If you deploy the Cloud Run GPU services and GCS bucket in your own Google Cloud project, remember to tear them down when finished experimenting:

export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"

gcloud run services delete cloudguard-adk-agent --region=${REGION} --quiet
gcloud run services delete gemma-vllm-service --region=${REGION} --quiet
gcloud storage rm --recursive gs://${PROJECT_ID}-gemma-weights
Enter fullscreen mode Exit fullscreen mode

Key Takeaways

  • Open-Weights Control + Cloud-Native Speed: Pairing Gemma 3 with Google ADK gives me complete model weight ownership and VPC data residency without sacrificing developer velocity.
  • Serverless GPU Economics: Combining vLLM, Cloud Run Serverless GPUs (nvidia-l4), and Cloud Storage FUSE eliminates both idle GPU VM waste and slow container cold starts.
  • Deterministic Governance & Evaluation: Using before_model_callback, before_tool_callback, and ADK Eval turns probabilistic LLM agents into auditable, production-grade software systems.

Companion GitHub Repository

All source code, Dockerfiles, Cloud Run Serverless GPU deployment manifests, golden evaluation datasets, and Vertex AI QLoRA fine-tuning scripts from my tutorial are open-source and available on GitHub:

Official Google Cloud Documentation & References