I've been having an issue that I cannot figure out or even properly debug. When developing locally with "cloud.google.com/go/datastore" on Kubernetes using an in-cluster configuration, I can write to Cloud Datastore just fine. However when I deploy it on my cluster, my programs hangs and never returns once .Put(... is called on my datastore client. I don't get any output whatsoever. I've been able to get rudimentary gdb access to a running process on my cluster but have not been able to figure out what is going wrong or where the code is getting stuck.

I have followed the directions here.

I have tried loading my service account file by these two methods.

client, err := datastore.NewClient(ctx, projectID, option.WithServiceAccountFile("/var/secrets/google/key.json"))
if err != nil {
	log.Fatalf("Failed to create client: %v", err)
}

client, err := datastore.NewClient(ctx, projectID)
if err != nil {
	log.Fatalf("Failed to create client: %v", err)
}

Both work in creating a valid client.

I also tried moving to new nodes with more permissions enabled with:

gcloud --project MY_PROJECT container node-pools create main-pool \
   --cluster my-cluster-us-cntrl1a \
   --zone us-central1-a \
   --enable-autoupgrade \
   --num-nodes 1 --machine-type n1-standard-2 \
   --enable-autoscaling --min-nodes=1 --max-nodes=6 \
   --scopes cloud-platform,datastore

The permissions to my cluster looks like this:

My service account has the role of Cloud Datastore User and Owner for good measure.

What are other things to check for when running on Kubernetes from within the cluster? Is there any good way to debug this to get logs as to what's happening?