Debugging With Skaffold

skaffold debug acts like skaffold dev, but it configures containers in pods for debugging as required for each container’s runtime technology. The associated debugging ports are exposed and labelled so that they can be port-forwarded to the local machine.

To debug in an IDE that uses Skaffold events to automatically configure debug sessions, you can use Google Cloud Code for VS Code, Cloud Code for IntelliJ, or, for an IDE in your browser, Cloud Code for Cloud Shell.

One notable difference from skaffold dev is that debug disables image rebuilding and syncing as it leads to users accidentally terminating debugging sessions by saving file changes. These behaviours can be re-enabled with the --auto-build, --auto-deploy, and --auto-sync flags.

Debugging is currently supported for five language runtimes.

• Go 1.13+ (runtime ID: go) using Delve
• NodeJS (runtime ID: nodejs) using the NodeJS Inspector (Chrome DevTools)
• Java and JVM languages (runtime ID: jvm) using JDWP
• Python 3.5+ (runtime ID: python) using debugpy (Debug Adapter Protocol) or pydevd
• .NET Core (runtime ID: netcore) using vsdbg

How It works

Enabling debugging has two phases:

1. Configuring: Skaffold automatically examines each built container image and attempts to recognize the underlying language runtime. Container images can be explicitly configured too.
2. Monitoring: Skaffold watches the cluster to detect when debuggable containers start execution.

Configuring container images for debugging

skaffold debug examines the built artifacts to determine the underlying language runtime technology. Kubernetes manifests that reference these artifacts are transformed on-the-fly to enable the language runtime’s debugging functionality. These transforms add or alter environment variables and entrypoints, and more.

Some language runtimes require additional support files to enable debugging. For these languages, a special set of runtime-specific images are configured as init-containers to populate a shared-volume that is mounted into each of the appropriate containers. These images are hosted at gcr.io/k8s-skaffold/skaffold-debug-support; alternative locations can be specified in Skaffold’s global configuration.

For images that are successfully recognized, Skaffold adds a debug.cloud.google.com/config annotation to the corresponding Kubernetes pod-spec that encode the debugging parameters.

Monitoring for debuggable containers

Once the application is deployed, debug monitors the cluster looking for debuggable pods with a debug.cloud.google.com/config annotation. For each new debuggable pod, Skaffold emits an event that can be used by tools like IDEs to establish a debug session.

Additional changes

debug makes some other adjustments to simplify the debug experience:

• Replica Counts: debug rewrites the replica counts to 1 for deployments, replica sets, and stateful sets. This results in requests being serialized so that one request is processed at a time.

• Kubernetes Probes: debug changes the timeouts on HTTP-based liveness, readiness, and startup probes to 600 seconds (10 minutes) from the default of 1 second. This change allows probes to be debugged, and avoids negative consequences from blocked probes when the app is already suspended during a debugging session. Failed liveness probes in particular result in the container being terminated and restarted.

  The probe timeout value can be set on a per-podspec basis by setting a debug.cloud.google.com/probe/timeouts annotation on the podspec’s metadata with a valid duration (see Go’s time.ParseDuration()). This probe timeout-rewriting can be skipped entirely by using skip. For example:

  metadata:
    annotations:
      debug.cloud.google.com/probe/timeouts: skip
  spec: ...
  

Supported Language Runtimes

This section describes how debug recognizes the language runtime used in a container image, and how the container image is configured for debugging.

Note that many debuggers may require additional information for the location of source files. We are looking for ways to identify this information and to pass it back if found.

Go (runtime: go, protocols: dlv)

Go-based applications are configured to run under Delve in its headless-server mode.

Go-based container images are recognized by:

• the presence of one of the standard Go runtime environment variables: GODEBUG, GOGC, GOMAXPROCS, or GOTRACEBACK, or
• the presence of the KO_DATA_PATH environment variable in container images built by ko, or
• is launching using dlv.

Unless you built your container image using ko, set one of the standard Go environment variables to allow Skaffold to detect your container as Go. GOTRACEBACK=single is the default setting for Go, so you can add this environment variable without changing behavior. Alternatively, GOTRACEBACK=all is a useful setting in many situtions.

On recognizing a Go-based container image, debug rewrites the container image’s entrypoint to invoke your application using dlv:

dlv exec --headless --continue --accept-multiclient --listen=:56268 --api-version=2 <app> -- <args> ...

Your application should be built with the -gcflags='all=-N -l' flag to disable optimizations and inlining. If you do not add this flag, debugging can becoming confusing due to seemingly-random execution jumps that happen due to statement reordering and inlining. Skaffold configures Docker builds with a SKAFFOLD_GO_GCFLAGS build argument that you should use in your Dockerfile:

FROM golang
ENV GOTRACEBACK=all
COPY . .
ARG SKAFFOLD_GO_GCFLAGS
RUN go build -gcflags="${SKAFFOLD_GO_GCFLAGS}" -o /app .

Note that the Alpine Linux-based golang container images do not include a C compiler which is required to build with -gcflags='all=-N -l'.

The ko builder adds -gcflags='all=-N -l' automatically when you use skaffold debug.

We recommend that you do not add the -trimpath flag when debugging, as this flag removes information from the resulting binary that aids debugging.

Skaffold debug using the VS Code Go extension

If you use the debug functionality of the VS Code Go extension to connect to dlv, you may need to configure local and remote source paths, via either the cwd and remotePath properties, or the substitutePath array property (reference).

The cwd or from properties should be the absolute path of the top-level directory of your source files and should generally match the artifact’s context directory in skaffold.yaml. You can represent this using the ${workspaceFolder} variable.

The remotePath or to properties should be set to the absolute path of the source location during compilation. For example, the golang images, which are often used in multi-stage builds, have a default working directory of /go. The following configuration works in this case:

{
    "name": "Skaffold Debug",
    "type": "go",
    "request": "attach",
    "mode": "remote",
    "host": "localhost",
    "port": 56268,
    "cwd": "${workspaceFolder}",
    "remotePath": "/go"
}

You can use the substitutePath property instead of cwd and remotePath:

{
    "name": "Skaffold Debug",
    "type": "go",
    "request": "attach",
    "mode": "remote",
    "host": "localhost",
    "port": 56268,
    "substitutePath": [
        {
            "from": "${workspaceFolder}",
            "to": "/go",
        },
    ],
}

If you use the ko local builder, the source path during compilation matches the source path in your IDE. In this case, the value for remotePath will match cwd. If the values of both cwd and remotePath are ${workspaceFolder}, you can omit these properties.

Java and Other JVM Languages (runtime: jvm, protocols: jdwp)

Java/JVM applications are configured to expose the JDWP agent using the JAVA_TOOL_OPTIONS environment variable. Note that the use of JAVA_TOOL_OPTIONS causes extra debugging output from the JVM on launch.

JVM application are recognized by:

• the presence of a JAVA_VERSION or JAVA_TOOL_OPTIONS environment variable, or
• the container command-line invokes java.

On recognizing a JVM-based container image, debug rewrites the container image’s environment to set:

JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y

NodeJS (runtime: nodejs, protocols: devtools)

NodeJS applications are configured to use the Chrome DevTools inspector via the --inspect argument.

NodeJS images are recognized by:

• the presence of a NODE_VERSION, NODEJS_VERSION, or NODE_ENV environment variable, or
• the container command-line invokes node or npm.

On recognizing a NodeJS-based container image, debug rewrites the container image’s entrypoint to invoke your application with --inspect:

node --inspect=9229 <app.js>

Note that a debugging client must first obtain the inspector UUID.

Python (runtime: python, protocols: dap or pydevd)

Python applications are configured to use either debugpy, or wrapper around pydevd. debugpy uses the debug adapter protocol (DAP) which is supported by Visual Studio Code, Eclipse LSP4e, and other editors and IDEs.

Python application are recognized by:

• the presence of a standard Python environment variable: PYTHON_VERSION, PYTHONVERBOSE, PYTHONINSPECT, PYTHONOPTIMIZE, PYTHONUSERSITE, PYTHONUNBUFFERED, PYTHONPATH, PYTHONUSERBASE, PYTHONWARNINGS, PYTHONHOME, PYTHONCASEOK, PYTHONIOENCODING, PYTHONHASHSEED, PYTHONDONTWRITEBYTECODE, or
• the container command-line invokes python, python2, or python3.

On recognizing a Python-based container image, debug rewrites the container image’s entrypoint to invoke Python using either the pydevd or debugpy modules:

python -m debugpy --listen 5678 <app>

or

python -m pydevd --server --port 5678 <app.py>

.NET Core (runtime: dotnet, protocols: vsdbg)

.NET Core applications are configured to be deployed along with vsdbg for VS Code.

.NET Core application are recognized by:

• the presence of a standard .NET environment variable: ASPNETCORE_URLS, DOTNET_RUNNING_IN_CONTAINER, DOTNET_SYSTEM_GLOBALIZATION_INVARIANT, or
• the container command-line invokes the dotnet cli

Furthermore, your app must be built with the --configuration Debug options to disable optimizations.

{
    "name": "Skaffold Debug",
    "type": "coreclr",
    "request": "attach",
    "processId" : 1,
    "justMyCode": true, // set to `true` in debug configuration and `false` in release configuration
    "pipeTransport": {
        "pipeProgram": "kubectl",
        "pipeArgs": [
            "exec",
            "-i",
            "<NAME OF YOUR POD>", // name of the pod you debug.
            "--"
        ],
        "pipeCwd": "${workspaceFolder}",
        "debuggerPath": "/dbg/netcore/vsdbg", // location where vsdbg binary installed.
        "quoteArgs": false
    },
    "sourceFileMap": {
        // Change this mapping if your app in not deployed in /src or /app in your docker image
        "/src": "${workspaceFolder}",
        "/app": "${workspaceFolder}"
        // May also be like this, depending of your repository layout
        // "/src": "${workspaceFolder}/src",
        // "/app": "${workspaceFolder}/src/<YOUR PROJECT TO DEBUG>"
    }
}

Troubleshooting

My container is not being made debuggable?

Was this image built by Skaffold? debug only works for images that were built by Skaffold so as to avoid affecting system- or infrastructure-level containers such as proxy sidecars.

Was Skaffold able to recognize the image? debug emits a warning when it is unable to configure an image for debugging:

WARN[0005] Image "image-name" not configured for debugging: unable to determine runtime for "image-name"

See the language runtime section details on how container images are recognized.

Can images be debugged without the runtime support images?

The special runtime-support images are provided as a convenience for automatic configuration. You can manually configure your images for debugging by:

1. Configure your container image to install and invoke the appropriate debugger.
2. Add a debug.cloud.google.com/config workload annotation on the pod-spec to describe the debug configuration of each container image in the pod, as described in Workload Annotations.

Limitations

skaffold debug has some limitations.

Unsupported Container Entrypoints

skaffold debug requires being able to examine and alter the command-line used in the container entrypoint. This transformation will not work with images that use intermediate launch scripts or binaries.

Supported Deployers

skaffold debug is only supported with the kubectl, kustomize, and helm deployers.

Deprecated Workload API Objects

skaffold debug does not support deprecated versions of Workload API objects:

• extensions/v1beta1 and apps/v1beta1 was deprecated in Kubernetes 1.8 and removed in Kubernetes 1.16.
• apps/v1beta2 was deprecated in Kubernetes 1.9 and removed in Kubernetes 1.16.

Applications should transition to the apps/v1 APIs, introduced in Kubernetes 1.9.

Appendix: IDE Support via Events and Metadata

debug provides additional support for IDEs to detect the debuggable containers and to determine appropriate configuration parameters.

Workload Annotations

Each transformed workload object carries a debug.cloud.google.com/config annotation with a JSON object describing the debug configurations for the pod’s containers (linebreaks for readability):

	debug.cloud.google.com/config={
		"<containerName>":{"runtime":"<runtimeId>",...},
		"<containerName>":{"runtime":"<runtimeId>",...},
		}

For example the following annotation indicates that the container named web is a Go application that is being debugged by a headless Delve session on port 56268 (linebreaks for readability):

debug.cloud.google.com/config={
  "web":{
    "artifact":"gcr.io/random/image",
    "runtime":"go",
    "ports":{"dlv":56268},
    "workingDir":"/some/path"}}

artifact is the corresponding artifact’s image name in the skaffold.yaml. runtime is the language runtime detected (one of: go, jvm, nodejs, python). ports is a list of debug ports keyed by the language runtime debugging protocol. workingDir is the working directory (if not an empty string).

API: Events

Each debuggable container being started or stopped raises a debug-container-event through Skaffold’s event mechanism (gRPC, REST).

{
  "result": {
    "timestamp": "2020-02-05T03:27:30.114354Z",
    "event": {
      "debuggingContainerEvent": {
        "status": "Started",
        "podName": "web-f6d56bcc5-6csgs",
        "containerName": "web",
        "namespace": "default",
        "artifact": "skaffold-jib",
        "runtime": "jvm",
        "debugPorts": {
          "jdwp": 5005
        }
      }
    },
    "entry": "Debuggable container started pod/web-f6d56bcc5-6csgs:web (default)"
  }
}

API: State

The API’s state (gRPC, REST) also includes a list of debuggable containers.

{
  "buildState": {
    "artifacts": {
      "skaffold-jib": "Complete"
    }
  },
  "deployState": {
    "status": "Complete"
  },
  "forwardedPorts": {
    "5005": {
      "localPort": 5005,
      "remotePort": 5005,
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "portName": "jdwp",
      "resourceType": "pod",
      "resourceName": "web-f6d56bcc5-6csgs",
      "address": "127.0.0.1"
    },
    "8080": {
      "localPort": 8080,
      "remotePort": 8080,
      "namespace": "default",
      "resourceType": "service",
      "resourceName": "web",
      "address": "127.0.0.1"
    },
    "8081": {
      "localPort": 8081,
      "remotePort": 8080,
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "resourceType": "pod",
      "resourceName": "web-f6d56bcc5-6csgs",
      "address": "127.0.0.1"
    }
  },
  "statusCheckState": {
    "status": "Not Started"
  },
  "fileSyncState": {
    "status": "Not Started"
  },
  "debuggingContainers": [
    {
      "status": "Started",
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "artifact": "skaffold-jib",
      "runtime": "jvm",
      "debugPorts": {
        "jdwp": 5005
      }
    }
  ]
}