Debugging With Skaffold ^beta

skaffold debug acts like skaffold dev, but it configures containers in pods for debugging as required for each container’s runtime technology. The associated debugging ports are exposed and labelled so that they can be port-forwarded to the local machine. IDEs like Google’s Cloud Code extensions use Skaffold’s events to automatically configure debug sessions.

One notable difference from skaffold dev is that debug disables image rebuilding and syncing as it leads to users accidentally terminating debugging sessions by saving file changes. These behaviours can be re-enabled with the --auto-build, --auto-deploy, and --auto-sync flags.

How It works

skaffold debug examines the built artifacts to determine the underlying language runtime technology. Kubernetes manifests that reference these artifacts are transformed on-the-fly to enable the language runtime’s debugging functionality. These transforms add or alter environment variables and entrypoints, and more.

Some language runtimes require additional support files to enable debugging. For these languages, a special set of runtime-specific images are configured as init-containers to populate a shared-volume that is mounted into each of the appropriate containers. These images are hosted at gcr.io/k8s-skaffold/skaffold-debug-support; alternative locations can be specified in Skaffold’s global configuration.

debug makes some other adjustments to simplify the debug experience:

Replica Counts: debug rewrites the replica counts to 1 for deployments, replica sets, and stateful sets. This results in requests being considered one at a time.
Kubernetes Probes: debug changes the timeouts on HTTP-based liveness, readiness, and startup probes to 600 seconds (10 minutes) from the default of 1 second. This change allows probes to be debugged, and avoids negative consequences from blocked probes when the app is already suspended during a debugging session. Failed liveness probes in particular result in the container being terminated and restarted.

The probe timeout value can be set on a per-podspec basis by setting a debug.cloud.google.com/probe/timeouts annotation on the podspec’s metadata with a valid duration (see Go’s time.ParseDuration()). This probe timeout-rewriting can be skipped entirely by using skip. For example:
```
metadata:
  annotations:
    debug.cloud.google.com/probe/timeouts: skip
spec: ...
```

Supported Language Runtimes

Debugging is currently supported for:

Go (runtime ID: go)
NodeJS (runtime ID: nodejs)
Java and JVM languages (runtime ID: jvm)
Python (runtime ID: python)
.NET Core (runtime ID: netcore)

Note that many debuggers may require additional information for the location of source files. We are looking for ways to identify this information and to pass it back if found.

Go

Go-based applications are configured to run under Delve in its headless-server mode. In order to configure your application for debugging, your app must be:

Identified as being Go-based by setting one of the standard Go runtime environment variables in the container, such as GODEBUG, GOGC, GOMAXPROCS, or GOTRACEBACK. GOTRACEBACK=single is the default setting for Go, and GOTRACEBACK=all is a generally useful configuration.
Built with the -gcflags='all=-N -l' options to disable optimizations and inlining. Debugging can be confusing otherwise due to seemingly-random execution jumps from statement reordering and inlining. Skaffold Profiles are a useful option. Note that the golang:NN-alpine container images do not include a C compiler which is required for -gcflags='all=-N -l'.

Note for users of VS Code’s debug adapter for Go: the debug adapter may require configuring both the local and remote source path prefixes via the cwd and remotePath properties. The cwd property should point to the top-level container of your source files and should generally match the artifact’s context directory in the skaffold.yaml. The remotePath path property should be set to the remote source location during compilation. For example, the golang images, which are often used in multi-stage builds, copy the source code to /go. The following remote launch configuration works in this case:

{
  "name": "Skaffold Debug",
  "type": "go",
  "request": "launch",
  "mode": "remote",
  "host": "localhost",
  "port": 56268,
  "cwd": "${workspaceFolder}",
  "remotePath": "/go/"
}

Java and Other JVM Languages

Java/JVM applications are configured to expose the JDWP agent using the JAVA_TOOL_OPTIONS environment variable.
Note that the use of JAVA_TOOL_OPTIONS causes extra debugging output from the JVM on launch.

NodeJS

NodeJS applications are configured to use the Chrome DevTools inspector via the --inspect argument.

Note that the client must first obtain the inspector UUID.

Note

Many applications use NodeJS-based tools as part of their launch, like npm, rather than invoke node directly. These intermediate node instances may interpret the –inspect arguments. Skaffold introduces a node wrapper that only invokes the real node with –inspect if running an application script, and skips scripts located in node_modules. For more details see the associated PR.

Python

Python applications are configured to use ptvsd, a wrapper around pydevd that uses the debug adapter protocol (DAP).

The DAP is supported by Visual Studio Code, Eclipse LSP4e, and other editors and IDEs. DAP is not yet supported by JetBrains IDEs like PyCharm.

.NET Core

.NET Core applications are configured to be deployed along with vsdbg.

In order to configure your application for debugging, your app must be:

Identified as being dotnet-based by having an entrypoint using dotnet cli or one of the following environment variables ASPNETCORE_URLS, DOTNET_RUNNING_IN_CONTAINER, DOTNET_SYSTEM_GLOBALIZATION_INVARIANT.
Built with the --configuration Debug options to disable optimizations.

Note for users of VS Code’s debug adapter for C#: the following configuration can be used to debug a container. It assumes that your code is deployed in /app or /src folder in the container. If that is not the case, the sourceFileMap property should be changed to match the correct folder. processId is usually 1 but might be different if you have an unusual entrypoint. You can also use "${command:pickRemoteProcess}" instead if supported by your base image. (// comments must be stripped.)

{
    "name": "Skaffold Debug",
    "type": "coreclr",
    "request": "attach",
    "processId" : 1, 
    "justMyCode": true, // set to `true` in debug configuration and `false` in release configuration
    "pipeTransport": {
        "pipeProgram": "kubectl",
        "pipeArgs": [
            "exec",
            "-i",
            "<NAME OF YOUR POD>", // name of the pod you debug.
            "--"
        ],
        "pipeCwd": "${workspaceFolder}",
        "debuggerPath": "/dbg/netcore/vsdbg", // location where vsdbg binary installed.
        "quoteArgs": false
    },
    "sourceFileMap": {
        // Change this mapping if your app in not deployed in /src or /app in your docker image
        "/src": "${workspaceFolder}",
        "/app": "${workspaceFolder}"
        // May also be like this, depending of your repository layout
        // "/src": "${workspaceFolder}/src",
        // "/app": "${workspaceFolder}/src/<YOUR PROJECT TO DEBUG>"
    }
}

IDE Support via Events and Metadata

debug provides additional support for IDEs to detect the debuggable containers and to determine appropriate configuration parameters.

Workload Annotations

Each transformed workload object carries a debug.cloud.google.com/config annotation with a JSON object describing the debug configurations for the pod’s containers (linebreaks for readability):

	debug.cloud.google.com/config={
		"<containerName>":{"runtime":"<runtimeId>",...},
		"<containerName>":{"runtime":"<runtimeId>",...},
		}

For example the following annotation indicates that the container named web is a Go application that is being debugged by a headless Delve session on port 56268 (linebreaks for readability):

debug.cloud.google.com/config={
  "web":{
    "artifact":"gcr.io/random/image",
    "runtime":"go",
    "ports":{"dlv":56268},
    "workingDir":"/some/path"}}

artifact is the corresponding artifact’s image name in the skaffold.yaml. runtime is the language runtime detected (one of: go, jvm, nodejs, python). ports is a list of debug ports keyed by the language runtime debugging protocol. workingDir is the working directory (if not an empty string).

API: Events

Each debuggable container being started or stopped raises a debug-container-event through Skaffold’s event mechanism (gRPC, REST).

`/v1/events` stream of `skaffold debug` within `examples/jib`

In this example, we do a skaffold debug, and then kill the deployed pod. The deployment starts a new pod. We get a terminated event for the container for the killed pod.

{
  "result": {
    "timestamp": "2020-02-05T03:27:30.114354Z",
    "event": {
      "debuggingContainerEvent": {
        "status": "Started",
        "podName": "web-f6d56bcc5-6csgs",
        "containerName": "web",
        "namespace": "default",
        "artifact": "skaffold-jib",
        "runtime": "jvm",
        "debugPorts": {
          "jdwp": 5005
        }
      }
    },
    "entry": "Debuggable container started pod/web-f6d56bcc5-6csgs:web (default)"
  }
}

API: State

The API’s state (gRPC, REST) also includes a list of debuggable containers.

The `/v1/state` listing debugging containers

{
  "buildState": {
    "artifacts": {
      "skaffold-jib": "Complete"
    }
  },
  "deployState": {
    "status": "Complete"
  },
  "forwardedPorts": {
    "5005": {
      "localPort": 5005,
      "remotePort": 5005,
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "portName": "jdwp",
      "resourceType": "pod",
      "resourceName": "web-f6d56bcc5-6csgs",
      "address": "127.0.0.1"
    },
    "8080": {
      "localPort": 8080,
      "remotePort": 8080,
      "namespace": "default",
      "resourceType": "service",
      "resourceName": "web",
      "address": "127.0.0.1"
    },
    "8081": {
      "localPort": 8081,
      "remotePort": 8080,
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "resourceType": "pod",
      "resourceName": "web-f6d56bcc5-6csgs",
      "address": "127.0.0.1"
    }
  },
  "statusCheckState": {
    "status": "Not Started"
  },
  "fileSyncState": {
    "status": "Not Started"
  },
  "debuggingContainers": [
    {
      "status": "Started",
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "artifact": "skaffold-jib",
      "runtime": "jvm",
      "debugPorts": {
        "jdwp": 5005
      }
    }
  ]
}

Limitations

skaffold debug has some limitations.

Unsupported Container Entrypoints

skaffold debug requires being able to examine and alter the command-line used in the container entrypoint. This transformation will not work with images that use intermediate launch scripts or binaries.

Supported Deployers

skaffold debug is only supported with the kubectl, kustomize, and helm deployers.

Note

Helm support requires using Helm v3.1.0 or greater.

Deprecated Workload API Objects

skaffold debug does not support deprecated versions of Workload API objects:

extensions/v1beta1 and apps/v1beta1 was deprecated in Kubernetes 1.8 and removed in Kubernetes 1.16.
apps/v1beta2 was deprecated in Kubernetes 1.9 and removed in Kubernetes 1.16.

Applications should transition to the apps/v1 APIs, introduced in Kubernetes 1.9.

Last modified April 27, 2021: release: v1.23.0 (e8f3c65)

Maturity	`beta`
Applicable	`debug`
Not applicable	`dev` `run` `build` `test` `deploy` `render`

Debugging With Skaffold beta

How It works

Supported Language Runtimes

Go

Java and Other JVM Languages

NodeJS

Note

Python

.NET Core

IDE Support via Events and Metadata

Workload Annotations

API: Events

API: State

Limitations

Unsupported Container Entrypoints

Supported Deployers

Note

Deprecated Workload API Objects

Debugging With Skaffold ^beta