Debugging With Skaffold beta
skaffold debug
acts like skaffold dev
, but it configures containers in pods
for debugging as required for each container’s runtime technology.
The associated debugging ports are exposed and labelled so that they can be port-forwarded to the
local machine. IDEs like Google’s Cloud Code extensions use Skaffold’s events
to automatically configure debug sessions.
One notable difference from skaffold dev
is that debug
disables image rebuilding and
syncing as it leads to users accidentally terminating debugging sessions by saving file changes.
These behaviours can be re-enabled with the --auto-build
, --auto-deploy
, and --auto-sync
flags.
How It Works
skaffold debug
examines the built artifacts to determine the underlying language runtime technology.
Kubernetes manifests that reference these artifacts are transformed on-the-fly to enable the
language runtime’s debugging functionality. These transforms add or alter environment variables
and entrypoints, and more.
Some language runtimes require additional support files to enable debugging.
For these languages, a special set of runtime-specific images
are configured as init-containers to populate a shared-volume that is mounted into
each of the appropriate containers. These images are hosted at
gcr.io/k8s-skaffold/skaffold-debug-support
; alternative locations can be
specified in Skaffold’s global configuration.
Supported Language Runtimes
Debugging is currently supported for:
- Go (runtime ID:
go
) - NodeJS (runtime ID:
nodejs
) - Java and JVM languages (runtime ID:
jvm
) - Python (runtime ID:
python
) - .NET Core (runtime ID:
netcore
)
Note that many debuggers may require additional information for the location of source files. We are looking for ways to identify this information and to pass it back if found.
Go
Go-based applications are configured to run under Delve in its headless-server mode. In order to configure your appliction for debugging, your app must be:
- Identified as being Go-based by setting one of the standard Go runtime
environment variables in the container, such as
GODEBUG
,GOGC
,GOMAXPROCS
, orGOTRACEBACK
.GOTRACEBACK=single
is the default setting for Go, andGOTRACEBACK=all
is a generally useful configuration. - Built with the
-gcflags='all=-N -l'
options to disable optimizations and inlining. Debugging can be confusing otherwise due to seemingly-random execution jumps from statement reordering and inlining. Skaffold Profiles are a useful option. Note that thegolang:NN-alpine
container images do not include a C compiler which is required for-gcflags='all=-N -l'
.
Note for users of VS Code’s debug adapter for Go: the debug adapter
may require configuring both the local and remote source path prefixes via the cwd
and remotePath
properties.
The cwd
property should point to the top-level container of your source files and should generally match
the artifact’s context
directory in the skaffold.yaml
. The remotePath
path property should be set to the
remote source location during compilation. For example, the golang
images, which are
often used in multi-stage builds,
copy the source code to /go
. The following
remote launch configuration
works in this case:
{
"name": "Skaffold Debug",
"type": "go",
"request": "launch",
"mode": "remote",
"host": "localhost",
"port": 56268,
"cwd": "${workspaceFolder}",
"remotePath": "/go/"
}
Java and Other JVM Languages
Java/JVM applications are configured to expose the JDWP agent using the JAVA_TOOL_OPTIONS
environment variable.
Note that the use of JAVA_TOOL_OPTIONS
causes extra debugging output from the JVM on launch.
NodeJS
NodeJS applications are configured to use the Chrome DevTools inspector via the --inspect
argument.
Note that the client must first obtain the inspector UUID.
Note
Many applications use NodeJS-based tools as part of their launch, like npm, rather than invoke node directly. These intermediate node instances may interpret the –inspect arguments. Skaffold introduces a node wrapper that only invokes the real node with –inspect if running an application script, and skips scripts located in node_modules. For more details see the associated PR.Python
Python applications are configured to use ptvsd
, a
wrapper around pydevd
that uses the
debug adapter protocol (DAP).
The DAP is supported by Visual Studio Code, Eclipse LSP4e, and other editors and IDEs. DAP is not yet supported by JetBrains IDEs like PyCharm.
.NET Core
.NET Core applications are configured to be deployed along with vsdbg
.
In order to configure your application for debugging, your app must be:
- Identified as being dotnet-based by having an entrypoint using dotnet cli
or one of the following environment variables
ASPNETCORE_URLS
,DOTNET_RUNNING_IN_CONTAINER
,DOTNET_SYSTEM_GLOBALIZATION_INVARIANT
. - Built with the
--configuration Debug
options to disable optimizations.
Note for users of VS Code’s debug adapter for C#:
the following configuration can be used to debug a container. It assumes that your code is deployed
in /app
or /src
folder in the container. If that is not the case, the sourceFileMap
property
should be changed to match the correct folder. processId
is usually 1 but might be different if you
have an unusual entrypoint. You can also use "${command:pickRemoteProcess}"
instead if supported by
your base image. (//
comments must be stripped.)
{
"name": "Skaffold Debug",
"type": "coreclr",
"request": "attach",
"processId" : 1,
"justMyCode": true, // set to `true` in debug configuration and `false` in release configuration
"pipeTransport": {
"pipeProgram": "kubectl",
"pipeArgs": [
"exec",
"-i",
"<NAME OF YOUR POD>", // name of the pod you debug.
"--"
],
"pipeCwd": "${workspaceFolder}",
"debuggerPath": "/dbg/netcore/vsdbg", // location where vsdbg binary installed.
"quoteArgs": false
},
"sourceFileMap": {
// Change this mapping if your app in not deployed in /src or /app in your docker image
"/src": "${workspaceFolder}",
"/app": "${workspaceFolder}"
// May also be like this, depending of your repository layout
// "/src": "${workspaceFolder}/src",
// "/app": "${workspaceFolder}/src/<YOUR PROJECT TO DEBUG>"
}
}
IDE Support via Events and Metadata
debug
provides additional support for IDEs to detect the debuggable containers and to determine
appropriate configuration parameters.
Workload Annotations
Each transformed workload object carries a debug.cloud.google.com/config
annotation with
a JSON object describing the debug configurations for the pod’s containers (linebreaks for readability):
debug.cloud.google.com/config={
"<containerName>":{"runtime":"<runtimeId>",...},
"<containerName>":{"runtime":"<runtimeId>",...},
}
For example the following annotation indicates that the container named web
is a Go application
that is being debugged by a headless Delve session on port 56268
(linebreaks for readability):
debug.cloud.google.com/config={
"web":{
"artifact":"gcr.io/random/image",
"runtime":"go",
"ports":{"dlv":56268},
"workingDir":"/some/path"}}
artifact
is the corresponding artifact’s image name in the skaffold.yaml
.
runtime
is the language runtime detected (one of: go
, jvm
, nodejs
, python
).
ports
is a list of debug ports keyed by the language runtime debugging protocol.
workingDir
is the working directory (if not an empty string).
API: Events
Each debuggable container being started or stopped raises a debug-container-event through Skaffold’s event mechanism (gRPC, REST).
`/v1/events` stream of `skaffold debug` within `examples/jib`
In this example, we do a skaffold debug
, and then kill the deployed pod. The deployment starts a new pod. We get a terminated event for the container for the killed pod.
{
"result": {
"timestamp": "2020-02-05T03:27:30.114354Z",
"event": {
"debuggingContainerEvent": {
"status": "Started",
"podName": "web-f6d56bcc5-6csgs",
"containerName": "web",
"namespace": "default",
"artifact": "skaffold-jib",
"runtime": "jvm",
"debugPorts": {
"jdwp": 5005
}
}
},
"entry": "Debuggable container started pod/web-f6d56bcc5-6csgs:web (default)"
}
}
API: State
The API’s state (gRPC, REST) also includes a list of debuggable containers.
The `/v1/state` listing debugging containers
{
"buildState": {
"artifacts": {
"skaffold-jib": "Complete"
}
},
"deployState": {
"status": "Complete"
},
"forwardedPorts": {
"5005": {
"localPort": 5005,
"remotePort": 5005,
"podName": "web-f6d56bcc5-6csgs",
"containerName": "web",
"namespace": "default",
"portName": "jdwp",
"resourceType": "pod",
"resourceName": "web-f6d56bcc5-6csgs",
"address": "127.0.0.1"
},
"8080": {
"localPort": 8080,
"remotePort": 8080,
"namespace": "default",
"resourceType": "service",
"resourceName": "web",
"address": "127.0.0.1"
},
"8081": {
"localPort": 8081,
"remotePort": 8080,
"podName": "web-f6d56bcc5-6csgs",
"containerName": "web",
"namespace": "default",
"resourceType": "pod",
"resourceName": "web-f6d56bcc5-6csgs",
"address": "127.0.0.1"
}
},
"statusCheckState": {
"status": "Not Started"
},
"fileSyncState": {
"status": "Not Started"
},
"debuggingContainers": [
{
"status": "Started",
"podName": "web-f6d56bcc5-6csgs",
"containerName": "web",
"namespace": "default",
"artifact": "skaffold-jib",
"runtime": "jvm",
"debugPorts": {
"jdwp": 5005
}
}
]
}
Limitations
skaffold debug
has some limitations.
Unsupported Container Entrypoints
skaffold debug
requires being able to examine and alter the
command-line used in the container entrypoint. This transformation
will not work with images that use intermediate launch scripts or
binaries.
Supported Deployers
skaffold debug
is only supported with the kubectl
, kustomize
, and helm
deployers.
Note
Helm support requires using Helm v3.1.0 or greater.Deprecated Workload API Objects
skaffold debug
does not support deprecated versions of Workload API objects:
extensions/v1beta1
andapps/v1beta1
was deprecated in Kubernetes 1.8 and removed in Kubernetes 1.16.apps/v1beta2
was deprecated in Kubernetes 1.9 and removed in Kubernetes 1.16.
Applications should transition to the apps/v1
APIs,
introduced in Kubernetes 1.9.