Step 7. Check that it works

This page shows you how to check that an installation works. There are two checks. Health asks every server, live, what its containers are doing now. Diagnostics collects one file that you can read later or send to another team.

Before you start

This section lists what must be ready before you check.

  • The installation is deployed. See Step 6. Deploy.

  • Deployment Manager can reach the servers over SSH. Both checks open an SSH session.

For the product words on this page, see the glossary.

Steps

This section shows how to check an installation with Health.

  1. Open the Installations tab.

  2. Find the card of your installation. Press Health. The dialog Health · (installation name) opens.

  3. Read the verdict at the top. It says if anything is wrong.

  4. Read the list of servers. A broken server is listed first.

  5. Read each container row. Each row shows one word, such as Up or Unhealthy.

  6. Optional: under Entry points, press Check reachability. This asks each address if it answers. See Checking those addresses is its own action.

  7. To ask again, press ↻ Refresh.

To collect a file for a ticket or another team, use Diagnostics:

  1. On the card, press Diagnostics. The button shows Collecting…​.

  2. Wait. The tool collects from every server and saves one file to your downloads. A message names the file.

What you see: Health

This section explains the Health check. Health opens an SSH session to every server bound to the installation, all at the same time. It asks Docker Compose for every container of the installation. Health is fast. It connects only when you press it. Nothing is kept between visits. If you close the dialog and open it again, it asks again.

It answers four questions, in this order:

  1. Is anything wrong? A verdict comes first, before any details.

  2. Which machine? The broken server is listed first.

  3. Which container? Each row shows one state. You do not need to compare three columns.

  4. How long has it been that way? Every reading shows its time.

This is the order a person asks in after a customer reports a problem. The time matters. A container list shows a server at one moment. Without a time, it looks live for as long as the dialog is open.

Per-server reachability comes first. A server that could not be reached is a different result from a server that answered with an empty container list. A container listing that ran and failed is also its own state. It is not the same as "no containers here". The dialog keeps these apart. One unreachable server does not hide the results of the others.

What a green verdict does and does not mean. A container counts as good when it is running and one of two things is true. It declares no health check, or it reports healthy. So for a service with no health check, a green row proves only that its process has not stopped. More important: Health reports what is there, not what is missing. It lists the containers that exist. It does not compare them with the containers the topology places on that server. So "All healthy" can mean "the four containers that exist are healthy" when six were expected. When this matters, compare with the service list on the deploy panel. An installation with no containers is never reported as good.

Health looks for the installation in more than one directory. A bundle carried to the site by hand is not always in the directory a deploy from here would use. A server may clearly be running but show an empty Health result. This usually means the install is in a directory that was never recorded.

Per-container state uses these words:

Word Meaning

Up

Running. Healthy, or it has no health check.

Starting

Normal. Still starting.

Unhealthy

Running, but its health probe is failing.

Exited

The container stopped.

Not created

The container does not exist yet.

Restarting

Docker is restarting it. Other Docker states are shown the same way, with a capital letter.

Docker’s own status line can separate a stable service from one that keeps crashing. For example, Up 3 hours (healthy) compared with Restarting (1) 12 seconds ago. The dialog shows that line. It does not reduce both to running. The image tag is also shown. It answers a different question: is the version this tool thinks it deployed the one actually on the machine?

A healthy container is not the same as a service that is ready

This section explains a limit of the health check. Docker’s healthcheck may answer while the service behind it is still not ready for traffic. A health endpoint can start and answer well before the service can serve a customer. So anything that starts as soon as a dependency reports healthy can start too early. This is one reason to run the entry-point check below together with the container list, not instead of it. That check sends a real request to the address a customer would use.

Entry points: the addresses an installation is reached on

This section explains the entry points. You find them in the Health dialog. An entry point is an address a customer or a support engineer would put in a browser or a script. It is not a port that is reachable only inside the system.

They are calculated, not typed in. The address comes from four facts:

  • The topology says which service runs on which host.

  • The compose fragment says what that service publishes.

  • The installation’s own variable values give the port.

  • The service’s contract says which path answers a health check.

Nobody maintains the address by hand. Three rules keep a calculated address correct:

  • A container port that is not published to the host is not an entry point. Only what the compose file actually maps out can be reached.

  • A port that uses only UDP is not an entry point either. The check uses HTTP.

  • A service whose healthcheck is not HTTP gives no address. An example is a database checked with its own client tool. There is no URL to open for it.

Each address shows where it came from. In the future, an owning team may declare an entry directly. That entry will be marked and used before one this tool calculated. Today, everything shown is calculated.

Two kinds of row. This applies when an installation’s servers are behind the customer’s own load balancer. There is one row per address the balancer answers on. It names the group of servers behind the balancer. There is also one row per server behind it, as usual. The balancer row is listed first, because that is the address given to the customer. But it never replaces the per-server rows. The balancer can answer while a server behind it is down. Only the per-server rows show which server, if any, is down.

Checking those addresses is its own action

This section explains the address check. The tool builds the list for free when Health opens. Asking each address if it answers is a separate action. It runs only when you press Check reachability. The check runs from where Deployment Manager itself is hosted. About half of customer sites cannot be reached from there. So there are three answers, not two:

Answer Meaning

Reachable

The address answered with some HTTP response. See below for what "some" means.

No answer

One of these: the request timed out, the port refused the connection, the name did not resolve, or the TLS handshake failed. The panel says which one in a sentence. It never shows raw text from the connection library.

Not checked

The button has not been pressed for this address. This is the answer for a site that cannot be reached from here. It is a normal, expected result. It is not a fault.

Any HTTP status counts as reached, including a 401 or 403. Those get their own label, Needs sign-in. The port is published, the firewall allows it, and the service replied. Whether the service is healthy is a separate question. The container list above answers it. Deployment Manager never signs in to make this check pass. Signing in would mean one of two things. Either this tool stores a customer’s production credential. Or it guesses a credential against their production service and fills their logs with failed logins. This check does neither, even for a green row.

This check does not verify certificates. A customer’s self-signed certificate must not show as a service that is down.

Diagnostics

This section explains Diagnostics. Diagnostics collects one file for the whole installation, with a section for each server. It uses the same kind of SSH session as Health. You can read it later. You can attach it to a ticket, or upload it back into Deployment Manager.

It collects, per server:

  • What every container is doing right now.

  • The last 300 log lines of each container.

  • The owner and permissions of every configuration file.

  • The docker versions in use.

  • Every setting in .env, with its value. A secret is shown as [MASKED], or as [EMPTY — no value was set] when it has no value.

Secrets are masked, not left out. Before the file is written, the value of every secret is replaced with [MASKED]. This includes a value a container printed into its own log by mistake. Whenever the tool can detect it, a password does not leave the customer’s site in this file.

When to use Diagnostics instead of Health: Health is fast. It answers "is it up right now" for a machine you can watch live. Collect Diagnostics when someone else needs to look. For example, you attach it to a ticket, give it to another team, or read it later. A live dialog cannot do any of these.

Good to know

This section explains two checks that run behind the scenes: the readiness checks before a deploy, and the probes after a failure.

The readiness checks, before any container starts

This section explains the readiness checks. Every deploy runs a set of checks on each server after the bundle is copied, but before any container starts or stops. This order is the key point. A readiness refusal means no running service on that machine was changed. So there is nothing to roll back. Fix what the check named and press Deploy again.

There are exactly four checks. A refusal always names one of them. There is no fifth cause behind a general "readiness failed".

Check What it proves What it does not prove

Open-file and process limits

The account the deploy connects as has limits at or above the minimums the installation needs.

What the containers will run with. Containers get their limits from the container daemon. If you raise only the login session’s limit, this check passes. But the services keep the old limit and fail later under load. So the offered repair has two named options: raise it in both places, or for login sessions only. The disruptive option is not selected by default, on purpose. Applying it restarts every container on that machine.

Name resolution

Every hostname the topology declares resolves on that server.

That it resolves inside a container. A container can resolve a name itself. The machine’s own hosts file is not the container’s. A fix made there can look correct and change nothing. A failure here usually explains the rest of the report too. The reachability check below uses the same names. Fix this first, then read the rest again.

Ports free

Nothing else is using a port this installation publishes.

Nothing more. The check tells apart three users of a port, and only one passes. A container of this installation passes, because a re-install replaces it. A container of another installation is refused, and the message names the container. In the past, "held by Docker" was treated as fine in this case, and the install then failed on the port seconds later. A process outside Docker is refused. A repair is offered only when the process can be identified.

Dependencies reachable

A TCP connection to each declared dependency succeeded within a few seconds.

Anything about credentials, schemas, or whether the right service is answering. A green result here does not rule out an authentication failure at deploy time. That is a completely different failure.

Three facts about how to read the result:

  • One line per refusing check, not per condition. A limits check can refuse on two different limits and still give one line. The number of lines is the number of failing checks.

  • A missing tool turns a check into a warning, not a failure. A minimal server image may lack the tools the port or reachability checks need. Then the check says so and the deploy does not stop. The summary counts warnings separately and asks you to read them.

  • On a multi-wave deploy the machine-level checks run once, at that server’s first visit. A later visit says the checks were done at the first one. This is normal. No check was skipped.

Only two failures can be repaired automatically: a low limit, and a port used by a process that can be identified. If no repair button appears, no safe automatic repair exists. It is not a missing feature. Stopping somebody else’s service is not this tool’s job. When a repair runs, it first checks the server again. If anything changed since the screen was shown, it refuses and changes nothing. It asks a process to stop before it forces it to stop.

Probes: the small set of questions the panel can ask a server

This section explains probes. When an install fails, the panel can ask the server a few specific questions. Nobody gets a shell. The set is fixed. There are six probes, and no option adds more.

Probe The question it answers Reads or writes

What came up on this server

Every container of this installation and what it is doing. The Health dialog uses the same reading, so the two screens cannot disagree.

Reads

Why one container is not up

What the container list cannot tell you: how many times it restarted, and whether it was killed for lack of memory. Also the container daemon’s own recorded error. A missing mount source is recorded there. The container’s log names only the file it could not write. And the last few health-probe results.

Reads

The last lines of one container’s log

The end of one container’s output.

Reads

Bring one container back up

Creates that one container again with the current configuration. Nothing else on the server changes.

Writes

Whether the APIs work, not only whether they are up

Asks the installation’s APIs a few questions. It changes nothing on the installation.

Reads

Disk space of the installation

Free space where this installation is stored.

Reads

  • A probe can only be aimed at a service of this installation. The list of services is read from the machine being probed. A name that is not on the list is refused. So a container of another customer in the same daemon cannot be reached this way.

  • Output is limited, and a cut is always reported. A request for a longer log tail than the limit is refused. It is not shortened without notice. So nobody thinks they read a full log when they did not.

  • The one probe that writes recreates the container, not restarts it. That is why a value corrected in the environment file actually takes effect. It is also why it does not touch anything that container depends on.

  • There is no probe that prints the resolved configuration, on purpose. That output contains the filled-in values.

  • A refusal is not a failure. The assistant can run a probe for an operator. Its permission lasts a short time and allows a limited number of calls. It is tied to one installation. A restart of the tool ends it. A second browser tab open on another customer cannot use it. The refusals it can return are a fixed set of sentences. One of them is "This tool only reads. A change has to be made by a person."

If something goes wrong

This section says where to look when a check shows a problem.