Debug a service that works only in your shell

Find why a Node app works in your terminal but fails under systemd by checking paths, identity, environment, permissions and service lifecycle.

A Node application that runs in your terminal but fails as a service usually has a hidden dependency on that terminal: its working directory, PATH, account, environment or writable folders. Debugging becomes predictable when you compare those inputs rather than repeatedly restarting.

This guide assumes an Ubuntu test host, systemd and a small existing Node application at /srv/demo-api/current/server.mjs. Replace the fictional unit and paths with your own reviewed test setup. Use sudo only for service administration. The command blocks are syntax-checked; no production unit was restarted to write this article. The VPS learning path links the surrounding setup.

Capture the failure before changing it

Ubuntu administrator — inspect the named service

sudo systemctl status demo-api.service --no-pager --full
sudo systemctl show demo-api.service \
  -p FragmentPath -p DropInPaths -p User -p Group \
  -p WorkingDirectory -p ExecStart -p Result -p ExecMainStatus
sudo journalctl -u demo-api.service -b -n 80 --no-pager

Read the first meaningful application error, not only the final restart-limit message. A missing executable, unreadable configuration and database authentication failure need different remedies. Review output privately: command arguments and application logs can contain sensitive values if the app was configured poorly.

Use FragmentPath and DropInPaths to find the configuration actually loaded. Editing a similarly named file in your home directory will not change the installed unit. Do not dump Environment or a process's complete environment into a support ticket.

Compare paths, not shell habits

Interactive Ubuntu shell — compare the known-good context

pwd
id
command -v node
node --version
readlink -f /srv/demo-api/current
namei -l /srv/demo-api/current/server.mjs

A system service does not automatically inherit your interactive .bashrc, activated version manager or current directory. Set an explicit working directory and verified absolute interpreter path. A relative import can work while a relative file read fails because those paths are resolved differently by the application.

Check directory traversal as well as the final file's mode. See application permissions. If a release symlink points to the wrong version, fix the deployment reference rather than copying random files into the running directory.

Make the service contract explicit

The following unit is a reference for a test application that listens on loopback and reads the shown settings. Do not overwrite an existing unit without preserving its current configuration.

/etc/systemd/system/demo-api.service

[Unit]
Description=Demo Node API
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=5
 
[Service]
Type=exec
User=demo-api
Group=demo-api
WorkingDirectory=/srv/demo-api/current
Environment=NODE_ENV=production
EnvironmentFile=/etc/demo-api.conf
ExecStart=/usr/bin/node /srv/demo-api/current/server.mjs
Restart=on-failure
RestartSec=5
TimeoutStopSec=20
NoNewPrivileges=true
PrivateTmp=true
 
[Install]
WantedBy=multi-user.target

Replace the Node path with the verified executable. Type=exec confirms successful execution before start completes; it still does not mean an HTTP route or database is ready. After=network.target describes ordering, not a working internet connection or healthy remote database. Build retries and readiness checks into the app. The systemd service reference explains those lifecycle distinctions.

The configuration file uses assignment syntax, not an arbitrary shell script:

/etc/demo-api.conf — non-secret example values

PORT=3001
HOST=127.0.0.1
UPLOAD_DIRECTORY=/var/lib/demo-api/uploads

The application must actually read these names. Do not use shell command substitutions or assume $OTHER_VARIABLE expands like a shell. Keep the real file root-owned and restricted; the system manager reads it before starting the service account. Environment-based secrets can still be available to the process and privileged administrators. systemd execution documentation describes the parsing and security limits.

Test identity without starting a duplicate writer

Ubuntu administrator — bounded access tests

sudo -u demo-api test -x /usr/bin/node
sudo -u demo-api test -r /srv/demo-api/current/server.mjs
sudo -u demo-api /usr/bin/node --version
sudo -u demo-api test -w /var/lib/demo-api/uploads

Inspect each exit status immediately if a command prints nothing. These checks do not run the application, send email or start a second database worker. Starting another copy “just to test” can consume queue jobs even when it uses a different HTTP port.

Ordinary access tests cannot reproduce every service restriction. If the application needs a file hidden by ProtectHome or a read-only sandbox, move the file to the intended service directory or make a narrow reviewed exception. Do not remove all hardening to make the error disappear.

Validate, reload and restart deliberately

Back up the exact unit or drop-in before editing it, recording the path and mode. Then:

Ubuntu administrator — apply a reviewed unit correction

sudo systemd-analyze verify /etc/systemd/system/demo-api.service
sudo systemctl daemon-reload
sudo systemctl restart demo-api.service
sudo systemctl is-active demo-api.service
curl --fail --max-time 5 http://127.0.0.1:3001/health
sudo journalctl -u demo-api.service --since "5 minutes ago" --no-pager

Stop if verification fails. daemon-reload makes systemd reread unit definitions; it does not replace the application process. Restart does. Changing an existing environment file's contents also needs a restart to affect a new process, even when the unit itself is unchanged.

If the unit has reached its start limit, first fix the cause; then reset only this unit's failed state with sudo systemctl reset-failed demo-api.service before starting again. Resetting the counter is not a repair.

Prove the fix and retain an escape route

Check the local listener and the public HTTPS route. Confirm expected application behaviour, not just HTTP 200 on a generic page. Rehearse boot recovery separately on a disposable host; enabled and active describe different things.

If the change fails, restore the exact backed-up unit/drop-in, run verification and daemon-reload, and restart this service only. Do not use a blanket “revert everything” command that might discard someone else's drop-ins. Follow Nginx 502 diagnosis if the service is healthy locally but the proxy is not.

A useful CodeKits handover documents the interpreter, account, paths, settings and health checks so the next operator does not have to reconstruct your shell session.