Troubleshooting & Command Reference

Almost always a database setting. Run sudo koolboxctl status to see what it thinks, then repair the configuration and bring it up:

sudo koolboxctl status
sudo koolboxctl setup && sudo koolboxctl migrate

For the full error:

sudo journalctl -u koolbox -n 50

On 1.2.8 and earlier, a koolbox-full install on RHEL 9 could leave the service failed with Connectivity: unreachable in koolboxctl status and nothing listening on port 3000 at all — curl: (7) … Connection refused rather than an error page. The cause was PostgreSQL 13's md5 default against the scram-sha-256 those releases wrote into pg_hba.conf.

Fixed from 1.3.0, and upgrading repairs an affected server on its own — it raises the cluster setting, re-hashes the role's password and starts the service:

sudo dnf upgrade koolbox koolbox-full
sudo koolboxctl status                    # expect: reachable / active

Change PORT in /etc/koolbox/koolbox.env and restart:

sudo nano /etc/koolbox/koolbox.env
sudo systemctl restart koolbox

• Check the service is running: sudo koolboxctl status.
• Check that HOSTNAME in /etc/koolbox/koolbox.env is 0.0.0.0 and not 127.0.0.1, which only allows local access.
• Check that port 3000 is open in your cloud provider's security group, and in the host firewall if there is one — sudo ufw status on Debian and Ubuntu, sudo firewall-cmd --list-ports on the RHEL family. Stock cloud images often ship with no host firewall at all (on the RHEL 9.6 image used for verification, firewalld was not even installed), in which case the security group is the only thing that can be blocking you.

The 3-day grace period has ended. Activate a license and everything returns immediately — nothing is deleted while the console is locked.

Expected in three cases:

• The operating mode is set to Air-gapped.
• The license is not yet activated — including during the grace period.
• Kai has not been configured.

Check Administration → Settings → Deployment.

Use one of the eight recovery codes saved when you enrolled: the 2FA challenge page offers Recovery Code alongside Authenticator and Security Key, and each code works once. Sign in that way and register a new authenticator from My Account.

A session outlived the timeout in your security policy. Sign in again. There is deliberately no fallback identity.

The unaccent and pg_trgm extensions are missing from the database. sudo koolboxctl setup adds them when it can; otherwise a database superuser must run:

CREATE EXTENSION unaccent;
CREATE EXTENSION pg_trgm;

If a restart does not change it, LibreOffice genuinely is not installed. It is optional, and deliberately not a hard requirement because some distributions do not ship it. Install libreoffice-core and libreoffice-writer to enable PDF and ODT output.

Either way, the service log states which of the two situations you are in at every start:

[KoolBox] LibreOffice not found — document generation will output DOCX only (no PDF conversion)

[KoolBox] LibreOffice detected: LibreOffice 24.2.7.2 420(Build:2) — PDF document generation available

On RHEL, Rocky Linux, AlmaLinux and Fedora this is SELinux rather than anything to do with KoolBox. Look in /var/log/nginx/error.log: if it says (13: Permission denied) connecting to 127.0.0.1:3000, nginx is not permitted to open outbound connections, and sudo ausearch -m avc -ts today will show a name_connect denial for dest=3000 — run it as sudo ausearch -m avc -ts recent </dev/null, because with no matching records it waits on your terminal and looks hung. One command fixes it permanently:

sudo setsebool -P httpd_can_network_connect 1

The proxy starts working immediately, with no restart. A genuinely stopped console reads connection refused in that log instead — check it with curl http://127.0.0.1:3000/api/health on the server itself. The full log lines are quoted in Put the console on a domain with HTTPS.

On Debian and Ubuntu, the stock default site is enabled and claims every request that does not match another server_name, so a test by IP address returns the distribution's welcome page and your KoolBox block is never consulted:

sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

Before DNS points at the server, test by asking for the name your block listens for: curl -I -H 'Host: console.example.com' http://127.0.0.1/ should answer 307 with location: /auth/login.

/etc/nginx/sites-available and sites-enabled do not exist on the RHEL family, so tee exits 1 and the symlink step fails with No such file or directory. Write the same server block to /etc/nginx/conf.d/koolbox.conf and skip the symlink entirely — nginx.conf already includes that directory.

Your reverse proxy is buffering. Add proxy_buffering off; as shown in the HTTPS setup.

Raise proxy_read_timeout in your reverse proxy. 600 seconds is a sensible starting point.

(Version 1.3.0 and later.) The console itself cannot install anything — it writes a request and a root-owned path unit acts on it. If that unit is not running, the request is never picked up, which is a different situation from the update having failed. Check it and restart it:

systemctl status koolbox-updater.path
sudo systemctl reset-failed koolbox-updater.path && sudo systemctl start koolbox-updater.path

(Version 1.3.0 and later.) The console's progress card survives its own restart, and usually picks up the result within about 23 seconds — but on one of three applies measured it sat on Installing packages for well over a minute after the update had already completed successfully. Check the outcome from outside the page before you do anything:

sudo koolboxctl status
sudo tail -n 20 /var/log/koolbox/update-*.log

The Update history list on the same page is also authoritative. Take the page's own advice and do not reboot the host: the privileged updater re-executes itself precisely so that it survives the console being replaced underneath it.

The notification for a successful rollback carries the right title and the wrong body: Rolled back to KoolBox 1.3.0 followed by “The update did not complete.” That is a cosmetic product bug. sudo koolboxctl status and the Update history row both report it correctly, and the command's own output ends ✓ KoolBox rolled back to … with Phase : rolled-back.

The console's offline upload card refuses to accept files and blames your browser. The browser is almost certainly fine: that card needs a secure context, so it only works over HTTPS or from localhost — and the installer banner tells administrators to use http://<address>:3000, which is not one.

Two ways out, and the second needs no infrastructure at all:

• Put the console behind HTTPS and use the page as intended.
• Stage the four files on the server and run sudo koolboxctl update --offline /path/to/dir instead — see Update an isolated server. On an air-gapped host this is the dependable route.

On Debian and Ubuntu, finish the interrupted operation and try again:

sudo dpkg --configure -a

Then re-run the update as described in Update from the terminal.

Both are read from the environment of the command, so sudo KOOLBOX_READY_TIMEOUT=300 koolboxctl migrate is the way to give a slow or heavily loaded server more time.

Run with a terminal attached, setup is a short interview: where PostgreSQL runs, the port, database name and role, then the console's port and bind address, every value pre-filled with what you already have, and a final Apply this configuration? [Y/n]. It then lists exactly what it is about to do before doing it — generate a JWT secret if one is missing, create the role, database and the two search extensions if they do not exist, raise the cluster's password hashing if it is still md5, enable password authentication for that one role on loopback only, and re-key the role if it cannot authenticate with the password in your configuration file. Existing databases, settings and data are never modified, and an external PostgreSQL server is never reconfigured at all — on a managed service or someone else's cluster it only checks and reports.

Run without a terminal — from a script, or with </dev/null — it asks nothing and does the same work, identically to setup --auto, in well under a second. It is safe to re-run as often as you like; on an already-configured server it reports what already exists and exits 0.

koolboxctl help lists every command and option on the release you have installed, which is the version worth trusting over any copy printed here.

Captured on 1.3.2. This is the reference worth trusting over anything printed here, because it is the release you actually have:

koolboxctl help

koolboxctl — KoolBox Private AI Control Panel administration

Usage: koolboxctl <command> [options]

Installation & upgrades
  setup                 Guided: asks where PostgreSQL runs, its connection
                        details and the console port, then creates/repairs the
                        configuration, database role, database and required
                        PostgreSQL extensions. Safe to re-run.
                        On a local cluster it also reconciles password hashing:
                        password_encryption, the role's stored hash and the
                        pg_hba.conf method are made to agree, and a role that
                        cannot authenticate is re-keyed from koolbox.env.
                        An external server is never reconfigured.
  setup --auto          The same work with no prompts — what the package
                        install scripts run. Exit 2 = configure the database by
                        hand (external PostgreSQL).
  migrate               Start or restart the service so it applies schema and
                        data upgrades, then wait until it reports ready.
  config [show]         Print effective configuration (secrets masked).
  config --merge-defaults
                        Append settings introduced by this release to
                        /etc/koolbox/koolbox.env without touching existing values.

Updates
  update                Guided: check for a release, show the notes, confirm,
                        apply, then verify. Takes a database backup first.
  update --check        Report only. Exit 0 = up to date, 10 = update
                        available, 1 = could not check.
  update --to X.Y.Z [--yes]
                        Install a named version. --yes skips the confirmation
                        only; the backup is always taken.
  update --offline <dir|file…> [--to X.Y.Z] [--yes]
                        Install from .deb/.rpm files already on this host, with
                        latest.json + latest.json.sig if you have them.
  rollback [--to X.Y.Z] [--yes]
                        Reinstall an earlier version. Does not restore the
                        database; without --to, the version of the newest
                        pre-upgrade backup is used.

                        Updates are applied by the privileged runner at
                        /usr/libexec/koolbox/koolbox-update-runner. The release
                        manifest is read from UPDATE_MANIFEST_URL in
                        /etc/koolbox/koolbox.env and its signature must verify
                        before anything is installed.

Operations
  status                Version, service, database and health summary.
  start | stop | restart
  logs [journalctl args]
  version

Data
  backup [file]         pg_dump the application database (gzip). Defaults to
                        /var/lib/koolbox/backups/, keeping the newest 5.
  restore <file> [--yes]
                        Replace the database contents with a backup. Destructive.
                        The dump is applied as the unprivileged koolbox account,
                        and one containing psql meta-commands is refused.

Environment
  KOOLBOX_BACKUP_KEEP   Backups to retain (default 5).
  KOOLBOX_READY_TIMEOUT Seconds to wait for readiness (default 120).
  KOOLBOX_UPDATE_TIMEOUT
                        Seconds allowed for each manifest download (default 20).
  NO_COLOR              Set to disable coloured output.

Documentation: https://koolify.ai

sudo koolboxctl config show

PORT=3000
HOSTNAME=0.0.0.0
DB_HOST=localhost
DB_PORT=5432
DB_NAME=koolbox
DB_USER=koolbox
DB_PASSWORD=********
DB_DIALECT=postgres
JWT_SECRET=********
JWT_EXPIRY=24h
NODE_ENV=production
NEXT_TELEMETRY_DISABLED=1
VERBOSE=false
LOG_CLEANUP_HOUR=2
PACKAGES_DIR=/var/lib/koolbox/packages
LICENSE_SERVER_URL=https://www.koolify.ai
PUBLIC_BASE_URL=
ASSISTANT_URL=
ASSISTANT_API_KEY=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
AWS_EC2_METADATA_DISABLED=

The two secrets really are masked rather than merely short — DB_PASSWORD and JWT_SECRET print as ******** against 36- and 64-character values — so this output is safe to paste into a support conversation. Values shown empty are settings this installation is not using.