Skip to content

TROUBLESHOOTING.md

Latest commit

 

History

History
474 lines (329 loc) · 13.4 KB

TROUBLESHOOTING.md

File metadata and controls

474 lines (329 loc) · 13.4 KB

Troubleshooting

If you encounter any issues while using Zerobyte, you can check the application logs for more information. These logs can help you identify and resolve common problems; you should also check existing and closed issues on GitHub. In case you need further assistance, feel free to open a new issue with detailed information about the problem you are facing and any relevant log entries.

Warning

Make sure to never share sensitive information such as passwords, access keys, or personal data in public issues so remove them from logs before posting.

Before troubleshooting, enable debug logging so the logs contain enough detail to diagnose issues.

services:
  zerobyte:
    environment:
      - LOG_LEVEL=debug

After adding LOG_LEVEL=debug, restart Zerobyte and then collect logs.

To view the logs, run the command below:

# replace 'zerobyte' with your container name if different
docker logs -f zerobyte

Table of Contents

Common Issues

Permission Denied Errors When Mounting Remote Shares

Mounting remote filesystems (such as SMB/CIFS) requires kernel-level privileges. When Zerobyte attempts to perform mounts from inside a container, additional permissions may be required.

Ensure that:

  • Remote share credentials are correct
  • The host kernel supports the target filesystem (e.g. CIFS module is available)
  • Docker is running in rootful mode (rootless Docker cannot perform kernel mounts)

In some environments, Linux security mechanisms such as AppArmor or seccomp may block mount-related operations even when the required capabilities are present.

Security Levels for Mounting Remote Shares

Zerobyte supports multiple deployment models depending on your security requirements and environment.

Permission Errors with Remote Shares

Problem: You're getting permission errors when Zerobyte tries to mount remote shares (SMB/CIFS, NFS, etc.).

Solution: Mount remote shares outside of Zerobyte (on the host) and point Zerobyte to an already-mounted local path. This avoids container permission issues entirely.

How to fix:

  1. Mount your remote share on the host first using systemd, autofs, or manual mount:

    sudo mount -t cifs //server/share /mnt/your-remote-share -o credentials=/path/to/creds

  2. Then mount that local path into Zerobyte:

    services:
  zerobyte:
    volumes:
      - /mnt/your-remote-share:/data

  3. Restart the container:

    docker compose down && docker compose up -d

Container Cannot Perform Mounts

Problem: Zerobyte shows "Operation not permitted" errors when trying to mount remote shares directly.

Solution: Grant the SYS_ADMIN capability to allow the container to perform mount operations.

Warning: This grants significant privileges to the container and should only be used when necessary.

services:
  zerobyte:
    cap_add:
      - SYS_ADMIN

⚠️ Granting SYS_ADMIN allows the container to perform mount operations and should be used only when strictly necessary.

AppArmor-Enabled Systems (Ubuntu/Debian)

On hosts using AppArmor, the default Docker profile (docker-default) may block mount operations even when SYS_ADMIN is present.

If mount operations fail with permission errors, you may need to disable AppArmor confinement for the container. Check first if AppArmor is enabled on your system and the profile of the container:

# check if AppArmor is enabled
sudo aa-status
# if next command returns 'docker-default', AppArmor is enabled on the container
docker inspect --format='{{.AppArmorProfile}}' zerobyte

If AppArmor is enabled, you can disable it for the Zerobyte container by adding the following to your docker-compose.yml:

services:
  zerobyte:
    cap_add:
      - SYS_ADMIN
    security_opt:
      - apparmor:unconfined

Seccomp-Restricted Environments

Docker's default seccomp profile may block mount-related syscalls required by filesystem operations.

If mount operations continue to fail, you may need to disable seccomp filtering for the container:

services:
  zerobyte:
    cap_add:
      - SYS_ADMIN
    security_opt:
      - seccomp:unconfined

SELinux-Enabled Systems (CentOS/Fedora)

On hosts using SELinux, you may need to adjust the security context to allow mount operations. If mount operations fail with permission errors, you can try adding the following label:

services:
  zerobyte:
    cap_add:
      - SYS_ADMIN
    security_opt:
      - label:type:container_runtime_t

or disable SELinux enforcement for the container:

services:
  zerobyte:
    cap_add:
      - SYS_ADMIN
    security_opt:
      - label:disable

Still Getting Permission Errors After SYS_ADMIN

Problem: Mount operations still fail even with SYS_ADMIN capability.

Solution (Last Resort): Run the container in privileged mode. This disables most container isolation mechanisms and significantly increases the attack surface.

Warning: Only use this as a last resort for troubleshooting. Remove and switch to a more secure solution once you identify the actual issue.

services:
  zerobyte:
    privileged: true

FUSE Mount Failures

Problem: FUSE-based mounts (sshfs, rclone mount) are failing with device access errors.

Cause: Access to /dev/fuse is required for FUSE-based filesystems.

Solution: Ensure /dev/fuse is mounted into the container. This is not required for SMB/CIFS mounts.

services:
  zerobyte:
    devices:
      - /dev/fuse:/dev/fuse

---

## Rclone Issues

### ⚠️ Critical: Test on Host First

**Before reporting any rclone-related issue, you MUST verify that rclone works correctly on your Docker host.**

Most rclone issues are due to misconfigured remotes, not Zerobyte bugs. Follow this checklist on your host machine:

```bash
# 1. List all configured remotes
rclone listremotes

# 2. Test listing a remote (replace 'myremote' with your remote name)
rclone lsd myremote:

# 3. Test reading from a remote
rclone ls myremote:path/to/test

# 4. For volume mounts: Test mounting on host first
mkdir -p /tmp/rclone-test
rclone mount myremote:path /tmp/rclone-test --daemon --vfs-cache-mode writes
ls /tmp/rclone-test
fusermount -u /tmp/rclone-test  # Unmount when done

If these commands fail on your host, fix your rclone configuration before using Zerobyte.

Common issues include:

  • Expired OAuth tokens (run rclone config to re-authenticate)
  • Incorrect credentials
  • Missing permissions on cloud provider side
  • Network/firewall issues

Pre-flight Checklist

If you're experiencing rclone issues, verify all of the following:

  • Rclone is installed and configured on the Docker host
  • rclone listremotes shows your remote
  • rclone lsd remote: successfully lists directories
  • The rclone config directory is mounted into the container

Common Rclone Errors

"No Remotes Available" in Dropdown

Cause: Zerobyte cannot find your rclone configuration file.

Diagnosis:

# Check which config file rclone will use inside the container
docker exec zerobyte sh -lc 'echo HOME=$HOME; rclone config file'

Solutions:

  1. Ensure you've mounted your host's rclone config:

    volumes:
  - ~/.config/rclone:/root/.config/rclone:ro

  2. For non-root containers (e.g., TrueNAS), set the correct path:

    environment:
  - RCLONE_CONFIG_DIR=/home/appuser/.config/rclone
volumes:
  - ~/.config/rclone:/home/appuser/.config/rclone:ro

  3. Restart the container after mounting the config:

    docker compose down
docker compose up -d

"Failed to Create File System" Error

Cause: Authentication failure with the cloud provider.

Solution:

  1. On your host, run: rclone config
  2. Re-authenticate the remote (especially for OAuth providers like Google Drive, Dropbox, OneDrive)
  3. Verify with: rclone lsd remote:
  4. Restart the Zerobyte container

EACCES Errors

Cause: AppArmor or seccomp is blocking rclone execution.

Solution: Disable security profiles as described in:

Rclone SFTP Repository Authentication Failures

When creating an rclone repository that uses an SFTP remote, you may encounter authentication errors even though:

  • The rclone config is mounted correctly
  • The SFTP remote appears in the dropdown
  • The same config works on the host

The issue: If your rclone SFTP remote uses key_file for SSH key authentication, the key file path in your rclone config points to a location on the host (e.g., ~/.ssh/id_rsa). When rclone runs inside the container, it cannot access that path.

Solutions:

Option 1: Mount your SSH keys (Recommended)

Add your SSH directory to the container volumes:

services:
  zerobyte:
    volumes:
      - ~/.config/rclone:/root/.config/rclone:ro
      - ~/.ssh:/root/.ssh:ro # Required for SFTP remotes using key_file

Option 2: Embed the SSH key in rclone config

Use key_pem instead of key_file to embed the private key directly in your rclone.conf:

# Convert your key to single-line format
awk '{printf "%s\\n", $0}' < ~/.ssh/id_rsa

Then update your rclone config:

[sftp-remote]
type = sftp
host = example.com
user = backup
key_pem = -----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5v...\n-----END OPENSSH PRIVATE KEY-----

Option 3: Use ssh-agent

Configure your rclone remote to use ssh-agent instead of a key file:

[sftp-remote]
type = sftp
host = example.com
user = backup
key_use_agent = true

Then mount the SSH agent socket and set the environment variable:

services:
  zerobyte:
    environment:
      - SSH_AUTH_SOCK=/ssh-agent
    volumes:
      - ~/.config/rclone:/root/.config/rclone:ro
      - ${SSH_AUTH_SOCK}:/ssh-agent

Rclone Volume Mount Issues

When using rclone as a volume backend (mounting cloud storage to back up from), additional requirements apply:

Prerequisites Check

If rclone volume mounting isn't working, verify these prerequisites on your system:

  • Linux host required: Windows/macOS hosts cannot use rclone volumes. If you're on these platforms, you'll need to use a different volume backend.
  • /dev/fuse device: Check if FUSE is available and properly passed to the container
  • SYS_ADMIN capability: Required for mount operations inside containers
  • FUSE support on host: Verify FUSE is installed and working on your Docker host

Common Mount Errors

"mount helper error: fusermount3: failed to open /dev/fuse: Permission denied"

Solutions in order of preference:

  1. Verify FUSE works on host first:

    # On host
rclone mount remote:path /tmp/test --daemon
ls /tmp/test
fusermount -u /tmp/test

  2. Ensure /dev/fuse is mounted in container:

    devices:
  - /dev/fuse:/dev/fuse

  3. Check user permissions inside container:

    docker exec zerobyte ls -la /dev/fuse

"mount helper error: fusermount3: mount failed: Operation not permitted"

This indicates the container lacks SYS_ADMIN capability:

cap_add:
  - SYS_ADMIN

Still having issues?

If you've verified rclone works on the host and followed all troubleshooting steps above, gather this information for your issue report:

# Host verification
rclone version
rclone listremotes
rclone lsd remote: 2>&1

# Container verification
docker exec zerobyte ls -la /root/.config/rclone/
docker exec zerobyte env | grep RCLONE
docker logs zerobyte 2>&1 | tail -50

# Check container capabilities
docker inspect zerobyte --format='{{.HostConfig.CapAdd}}'
docker inspect zerobyte --format='{{.HostConfig.Devices}}'

Include the output (with sensitive data redacted) in your GitHub issue.