Custom Advanced Parameter Function for Rclone Cloud Sync

Feature Requests
1

Issues / Rationale

The cloud sync feature of TrueNAS is underpinned by Rclone to enable cross-storage data synchronization (e.g., between local storage and S3-compliant object storage). However, it currently only supports extension via pre-scripts/post-scripts and lacks a native configuration entry for customizing advanced Rclone parameters. This creates critical pain points for users in complex synchronization scenarios (such as S3 protocol compatibility, special permission handling, error avoidance, and refined log configuration), including the following:

  • Inability to precisely resolve protocol compatibility issues: For example, the “incompatible remotes” error that occurs when syncing between S3 and local storage requires disabling implicit operations using native Rclone parameters like and . Yet the existing script functionality cannot directly embed these parameters into the core sync process, leading to recurrent errors.--no-move --ignore-errors
  • Inefficient logging and debugging: Troubleshooting sync errors requires manual modification of scripts or configuration files to add parameters such as and . This process is cumbersome, error-prone, and cannot be configured quickly via the graphical user interface (GUI).--log-level=DEBUG --log-format
  • Unmet requirements for special scenarios: Needs such as skipping symbolic links during sync (), preserving file metadata (), and limiting transfer rates () cannot be fulfilled. The existing pre-/post-scripts only execute independent commands and cannot intervene in the core logic of Rclone synchronization.--skip-links --preserve-modtime --bwlimit
  • Limitations of the script functionality: Pre-/post-scripts are designed for auxiliary operations before and after synchronization (e.g., directory creation, cache cleanup), rather than parameter injection during the sync process. They cannot replace the direct control of sync behavior enabled by advanced Rclone parameters.

Impacts

1. Core Benefits and Advantages

  • For users in complex scenarios (e.g., S3 sync users): Deliver flexible parameter configuration capabilities, allowing protocol compatibility, error avoidance, and log debugging to be resolved directly via the GUI—without reliance on the command line or scripts, lowering technical barriers.
  • For regular users: No disruption to existing usage habits; the original sync logic is retained by default, with only an optional Advanced Parameters configuration item added for on-demand enablement.
  • For the TrueNAS product ecosystem: Enhance the flexibility and compatibility of the cloud sync feature to adapt to the special requirements of more storage protocols (e.g., S3, WebDAV, FTP), improving the product’s practicality in enterprise-level scenarios.
  • Efficiency gains: Reduce script writing and configuration file modifications required for user parameter configuration, and shorten troubleshooting cycles (e.g., obtaining detailed logs quickly via ).--log-level

2. Potential Disadvantages and Mitigation Solutions

  • Risk of parameter misuse: Certain Rclone parameters (e.g., ) may pose data risks. Mitigate this by adding links to parameter documentation and risk prompt popups in the interface, and advising users to configure parameters with reference to the official Rclone documentation.--delete-all
  • Increased interface complexity: New configuration items may confuse novice users. Resolve this by placing the Advanced Parameters in a collapsible Advanced Options panel, hidden by default to avoid interfering with basic user operations.
  • Compatibility maintenance costs: Rclone parameters iterate rapidly, requiring the TrueNAS team to adapt to new parameters regularly. However, core parameters (e.g., those for logging, error handling, and transfer control) have high stability, making maintenance costs manageable.

User Stories

Scenario 1: Resolve the “incompatible remotes” Error in S3 Sync

User Role: TrueNAS administrator who needs to sync a local directory to S3-compliant object storage.User Goal: Avoid the “can’t move object - incompatible remotes” error by disabling Rclone’s implicit move operation.Steps:

  1. Log in to the TrueNAS Web UI and navigate to Cloud Sync → Create/Edit Sync Task.
  2. After completing basic configurations (source/target paths, selecting Copy as the operation type), expand the Advanced Options panel.
  3. Locate the Rclone Advanced Parameters input box and enter the parameters: .--no-move --ignore-errors --log-level=DEBUG
  4. Save the task and trigger synchronization manually; the system automatically passes the parameters to the core Rclone command.
  5. After synchronization is complete, verify that the error is resolved via logs and confirm that files are copied to S3 successfully.

Scenario 2: Configure Refined Sync Logs

User Role: TrueNAS user who needs to troubleshoot I/O errors during synchronization.User Goal: Obtain detailed logs with millisecond-level timestamps to locate problematic files.Steps:

  1. Navigate to Advanced Options → Rclone Advanced Parameters for the sync task.
  2. Enter log-related parameters: .--log-file=/var/log/rclone/s3_sync.log --log-format=date,time,millis,level,msg --log-level=DEBUG
  3. After running the sync task, execute in the TrueNAS terminal to view real-time detailed logs with precise timestamps.tail -f /var/log/rclone/s3_sync.log
  4. Adjust permissions or path configurations targetedly based on error information in the logs (e.g., I/O errors for specific files).

Scenario 3: Limit Sync Transfer Rates

User Role: Enterprise user who needs to sync large files during working hours without occupying excessive bandwidth.User Goal: Limit the maximum transfer rate of Rclone synchronization to avoid impacting other business operations.Steps:

  1. Edit the cloud sync task and enter the parameter in Rclone Advanced Parameters: (limit to 10MB/s).--bwlimit=10M
  2. Save the configuration and start synchronization; the system automatically transfers files at the configured rate.
  3. Monitor the transfer status via the task progress bar during synchronization and confirm that bandwidth usage meets expectations.
2

Vote for your own request …

1 Like
3

Thanks for the reminder!

4

I doubt that this will get accepted since this feature request got denied

5

Your vote for this request would be greatly appreciated？

6
7

Maybe try writing up a detailed report of how you fixed your S3 issue. If all S3 require those settings, that may be possible for a preset or something. If you need to change the title, DM someone from the Moderators group. You might want to update your ‘problem’ thread with your results and fixes, also.
https://forums.truenas.com/g/moderators

8

Actually, fixing S3-related issues often just requires using rclone’s advanced custom parameters. There are so many different S3-compatible object storage services out there, and many compatibility problems can arise. However, rclone’s advanced parameters allow you to customize settings for different scenarios, which can resolve at least 90% of these issues.

9

Hello, I mean no harm, but my post has clearly stated why certain functional requirements are necessary. For example, the --log-level=DEBUG log information is needed for debugging. Why did you turn a blind eye to it?

10

I thought you would update your other thread with what you went through and how you did your troubleshooting. It was written pretty well here and there were no replies there. I figure that thread may be brought up in search for other users.

This one will probably be marked Rejected like the other Feature Request and locked. TrueNAS dev has already made the decisions on taming choices in the GUI. Open ended Custom Parameters just isn’t a design choice. Your Limit Sync Transfer Rates might make a good FR by itself.

You were given constructive criticism to narrow your request or refine it by two others.

11

Thank you sincerely for your explanation and suggestions. I understand now. Based on my analysis, the UI features currently especially needed are:

  1. --log-level – A log level selector for easier debugging.
  2. --no-move – For better compatibility with different S3 vendors.
12

Feature needed --ignore-size

necessary for Onedrive synchronization Office files.

13

You can talk to the TrueNAS middleware API via CLI. This makes it possible to add args/parameters to your rclone tasks.

Example:

midclt call cloudsync.update <cloud sync task number, see table in Data Protection, starts with 1> '{"args": "<rclone args>"}'
midclt call cloudsync.update 2 '{"args": "--modify-window 1s"}'
midclt call cloudsync.update 2 '{"args": "--fast-list --onedrive-delta --modify-window 1s"}'
(via TrueNAS shell)

I have to use --fast-list --onedrive-delta --modify-window 1s for my OneDrive syncs. This reduced the task from 27min to just 3min.

It’s ridiculous how complicated this is, but at least it’s somehow possible.

14

All parameters of rclone are in the file /usr/lib/python3/dist-packages/middlewared/plugins/cloud_sync.py.

In section

# Use a temporary file to store rclone file
async with RcloneConfig(cloud_sync) as config:
args = [

you can add any parameter.

You will need “sudo” rights and SSH connect.

sudo mount -o remount,rw /usr

sudo nano /usr/lib/python3/dist-packages/middlewared/plugins/cloud_sync.py

edit file and restart TrueNAS