I have two locations where I have upgraded TN 25.04.2.3 to 25.10. In both instances that drive performance has dropped through the floor. In some instances it has dropped as much as 98%.
Both are virtual machines (XCP-NG): one is running to a passed through
truenas-tests.txt (4.7 KB)
LSI controller and one are just virtual drives. Both have worked great since version 22xxxx.
In both instances, if you click Reporting > disks…the screen just spins.
Shown below are the tests from FIO (Flexible I/O Tester)(I copied them from a text file and may not be useful). I’m also attaching a text file with the tests.
Any ideas?
Thanks!
| Computer #1 | ||
|---|---|---|
| Processor | Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz | |
| RAM | 16 GB | |
| VDEV type | RaidZ1 | |
| SLOG | nvme | |
| TESTS | ||
| Random write test | ||
| test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | WRITE: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=4096MiB (4295MB), run=17884-17884msec | |
| After | WRITE: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=4096MiB (4295MB), run=22698-22698msec | |
| Decrease | -21.40% | |
| Random Read test | ||
| test: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | READ: bw=1341MiB/s (1406MB/s), 1341MiB/s-1341MiB/s (1406MB/s-1406MB/s), io=4096MiB (4295MB), run=3055-3055msec | |
| After | READ: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s-54.1MB/s), io=2821MiB (2958MB), run=54716-54716msec | |
| Decrease | -96.15% | |
| Mixed Random Workload test | ||
| test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | READ: bw=440MiB/s (461MB/s), 440MiB/s-440MiB/s (461MB/s-461MB/s), io=1992MiB (2089MB), run=4532-4532msec | |
| After | READ: bw=60.9MiB/s (63.8MB/s), 60.9MiB/s-60.9MiB/s (63.8MB/s-63.8MB/s), io=254MiB (266MB), run=4173-4173msec | |
| Decrease | -86.16% | |
| Before | WRITE: bw=464MiB/s (487MB/s), 464MiB/s-464MiB/s (487MB/s-487MB/s), io=2104MiB (2206MB), run=4532-4532msec | |
| After | WRITE: bw=73.6MiB/s (77.1MB/s), 73.6MiB/s-73.6MiB/s (77.1MB/s-77.1MB/s), io=307MiB (322MB), run=4173-4173msec | |
| Decrease | -84.14% | |
| Sequential write test | ||
| test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | WRITE: bw=243MiB/s (254MB/s), 243MiB/s-243MiB/s (254MB/s-254MB/s), io=4096MiB (4295MB), run=16889-16889msec | |
| After | WRITE: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=4096MiB (4295MB), run=23065-23065msec | |
| Decrease | -26.75% | |
| Sequential Read test | ||
| test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | READ: bw=1375MiB/s (1442MB/s), 1375MiB/s-1375MiB/s (1442MB/s-1442MB/s), io=4096MiB (4295MB), run=2979-2979msec | |
| After | READ: bw=250MiB/s (263MB/s), 250MiB/s-250MiB/s (263MB/s-263MB/s), io=4096MiB (4295MB), run=16353-16353msec | |
| Decrease | -81.82% | |
| Computer #2 | ||
| Processor | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz | |
| RAM | 60 GB | |
| VDEV type | Mirror | |
| TESTS | ||
| Random write test | ||
| test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | WRITE: bw=1140MiB/s (1196MB/s), 1140MiB/s-1140MiB/s (1196MB/s-1196MB/s), io=4096MiB (4295MB), run=3592-3592msec | |
| After | WRITE: bw=55.1MiB/s (57.8MB/s), 55.1MiB/s-55.1MiB/s (57.8MB/s-57.8MB/s), io=2865MiB (3004MB), run=51950-51950msec | |
| Decrease | -99.55% | |
| Random Read test | ||
| test: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | READ: bw=2693MiB/s (2824MB/s), 2693MiB/s-2693MiB/s (2824MB/s-2824MB/s), io=4096MiB (4295MB), run=1521-1521msec | |
| After | READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=4096MiB (4295MB), run=18081-18081msec | |
| Decrease | -91.57% | |
| Mixed Random Workload test | ||
| test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=1992MiB (2089MB), run=19285-19285msec | |
| After | READ: bw=10.3MiB/s (10.8MB/s), 10.3MiB/s-10.3MiB/s (10.8MB/s-10.8MB/s), io=1902MiB (1994MB), run=184480-184480msec | |
| Decrease | -90.00% | |
| Before | WRITE: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=2104MiB (2206MB), run=19285-19285msec | |
| After | WRITE: bw=10.9MiB/s (11.4MB/s), 10.9MiB/s-10.9MiB/s (11.4MB/s-11.4MB/s), io=2014MiB (2112MB), run=184480-184480msec | |
| Decrease | -90.00% | |
| Sequential write test | ||
| test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | WRITE: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=1105MiB (1159MB), run=7082-7082msec | |
| After | WRITE: bw=50.9MiB/s (53.3MB/s), 50.9MiB/s-50.9MiB/s (53.3MB/s-53.3MB/s), io=2332MiB (2445MB), run=45856-45856msec | |
| Decrease | -67.37% | |
| Sequential Read test | ||
| test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 | ||
| Before | READ: bw=2626MiB/s (2753MB/s), 2626MiB/s-2626MiB/s (2753MB/s-2753MB/s), io=4096MiB (4295MB), run=1560-1560msec | |
| After | READ: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=4096MiB (4295MB), run=15304-15304msec | |
| Decrease | -89.79% |