Video Playback Stalling

Not usually a problem, but I was playing a movie the other night, and the stream kept stalling, so I was wondering what might be going on, and how I could remedy this problem, or at least investigate it better.

I was viewing the movie via VLC running on my Google TV box, connected by 1 Gb ethernet to my switch.

My TrueNAS is a Supermicro H13SSL-NT with 10 Gb ethernet, using an AMD EPYC 9015 with 64 GB of RAM, and a VDEV with eight 8 TB disks, 3 of which are parity.

There is an SMB share running on the VDEV, which is how the Google TV box sees it. There has never been a problem with stalling before, but this whole movie was constantly stalling. The movie was a 68 GB MKV file, Oceans 11.

Watching the ARC there were a lot of cache misses, but I am a newbie with ZFS, so what do I know. The ARC is using most of my RAM.

  1. VLC could have done a better job of buffering, but there seems to be no way to configure this.
  2. Is there a way to configure ZFS/TrueNAS to prefetch from a file being read sequentially?
  3. Is there something else I can do to reduce such stalling?

While I have PLEX, I cannot seem to get it working. It refuses to scan the filesystem with my media. It can see it, but refuses to scan it.

Overall, I find PLEX insanely more complicated than it needs to be. It’s like trying to build a website with Spring, which is insanely more complicated than it needs to be. After six hours of fussing with PLEX, I gave up.

However, I suspect PLEX does have a way to do better buffering/prefetching, and it would be nice to have the option of transcoding.

Hi, you mention that this is not normally an issue so do you find the other films you watch are much smaller than 68GB? If so what’s a typical size?

Could you post the output of arc_summary.

ARC resets after reboot so if you turn the system off regularly you’ll need to play the film and then after run the command to get the most accurate and helpful output.

Sounds like you have a single vdev in your pool an 8disk Z3 does that sound right? Feel free to also post the output of zpool status.

I can experiment with some other large video files…

I hope this helps. I have not rebooted the system since watching the movie.

Summary

ZFS Subsystem Report Tue Feb 04 07:40:37 2025
Linux 6.6.44-production+truenas 2.2.99-1
Machine: truenas (x86_64) 2.2.99-1

ARC status:
Total memory size: 62.4 GiB
Min target size: 3.1 % 2.0 GiB
Max target size: 98.4 % 61.4 GiB
Target size (adaptive): 82.8 % 50.9 GiB
Current size: 82.8 % 50.8 GiB
Free memory size: 8.1 GiB
Available memory size: 5.9 GiB

ARC structural breakdown (current size): 50.8 GiB
Compressed size: 95.7 % 48.7 GiB
Overhead size: 3.5 % 1.8 GiB
Bonus size: 0.1 % 26.9 MiB
Dnode size: 0.2 % 104.8 MiB
Dbuf size: 0.1 % 69.6 MiB
Header size: 0.3 % 181.8 MiB
L2 header size: 0.0 % 0 Bytes
ABD chunk waste size: < 0.1 % 4.9 MiB

ARC types breakdown (compressed + overhead): 50.5 GiB
Data size: 97.0 % 49.0 GiB
Metadata size: 3.0 % 1.5 GiB

ARC states breakdown (compressed + overhead): 50.5 GiB
Anonymous data size: 0.0 % 0 Bytes
Anonymous metadata size: < 0.1 % 557.5 KiB
MFU data target: 53.9 % 27.2 GiB
MFU data size: 56.0 % 28.3 GiB
MFU evictable data size: 55.2 % 27.8 GiB
MFU ghost data size: 9.4 GiB
MFU metadata target: 6.8 % 3.4 GiB
MFU metadata size: 1.1 % 546.0 MiB
MFU evictable metadata size: 0.8 % 438.1 MiB
MFU ghost metadata size: 0 Bytes
MRU data target: 32.5 % 16.4 GiB
MRU data size: 41.0 % 20.7 GiB
MRU evictable data size: 36.3 % 18.3 GiB
MRU ghost data size: 14.9 GiB
MRU metadata target: 6.8 % 3.4 GiB
MRU metadata size: 1.9 % 995.1 MiB
MRU evictable metadata size: 0.8 % 437.7 MiB
MRU ghost metadata size: 0 Bytes
Uncached data size: 0.0 % 0 Bytes
Uncached metadata size: 0.0 % 0 Bytes

ARC hash breakdown:
Elements max: 906.4k
Elements current: 85.9 % 778.5k
Collisions: 2.5M
Chain max: 4
Chains: 34.1k

ARC misc:
Memory throttles: 0
Memory direct reclaims: 410420
Memory indirect reclaims: 1869281
Deleted: 19.8M
Mutex misses: 244
Eviction skips: 4.8k
Eviction skips due to L2 writes: 0
L2 cached evictions: 0 Bytes
L2 eligible evictions: 2.3 TiB
L2 eligible MFU evictions: 8.2 % 197.3 GiB
L2 eligible MRU evictions: 91.8 % 2.2 TiB
L2 ineligible evictions: 105.5 GiB

ARC total accesses: 473.5M
Total hits: 98.8 % 467.8M
Total I/O hits: < 0.1 % 58.0k
Total misses: 1.2 % 5.6M

ARC demand data accesses: 5.4 % 25.5M
Demand data hits: 90.8 % 23.2M
Demand data I/O hits: 0.2 % 49.0k
Demand data misses: 9.0 % 2.3M

ARC demand metadata accesses: 93.7 % 443.8M
Demand metadata hits: 100.0 % 443.8M
Demand metadata I/O hits: < 0.1 % 295
Demand metadata misses: < 0.1 % 34.0k

ARC prefetch data accesses: 0.8 % 3.9M
Prefetch data hits: 15.7 % 613.0k
Prefetch data I/O hits: < 0.1 % 785
Prefetch data misses: 84.3 % 3.3M

ARC prefetch metadata accesses: < 0.1 % 235.5k
Prefetch metadata hits: 92.9 % 218.8k
Prefetch metadata I/O hits: 3.4 % 8.0k
Prefetch metadata misses: 3.7 % 8.7k

ARC predictive prefetches: 99.6 % 4.1M
Demand hits after predictive: 66.6 % 2.7M
Demand I/O hits after predictive: 1.9 % 78.9k
Never demanded after predictive: 31.5 % 1.3M

ARC prescient prefetches: 0.4 % 15.1k
Demand hits after prescient: 98.5 % 14.8k
Demand I/O hits after prescient: 0.9 % 141
Never demanded after prescient: 0.6 % 85

ARC states hits of all accesses:
Most frequently used (MFU): 94.4 % 447.2M
Most recently used (MRU): 4.4 % 20.6M
Most frequently used (MFU) ghost: 0.1 % 317.8k
Most recently used (MRU) ghost: 0.1 % 289.5k
Uncached: 0.0 % 0

DMU predictive prefetcher calls: 16.3M
Stream hits: 42.2 % 6.9M
Hits ahead of stream: 28.0 % 4.6M
Hits behind stream: 10.4 % 1.7M
Stream misses: 19.3 % 3.1M
Streams limit reached: 13.6 % 426.6k
Stream strides: 220.7k
Prefetches issued 4.0M

L2ARC not detected, skipping section

Solaris Porting Layer (SPL):
spl_hostid 0
spl_hostid_path /etc/hostid
spl_kmem_alloc_max 16777216
spl_kmem_alloc_warn 65536
spl_kmem_cache_kmem_threads 4
spl_kmem_cache_magazine_size 0
spl_kmem_cache_max_size 32
spl_kmem_cache_obj_per_slab 8
spl_kmem_cache_slab_limit 16384
spl_panic_halt 1
spl_schedule_hrtimeout_slack_us 0
spl_taskq_kick 0
spl_taskq_thread_bind 0
spl_taskq_thread_dynamic 1
spl_taskq_thread_priority 1
spl_taskq_thread_sequential 4
spl_taskq_thread_timeout_ms 5000

Tunables:
brt_zap_default_bs 12
brt_zap_default_ibs 12
brt_zap_prefetch 1
dbuf_cache_hiwater_pct 10
dbuf_cache_lowater_pct 10
dbuf_cache_max_bytes 18446744073709551615
dbuf_cache_shift 5
dbuf_metadata_cache_max_bytes 18446744073709551615
dbuf_metadata_cache_shift 6
dbuf_mutex_cache_shift 0
ddt_zap_default_bs 15
ddt_zap_default_ibs 15
dmu_ddt_copies 0
dmu_object_alloc_chunk_shift 7
dmu_prefetch_max 134217728
icp_aes_impl cycle [fastest] generic x86_64 aesni
icp_gcm_avx_chunk_size 32736
icp_gcm_impl cycle [fastest] avx generic pclmulqdq
ignore_hole_birth 1
l2arc_exclude_special 0
l2arc_feed_again 1
l2arc_feed_min_ms 200
l2arc_feed_secs 1
l2arc_headroom 8
l2arc_headroom_boost 200
l2arc_meta_percent 33
l2arc_mfuonly 0
l2arc_noprefetch 1
l2arc_norw 0
l2arc_rebuild_blocks_min_l2size 1073741824
l2arc_rebuild_enabled 1
l2arc_trim_ahead 0
l2arc_write_boost 33554432
l2arc_write_max 33554432
metaslab_aliquot 1048576
metaslab_bias_enabled 1
metaslab_debug_load 0
metaslab_debug_unload 0
metaslab_df_max_search 16777216
metaslab_df_use_largest_segment 0
metaslab_force_ganging 16777217
metaslab_force_ganging_pct 3
metaslab_fragmentation_factor_enabled 1
metaslab_lba_weighting_enabled 1
metaslab_preload_enabled 1
metaslab_preload_limit 10
metaslab_preload_pct 50
metaslab_unload_delay 32
metaslab_unload_delay_ms 600000
raidz_expand_max_copy_bytes 167772160
raidz_expand_max_reflow_bytes 0
raidz_io_aggregate_rows 4
send_holes_without_birth_time 1
spa_asize_inflation 24
spa_config_path /etc/zfs/zpool.cache
spa_cpus_per_allocator 4
spa_load_print_vdev_tree 0
spa_load_verify_data 1
spa_load_verify_metadata 1
spa_load_verify_shift 4
spa_num_allocators 4
spa_slop_shift 5
spa_upgrade_errlog_limit 0
vdev_file_logical_ashift 9
vdev_file_physical_ashift 9
vdev_removal_max_span 32768
vdev_validate_skip 0
zap_iterate_prefetch 1
zap_micro_max_size 131072
zap_shrink_enabled 1
zfetch_hole_shift 2
zfetch_max_distance 67108864
zfetch_max_idistance 67108864
zfetch_max_reorder 16777216
zfetch_max_sec_reap 2
zfetch_max_streams 8
zfetch_min_distance 4194304
zfetch_min_sec_reap 1
zfs_abd_scatter_enabled 1
zfs_abd_scatter_max_order 13
zfs_abd_scatter_min_size 1536
zfs_active_allocator dynamic
zfs_admin_snapshot 0
zfs_allow_redacted_dataset_mount 0
zfs_arc_average_blocksize 8192
zfs_arc_dnode_limit 0
zfs_arc_dnode_limit_percent 10
zfs_arc_dnode_reduce_percent 10
zfs_arc_evict_batch_limit 10
zfs_arc_eviction_pct 200
zfs_arc_grow_retry 0
zfs_arc_lotsfree_percent 10
zfs_arc_max 0
zfs_arc_meta_balance 500
zfs_arc_min 0
zfs_arc_min_prefetch_ms 0
zfs_arc_min_prescient_prefetch_ms 0
zfs_arc_pc_percent 300
zfs_arc_prune_task_threads 1
zfs_arc_shrink_shift 0
zfs_arc_shrinker_limit 0
zfs_arc_shrinker_seeks 2
zfs_arc_sys_free 0
zfs_async_block_max_blocks 18446744073709551615
zfs_autoimport_disable 1
zfs_bclone_enabled 1
zfs_bclone_wait_dirty 0
zfs_blake3_impl cycle [fastest] generic sse2 sse41 avx2 avx512
zfs_btree_verify_intensity 0
zfs_checksum_events_per_second 20
zfs_commit_timeout_pct 10
zfs_compressed_arc_enabled 1
zfs_condense_indirect_commit_entry_delay_ms 0
zfs_condense_indirect_obsolete_pct 25
zfs_condense_indirect_vdevs_enable 1
zfs_condense_max_obsolete_bytes 1073741824
zfs_condense_min_mapping_bytes 131072
zfs_dbgmsg_enable 1
zfs_dbgmsg_maxsize 4194304
zfs_dbuf_state_index 0
zfs_ddt_data_is_special 1
zfs_deadman_checktime_ms 60000
zfs_deadman_enabled 1
zfs_deadman_events_per_second 1
zfs_deadman_failmode wait
zfs_deadman_synctime_ms 600000
zfs_deadman_ziotime_ms 300000
zfs_dedup_log_flush_entries_min 1000
zfs_dedup_log_flush_flow_rate_txgs 10
zfs_dedup_log_flush_min_time_ms 1000
zfs_dedup_log_flush_passes_max 8
zfs_dedup_log_mem_max 670174617
zfs_dedup_log_mem_max_percent 1
zfs_dedup_log_txg_max 8
zfs_dedup_prefetch 0
zfs_default_bs 9
zfs_default_ibs 15
zfs_delay_min_dirty_percent 60
zfs_delay_scale 500000
zfs_delete_blocks 20480
zfs_dirty_data_max 4294967296
zfs_dirty_data_max_max 4294967296
zfs_dirty_data_max_max_percent 25
zfs_dirty_data_max_percent 10
zfs_dirty_data_sync_percent 20
zfs_disable_ivset_guid_check 0
zfs_dmu_offset_next_sync 1
zfs_embedded_slog_min_ms 64
zfs_expire_snapshot 300
zfs_fallocate_reserve_percent 110
zfs_flags 0
zfs_fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3 avx2 avx512f avx512bw
zfs_free_bpobj_enabled 1
zfs_free_leak_on_eio 0
zfs_free_min_time_ms 1000
zfs_history_output_max 1048576
zfs_immediate_write_sz 32768
zfs_initialize_chunk_size 1048576
zfs_initialize_value 16045690984833335022
zfs_keep_log_spacemaps_at_export 0
zfs_key_max_salt_uses 400000000
zfs_livelist_condense_new_alloc 0
zfs_livelist_condense_sync_cancel 0
zfs_livelist_condense_sync_pause 0
zfs_livelist_condense_zthr_cancel 0
zfs_livelist_condense_zthr_pause 0
zfs_livelist_max_entries 500000
zfs_livelist_min_percent_shared 75
zfs_lua_max_instrlimit 100000000
zfs_lua_max_memlimit 104857600
zfs_max_async_dedup_frees 100000
zfs_max_dataset_nesting 50
zfs_max_log_walking 5
zfs_max_logsm_summary_length 10
zfs_max_missing_tvds 0
zfs_max_nvlist_src_size 0
zfs_max_recordsize 16777216
zfs_metaslab_find_max_tries 100
zfs_metaslab_fragmentation_threshold 70
zfs_metaslab_max_size_cache_sec 3600
zfs_metaslab_mem_limit 25
zfs_metaslab_segment_weight_enabled 1
zfs_metaslab_switch_threshold 2
zfs_metaslab_try_hard_before_gang 0
zfs_mg_fragmentation_threshold 95
zfs_mg_noalloc_threshold 0
zfs_min_metaslabs_to_flush 1
zfs_multihost_fail_intervals 10
zfs_multihost_history 0
zfs_multihost_import_intervals 20
zfs_multihost_interval 1000
zfs_multilist_num_sublists 0
zfs_no_scrub_io 0
zfs_no_scrub_prefetch 0
zfs_nocacheflush 0
zfs_nopwrite_enabled 1
zfs_object_mutex_size 64
zfs_obsolete_min_time_ms 500
zfs_override_estimate_recordsize 0
zfs_pd_bytes_max 52428800
zfs_per_txg_dirty_frees_percent 30
zfs_prefetch_disable 0
zfs_read_history 0
zfs_read_history_hits 0
zfs_rebuild_max_segment 1048576
zfs_rebuild_scrub_enabled 1
zfs_rebuild_vdev_limit 67108864
zfs_reconstruct_indirect_combinations_max 4096
zfs_recover 0
zfs_recv_best_effort_corrective 0
zfs_recv_queue_ff 20
zfs_recv_queue_length 16777216
zfs_recv_write_batch_size 1048576
zfs_removal_ignore_errors 0
zfs_removal_suspend_progress 0
zfs_remove_max_segment 16777216
zfs_resilver_disable_defer 0
zfs_resilver_min_time_ms 3000
zfs_scan_blkstats 0
zfs_scan_checkpoint_intval 7200
zfs_scan_fill_weight 3
zfs_scan_ignore_errors 0
zfs_scan_issue_strategy 0
zfs_scan_legacy 0
zfs_scan_max_ext_gap 2097152
zfs_scan_mem_lim_fact 20
zfs_scan_mem_lim_soft_fact 20
zfs_scan_report_txgs 0
zfs_scan_strict_mem_lim 0
zfs_scan_suspend_progress 0
zfs_scan_vdev_limit 16777216
zfs_scrub_after_expand 1
zfs_scrub_error_blocks_per_txg 4096
zfs_scrub_min_time_ms 1000
zfs_send_corrupt_data 0
zfs_send_no_prefetch_queue_ff 20
zfs_send_no_prefetch_queue_length 1048576
zfs_send_queue_ff 20
zfs_send_queue_length 16777216
zfs_send_unmodified_spill_blocks 1
zfs_sha256_impl cycle [fastest] generic x64 ssse3 avx avx2 shani
zfs_sha512_impl cycle [fastest] generic x64 avx avx2
zfs_slow_io_events_per_second 20
zfs_snapshot_history_enabled 1
zfs_spa_discard_memory_limit 16777216
zfs_special_class_metadata_reserve_pct 25
zfs_sync_pass_deferred_free 2
zfs_sync_pass_dont_compress 8
zfs_sync_pass_rewrite 2
zfs_traverse_indirect_prefetch_limit 32
zfs_trim_extent_bytes_max 134217728
zfs_trim_extent_bytes_min 32768
zfs_trim_metaslab_skip 0
zfs_trim_queue_limit 10
zfs_trim_txg_batch 32
zfs_txg_history 100
zfs_txg_timeout 5
zfs_unflushed_log_block_max 131072
zfs_unflushed_log_block_min 1000
zfs_unflushed_log_block_pct 400
zfs_unflushed_log_txg_max 1000
zfs_unflushed_max_mem_amt 1073741824
zfs_unflushed_max_mem_ppm 1000
zfs_unlink_suspend_progress 0
zfs_user_indirect_is_special 1
zfs_vdev_aggregation_limit 1048576
zfs_vdev_aggregation_limit_non_rotating 131072
zfs_vdev_async_read_max_active 3
zfs_vdev_async_read_min_active 1
zfs_vdev_async_write_active_max_dirty_percent 60
zfs_vdev_async_write_active_min_dirty_percent 30
zfs_vdev_async_write_max_active 10
zfs_vdev_async_write_min_active 2
zfs_vdev_def_queue_depth 32
zfs_vdev_default_ms_count 200
zfs_vdev_default_ms_shift 29
zfs_vdev_disk_classic 0
zfs_vdev_disk_max_segs 0
zfs_vdev_failfast_mask 1
zfs_vdev_initializing_max_active 1
zfs_vdev_initializing_min_active 1
zfs_vdev_max_active 1000
zfs_vdev_max_auto_ashift 14
zfs_vdev_max_ms_shift 34
zfs_vdev_min_auto_ashift 9
zfs_vdev_min_ms_count 16
zfs_vdev_mirror_non_rotating_inc 0
zfs_vdev_mirror_non_rotating_seek_inc 1
zfs_vdev_mirror_rotating_inc 0
zfs_vdev_mirror_rotating_seek_inc 5
zfs_vdev_mirror_rotating_seek_offset 1048576
zfs_vdev_ms_count_limit 131072
zfs_vdev_nia_credit 5
zfs_vdev_nia_delay 5
zfs_vdev_open_timeout_ms 1000
zfs_vdev_queue_depth_pct 1000
zfs_vdev_raidz_impl cycle [fastest] original scalar sse2 ssse3 avx2 avx512f avx512bw
zfs_vdev_read_gap_limit 32768
zfs_vdev_rebuild_max_active 3
zfs_vdev_rebuild_min_active 1
zfs_vdev_removal_max_active 2
zfs_vdev_removal_min_active 1
zfs_vdev_scheduler unused
zfs_vdev_scrub_max_active 3
zfs_vdev_scrub_min_active 1
zfs_vdev_sync_read_max_active 10
zfs_vdev_sync_read_min_active 10
zfs_vdev_sync_write_max_active 10
zfs_vdev_sync_write_min_active 10
zfs_vdev_trim_max_active 2
zfs_vdev_trim_min_active 1
zfs_vdev_write_gap_limit 4096
zfs_vnops_read_chunk_size 1048576
zfs_wrlog_data_max 8589934592
zfs_xattr_compat 0
zfs_zevent_len_max 512
zfs_zevent_retain_expire_secs 900
zfs_zevent_retain_max 2000
zfs_zil_clean_taskq_maxalloc 1048576
zfs_zil_clean_taskq_minalloc 1024
zfs_zil_clean_taskq_nthr_pct 100
zfs_zil_saxattr 1
zil_maxblocksize 131072
zil_maxcopied 7680
zil_nocacheflush 0
zil_replay_disable 0
zil_slog_bulk 67108864
zio_deadman_log_all 0
zio_dva_throttle_enabled 1
zio_requeue_io_start_cut_in_line 1
zio_slow_io_ms 30000
zio_taskq_batch_pct 80
zio_taskq_batch_tpq 0
zio_taskq_read fixed,1,8 null scale null
zio_taskq_write sync null scale null
zio_taskq_write_tpq 16
zstd_abort_size 131072
zstd_earlyabort_pass 1
zvol_blk_mq_blocks_per_thread 8
zvol_blk_mq_queue_depth 128
zvol_enforce_quotas 1
zvol_inhibit_dev 0
zvol_major 230
zvol_max_discard_blocks 16384
zvol_num_taskqs 0
zvol_open_timeout_ms 1000
zvol_prefetch_bytes 131072
zvol_request_sync 0
zvol_threads 0
zvol_use_blk_mq 0
zvol_volmode 2

ZIL committed transactions: 6.8M
Commit requests: 233.1k
Flushes to stable storage: 232.7k
Transactions to SLOG storage pool: 0 Bytes 0
Transactions to non-SLOG storage pool: 6.8 GiB 206.4k

I have two VDEVs, ‘archive’ as mentioned, and ‘scratch’ with is four 500 GB SSDs with no parity, used for stuff I don’t mind losing.

Maybe the Google TV box lacks the hardware to deal with a 68GB mkv.
Do you have access to more powerful hardware you can try playing the file, as a comparison? Also using VLC and accessing the share in the same manner.

This would help rule out some issues at the source.

I think you are right. VLC running on my M3 MacBook Pro via a 4K monitor has no trouble. Also, the video is 2160, whereas my TV is 1080, so I guess VLC on Google TV has extra work to do, with too few resources. Both are running off the same SMB share.

@neofusion Thanks for the good advice.

Transcoding is a paid option in Plex, requiring their “Plex Pass” licensing; however, the free and open source alternative Jellyfin does not require this. Just food for thought. :wink:

2 Likes

I would not mind paying for PLEX if it did not have such astonishingly bad user experience. I will have to try Jellyfin to see if I can get it to work.

Okay, finally figured out why I was having so much trouble getting Plex and Jellyfin to work… Will ponder the best way to describe this in another post.

I am having a similar problem and would love to know what you found.