Supermicro A2SDi-8C-HLN4F not reaching higher C-States

thomas-hn · October 11, 2024, 1:02pm

Hello,

my system has a Supermicro A2SDi-8C-HLN4F, an Intel X710-DA2 in the PCIe slot and drives connected to the onboard SATA connectors (SATA plugs + mini-SAS plugs) via a Supermicro BPN-SAS3-743A backplane (this backplane is not much more than a bunch of wires - not containing any controller).

When running powertop no higher C-States are reached.

           Pkg(HW)  |            Core(HW) |            CPU(OS) 0
                    |                     | C0 active  16.0%
                    |                     | POLL        0.1%    0.0 ms
                    |                     | C1          0.0%    0.0 ms
C2 (pc2)    0.0%    |                     | C1E        76.4%    0.7 ms
C3 (pc3)    0.0%    | C3 (cc3)    0.0%    |
C6 (pc6)    0.0%    | C6 (cc6)    0.0%    | C6          0.0%    0.0 ms
C7 (pc7)    0.0%    | C7 (cc7)    0.0%    |
C8 (pc8)    0.0%    |                     |
C9 (pc9)    0.0%    |                     |
C10 (pc10)  0.0%    |                     |

                    |            Core(HW) |            CPU(OS) 1
                    |                     | C0 active  15.2%
                    |                     | POLL        0.1%    0.0 ms
                    |                     | C1          0.0%    0.0 ms
                    |                     | C1E        75.5%    0.7 ms
                    | C3 (cc3)    0.0%    |
                    | C6 (cc6)    0.0%    | C6          0.0%    0.0 ms
                    | C7 (cc7)    0.0%    |
                    |                     |
                    |                     |
                    |                     |
...

I tried to run powertop --auto-tune without any effect.

When checking via lspci I can see that some devices do not support ASPM.

sudo lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM |Disabled;)'
00:06.0 PCI bridge: Intel Corporation Atom Processor C3000 Series Integrated QAT Root Port (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:09.0 PCI bridge: Intel Corporation Atom Processor C3000 Series PCI Express Root Port #0 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #9, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:10.0 PCI bridge: Intel Corporation Atom Processor C3000 Series PCI Express Root Port #6 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #16, Speed 8GT/s, Width x2, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
00:11.0 PCI bridge: Intel Corporation Atom Processor C3000 Series PCI Express Root Port #7 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
00:16.0 PCI bridge: Intel Corporation Atom Processor C3000 Series Integrated LAN Root Port #0 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:17.0 PCI bridge: Intel Corporation Atom Processor C3000 Series Integrated LAN Root Port #1 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
01:00.0 Co-processor: Intel Corporation Atom Processor C3000 Series QuickAssist Technology (rev 11)
                LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
07:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
07:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
09:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
09:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+

The devices are:

00:10.0 Intel Corporation Atom Processor C3000 Series PCI Express Root Port #6
00:11.0PCI bridge: Intel Corporation Atom Processor C3000 Series PCI Express Root Port #7
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge

Has someone an idea why those onboard devices of the A2SDi-8C-HLN4F report to not support ASPM and might be the root cause to not reach higher C-States?

My current assumption is that it might have something to do with the onboard SATA ports. The Link Power Management (LPM) of those ports is disabled in the BIOS. Could this be the reason for the issue?

Would it be safe to enable LPM in the BIOS? I read a bit about LPM and the general information was that it might cause harddrives to spin-down regularly if they do not properly support the feature.

Do you have any ideas what is blocking my system from reaching higher C-States and how I could solve it?

My BIOS configuration is attached as screenshots. Please let me know if there is something I should optimize.

Thanks a lot in advance,

Thomas

thomas-hn · October 12, 2024, 7:16am

Is really noone with experience on this topic here?

etorix · October 12, 2024, 8:26am

I admit I have not looked into that at all in my (CORE) sytems on account that C3758 is low poser enough and that the spinning drives use more power than the SoC anyway.

awasb · October 12, 2024, 8:57am

I’m running the exact same board as a test system (see sig) with CORE (powerd enabled). It consumes about 3 W less on average (26/27 vs. 29/30) with lower spikes (36 vs. 40) compared to Scale (DF 24.04.2 tested). No low c-states either. BIOS settings all the same. Bare ZFS filer. No vms activated.

The cpupower output with linux-kernel was:

admin@truenas ~ % sudo cpupower idle-info                                                     [1]
CPUidle driver: intel_idle
CPUidle governor: menu
analyzing CPU 0:

Number of idle states: 4
Available idle states: POLL C1 C1E C6
POLL:
Flags/Description: CPUIDLE CORE POLL IDLE
Latency: 0
Usage: 1538144
Duration: 36722164
C1 (DISABLED) :
Flags/Description: MWAIT 0x00
Latency: 2
Usage: 0
Duration: 0
C1E:
Flags/Description: MWAIT 0x01
Latency: 10
Usage: 81692503
Duration: 474475939689
C6 (DISABLED) :
Flags/Description: MWAIT 0x20
Latency: 50
Usage: 0
Duration: 0

BSD uses C1E. Frequency scaling works fine on both. My guess is, that lower c-states don’t get used, because the system is never really idle. They are not disabled on the kernel side AFAICS.

Okedokey · October 12, 2024, 1:46pm

tried with adaptive throttling disabled?

etorix · October 12, 2024, 4:53pm

It is also possible that higher C states are simply not implemented in BIOS. Supermicro is known for not letting its motherboards sleep.

thomas-hn · October 12, 2024, 5:12pm

Do you mean for the “TM2 Mode” in the BIOS?
The options for the TM2 Mode are “LFM throttling” and “Adaptive Throttling”.
What I found on Difference between LFM and adaptive throttling for TM2 thermal management states that this only configures “what the CPU should do when overheating”.

awasb · October 12, 2024, 5:18pm

@etorix The option for defining „upper“ (better: deep down) c-state levels is there. In the a2sdi’s BIOS. It‘s just not used. I guess it depends on package vs. core scaling (and maybe the scheduler), too.

awasb · October 12, 2024, 5:21pm

What is your system’s power draw at the moment?

thomas-hn · October 12, 2024, 5:55pm

The system draws at the moment around 88 watts from the socket.
This should be roughly:

42 Watt for 7 HDDs
6 Watt for 2 SSDs
4 Watt for the idling Intel X710-DA2
8 Watt for 7 fans
If the power supply has 85% efficiency at this low load, the power consumption of the power supply is around 13 Watt.
This means that the motherboard (incl. CPU) are drawing around 15 Watt at the load attached in the screenshot.

I think the overall efficiency of the Atom system is absolutely fine and nothing to worry about, but reaching higher C-States would be nice (and by the way - I would like to understand why it is not possible).

joeschmuck · October 12, 2024, 5:56pm

I do not have that motherboard but I will offer a suggestion, even if it does not work. Have you fooled around with: processor.max_cstate=X and intel_idle.max_cstate=0 at all?

This should give you the max C state you can reach:

cat /sys/module/intel_idle/parameters/max_cstate

I realize you are already running in C State C1E 75% of the time and while I know it is not C6, with respect to real world impact, that has got to be extremely minor. But if this is just to gain some knowledge as many of us do, not because it is practical, that is cool and I hope you find the answer.

thomas-hn · October 12, 2024, 6:10pm

This gives me a 9.

awasb · October 12, 2024, 7:02pm

15 Watts are fantastic IMHO. Don’t forget the IPMI-chip; it takes another 7 to 8 Watts.

Edith forgot: According to JEDEC specification every memory module adds another 3 Watts (1.2V * 2.5A). So …

thomas-hn · October 13, 2024, 9:12pm

I did some more testing.
Attached you can see the CPU Load with the Virtual Machines on SCALE running and with the VMs shutdown.

As you can see, with the VMs running the CPU only reaches C1E for around 72%. With the VMs shut-down, the system reaches C1E for 92% and also C2 for 23%.

I am still wondering how this “light load” on the system (on average the CPU is idle for around 80%) can prevent higher C-States. Isn’t there still a lot of idle time on the CPU in this case?

kai-uwe · December 8, 2024, 12:18pm

With cpupower idle-set -E I was able to enable at least up to the C6 state. You may check before/after with cpupower idle-info.

Jaywalker · December 8, 2024, 9:39pm

Did you measure power consumption of the system before and after enabling higher C states? When I tried this on my A2SDI board, it did not make any difference in power consumption, so I guess the default setting that only goes to C2 makes sense.

kai-uwe · December 13, 2024, 6:08am

I did not measure any significant difference, but there are several other devices included in the power measurement aswell. Maybe CPU temperature is slightly lower with enabled C-states.