800% cpu usage on Nginx

你们好!
truenas scale中,我突然发现cpu占用率一直百分之五十以上,通过top命令查看到是nginx的进程在一直占用我的cpu。一开始我以为是我安装了NPM去反向代理我的其他服务,但是我直接卸载掉NPM应用之后,仍然是如此。

后来我联想到我开启了http的本地代理,如下图所示,后面去掉了之后仍然是nginx占用比较高的CPU。
看上去是把核心全都给使用了

image

请问有大神可以帮我看看是怎么回事吗?万分感谢!

hello!

In the truenas scale, I suddenly noticed that the CPU usage rate has been consistently above 50%. Through the top command, I found that it is nginx processes that have been occupying my CPU. At first, I thought I had installed NPM to reverse proxy my other services, but even after uninstalling the NPM application, it remained the same.

Later on, I remembered that I had enabled a local proxy for HTTP, as shown in the following picture. Even after removing it, nginx still consumes a relatively high amount of CPU.

It seems that the core has been fully utilized

thanks!

Actually, looking at this more.
Why is this nginx process running as admin and not root?

I think you might have picked up an XMR miner or similar. Can you kill the process and confirm whether you’re still able to access the WebUI?

我从外网通过https访问到家里的nas,如果我强行把这个进程相关的都kill掉,我从外网通过端口转发的方式就访问不了我的WebUI

目前我在外地,没有办法通过内网的方式去访问到我的nas设备

I access my home NAS through HTTPS from the external network. If I forcefully kill all the processes related to this process, I won’t be able to access my WebUI through port forwarding from the external network

I am currently out of town and unable to access my NAS device through the internal network

Oooh. Its not safe to expose the TrueNAS web port to the internet. Even if you are behind a reverse proxy.

1 Like

即使是https,也是不安全的吗?我现在还没开始大规模使用我的NAS,这种情况是不是只能重装一下Truenas Scale了呢?
Even HTTPS is not secure? I haven’t started using my NAS storage space on a large scale yet. In this situation, can I just reinstall Truenas Scale?

If you want to access your NAS from outside your LAN, the only safe way to do it is to use a VPN.

Tailscale, Wireguard, OpenVPN etc.

If you want to access web based services etc that are hosted on your NAS, it is not safe to port forward the internet facing web ports to a reverse proxy port that also proxies to internal web services… as they can then be accessed from the WAN.

Once you expose a web service to the internet it is exposed to the world… and the world is not a nice place for innocent little web ports.

I don’t know what is running on your system, it could just be a bug in nginx which is spinning.

Have you tried killing the process? Or restarting?

Are there any situations in which the nginx process doesn’t run as root? Considering it is running as a user process it’s probably something disguising itself as nginx :worried:

@cokezhu Do you run anything like qBittorrent with external access enabled? It is a very common attack vector.
I’d check crontab to see if you have anything unusual configured to start on boot. Probably also worth seeing if find /tmp/* | grep nginx turns up anything.

1 Like

thank you very much to teach me so lot.
Is it also unsafe for me to use npm to reverse proxy my Nextcloud so that I can access my Nextcloud application through domain+https from outside?

just nextcloud ,netdata, home assistant is installed on this system

i tried to kill this proccess, but it restarted a few seconds later.

Without properly configured access controls, it’s a risk to expose anything to the open internet, which is why VPN solutions are the go-to recommendation.

1 Like

no qbittorrent,just a few applications installed on the system
find nothing through the cmd
image

Are you able to confirm the process path via ll /proc/<PID>/exe (replacing <PID> with this nginx process PID)

看上去我是被这个.redtail给攻击了,我在网上搜了一下,这个是加密货币挖矿程序
It seems that I was attacked by this. reddmail. I searched online and found that it is a cryptocurrency mining program

2 Likes

Right on the money, I’m aware of RedTail.

This will have been deployed via something you’ve exposed to the internet with bad defaults.

Typical deployment for this is to add persistence via cron (crontab -e). Remove it, the executable, and lock down your remote access. Once you’re able to, I’d recommended reinstalling TrueNAS for good measure.

Note: Something I wasn’t aware of about this though was that it is also used to steal SSH credentials (thanks to the folks at Forescout for the writeup). This could allow lateral movement within your network, meaning TrueNAS could have been compromised from another device. I would also scrutinize any device you have used to connect to TN via SSH.

3 Likes

Its not the reverse proxy that is unsafe… its forwarding WAN facing web ports to the reverse proxy that is unsafe.

Ie, you should be using a VPN to access your LAN/NAS from outside your LAN.

Then you can access all the web ports on your NAS that you want without exposing your NAS’ web ports to the internet.

Ie, to access next cloud, first logon to your VPN… then access nextcloud.

The VPN software will even have clients for your phone/tablets.

2 Likes

i got it !thank you and love you so much! :+1:

thank you! i will do this after reinstall the truenas scale , love you too :clap: