Afserver crashes
Re: Afserver crashes
We have about 200 dedicated rendernodes and 40 workstations with 25 monitors. But as soon as I add the 40 workstations afserver starts to spit weird glibc double free errors.... We had trouble with afserver crashing before but it was never so obvious.
It's hard to debug in full production right now but I will try.
It's hard to debug in full production right now but I will try.
Re: Afserver crashes
Our server just crashed and I got this in 'dmesg':
Code: Select all
afserver[22218]: segfault at 7f88e4ff9d13 ip 00007f894a982e8f sp 00007f89420d8bc0 error 4 in libpthread-2.23.so[7f894a97b000+18000]
Re: Afserver crashes
Have same trouble with afserver 2.2.1 on openSUSE Leap 42.2(vm on vmware host). Crashes every 2-3 days about.
Libthreads become dead and afserver service unloaded...
Libthreads become dead and afserver service unloaded...
Re: Afserver crashes
Hi.
I noticed pthread bug on ubuntu16lts (gcc5.4.0) too.
On a test imitating very heady load.
But at work on ubuntu14tls (gcc4.8.4) i can`t reproduce it.
Almost needed threads afserver spawns at start (one thread to write files, one thread to write SQL and so on).
After the start the only reason to raise a thread is to read a socket of each incoming connection.
So it crashed here:
https://github.com/CGRU/cgru/blob/maste ... t.cpp#L222
But for now i do not why.
Also it seems that kernel&gcc versions plays some role (probability).
I will dig this issue deeper.
I noticed pthread bug on ubuntu16lts (gcc5.4.0) too.
On a test imitating very heady load.
But at work on ubuntu14tls (gcc4.8.4) i can`t reproduce it.
Almost needed threads afserver spawns at start (one thread to write files, one thread to write SQL and so on).
After the start the only reason to raise a thread is to read a socket of each incoming connection.
So it crashed here:
https://github.com/CGRU/cgru/blob/maste ... t.cpp#L222
But for now i do not why.
Also it seems that kernel&gcc versions plays some role (probability).
I will dig this issue deeper.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
Re: Afserver crashes
How do you imitate heavy load? I'm trying my best to get afanasy to crash on a test system, without any luck.
I'm trying to connect hundreds of dummy renders while adding jobs in an endless loop.
I'm trying to connect hundreds of dummy renders while adding jobs in an endless loop.
Re: Afserver crashes
Just launch 200 renders on the same host with afserver.
On ubuntu 16lts it craches in a minute.
On ubuntu 14lts i did not wait crash.
On ubuntu 16lts it craches in a minute.
On ubuntu 14lts i did not wait crash.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
Re: Afserver crashes
I launched about 1400 clients on Ubuntu 16.04 and it didn't crash.
This was using a self compiled version from latest git on my laptop. Did your test machine have SSD?
Re: Afserver crashes
No. My laptop has HDD.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
Re: Afserver crashes
did you get around to dig into this? It is getting annoying.