TRB500: Out of memory: Kill process ipacm

[VitaliiK]

Hello All,

I’ve gotten 2 pairs of the “Out of memory: Kill process xxx (ipacm)” occurrences in last a few hours on my new TRB500 device.
The OOM-Killer invoked in ~2-3 hours after reboot…
The UI unable to handle modem/mobile connection after the ipacm kills.

Can you help me to solve this critical for me issue?

The part of syslog messages are below:

Apr 25 21:08:41 gateway-mgmt kernel: [ 9364.206020] ledman invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null),  order=0, oom_score_adj=0
...
Apr 25 21:08:41 gateway-mgmt kernel: [ 9364.532008] [  360]     0   360    33159    31508      67       0        0             0 ipacm
...
Apr 25 21:08:41 gateway-mgmt kernel: [ 9364.923602] Out of memory: Kill process 360 (ipacm) score 550 or sacrifice child
Apr 25 21:08:41 gateway-mgmt kernel: [ 9364.931952] Killed process 360 (ipacm) total-vm:132636kB, anon-rss:126032kB, file-rss:0kB, shmem-rss:0kB
Apr 25 21:08:41 gateway-mgmt kernel: [ 9365.055179] oom_reaper: reaped process 360 (ipacm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Apr 25 21:15:00 gateway-mgmt root: [ 9364.206020] ledman invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null),  order=0, oom_score_adj=0
...
Apr 25 21:15:00 gateway-mgmt root: [ 9364.532008] [  360]     0   360    33159    31508      67       0        0             0 ipacm
...
Apr 25 21:15:00 gateway-mgmt root: [ 9364.923602] Out of memory: Kill process 360 (ipacm) score 550 or sacrifice child
Apr 25 21:15:00 gateway-mgmt root: [ 9364.931952] Killed process 360 (ipacm) total-vm:132636kB, anon-rss:126032kB, file-rss:0kB, shmem-rss:0kB
Apr 25 21:15:00 gateway-mgmt root: [ 9365.055179] oom_reaper: reaped process 360 (ipacm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Apr 26 13:09:27 gateway-mgmt kernel: [ 8156.892369] dnsmasq invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null),  order=1, oom_score_adj=0
...
Apr 26 13:09:27 gateway-mgmt kernel: [ 8157.626599] [  366]     0   366    33299    31669      68       0        0             0 ipacm
...
Apr 26 13:09:28 gateway-mgmt kernel: [ 8158.428552] Out of memory: Kill process 366 (ipacm) score 552 or sacrifice child
Apr 26 13:09:28 gateway-mgmt kernel: [ 8158.429627] Killed process 366 (ipacm) total-vm:133196kB, anon-rss:126620kB, file-rss:0kB, shmem-rss:0kB
Apr 26 13:09:28 gateway-mgmt kernel: [ 8158.631602] oom_reaper: reaped process 366 (ipacm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 26 13:15:01 gateway-mgmt root: [ 8156.892369] dnsmasq invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null),  order=1, oom_score_adj=0
...
Apr 26 13:15:01 gateway-mgmt root: [ 8157.626599] [  366]     0   366    33299    31669      68       0        0             0 ipacm
...
Apr 26 13:15:01 gateway-mgmt root: [ 8158.428552] Out of memory: Kill process 366 (ipacm) score 552 or sacrifice child
Apr 26 13:15:01 gateway-mgmt root: [ 8158.429627] Killed process 366 (ipacm) total-vm:133196kB, anon-rss:126620kB, file-rss:0kB, shmem-rss:0kB
Apr 26 13:15:01 gateway-mgmt root: [ 8158.631602] oom_reaper: reaped process 366 (ipacm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Regards,
Vitalii

[VitaliiK]

Hot update:

gateway:~# uptime
 20:03:52 up 50 min,  load average: 3.29, 2.81, 2.55

Part of the top output:

Mem: 205388K used, 24180K free, 284K shrd, 8652K buff, 58116K cached
CPU:  54% usr  37% sys   0% nic   0% idle   0% io   0% irq   7% sirq
Load average: 2.96 2.69 2.51 3/266 20249
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  360     1 root     S    63740  28%  75% /usr/bin/ipacm
 1276     2 root     DW       0   0%   7% [kworker/u2:12]
 1273     2 root     DW       0   0%   4% [kworker/u2:11]
 6255     1 dnsmasq  S     1340   1%   1% /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid

[VitaliiK]

I’ve found the ipacm memory leaking cause: the /etc/data/ipa/IPACM_cfg.xml file was empty,
Issue disappeared when I copied that file content from other same device.
Unfortunately, I can’t find cause of that file truncation/clearing.