Modify

Opened 3 years ago

Closed 13 months ago

#14453 closed defect (fixed)

lantiq ar9/vr9 insmod memory leak

Reported by: fld@… Owned by: developers
Priority: high Milestone: Chaos Calmer 15.05
Component: kernel Version: Trunk
Keywords: lantiq, ar9, vr9, xway, xrx200, insmod, memory leak Cc:

Description

Trying to insmod a kernel module that is already loaded causes varying size (size depending on the module being loaded) kmallocs to be created that do not get kfree'd

Let me demonstrate:

root@OpenWrt:/etc# lsmod
arc4                    1360  0
ath                    16441  3 ath9k
ath9k                 103241  0
ath9k_common            1563  1 ath9k
ath9k_hw              380748  2 ath9k
atm                    37426  4 ltq_atm_vr9
br2684                  6848  1
cfg80211              179761  3 ath9k
compat                   651  5 ath9k
crc_ccitt               1019  0
crypto_blkcipher       10695  2 ltq_deu_vr9
drv_dsl_cpe_api       149208  0
drv_ifxos              14874  2 drv_dsl_cpe_api
drv_mei_cpe            97192  2 ltq_atm_vr9
gpio_button_hotplug     4784  0
ip_tables               9421  4 iptable_nat
ipt_MASQUERADE          1072  1
ipt_REJECT              1840  2
iptable_filter           672  1
iptable_mangle           912  1
iptable_nat             1696  1
iptable_raw              640  1
ipv6                  261526 12
ledtrig_usbdev          1952  0
ltq_atm_vr9            29117  1
ltq_deu_vr9            26368  2
ltq_hcd_vr9            47416  0
mac80211              334664  1 ath9k
nf_conntrack           46731 12 iptable_nat
nf_conntrack_ftp        5312  1 nf_nat_ftp
nf_conntrack_ipv4       4772  4
nf_conntrack_irc        2832  1 nf_nat_irc
nf_defrag_ipv4           742  1 nf_conntrack_ipv4
nf_nat                  9679  7 iptable_nat
nf_nat_ftp              1248  0
nf_nat_ipv4             2478  1 iptable_nat
nf_nat_irc               928  0
x_tables               11061 20 iptable_raw
xt_CT                   2656  0
xt_LOG                  6432  0
xt_REDIRECT             1104  0
xt_TCPMSS               2032  1
xt_comment               480 19
xt_conntrack            2144  3
xt_limit                1344  1
xt_mac                   608  0
xt_mark                  672  0
xt_multiport            1184  0
xt_nat                  1088  0
xt_state                 704  0
xt_tcpudp               1680  4
xt_time                 1760  0

root@OpenWrt:/etc# cat /proc/meminfo
MemTotal:          61184 kB
MemFree:           37936 kB
Buffers:               0 kB
Cached:            12988 kB
SwapCached:            0 kB
Active:             6236 kB
Inactive:           7956 kB
Active(anon):       1216 kB
Inactive(anon):       24 kB
Active(file):       5020 kB
Inactive(file):     7932 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          1224 kB
Mapped:             1168 kB
Shmem:                36 kB
Slab:               4856 kB
SReclaimable:        924 kB
SUnreclaim:         3932 kB
KernelStack:         248 kB
PageTables:          168 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       30592 kB
Committed_AS:       9332 kB
VmallocTotal:    1048372 kB
VmallocUsed:        1072 kB
VmallocChunk:    1037840 kB

root@OpenWrt:/etc# for i in `seq 50`; do /sbin/insmod mac80211 2>/dev/null; done

root@OpenWrt:/etc# cat /proc/meminfo
MemTotal:          61184 kB
MemFree:           32716 kB
Buffers:               0 kB
Cached:            12988 kB
SwapCached:            0 kB
Active:             6236 kB
Inactive:           7956 kB
Active(anon):       1216 kB
Inactive(anon):       24 kB
Active(file):       5020 kB
Inactive(file):     7932 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          1224 kB
Mapped:             1168 kB
Shmem:                36 kB
Slab:               8092 kB
SReclaimable:        924 kB
SUnreclaim:         7168 kB
KernelStack:         248 kB
PageTables:          168 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       30592 kB
Committed_AS:       9332 kB
VmallocTotal:    1048372 kB
VmallocUsed:        3272 kB
VmallocChunk:    1013840 kB

root@OpenWrt:/etc# for i in `seq 50`; do /sbin/insmod mac80211 2>/dev/null; done

root@OpenWrt:/etc# cat /proc/meminfo
MemTotal:          61184 kB
MemFree:           27492 kB
Buffers:               0 kB
Cached:            12988 kB
SwapCached:            0 kB
Active:             6236 kB
Inactive:           7956 kB
Active(anon):       1216 kB
Inactive(anon):       24 kB
Active(file):       5020 kB
Inactive(file):     7932 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          1224 kB
Mapped:             1168 kB
Shmem:                36 kB
Slab:              11292 kB
SReclaimable:        924 kB
SUnreclaim:        10368 kB
KernelStack:         248 kB
PageTables:          168 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       30592 kB
Committed_AS:       9332 kB
VmallocTotal:    1048372 kB
VmallocUsed:        5472 kB
VmallocChunk:    1011920 kB

With kmemleak debugger it looks like this:

unreferenced object 0x824ec000 (size 2048):
  comm "insmod", pid 1776, jiffies 67025 (age 1442.448s)
  hex dump (first 32 bytes):
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
  backtrace:
    [<8009c494>] __kmalloc+0x1c0/0x224
    [<800196b8>] module_frob_arch_sections+0xc0/0x120
    [<8005b480>] load_module+0x448/0x1e68
    [<8005cf68>] SyS_init_module+0xc8/0xd0
    [<80008144>] stack_done+0x20/0x40
unreferenced object 0x8265f800 (size 2048):
  comm "insmod", pid 2700, jiffies 209747 (age 871.564s)
  hex dump (first 32 bytes):
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
  backtrace:
    [<8009c494>] __kmalloc+0x1c0/0x224
    [<800196b8>] module_frob_arch_sections+0xc0/0x120
    [<8005b480>] load_module+0x448/0x1e68
    [<8005cf68>] SyS_init_module+0xc8/0xd0
    [<80008144>] stack_done+0x20/0x40
unreferenced object 0x82666800 (size 2048):
  comm "insmod", pid 2896, jiffies 240012 (age 750.504s)
  hex dump (first 32 bytes):
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
  backtrace:
    [<8009c494>] __kmalloc+0x1c0/0x224
    [<800196b8>] module_frob_arch_sections+0xc0/0x120
    [<8005b480>] load_module+0x448/0x1e68
    [<8005cf68>] SyS_init_module+0xc8/0xd0
    [<80008144>] stack_done+0x20/0x40
unreferenced object 0x8255ac00 (size 512):
  comm "insmod", pid 2917, jiffies 244022 (age 734.464s)
  hex dump (first 32 bytes):
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
  backtrace:
    [<8009c494>] __kmalloc+0x1c0/0x224
    [<800196b8>] module_frob_arch_sections+0xc0/0x120
    [<8005b480>] load_module+0x448/0x1e68
    [<8005cf68>] SyS_init_module+0xc8/0xd0
    [<80008144>] stack_done+0x20/0x40
unreferenced object 0x8266e000 (size 8192):
  comm "insmod", pid 2922, jiffies 244104 (age 734.136s)
  hex dump (first 32 bytes):
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
  backtrace:
    [<8009c494>] __kmalloc+0x1c0/0x224
    [<800196b8>] module_frob_arch_sections+0xc0/0x120
    [<8005b480>] load_module+0x448/0x1e68
    [<8005cf68>] SyS_init_module+0xc8/0xd0
    [<80008144>] stack_done+0x20/0x40
unreferenced object 0x82bc0680 (size 256):
  comm "insmod", pid 2923, jiffies 244243 (age 733.580s)
  hex dump (first 32 bytes):
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
  backtrace:
    [<8009c494>] __kmalloc+0x1c0/0x224
    [<800196b8>] module_frob_arch_sections+0xc0/0x120
    [<8005b480>] load_module+0x448/0x1e68
    [<8005cf68>] SyS_init_module+0xc8/0xd0
    [<80008144>] stack_done+0x20/0x40
unreferenced object 0x826b3800 (size 2048):
  comm "insmod", pid 2924, jiffies 244344 (age 733.176s)
  hex dump (first 32 bytes):
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
    5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
  backtrace:
    [<8009c494>] __kmalloc+0x1c0/0x224
    [<800196b8>] module_frob_arch_sections+0xc0/0x120
    [<8005b480>] load_module+0x448/0x1e68
    [<8005cf68>] SyS_init_module+0xc8/0xd0
    [<80008144>] stack_done+0x20/0x40
unreferenced object 0x8259ee00 (size 512):

System info:

root@OpenWrt:/etc# cat /proc/version
Linux version 3.10.18 (fld@HA) (gcc version 4.6.4 (OpenWrt/Linaro GCC 4.6-2013.05 r38578) ) #31 Sun Nov 10 10:56:34 EET 2013

root@OpenWrt:/etc# cat /proc/cpuinfo
system type             : VR9 rev 1.2
machine                 : TDW8970 - TP-LINK TD-W8970
processor               : 0
cpu model               : MIPS 34Kc V5.6
BogoMIPS                : 332.54
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 16
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 4, address/irw mask: [0x0000, 0x0260, 0x0010, 0x0200]
isa                     : mips1 mips2 mips32r1 mips32r2
ASEs implemented        : mips16 dsp mt
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

Attachments (0)

Change History (8)

comment:1 Changed 3 years ago by Catalin Patulea <cat@…>

On MIPS, module_frob_arch_sections appears to come from source:trunk/target/linux/generic/patches-3.10/305-mips_module_reloc.patch:

+int module_frob_arch_sections(Elf_Ehdr *hdr, Elf_Shdr *sechdrs,
+                             char *secstrings, struct module *mod)
+{
+       unsigned int symindex = 0;
+       unsigned int core_size, init_size;
+       int i;
[...]
+       mod->arch.phys_plt_tbl = __module_alloc(core_size + init_size, 1);
+       if (!mod->arch.phys_plt_tbl)
[...]
+       mod->arch.virt_plt_tbl = __module_alloc(core_size + init_size, 0);
+       if (!mod->arch.virt_plt_tbl) {

I've seen problems with the MIPS module loader on ar71xx with CONFIG_KALLSYMS_ALL=y (default is n). Memory corruption at load time due to some wrong offset calculations caused by the extra symbol sections. Maybe there are some larger bugs in this area.

No idea why your leak would only affect ar9. Seems like this patch would apply to all MIPS targets.

comment:2 Changed 3 years ago by Catalin Patulea <cat@…>

Do you get, for example, "insmod: can't insert 'ext4': File exists" when insmod'ing the second time?

If your mod is already loaded, add_unformed_module should veto the load before load_module even runs:
http://lxr.linux.no/linux+v3.10.9/kernel/module.c#L3222

comment:3 Changed 3 years ago by fld@…

Yes, insmod gives the can't insert: File exists" error, yet resources keep getting wasted every time.

comment:4 Changed 16 months ago by anonymous

The problem is that layout_and_allocate (which then calls module_frob_arch_sections) is done before add_unformed_module.

This also means that module_deallocate is called but it seems none of the weak functions called in by function is overwritten by this mips specific patch. The best candidate seems to be module_arch_freeing_init which should free the stuff similar to module_arch_cleanup.

comment:5 Changed 16 months ago by anonymous

Btw. first the change https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d453cded05ee219b77815ea194dc36efa5398bca must integrated to have the hook module_arch_freeing_init.

And it seems that the mod->arch.phys_plt_tbl + mod->arch.virt_plt_tbl free should be completely moved from module_arch_cleanup to module_arch_freeing_init

comment:6 Changed 16 months ago by anonymous

The patches I use can be found at

https://gist.githubusercontent.com/ecsv/222d824eb163d069820f/raw/05cb2585ed4ed4875061b7f9a7851d4b3c3a6563/311-module_arch_freeing_init-new-hook-for-archs-before-m.patch
https://gist.githubusercontent.com/ecsv/353cffef547c6eb45675/raw/ea0bd27ab184136d31f3704f99c9ccdd02783646/312-mips_module_dealloc.patch

You have to place them in the correct generic kernel patches folder like target/linux/generic/patches-3.18/

I have tested it with:

free -m
for i in `seq 1 100`; do 
insmod /lib/modules/*/sch_htb.ko >& /dev/null
insmod /lib/modules/*/cls_fw.ko >& /dev/null
insmod /lib/modules/*/cls_u32.ko >& /dev/null
insmod /lib/modules/*/act_mirred.ko >& /dev/null
done
free -m

The lost memory (before applying the patches) with each test (100 x 4 insmods) was a little bit over 2 MB of memory. This problem seems to be gone with the patches applied.

comment:7 Changed 15 months ago by anonymous

Complete patch was submitted: http://patchwork.ozlabs.org/patch/488475/

comment:8 Changed 13 months ago by nbd

  • Resolution set to fixed
  • Status changed from new to closed

fixed in r46247 (backported in r47004)

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.