Is this still the case with the recent NSD 4.1.18 and later, which has improved memory usage (less memory usage) with 18% and with an aggressive option even 35%? It is a honest question, as we did not compare the memory usage of NSD, Knot and BIND.

For a first idea of usage I'd choose the "resident set size" measure - either VmRSS (current) or VmHWM (peak).  (I'm assuming no noticeable swapping happens here.)

For Knot-DNS itself, there have also been some improvements in memory usage since 2.3.4 IIRC, but I don't know details there (Dan should know best).

Hi Aleš, I would agree with Vladimír. Virtual memory consumption is not important. We should focus on physical memory consuption (VmRSS). But in your case, these values are almost equal. It's very strange! I did a quick measurement:                 TLD with DNSSEC | 10k zones with DNSSEC                 VIRT    RES    | VIRT  RES --------------------------------+----------------- Bind 9.12.1    656M    419M    | 1269M    152M Knot 2.6.5    4384M    360M    | 4588M    39440 NSD 4.1.20    124M    87216    | 147M    108M               1759M   1732M    | 115M    89124 Note1: I don't know how NSD work, so I printed 2 top level processes. Note2: Bind was compiled with --enable-threads. The virtual memory consumption by Knot can be significantly reduced by setting max-timer-db-size, max-journal-db-size, max-kasp-db-size, and --with-conf-mapsize configure option (if compiled from the source code). Best, Daniel On 2018-03-15 13:37, Aleš Rygl wrote: > Dear all, > > While trying to migrate or DNS to Knot I have noticed that a slave > server with 2GB RAM is facing memory exhaustion. I am running > 2.6.5-1+0~20180216080324.14+stretch~1.gbp257446. There is 141 zones > having around 1MB in total. Knot is acting as pure slave server with > minimal configuration. > > There is nearly 1.7GB of memory consumed by Knot on a freshly > rebooted server: > > root@eira:/proc/397# cat status > Name:   knotd > Umask:  0007 > State:  S (sleeping) > Tgid:   397 > Ngid:   0 > Pid:    397 > PPid:   1 > TracerPid:      0 > Uid:    108     108     108     108 > Gid:    112     112     112     112 > FDSize: 64 > Groups: 112 > NStgid: 397 > NSpid:  397 > NSpgid: 397 > NSsid:  397 > VmPeak: 24817520 kB > VmSize: 24687160 kB > VmLck:         0 kB > VmPin:         0 kB > VmHWM:   1743400 kB > VmRSS:   1743272 kB > RssAnon:         1737088 kB > RssFile:            6184 kB > RssShmem:              0 kB > VmData:  1781668 kB > VmStk:       132 kB > VmExe:       516 kB > VmLib:     11488 kB > VmPTE:      3708 kB > VmPMD:        32 kB > VmSwap:        0 kB > HugetlbPages:          0 kB > Threads:        21 > SigQ:   0/7929 > SigPnd: 0000000000000000 > ShdPnd: 0000000000000000 > SigBlk: fffffffe7bfbbefc > SigIgn: 0000000000000000 > SigCgt: 0000000180007003 > CapInh: 0000000000000000 > CapPrm: 0000000000000000 > CapEff: 0000000000000000 > CapBnd: 0000003fffffffff > CapAmb: 0000000000000000 > Seccomp:        0 > Cpus_allowed:   f > Cpus_allowed_list:      0-3 > Mems_allowed:   00000000,00000001 > Mems_allowed_list:      0 > voluntary_ctxt_switches:        260 > nonvoluntary_ctxt_switches:     316 > root@eira:/proc/397# > > Config: > > server: >     listen: 0.0.0.0@53 >     listen: ::@53 >     user: knot:knot > > log: >   - target: syslog >     any: info > mod-rrl: >   - id: rrl-10 >     rate-limit: 10    # Allow 200 resp/s for each flow >     slip: 2           # Every other response slips > > mod-stats: >   - id: custom >     edns-presence: on >     query-type: on >     request-protocol: on >     server-operation: on >     request-bytes: on >     response-bytes: on >     edns-presence: on >     flag-presence: on >     response-code: on >     reply-nodata: on >     query-type: on >     query-size: on >     reply-size: on > > > template: >   - id: default >     storage: "/var/lib/knot" >     module: mod-rrl/rrl-10 >     module: mod-stats/custom >     acl: [allowed_transfer] >     disable-any: on >     master: idunn > > I was pretty sure that a VM with 2GB RAM is enough for my setup :-) > > BR > > Ales

Aleš, On 03/16/2018 10:32 AM, Aleš Rygl wrote: Probably I don't understand. Knot consumed less physical memory than Bind during my test :-) To be honest, I don't understand why Knot is consuming such amount of memory in your case! Could you please try disabling the statistics module? Statistics per zone with all metrics consume some memory. Daniel > > Thanks for memory saving tips. > > BR > Ales > > > > On 15.3.2018 21:16, daniel.salzman(a)nic.cz wrote: >> Hi Aleš, >> >> I would agree with Vladimír. Virtual memory consumption is not >> important. We should focus on physical memory consuption (VmRSS). >> But in your case, these values are almost equal. It's very strange! >> >> I did a quick measurement: >> >>                 TLD with DNSSEC | 10k zones with DNSSEC >>                 VIRT    RES    | VIRT  RES >> --------------------------------+----------------- >> Bind 9.12.1    656M    419M    | 1269M    152M >> Knot 2.6.5    4384M    360M    | 4588M    39440 >> NSD 4.1.20    124M    87216    | 147M    108M >>               1759M   1732M    | 115M    89124 >> >> Note1: I don't know how NSD work, so I printed 2 top level processes. >> Note2: Bind was compiled with --enable-threads. >> >> The virtual memory consumption by Knot can be significantly reduced by >> setting max-timer-db-size, max-journal-db-size, max-kasp-db-size, and >> --with-conf-mapsize configure option (if compiled from the source code). >> >> Best, >> Daniel >> >> On 2018-03-15 13:37, Aleš Rygl wrote: >>> Dear all, >>> >>> While trying to migrate or DNS to Knot I have noticed that a slave >>> server with 2GB RAM is facing memory exhaustion. I am running >>> 2.6.5-1+0~20180216080324.14+stretch~1.gbp257446. There is 141 zones >>> having around 1MB in total. Knot is acting as pure slave server with >>> minimal configuration. >>> >>> There is nearly 1.7GB of memory consumed by Knot on a freshly rebooted server: >>> >>> root@eira:/proc/397# cat status >>> Name:   knotd >>> Umask:  0007 >>> State:  S (sleeping) >>> Tgid:   397 >>> Ngid:   0 >>> Pid:    397 >>> PPid:   1 >>> TracerPid:      0 >>> Uid:    108     108     108     108 >>> Gid:    112     112     112     112 >>> FDSize: 64 >>> Groups: 112 >>> NStgid: 397 >>> NSpid:  397 >>> NSpgid: 397 >>> NSsid:  397 >>> VmPeak: 24817520 kB >>> VmSize: 24687160 kB >>> VmLck:         0 kB >>> VmPin:         0 kB >>> VmHWM:   1743400 kB >>> VmRSS:   1743272 kB >>> RssAnon:         1737088 kB >>> RssFile:            6184 kB >>> RssShmem:              0 kB >>> VmData:  1781668 kB >>> VmStk:       132 kB >>> VmExe:       516 kB >>> VmLib:     11488 kB >>> VmPTE:      3708 kB >>> VmPMD:        32 kB >>> VmSwap:        0 kB >>> HugetlbPages:          0 kB >>> Threads:        21 >>> SigQ:   0/7929 >>> SigPnd: 0000000000000000 >>> ShdPnd: 0000000000000000 >>> SigBlk: fffffffe7bfbbefc >>> SigIgn: 0000000000000000 >>> SigCgt: 0000000180007003 >>> CapInh: 0000000000000000 >>> CapPrm: 0000000000000000 >>> CapEff: 0000000000000000 >>> CapBnd: 0000003fffffffff >>> CapAmb: 0000000000000000 >>> Seccomp:        0 >>> Cpus_allowed:   f >>> Cpus_allowed_list:      0-3 >>> Mems_allowed:   00000000,00000001 >>> Mems_allowed_list:      0 >>> voluntary_ctxt_switches:        260 >>> nonvoluntary_ctxt_switches:     316 >>> root@eira:/proc/397# >>> >>> Config: >>> >>> server: >>>     listen: 0.0.0.0@53 >>>     listen: ::@53 >>>     user: knot:knot >>> >>> log: >>>   - target: syslog >>>     any: info >>> mod-rrl: >>>   - id: rrl-10 >>>     rate-limit: 10    # Allow 200 resp/s for each flow >>>     slip: 2           # Every other response slips >>> >>> mod-stats: >>>   - id: custom >>>     edns-presence: on >>>     query-type: on >>>     request-protocol: on >>>     server-operation: on >>>     request-bytes: on >>>     response-bytes: on >>>     edns-presence: on >>>     flag-presence: on >>>     response-code: on >>>     reply-nodata: on >>>     query-type: on >>>     query-size: on >>>     reply-size: on >>> >>> >>> template: >>>   - id: default >>>     storage: "/var/lib/knot" >>>     module: mod-rrl/rrl-10 >>>     module: mod-stats/custom >>>     acl: [allowed_transfer] >>>     disable-any: on >>>     master: idunn >>> >>> I was pretty sure that a VM with 2GB RAM is enough for my setup :-) >>> >>> BR >>> >>> Ales

Aleš, On 03/16/2018 10:32 AM, Aleš Rygl wrote: Probably I don't understand. Knot consumed less physical memory than Bind during my test :-) To be honest, I don't understand why Knot is consuming such amount of memory in your case! Could you please try disabling the statistics module? Statistics per zone with all metrics consume some memory.

Hello Daniel to my big surprise, when mod-rrl is disabled, memory consumption is nearly negligible! root@eira:~# uptime  15:59:34 up 0 min,  1 user,  load average: 0.15, 0.03, 0.01 root@eira:~# root@eira:~# free -h               total        used        free      shared buff/cache   available Mem:           2.0G         54M        1.8G 2.9M         78M        1.8G Swap:            0B          0B          0B root@eira:~# root@eira:~# systemctl start knot root@eira:~# root@eira:~# free -h               total        used        free      shared buff/cache   available Mem:           2.0G         56M        1.8G 2.9M         81M        1.8G Swap:            0B          0B          0B root@eira:~# From proces status: VmPeak: 22952964 kB VmSize: 22952964 kB VmLck:         0 kB VmPin:         0 kB VmHWM:      9128 kB VmRSS:      8984 kB RssAnon:            2820 kB RssFile:            6164 kB RssShmem:              0 kB VmData:    47496 kB VmStk:       132 kB VmExe:       516 kB VmLib:     11488 kB VmPTE:       320 kB VmPMD:        24 kB VmSwap:        0 kB What do you think about it? BR Ales On 16.3.2018 15:47, Aleš Rygl wrote:

Hello Aleš, I have verified that one instance of the RRL module takes 12.5 MiB, as it contains a big hash table. So it's reasonable to configure it as a global-module in the default template. Also I have found insufficiently optimized memory allocation in the statistics module for disabled metrics. This will be improved in the next release. Daniel On 03/16/2018 04:12 PM, Daniel Salzman wrote: > Hi Aleš, > > Interesting :-) I will look at it! > > I think you could configure the modules (at least rrl) common for all zones, see https://www.knot-dns.cz/docs/2.6/singlehtml/index.html#global-module > >

Just some comments from a user: It is hard to guess if the consumed memory is something to care or not. I do not know if all the virtual memory may be used and some time may cause problems. If my monitoring tells me "100 % memory used" I am concerned, as I do not know if this is a real problem or just some irrelevant buffers. Also I do not know how the name server will behave when running out of memory (crash, serving old data, serving broken data, ...). Hence, I have to add more memory or choose a different name server. I have a server with limited memory. NSD was running out of memory. So I tried Knot. Knot consumed similar memory (from a non-developer view, not capably of analyzing virtual vs resistent vs ... memory). So, I switched the Bind and was happy because my monitoring now tells me "50% memory used" - and I am not worried about memory anymore. Sometimes I have time to investigate and analyze problems, but most of the time I do not have time for debugging and just need a solution for my problem, hence I still use Bind on this server. Regards Klaus