[msgs:/var/log/messages]
<...SKIPPED...>
Sep 22 04:27:07 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader) e3 handle_command mon_command({"prefix": "osd blocklist ls", "format": "json"} v 0)
Sep 22 04:27:07 cephnode01 ceph-mon[2323]: log_channel(audit) log [DBG] : from='mgr.44099 10.4.20.31:0/99874581' entity='mgr.cephnode01.kxbnqe' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch
Sep 22 04:27:08 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader).osd e784 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 343932928 full_alloc: 348127232 kv_alloc: 318767104
Sep 22 04:27:09 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018542: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 85 B/s rd, 0 op/s
Sep 22 04:27:11 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018543: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 170 B/s rd, 0 op/s
<...CURRENT...>
Sep 22 04:27:13 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader).osd e784 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 343932928 full_alloc: 348127232 kv_alloc: 318767104
Sep 22 04:27:13 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018544: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 170 B/s rd, 0 op/s
Sep 22 04:27:15 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018545: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 1 op/s
Sep 22 04:27:17 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018546: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 37 KiB/s rd, 9.7 KiB/s wr, 4 op/s
Sep 22 04:27:18 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader).osd e784 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 343932928 full_alloc: 348127232 kv_alloc: 318767104
<...SKIPPED...>
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018587: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 85 B/s rd, 170 B/s wr, 0 op/s
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:39 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:40 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:40 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:40 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:40 cephnode01 ceph-mon[2323]: cephx server client.crash.cephnode03: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]
Sep 22 04:28:41 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018588: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 170 B/s rd, 255 B/s wr, 0 op/s
Sep 22 04:28:42 cephnode01 systemd[1]: meshagent.service: Watchdog timeout (limit 1min)!
Sep 22 04:28:42 cephnode01 systemd[1]: meshagent.service: Killing process 3455507 (meshagent) with signal SIGABRT.
Sep 22 04:28:42 cephnode01 systemd[1]: Started Process Core Dump (PID 3456188/UID 0).
Sep 22 04:28:42 cephnode01 systemd-coredump[3456189]: Removed old coredump core.meshagent.0.e9e6a7da041a4305b969d609322cb985.883169.1789866296000000.zst.
Sep 22 04:28:43 cephnode01 systemd-coredump[3456189]: Process 3455507 (meshagent) of user 0 dumped core.#012#012Stack trace of thread 3455507:#012#0  0x00007ffbc6b05c7d __select (libc.so.6 + 0x105c7d)#012#1  0x000000000041ed7e n/a (meshagent + 0x1ed7e)#012#2  0x00000000004cb007 n/a (meshagent + 0xcb007)#012#3  0x00000000004d4b11 n/a (meshagent + 0xd4b11)#012#4  0x00007ffbc6a2a610 __libc_start_call_main (libc.so.6 + 0x2a610)#012#5  0x00007ffbc6a2a6c0 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a6c0)#012#6  0x000000000040dd01 n/a (meshagent + 0xdd01)#012ELF object binary architecture: AMD x86-64
Sep 22 04:28:43 cephnode01 systemd[1]: systemd-coredump@62396-3456188-0.service: Deactivated successfully.
Sep 22 04:28:43 cephnode01 systemd[1]: meshagent.service: Main process exited, code=dumped, status=6/ABRT
Sep 22 04:28:43 cephnode01 systemd[1]: meshagent.service: Failed with result 'watchdog'.
Sep 22 04:28:43 cephnode01 systemd[1]: meshagent.service: Consumed 2.769s CPU time, 7.7M memory peak.
Sep 22 04:28:43 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader).osd e784 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 343932928 full_alloc: 348127232 kv_alloc: 318767104
Sep 22 04:28:43 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018589: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 170 B/s rd, 255 B/s wr, 0 op/s
Sep 22 04:28:45 cephnode01 ceph-mgr[2360]: [pg_autoscaler INFO pg_autoscaler] _maybe_adjust
Sep 22 04:28:45 cephnode01 ceph-mgr[2360]: [pg_autoscaler INFO pg_autoscaler] effective_target_ratio 0.0 0.0 0 48009345761280
Sep 22 04:28:45 cephnode01 ceph-mgr[2360]: [pg_autoscaler INFO pg_autoscaler] Pool '.mgr' root_id -1 using 4.6370817279403087e-07 of space, bias 1.0, pg target 0.00018548326911761235 quantized to 1 (current 1)
Sep 22 04:28:45 cephnode01 ceph-mgr[2360]: [pg_autoscaler INFO pg_autoscaler] effective_target_ratio 0.0 0.0 0 48009345761280
Sep 22 04:28:45 cephnode01 ceph-mgr[2360]: [pg_autoscaler INFO pg_autoscaler] Pool 'rbd' root_id -1 using 0.0011250429977648574 of space, bias 1.0, pg target 0.45001719910594296 quantized to 32 (current 32)
Sep 22 04:28:45 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018590: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 170 B/s rd, 255 B/s wr, 0 op/s
Sep 22 04:28:46 cephnode01 systemd[1]: meshagent.service: Scheduled restart job, restart counter is at 62397.
Sep 22 04:28:46 cephnode01 systemd[1]: Stopped meshagent background service.
Sep 22 04:28:46 cephnode01 systemd[1]: meshagent.service: Consumed 2.769s CPU time, 7.7M memory peak.
Sep 22 04:28:46 cephnode01 systemd[1]: Started meshagent background service.
Sep 22 04:28:47 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018591: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 170 B/s rd, 255 B/s wr, 0 op/s
Sep 22 04:28:48 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader).osd e784 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 343932928 full_alloc: 348127232 kv_alloc: 318767104
Sep 22 04:28:49 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018592: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 170 B/s rd, 597 B/s wr, 0 op/s
Sep 22 04:28:51 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018593: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 85 B/s rd, 3.6 KiB/s wr, 0 op/s
Sep 22 04:28:52 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader) e3 handle_command mon_command({"prefix": "osd blocklist ls", "format": "json"} v 0)
Sep 22 04:28:52 cephnode01 ceph-mon[2323]: log_channel(audit) log [DBG] : from='mgr.44099 10.4.20.31:0/99874581' entity='mgr.cephnode01.kxbnqe' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch
Sep 22 04:28:52 cephnode01 ceph-mgr[2360]: [mgr INFO mgr] scanning for idle connections..
Sep 22 04:28:52 cephnode01 ceph-mgr[2360]: [mgr INFO mgr] cleaning up connections: []
Sep 22 04:28:52 cephnode01 ceph-mgr[2360]: [mgr INFO mgr] scanning for idle connections..
Sep 22 04:28:52 cephnode01 ceph-mgr[2360]: [mgr INFO mgr] cleaning up connections: []
Sep 22 04:28:52 cephnode01 ceph-mgr[2360]: [mgr INFO mgr] scanning for idle connections..
Sep 22 04:28:52 cephnode01 ceph-mgr[2360]: [mgr INFO mgr] cleaning up connections: []
Sep 22 04:28:53 cephnode01 ceph-mon[2323]: mon.cephnode01@0(leader).osd e784 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 343932928 full_alloc: 348127232 kv_alloc: 318767104
Sep 22 04:28:53 cephnode01 ceph-mgr[2360]: log_channel(cluster) log [DBG] : pgmap v2018594: 33 pgs: 33 active+clean; 17 GiB data, 64 GiB used, 44 TiB / 44 TiB avail; 85 B/s rd, 12 KiB/s wr, 1 op/s