Sun May 24 22:28:08 AEST 2026 - Log files ok
No notable entries in /var/log/messages
Full log: /var/log/messages
<...SKIPPED...>
May 24 22:26:10 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:15 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:20 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:25 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:25 cephnode03 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
<...CURRENT...>
May 24 22:26:30 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:35 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:40 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:45 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
May 24 22:26:50 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
<...SKIPPED...>
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 10575] Flush lasted 9908 microseconds, and 6482 cpu microseconds.
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.723947) [db/flush_job.cc:967] [default] [JOB 10575] Level-0 flush table #21302: 1258033 bytes OK
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.723979) [db/memtable_list.cc:519] [default] Level-0 commit table #21302 started
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.724879) [db/memtable_list.cc:722] [default] Level-0 commit table #21302: memtable #1 done
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.724927) EVENT_LOG_v1 {"time_micros": 1779625684724918, "job": 10575, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0}
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.724953) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 10575] Try to delete WAL files size 1992878, prev total WAL file size 1992878, number of live WAL files 2.
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-cephnode03/store.db/021298.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.725603) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F7300383934383135' seq:72057594037927935, type:22 .. '7061786F7300383935303637' seq:0, type:0; will stop at (end)
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 10576] Compacting 1@0 + 1@6 files to L6, score -1.00
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 10575 Base level 0, inputs: [21302(1228KB)], [21300(29MB)]
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1779625684725660, "job": 10576, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [21302], "files_L6": [21300], "score": -1, "input_data_size": 32479514, "oldest_snapshot_seqno": -1}
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 10576] Generated table #21303: 23492 keys, 31811738 bytes, temperature: kUnknown
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1779625684884557, "cf_name": "default", "job": 10576, "event": "table_file_creation", "file_number": 21303, "file_size": 31811738, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 31681182, "index_size": 70727, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 58757, "raw_key_size": 645661, "raw_average_key_size": 27, "raw_value_size": 31295240, "raw_average_value_size": 1332, "num_data_blocks": 2472, "num_entries": 23492, "num_filter_entries": 23492, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1778820315, "oldest_key_time": 0, "file_creation_time": 1779625684, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "55250332-332c-4e74-b612-d0ef2ccbeb68", "db_session_id": "OYQ1FS82N4AGHFFRB9LC", "orig_file_number": 21303, "seqno_to_time_mapping": "N/A"}}
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.885029) [db/compaction/compaction_job.cc:1663] [default] [JOB 10576] Compacted 1@0 + 1@6 files to L6 => 31811738 bytes
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.886151) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 204.4 rd, 200.1 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.2, 29.8 +0.0 blob) out(30.3 +0.0 blob), read-write-amplify(51.1) write-amplify(25.3) OK, records in: 24327, records dropped: 835 output_compression: NoCompression
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.886174) EVENT_LOG_v1 {"time_micros": 1779625684886164, "job": 10576, "event": "compaction_finished", "compaction_time_micros": 158940, "compaction_time_cpu_micros": 90174, "output_level": 6, "num_output_files": 1, "total_output_size": 31811738, "num_input_records": 24327, "num_output_records": 23492, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]}
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-cephnode03/store.db/021302.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1779625684886451, "job": 10576, "event": "table_file_deletion", "file_number": 21302}
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-cephnode03/store.db/021300.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1779625684889153, "job": 10576, "event": "table_file_deletion", "file_number": 21300}
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.725522) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.889240) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.889261) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.889266) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.889269) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
May 24 22:28:04 cephnode03 ceph-mon[2631]: rocksdb: (Original Log Time 2026/05/24-12:28:04.889273) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
May 24 22:28:05 cephnode03 ceph-mon[2631]: mon.cephnode03@2(peon).osd e229 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
|