Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Regarding the issue of writing stalls
See original GitHub issueRegarding the issue of stalling, I hope to get some help, I have spent 3 weeks on this issue and still have not been resolved. I understand that the issue should not be posted here, but I have not been resolved when I ask for help elsewhere，Please allow me to spend a few minutes to introduce my rocksdb problem
I used to put rocksdb on the ssd disk, but there will be a problem of slower and slower writing. In order to solve the problem, I expanded the memory of the server to 256GB, and put all the rocksdb data in the memory, using the tmpfs memory disk of linux , But the problem still exists, and the speed will be slower and slower，I suspect it is caused by the use of batch writing, because the separate put method I use will not have this problem，Below is my parameter configuration
2020/11/28-02:17:08.891198 7f5b14ed9d48 RocksDB version: 5.18.4
2020/11/28-02:17:08.891235 7f5b14ed9d48 Git sha rocksdb_build_git_sha:3d5054f358faad1a4e76497547b023defd79cc6c
2020/11/28-02:17:08.891238 7f5b14ed9d48 Compile date Oct 1 2020
2020/11/28-02:17:08.891241 7f5b14ed9d48 DB SUMMARY
2020/11/28-02:17:08.891321 7f5b14ed9d48 CURRENT file: CURRENT
2020/11/28-02:17:08.891324 7f5b14ed9d48 IDENTITY file: IDENTITY
2020/11/28-02:17:08.891330 7f5b14ed9d48 MANIFEST file: MANIFEST-251092 size: 1542439 Bytes
2020/11/28-02:17:08.891333 7f5b14ed9d48 SST files in /data/addressdb/ dir, Total Num: 28, files: 245408.sst 24
5409.sst 245410.sst 245411.sst 245412.sst 245413.sst 245414.sst 245415.sst 245416.sst
2020/11/28-02:17:08.891376 7f5b14ed9d48 Write Ahead Log file in /data/addressdb: 266325.log size: 268740975 ;
266321.log size: 268719865 ;
2020/11/28-02:17:08.891383 7f5b14ed9d48 Options.error_if_exists: 0
2020/11/28-02:17:08.891386 7f5b14ed9d48 Options.create_if_missing: 1
2020/11/28-02:17:08.891389 7f5b14ed9d48 Options.paranoid_checks: 1
2020/11/28-02:17:08.891392 7f5b14ed9d48 Options.env: 0x7f5b13c009e0
2020/11/28-02:17:08.891395 7f5b14ed9d48 Options.info_log: 0x55bd13a1f140
2020/11/28-02:17:08.891397 7f5b14ed9d48 Options.max_file_opening_threads: 16
2020/11/28-02:17:08.891400 7f5b14ed9d48 Options.statistics: 0
2020/11/28-02:17:08.891402 7f5b14ed9d48 Options.use_fsync: 1
2020/11/28-02:17:08.891404 7f5b14ed9d48 Options.max_log_file_size: 0
2020/11/28-02:17:08.891407 7f5b14ed9d48 Options.max_manifest_file_size: 1073741824
2020/11/28-02:17:08.891409 7f5b14ed9d48 Options.log_file_time_to_roll: 0
2020/11/28-02:17:08.891412 7f5b14ed9d48 Options.keep_log_file_num: 1000
2020/11/28-02:17:08.891414 7f5b14ed9d48 Options.recycle_log_file_num: 0
2020/11/28-02:17:08.891417 7f5b14ed9d48 Options.allow_fallocate: 1
2020/11/28-02:17:08.891420 7f5b14ed9d48 Options.allow_mmap_reads: 0
2020/11/28-02:17:08.891422 7f5b14ed9d48 Options.allow_mmap_writes: 0
2020/11/28-02:17:08.891424 7f5b14ed9d48 Options.use_direct_reads: 0
2020/11/28-02:17:08.891427 7f5b14ed9d48 Options.use_direct_io_for_flush_and_compaction: 0
2020/11/28-02:17:08.891429 7f5b14ed9d48 Options.create_missing_column_families: 0
2020/11/28-02:17:08.891432 7f5b14ed9d48 Options.db_log_dir:
2020/11/28-02:17:08.891434 7f5b14ed9d48 Options.wal_dir: /data/addressdb
2020/11/28-02:17:08.891436 7f5b14ed9d48 Options.table_cache_numshardbits: 6
2020/11/28-02:17:08.891439 7f5b14ed9d48 Options.max_subcompactions: 1
2020/11/28-02:17:08.891441 7f5b14ed9d48 Options.max_background_flushes: -1
2020/11/28-02:17:08.891443 7f5b14ed9d48 Options.WAL_ttl_seconds: 0
2020/11/28-02:17:08.891447 7f5b14ed9d48 Options.WAL_size_limit_MB: 0
2020/11/28-02:17:08.891449 7f5b14ed9d48 Options.manifest_preallocation_size: 4194304
2020/11/28-02:17:08.891451 7f5b14ed9d48 Options.is_fd_close_on_exec: 1
2020/11/28-02:17:08.891454 7f5b14ed9d48 Options.advise_random_on_open: 1
2020/11/28-02:17:08.891456 7f5b14ed9d48 Options.db_write_buffer_size: 0
2020/11/28-02:17:08.891458 7f5b14ed9d48 Options.write_buffer_manager: 0x55bd13a1ec00
2020/11/28-02:17:08.891461 7f5b14ed9d48 Options.access_hint_on_compaction_start: 1
2020/11/28-02:17:08.891463 7f5b14ed9d48 Options.new_table_reader_for_compaction_inputs: 0
2020/11/28-02:17:08.891465 7f5b14ed9d48 Options.random_access_max_buffer_size: 1048576
2020/11/28-02:17:08.891468 7f5b14ed9d48 Options.use_adaptive_mutex: 0
2020/11/28-02:17:08.891470 7f5b14ed9d48 Options.rate_limiter: 0
2020/11/28-02:17:08.891475 7f5b14ed9d48 Options.sst_file_manager.rate_bytes_per_sec: 0
2020/11/28-02:17:08.891477 7f5b14ed9d48 Options.wal_recovery_mode: 2
2020/11/28-02:17:08.891480 7f5b14ed9d48 Options.enable_thread_tracking: 0
2020/11/28-02:17:08.891482 7f5b14ed9d48 Options.enable_pipelined_write: 0
2020/11/28-02:17:08.891484 7f5b14ed9d48 Options.allow_concurrent_memtable_write: 1
2020/11/28-02:17:08.891486 7f5b14ed9d48 Options.enable_write_thread_adaptive_yield: 1
2020/11/28-02:17:08.891489 7f5b14ed9d48 Options.write_thread_max_yield_usec: 100
2020/11/28-02:17:08.891491 7f5b14ed9d48 Options.write_thread_slow_yield_usec: 3
2020/11/28-02:17:08.891493 7f5b14ed9d48 Options.row_cache: None
2020/11/28-02:17:08.891495 7f5b14ed9d48 Options.wal_filter: None
2020/11/28-02:17:08.891498 7f5b14ed9d48 Options.avoid_flush_during_recovery: 0
2020/11/28-02:17:08.891500 7f5b14ed9d48 Options.allow_ingest_behind: 0
2020/11/28-02:17:08.891503 7f5b14ed9d48 Options.preserve_deletes: 0
2020/11/28-02:17:08.891506 7f5b14ed9d48 Options.two_write_queues: 0
2020/11/28-02:17:08.891508 7f5b14ed9d48 Options.manual_wal_flush: 0
2020/11/28-02:17:08.891510 7f5b14ed9d48 Options.max_background_jobs: 2
2020/11/28-02:17:08.891513 7f5b14ed9d48 Options.max_background_compactions: -1
2020/11/28-02:17:08.891515 7f5b14ed9d48 Options.avoid_flush_during_shutdown: 0
2020/11/28-02:17:08.891517 7f5b14ed9d48 Options.writable_file_max_buffer_size: 1048576
2020/11/28-02:17:08.891520 7f5b14ed9d48 Options.delayed_write_rate : 16777216
2020/11/28-02:17:08.891522 7f5b14ed9d48 Options.max_total_wal_size: 0
2020/11/28-02:17:08.891535 7f5b14ed9d48 Options.delete_obsolete_files_period_micros: 21600000000
2020/11/28-02:17:08.891538 7f5b14ed9d48 Options.stats_dump_period_sec: 600
2020/11/28-02:17:08.891540 7f5b14ed9d48 Options.max_open_files: -1
2020/11/28-02:17:08.891543 7f5b14ed9d48 Options.bytes_per_sync: 0
2020/11/28-02:17:08.891546 7f5b14ed9d48 Options.wal_bytes_per_sync: 0
2020/11/28-02:17:08.891548 7f5b14ed9d48 Options.compaction_readahead_size: 0
2020/11/28-02:17:08.891550 7f5b14ed9d48 Compression algorithms supported:
2020/11/28-02:17:08.891552 7f5b14ed9d48 kZSTDNotFinalCompression supported: 1
2020/11/28-02:17:08.891555 7f5b14ed9d48 kZSTD supported: 1
2020/11/28-02:17:08.891558 7f5b14ed9d48 kXpressCompression supported: 0
2020/11/28-02:17:08.891560 7f5b14ed9d48 kLZ4HCCompression supported: 1
2020/11/28-02:17:08.891562 7f5b14ed9d48 kLZ4Compression supported: 1
2020/11/28-02:17:08.891565 7f5b14ed9d48 kBZip2Compression supported: 1
2020/11/28-02:17:08.891567 7f5b14ed9d48 kZlibCompression supported: 1
2020/11/28-02:17:08.891569 7f5b14ed9d48 kSnappyCompression supported: 1
2020/11/28-02:17:08.891577 7f5b14ed9d48 Fast CRC32 supported: Supported on x86
2020/11/28-02:17:08.891642 7f5b14ed9d48 [/version_set.cc:3507] Recovering from manifest file: MANIFEST-251092
2020/11/28-02:17:08.891974 7f5b14ed9d48 [/column_family.cc:472] --------------- Options for column family [defa
ult]:
2020/11/28-02:17:08.891977 7f5b14ed9d48 Options.comparator: leveldb.BytewiseComparator
2020/11/28-02:17:08.891980 7f5b14ed9d48 Options.merge_operator: None
2020/11/28-02:17:08.891982 7f5b14ed9d48 Options.compaction_filter: None
2020/11/28-02:17:08.891984 7f5b14ed9d48 Options.compaction_filter_factory: None
2020/11/28-02:17:08.891987 7f5b14ed9d48 Options.memtable_factory: SkipListFactory
2020/11/28-02:17:08.891989 7f5b14ed9d48 Options.table_factory: BlockBasedTable
2020/11/28-02:17:08.892024 7f5b14ed9d48 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bd137dab80)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x55bd13a1bd30
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
memory_allocator : None
high_pri_pool_ratio: 0.000
block_cache_compressed: 0
persistent_cache: 0
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: nullptr
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0
2020/11/28-02:17:08.892029 7f5b14ed9d48 Options.write_buffer_size: 67108864
2020/11/28-02:17:08.892032 7f5b14ed9d48 Options.max_write_buffer_number: 2
2020/11/28-02:17:08.892034 7f5b14ed9d48 Options.compression: Snappy
2020/11/28-02:17:08.892036 7f5b14ed9d48 Options.bottommost_compression: Disabled
2020/11/28-02:17:08.892039 7f5b14ed9d48 Options.prefix_extractor: nullptr
2020/11/28-02:17:08.892041 7f5b14ed9d48 Options.memtable_insert_with_hint_prefix_extractor: nullptr
2020/11/28-02:17:08.892043 7f5b14ed9d48 Options.num_levels: 7
2020/11/28-02:17:08.892046 7f5b14ed9d48 Options.min_write_buffer_number_to_merge: 1
2020/11/28-02:17:08.892048 7f5b14ed9d48 Options.max_write_buffer_number_to_maintain: 0
2020/11/28-02:17:08.892050 7f5b14ed9d48 Options.bottommost_compression_opts.window_bits: -14
2020/11/28-02:17:08.892053 7f5b14ed9d48 Options.bottommost_compression_opts.level: 32767
2020/11/28-02:17:08.892055 7f5b14ed9d48 Options.bottommost_compression_opts.strategy: 0
2020/11/28-02:17:08.892069 7f5b14ed9d48 Options.bottommost_compression_opts.max_dict_bytes: 0
2020/11/28-02:17:08.892071 7f5b14ed9d48 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2020/11/28-02:17:08.892073 7f5b14ed9d48 Options.bottommost_compression_opts.enabled: false
2020/11/28-02:17:08.892075 7f5b14ed9d48 Options.compression_opts.window_bits: -14
2020/11/28-02:17:08.892078 7f5b14ed9d48 Options.compression_opts.level: 32767
2020/11/28-02:17:08.892080 7f5b14ed9d48 Options.compression_opts.strategy: 0
2020/11/28-02:17:08.892082 7f5b14ed9d48 Options.compression_opts.max_dict_bytes: 0
2020/11/28-02:17:08.892084 7f5b14ed9d48 Options.compression_opts.zstd_max_train_bytes: 0
2020/11/28-02:17:08.892086 7f5b14ed9d48 Options.compression_opts.enabled: false
2020/11/28-02:17:08.892088 7f5b14ed9d48 Options.level0_file_num_compaction_trigger: 4
2020/11/28-02:17:08.892090 7f5b14ed9d48 Options.level0_slowdown_writes_trigger: 20
2020/11/28-02:17:08.892093 7f5b14ed9d48 Options.level0_stop_writes_trigger: 36
2020/11/28-02:17:08.892095 7f5b14ed9d48 Options.target_file_size_base: 33554432
2020/11/28-02:17:08.892097 7f5b14ed9d48 Options.target_file_size_multiplier: 1
2020/11/28-02:17:08.892100 7f5b14ed9d48 Options.max_bytes_for_level_base: 268435456
2020/11/28-02:17:08.892102 7f5b14ed9d48 Options.level_compaction_dynamic_level_bytes: 0
2020/11/28-02:17:08.892104 7f5b14ed9d48 Options.max_bytes_for_level_multiplier: 10.000000
2020/11/28-02:17:08.892106 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2020/11/28-02:17:08.892109 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2020/11/28-02:17:08.892111 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2020/11/28-02:17:08.892113 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2020/11/28-02:17:08.892116 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2020/11/28-02:17:08.892118 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2020/11/28-02:17:08.892122 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2020/11/28-02:17:08.892124 7f5b14ed9d48 Options.max_sequential_skip_in_iterations: 8
2020/11/28-02:17:08.892126 7f5b14ed9d48 Options.max_compaction_bytes: 838860800
2020/11/28-02:17:08.892128 7f5b14ed9d48 Options.arena_block_size: 8388608
2020/11/28-02:17:08.892131 7f5b14ed9d48 Options.soft_pending_compaction_bytes_limit: 68719476736
2020/11/28-02:17:08.892133 7f5b14ed9d48 Options.hard_pending_compaction_bytes_limit: 274877906944
2020/11/28-02:17:08.892135 7f5b14ed9d48 Options.rate_limit_delay_max_milliseconds: 100
2020/11/28-02:17:08.892137 7f5b14ed9d48 Options.disable_auto_compactions: 0
2020/11/28-02:17:08.892140 7f5b14ed9d48 Options.compaction_style: kCompactionStyleLevel
2020/11/28-02:17:08.892142 7f5b14ed9d48 Options.compaction_pri: kByCompensatedSize
2020/11/28-02:17:08.892144 7f5b14ed9d48 Options.compaction_options_universal.size_ratio: 1
2020/11/28-02:17:08.892147 7f5b14ed9d48 Options.compaction_options_universal.min_merge_width: 2
2020/11/28-02:17:08.892149 7f5b14ed9d48 Options.compaction_options_universal.max_merge_width: 4294967295
2020/11/28-02:17:08.892152 7f5b14ed9d48 Options.compaction_options_universal.max_size_amplification_percent: 200
2020/11/28-02:17:08.892154 7f5b14ed9d48 Options.compaction_options_universal.compression_size_percent: -1
2020/11/28-02:17:08.892156 7f5b14ed9d48 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2020/11/28-02:17:08.892158 7f5b14ed9d48 Options.compaction_options_fifo.max_table_files_size: 1073741824
2020/11/28-02:17:08.892161 7f5b14ed9d48 Options.compaction_options_fifo.allow_compaction: 0
2020/11/28-02:17:08.892163 7f5b14ed9d48 Options.compaction_options_fifo.ttl: 0
2020/11/28-02:17:08.892165 7f5b14ed9d48 Options.table_properties_collectors:
2020/11/28-02:17:08.892167 7f5b14ed9d48 Options.inplace_update_support: 0
2020/11/28-02:17:08.892169 7f5b14ed9d48 Options.inplace_update_num_locks: 10000
2020/11/28-02:17:08.892172 7f5b14ed9d48 Options.memtable_prefix_bloom_size_ratio: 0.000000
2020/11/28-02:17:08.892174 7f5b14ed9d48 Options.memtable_huge_page_size: 0
2020/11/28-02:17:08.892177 7f5b14ed9d48 Options.bloom_locality: 0
2020/11/28-02:17:08.892179 7f5b14ed9d48 Options.max_successive_merges: 0
2020/11/28-02:17:08.892181 7f5b14ed9d48 Options.optimize_filters_for_hits: 0
2020/11/28-02:17:08.892183 7f5b14ed9d48 Options.paranoid_file_checks: 0
2020/11/28-02:17:08.892185 7f5b14ed9d48 Options.force_consistency_checks: 0
2020/11/28-02:17:08.892187 7f5b14ed9d48 Options.report_bg_io_stats: 0
2020/11/28-02:17:08.892189 7f5b14ed9d48 Options.ttl: 0
In order to ensure the consistency of writing, I used batch writing and printing time, but this time will get longer and longer. It only needs 0:00:00.00728 to start the program at the beginning, as the program runs time It will slowly become 00:00:10 or even longer，Cause me to restart the program regularly，I don’t understand what I did wrong. I have replaced all of them with memory, but the efficiency cannot be improved. Please help me, thank you very much
starttime = datetime.datetime.now()
self.DB.write(rocksdb.WriteBatch())
endtime = datetime.datetime.now()
print(endtime - starttime)
This is the state that costs 00:00:07.128520
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 37.57 MB 0.5 0.0 0.0 0.0 5.0 5.0 0.0 1.0 0.0 6.1 843 364 2.316 0 0
L1 3/0 82.26 MB 0.3 12.0 5.0 7.0 7.0 0.0 0.0 1.4 92.3 53.9 133 91 1.462 300M 124M
L2 2/0 55.75 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L5 3/0 1.72 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L6 22/0 135.73 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Sum 32/0 313.03 MB 0.0 12.0 5.0 7.0 12.0 5.0 0.0 2.4 12.6 12.6 976 455 2.146 300M 124M
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Uptime(secs): 15217.4 total, 0.0 interval
Flush(GB): cumulative 5.033, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 12.04 GB write, 0.81 MB/s write, 11.99 GB read, 0.81 MB/s read, 976.2 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_com
Issue Analytics
- State:
- Created 3 years ago
- Comments:9
Top Results From Across the Web
What to Do When Your Writing Stalls Out - Fiction University
Production stalls out. Panic sets in. If you're a career writer like I am, that panic is compounded by the fact that writing...
Read more >How to Keep Your Story From Stalling - Writers Store
No matter what your particular story stall issue, what helps in every case is to remember that exposition is not meant to stand...
Read more >9 Ways to Stop Your Novel from Stalling - Writer's Digest
Let's begin by identifying the warning signs that your writing may be about to stall out. Consider current or previous writing resolutions ......
Read more >8 Reasons Writing Stalls — and How to Turn Them Around
Writers for centuries have struggled with time constraints, the challenge of prioritizing writing and their own creative selves, and ...
Read more >How to Avoid Stalling During the Writing Process: Some Tips ...
Writers usually stall during the writing process because they don't know what happens next in their story. After all, you can't write what...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
No! Just this:
thank you