新的设置 `allow_asynchronous_read_from_io_pool_for_merge_tree` 允许读取线程(流)的数量高于查询执行管道中其余线程的数量。
同步数据读取
新的设置 allow_asynchronous_read_from_io_pool_for_merge_tree 允许读取线程(流)的数量高于查询执行管道中其余线程的数量。
通常,max_threads 设置 控制 并行读取线程和并行查询处理线程的数量
数据按列顺序从磁盘读取。
异步数据读取
新的设置 allow_asynchronous_read_from_io_pool_for_merge_tree 允许读取线程(流)的数量高于查询执行管道中其余线程的数量,以加速低 CPU ClickHouse Cloud 服务的冷查询,并提高 I/O 密集型查询的性能。当启用该设置时,读取线程的数量由 max_streams_for_merge_tree_reading 设置控制
数据以异步方式读取,从不同的列并行读取。
请注意,还有 max_streams_to_max_threads_ratio 设置用于配置读取线程(流)数量与查询执行管道中其余线程数量之间的比例。但是,在基准测试中,它并没有像 max_streams_for_merge_tree_reading 设置那样提供很大的帮助
关于 optimize_read_in_order 呢?
使用 optimize_read_in_order 优化,如果查询排序顺序反映磁盘上数据的物理顺序,ClickHouse 可以 跳过 内存中的数据排序,但这需要按顺序读取数据(与异步读取相反)
optimize_read_in_order 优先于异步读取
当 ClickHouse 看到可以应用 optimize_read_in_order 优化 时,allow_asynchronous_read_from_io_pool_for_merge_tree 设置将被忽略/禁用。
演示上述所有内容的示例
SELECT getSetting('max_threads');
┌─getSetting('max_threads')─┐
│ 10 │
└───────────────────────────┘
EXPLAIN PIPELINE
SELECT *
FROM uk_price_paid;
┌─explain──────────────────────┐
│ (Expression) │
│ ExpressionTransform × 10 │
│ (ReadFromMergeTree) │
│ MergeTreeThread × 10 0 → 1 │
└──────────────────────────────┘
- 检查使用 60 个异步读取线程和默认线程数进行其余查询执行管道的查询管道
EXPLAIN PIPELINE
SELECT *
FROM uk_price_paid
SETTINGS
allow_asynchronous_read_from_io_pool_for_merge_tree = 1,
max_streams_for_merge_tree_reading = 60;
┌─explain────────────────────────┐
│ (Expression) │
│ ExpressionTransform × 10 │
│ (ReadFromMergeTree) │
│ Resize 60 → 10 │
│ MergeTreeThread × 60 0 → 1 │
└────────────────────────────────┘
- 检查使用 20 个线程进行数据读取和处理的查询管道
EXPLAIN PIPELINE
SELECT *
FROM uk_price_paid
SETTINGS
max_threads = 20;
┌─explain──────────────────────┐
│ (Expression) │
│ ExpressionTransform × 20 │
│ (ReadFromMergeTree) │
│ MergeTreeThread × 20 0 → 1 │
└──────────────────────────────┘
- 检查使用 60 个异步读取线程和 20 个线程进行其余查询执行管道的查询管道
EXPLAIN PIPELINE
SELECT *
FROM uk_price_paid
SETTINGS
max_threads = 20,
allow_asynchronous_read_from_io_pool_for_merge_tree = 1,
max_streams_for_merge_tree_reading = 60;
┌─explain────────────────────────┐
│ (Expression) │
│ ExpressionTransform × 20 │
│ (ReadFromMergeTree) │
│ Resize 60 → 20 │
│ MergeTreeThread × 60 0 → 1 │
└────────────────────────────────┘
- 检查当可以应用
optimize_read_in_order 优化 时,使用 60 个异步读取线程和 20 个线程进行其余查询执行管道的查询管道
EXPLAIN PIPELINE
SELECT *
FROM uk_price_paid
ORDER BY postcode1, postcode2
SETTINGS
max_threads = 20,
allow_asynchronous_read_from_io_pool_for_merge_tree= 1,
max_streams_for_merge_tree_reading= 60;
┌─explain───────────────────────────┐
│ (Expression) │
│ ExpressionTransform │
│ (Sorting) │
│ MergingSortedTransform 20 → 1 │
│ (Expression) │
│ ExpressionTransform × 20 │
│ (ReadFromMergeTree) │
│ MergeTreeInOrder × 20 0 → 1 │
└───────────────────────────────────┘
-- note that this is equivalent to disabling allow_asynchronous_read_from_io_pool_for_merge_tree
EXPLAIN PIPELINE
SELECT *
FROM uk_price_paid
ORDER BY postcode1, postcode2
SETTINGS
max_threads = 20,
allow_asynchronous_read_from_io_pool_for_merge_tree = 0,
max_streams_for_merge_tree_reading = 0;
┌─explain───────────────────────────┐
│ (Expression) │
│ ExpressionTransform │
│ (Sorting) │
│ MergingSortedTransform 20 → 1 │
│ (Expression) │
│ ExpressionTransform × 20 │
│ (ReadFromMergeTree) │
│ MergeTreeInOrder × 20 0 → 1 │
└───────────────────────────────────┘
-- note that you can enforce allow_asynchronous_read_from_io_pool_for_merge_tree by disabling optimize_read_in_order
EXPLAIN PIPELINE
SELECT *
FROM uk_price_paid
ORDER BY
postcode1 ASC,
postcode2 ASC
SETTINGS
max_threads = 20,
allow_asynchronous_read_from_io_pool_for_merge_tree = 1,
max_streams_for_merge_tree_reading = 60,
optimize_read_in_order = 0;
┌─explain──────────────────────────────┐
│ (Expression) │
│ ExpressionTransform │
│ (Sorting) │
│ MergingSortedTransform 20 → 1 │
│ MergeSortingTransform × 20 │
│ (Expression) │
│ ExpressionTransform × 20 │
│ (ReadFromMergeTree) │
│ Resize 60 → 20 │
│ MergeTreeThread × 60 0 → 1 │
└──────────────────────────────────────┘
·阅读时间:4 分钟