背景

cratedb突然大量报错

 1 ### Error updating database.  Cause: io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
 2 ### The error may involve defaultParameterMap
 3 ### The error occurred while setting parameters
 4 ### SQL: insert into tableName (.......
 5 ......
 6 ......
 7 ### Cause: io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
 8 ; uncategorized SQLException for SQL []; SQL state [XX000]; error code [0]; ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];; nested exception is io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
 9 org.springframework.jdbc.UncategorizedSQLException:
10 ### Error updating database.  Cause: io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
11 ### The error may involve defaultParameterMap
12 ### The error occurred while setting parameters
13 ### SQL: insert into tableName ....
14 ....
15 ...
16 ### Cause: io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
17 ; uncategorized SQLException for SQL []; SQL state [XX000]; error code [0]; ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];; nested exception is io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
18         at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:90)
19         at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
20         at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
21         at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:75)
22         at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:447)
23         at com.sun.proxy.$Proxy46.insert(Unknown Source)
24         at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:279)
25         at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:57)
26         at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:53)
27         at com.sun.proxy.$Proxy58.commonBatchInsert(Unknown Source)
28         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
29         at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:65)
30         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
31         at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
32         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
33         at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:73)
34         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
35         at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:112)
36         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
37         at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
38         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
39         at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
40         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
41         at com.alibaba.dubbo.registry.filter.AdminZkServiceFilter.invoke(AdminZkServiceFilter.java:40)
42         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
43         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
44         at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
45         at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:94)
46         at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:186)
47         at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:140)
48         at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
49         at com.alibaba.dubbo.remoting.transport.dispather.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
50         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
51         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
52         at java.lang.Thread.run(Thread.java:748)
53 Caused by: io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
54         at io.crate.shade.org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2458)
55         at io.crate.shade.org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2158)
56         at io.crate.shade.org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:291)
57         at io.crate.shade.org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:433)
58         at io.crate.shade.org.postgresql.jdbc.PgStatement.execute(PgStatement.java:359)
59         at io.crate.shade.org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
60         at io.crate.shade.org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:158)

异常栈

原因

当cratedb集群中有节点的磁盘使用率达到 cluster.routing.allocation.disk.watermark.flood_stage 设置的值(默认为95%)时 cratedb会让受影响的index都设置为只读且可删除状态。

cluster.routing.allocation.disk.watermark.flood_stage

Default: 95%

Runtime: yes

Defines the threshold on which CrateDB enforces a read-only block on every index that has at least one shard allocated on a node with at least one disk exceeding the flood stage.

使用命令 df -h 发现确实有节点磁盘使用率超过了 95%

也可以在AdminUI中查看各个节点的磁盘使用情况

清理空间重启服务

机器空间扩大后，重启cratedb服务，却没有重启成功。

[ERROR][o.e.b.BootstrapProxy     ] [nodeName] node validation exception
[1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

根据提示修改参数 vm.max_map_count

在文件 /etc/sysctl.conf 中添加

vm.max_map_count = 262144

重启cratedb服务成功

./bin/crate -d

后续

节点也扩容了，服务也重启了，报错依然存在。。。。

认真看，参数 cluster.routing.allocation.disk.watermark.flood_stage 后面还有个提示

Note

Read-only blocks are not automatically removed from the indices if the disk space is freed and the threshold is undershot. To remove the block, execute ALTER TABLE ... SET ("blocks.read_only_allow_delete" = FALSE) for affected tables (see blocks.read_only_allow_delete).

(⊙o⊙)… 空间虽然释放了，但是index的只读设置并不会自动移除，必须自己在 crash或者AdminUI中在受影响的表上只执行命令

ALTER TABLE tableName SET ("blocks.read_only_allow_delete" = FALSE)

关于参数blocks.read_only_allow_delete

blocks.read_only_allow_delete¶

Allows to have a read only table that additionally can be deleted.

value

Table is read only and can be deleted if value set to true. Allows writes and table settings changes if set to false.

When a disk on a node exceeds the cluster.routing.allocation.disk.watermark.flood_stage threshold, this block is applied (set to true) to all tables on that affected node. Once you’ve freed disk space again and the threshold is undershot, you need to set the blocks.read_only_allow_delete table setting to false.