question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Not able to deserialize ClickHouseBitmap for too many bitmaps(>4294967295)!

See original GitHub issue

clickhouse-server version:22.2.2.1 clickhouse-jdbc version:0.3.2-patch8

create search

an error occurs when I fetch the bitmap of the next. even if I adjust the order of index_val, the second data will still report an error.

jdbc sql: select index_name, index_val, length(bitmapToArray(cst_bitmap)) as bitmap_count, cst_bitmap fromindex_bitmap_mergetree orderbyindex_name,index_val female jdbc sql: select index_name, index_val, length(bitmapToArray(cst_bitmap)) as bitmap_count, cst_bitmap fromindex_bitmap_mergetree orderbyindex_name,index_val desc man

test code

  public static void main(String[] args) throws SQLException {
      String url = "jdbc:clickhouse://192.168.70.131:8123";
      Properties properties = new Properties();
      ClickHouseDataSource dataSource = new ClickHouseDataSource(url, properties);
//        insert(dataSource, 2000);
      searchBitmap(dataSource);
  }

  public static void searchBitmap(ClickHouseDataSource dataSource) {
      try (Connection connection = dataSource.getConnection("default", "");
           Statement statement = connection.createStatement();
           ResultSet resultSet = statement.executeQuery("" +
                   "select " +
                   "  index_name," +
                   "  index_val," +
                   "  length(bitmapToArray(cst_bitmap)) as bitmap_count, " +
                   "  cst_bitmap " +
                   "from index_bitmap_mergetree " +
                   "order by index_name, index_val desc")) {
          while (resultSet.next()) {
              String indexName = resultSet.getString(1);
              String indexVal = resultSet.getString(2);
              int bitmapCount = resultSet.getInt(3);
              Roaring64NavigableMap roaringBitmap = (Roaring64NavigableMap) resultSet.getObject(4, ClickHouseBitmap.class).unwrap();
              System.out.printf("indexName:%s, indexVal:%s, size:%d%n", indexName, indexVal, bitmapCount);
              roaringBitmap.stream().forEach(i -> System.out.print(i + ","));
              System.out.println();
          }
      } catch (SQLException e) {
          e.printStackTrace();
      }
  }

error

Exception in thread "main" java.lang.IllegalStateException: Not able to deserialize ClickHouseBitmap for too many bitmaps(>4294967295)!
	at com.clickhouse.client.data.ClickHouseBitmap.deserialize(ClickHouseBitmap.java:372)
	at com.clickhouse.client.data.ClickHouseBitmap.deserialize(ClickHouseBitmap.java:339)
	at com.clickhouse.client.data.BinaryStreamUtils.readBitmap(BinaryStreamUtils.java:210)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor$MappedFunctions.lambda$buildMappingsForAggregateFunctions$0(ClickHouseRowBinaryProcessor.java:178)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor$MappedFunctions.lambda$buildMappingsForAggregateFunctions$2(ClickHouseRowBinaryProcessor.java:185)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor$MappedFunctions.deserialize(ClickHouseRowBinaryProcessor.java:464)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor.readAndFill(ClickHouseRowBinaryProcessor.java:496)
	at com.clickhouse.client.ClickHouseDataProcessor$1.next(ClickHouseDataProcessor.java:122)
	at com.clickhouse.client.ClickHouseDataProcessor$1.next(ClickHouseDataProcessor.java:108)
	at com.clickhouse.jdbc.ClickHouseResultSet.next(ClickHouseResultSet.java:708)
	at com.yatop.clickhouse.JdbcDemo.searchBitmap(JdbcDemo.java:39)
	at com.yatop.clickhouse.JdbcDemo.main(JdbcDemo.java:25)

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:10 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
wzqiang1332commented, Jul 27, 2022

@zhicwu I have tested it in my local environment. The bug is caused by the bitmap reading 3 bytes too many, resulting in the following field reading misalignment

0reactions
zhicwucommented, Jul 27, 2022

Thanks for confirming @wzqiang1332! It will be part of 0.3.2-patch11 which hopefully can be released by Friday, before I start to play Xenoblade 3 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

AggregateFunction(groupBitmap,UInt64) support #698 - GitHub
No able to deserialize ClickhouseBitmap for too many bitmaps ... it more than 4,294,967,295? And which version of ClickHouse are you using?
Read more >
Bitmap Functions | ClickHouse Docs
Bitmap functions work for two bitmaps Object value calculation, it is to return new bitmap or cardinality while using formula calculation, such as...
Read more >
ClickHouse Documentation
If a query uses a primary key and does not select too many rows to process (hundreds of thousands), and does not use...
Read more >
Not able to deserialize ClickHouseBitmap for too ... - bytemeta
IllegalStateException: Not able to deserialize ClickHouseBitmap for too many bitmaps(>4294967295)! at com.clickhouse.client.data.
Read more >
How can i perform AND operation on multiple bitmaps
1 Answer 1 · I found that groupbitmapand function can be implemented clickhouse.com/docs/en/sql-reference/aggregate-functions/… – lant. Apr 7 at ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found