question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] Setting the maxmemory flag on Redis, causes the platform pods to crash loop once Redis reaches max memory

See original GitHub issue

Description

Setting the maxmemory flag on redis, causes the platform pods to crash loop once redis reaches max memory

ReplyError: OOM command not allowed when used memory > 'maxmemory'.
    at parseError (/opt/opencti/build/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/opt/opencti/build/node_modules/redis-parser/lib/parser.js:302:14)

Environment

  1. OS (where OpenCTI server runs): Cent OS Stream 8
  2. OpenCTI version: 5.4.0
  3. OpenCTI client: frontend
  4. Other environment details: Kubernetes Deployment

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Start Redis server with redis-server --maxmemory 6442450944 --maxmemory-policy volatile-lru
  2. Wait for memory usage to balloon in the rdis container
  3. Once it levels of at ~6GB (Or what you set above ^), the platform pods will start to crash after some time

Expected Output

Better management ofd Redis connection handing. Ive got kinda a catch 22 here:

On one hand, If I dont set the maxmemeory Kubernetes will OOMKill Redis (This its self is fine), but when that happens OpenCTI Stalls, CPU usage drops to near 0 and the WebUI locks up

On the other hand If I cap Redis, Kubernetes wont OOMKill it, but the OpenCTI platform pods will throw the error in this issue and crash.

So since clustering isnt supported, this makes it where the platform is requiring manual interaction to keep running every 12-36 hours

Actual Output

ReplyError: OOM command not allowed when used memory > 'maxmemory'.
    at parseError (/opt/opencti/build/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/opt/opencti/build/node_modules/redis-parser/lib/parser.js:302:14)

Additional information

Please let me know if I can help testing here, as this is a big stability issue at the moment

Screenshots (optional)

Issue Analytics

  • State:open
  • Created 10 months ago
  • Comments:12 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
richard-juliencommented, Nov 29, 2022

We need to take a look about what is currently taking space in your redis. I can recommand to use redisinsight that will provide you a nice UI to explore the current redis data and try to find that keys or stream are currently taking so much space.

0reactions
MaxwellDPScommented, Dec 13, 2022

It does not as of now

Do you ingest in your platform a feed that give you artifact with payload bin ?

Read more comments on GitHub >

github_iconTop Results From Across the Web

PHP Fatal error when Redis maxmemory is reached
Hi, Litespeed recently created a PHP Fatal error when Redis reached its 10 GB Memory Limit. Redis memory is now set to 20...
Read more >
MISCONF Redis is configured to save RDB snapshots
This error occurs due to starting the redis server in a directory where redis does not have permissions. I recommend reverting back to...
Read more >
Redis 6.0 release notes - GitHub
An integer overflow bug in Redis version 2.2 or newer can be exploited using the ... The vulnerability involves changing the default set-max-intset-entries ......
Read more >
Collection of alerting rules - Awesome Prometheus alerts
Prometheus AlertManager configuration reload failure ... for: 2m labels: severity: warning annotations: summary: Redis out of configured maxmemory (instance ...
Read more >
Memory management best practices | Memorystore for Redis
Maxmemory is a Redis configuration that allows you to set the memory limit at ... running out of memory and eventually causes the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found