Error "No space left on device" observed while running warp run I/O

Environment info

Noobaa version

[root@hpo-app1 ~]# noobaa version
INFO[0000] CLI version: 5.11.0
INFO[0000] noobaa-image: noobaa/noobaa-core:5.10.0-20220120
INFO[0000] operator-image: noobaa/noobaa-operator:5.11.0

OC version

[root@hpo-app1 ~]# oc version
Client Version: 4.11.0
Kustomize Version: v4.5.4
Server Version: 4.11.0
Kubernetes Version: v1.24.0+9546431

ODF version

[root@hpo-app1 ~]# oc get csv
NAME                                   DISPLAY                       VERSION               REPLACES                          PHASE
mcg-operator.v4.11.1                   NooBaa Operator               4.11.1                mcg-operator.v4.11.0              Succeeded
metallb-operator.4.11.0-202208300306   MetalLB Operator              4.11.0-202208300306                                     Succeeded
ocs-operator.v4.11.1                   OpenShift Container Storage   4.11.1                ocs-operator.v4.11.0              Succeeded
odf-csi-addons-operator.v4.11.1        CSI Addons                    4.11.1                odf-csi-addons-operator.v4.11.0   Succeeded
odf-operator.v4.11.1                   OpenShift Data Foundation     4.11.1                odf-operator.v4.11.0              Succeeded

When running warp consecutively on 3 IP’s, the run fails due to error “warp: <ERROR> upload error: We encountered an internal error. Please try again”. On checking end-points logs, found that error i.e "upload_stream had error: [Error: No space left on device] { code: 'ENOSPC' } is returned for all runs.

Actual behavior

1. Warp run is failing

Expected behavior

1.Warp run should execute with any error.

Steps to reproduce

1. Setup BM with CNSA/CSI RC4, ODF 4.11.1, latest DAS operator.
2. Run warp on all three metallb IPs with 3 different users.

More information - Screenshots / Logs / Other output

noobaa_diagnostics_1663759506.tar.gz

[rkomandu]

This is the current status on the hpofvt1 machine (neesha m/c) atm

for i in `oc get pods  |grep noobaa-endpoint | awk ' {print $1}'`; do oc rsh $i ulimit -n; oc rsh $i lsof |wc -l; echo $i is done; done
1048576
505
noobaa-endpoint-5c77fc9bd5-2g5zc is done
1048576
494
noobaa-endpoint-5c77fc9bd5-8tc4f is done
1048576
516
noobaa-endpoint-5c77fc9bd5-brx4w is done
1048576
483
noobaa-endpoint-5c77fc9bd5-ckhwd is done
1048576
494
noobaa-endpoint-5c77fc9bd5-hq2j8 is done
1048576
483
noobaa-endpoint-5c77fc9bd5-l7wrl is done
1048576
494
noobaa-endpoint-5c77fc9bd5-nlkp7 is done
1048576
483
noobaa-endpoint-5c77fc9bd5-pkrwm is done
1048576
483
noobaa-endpoint-5c77fc9bd5-zv7ld is done

[romayalon]

closing it per @rkomandu last comment