[Issue] Problem loading big amount of domains

I’m trying to add a big program, around 5.5 million domains, the inscope and outscope are big also.

I don’t know why it’s failing, this is what I tried:

cat domains-big.txt | bbrf domain add - -s subfinder --show-new

Doesn’t add any domain nor output any error.

If I do this sed -n '50001 ,100000p' domains-big.txt| bbrf domain add - -s subfinder --show-new

I get this error “[ERROR] local variable ‘success’ referenced before assignment”

The same error doing:

awk 'NR > 50000 && NR <= 100000' domains-big.txt | bbrf domain add - -s subfinder --show-new

Any idea why this might happen?

[z0mb13s3c]

Yup, 1.1.2 & 1.1.3 give me both the same error. At first I thought it’s something to do with subfinder but then I tried chaos, enabled the debug and figured it’s bbrf. I’m not sure if it’s the client or the server though. Here’s the full output, maybe it helps.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): xxx.xxx.xxx.xxx:1234
DEBUG:urllib3.connectionpool:https://xxx.xxx.xxx.xxx:1234 "GET /bbrf/program HTTP/1.1" 200 862
[DEBUG] getting program scope
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): xxx.xxx.xxx.xxx:12345
DEBUG:urllib3.connectionpool:https://xxx.xxx.xxx.xxx:1234 "GET /bbrf/program HTTP/1.1" 200 862
[DEBUG] getting program blacklist
[DEBUG] adding document in bulk
DEBUG:urllib3.connectionpool:https://xxx.xxx.xxx.xxx:1234 "POST /bbrf/_bulk_docs HTTP/1.1" 400 91
[ERROR] string indices must be integers

[honoki]

There was a bug in the --show-new flag that I have silently fixed yesterday in v1.1.3

However, that is probably unrelated to the large input failing. I’ll run some local tests to see if there’s a size limit on couchdb. I can imagine maybe it times out when the input is too large.