BUILD_LIBRDKAFKA flag doesn't work

Environment Information

• OS [e.g. Mac, Arch, Windows 10]: Debian
• Node Version [e.g. 8.2.1]: 8/11/12
• NPM Version [e.g. 5.4.2]: *
• C++ Toolchain [e.g. Visual Studio, llvm, g++]: see below
• node-rdkafka version [e.g. 2.3.3]: v2.7.0

Steps to Reproduce

• Build docker: docker build -t test-native-librdkafka .

FROM node:11.5

ENV LIBRDKAFKA_VERSION=0.11.6
ENV LIBRDKAFKA_HOME=/usr/local
ENV BUILD_LIBRDKAFKA=0
ARG BUILD_LIBRDKAFKA=0

WORKDIR /usr/src/app
COPY . /usr/src/app/

RUN set -ex; \
    apt-get -y update; \
    apt-get purge -y libssl-dev; \
    apt-get install -y --no-install-recommends \
    ca-certificates \
    git \
    build-essential \
    python \
    libsasl2-dev \
    libssl-dev \
    openssl \
    libc-bin \
    libc6 \
    libstdc++6; \
    apt-get clean; \
    rm -rf /var/lib/apt/lists/*; \
    \
    mkdir -p ${LIBRDKAFKA_HOME}; \
    curl -s -Lk -o ${LIBRDKAFKA_HOME}/librdkafka-${LIBRDKAFKA_VERSION}.tar.gz https://github.com/edenhill/librdkafka/archive/v${LIBRDKAFKA_VERSION}.tar.gz; \
    tar -xzf ${LIBRDKAFKA_HOME}/librdkafka-${LIBRDKAFKA_VERSION}.tar.gz -C ${LIBRDKAFKA_HOME}; \
    mv ${LIBRDKAFKA_HOME}/librdkafka-${LIBRDKAFKA_VERSION} ${LIBRDKAFKA_HOME}/librdkafka; \
    ls -la ${LIBRDKAFKA_HOME}/librdkafka/; \
    cd ${LIBRDKAFKA_HOME}/librdkafka; \
    ./configure --prefix=${LIBRDKAFKA_HOME}/librdkafka; \
    make; \
    make install; \
    ldconfig; \
    make clean; \
    ./configure --clean; \
    ls -la

RUN set -ex; \
    yarn install; \
    yarn run compile; \
    rm -rf ./node_modules; \
    yarn install --prod; \
    yarn cache clean

ENV NODE_ENV=production
CMD [ "yarn", "start" ]

node-rdkafka Configuration Settings [none]

Actual behavior The node-rdkafka installation rebuilds librdkafka deps every time.

Expected behavior The node-rdkafka uses native installation.

[nervgh]

I have the same issue.

My configuration is:

librdkafka-dev@1.1.0-r0 https://pkgs.alpinelinux.org/packages?name=librdkafka-dev&branch=edge

node-rdkafka@2.7.1-2

# The image from Docker Hub
FROM node:10.16-alpine

# Install global dependencies
RUN apk add --no-cache git openssh librdkafka-dev tzdata

# Install node-rdkafka dependecies
RUN apk add --no-cache g++ make python

## Define and set defaults for environment variables
ENV BUILD_LIBRDKAFKA=0

# It installs node-rdkafka@2.7.1-2 from NPM registry
RUN npm ci

And error log is:

> node-rdkafka@v2.7.2 install /usr/src/app/node_modules/node-rdkafka
> node-gyp rebuild

gyp info it worked if it ends with ok
gyp info using node-gyp@3.8.0
gyp info using node@10.16.0 | linux | x64
gyp http GET https://nodejs.org/download/release/v10.16.0/node-v10.16.0-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v10.16.0/node-v10.16.0-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v10.16.0/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v10.16.0/SHASUMS256.txt
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/src/app/node_modules/node-rdkafka/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/local/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/root/.node-gyp/10.16.0/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/root/.node-gyp/10.16.0',
gyp info spawn args   '-Dnode_gyp_dir=/usr/local/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/root/.node-gyp/10.16.0/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/usr/src/app/node_modules/node-rdkafka',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/usr/src/app/node_modules/node-rdkafka/build'
  CXX(target) Release/obj.target/node-librdkafka/src/binding.o
In file included from /usr/src/app/node_modules/node-rdkafka/src/binding.h:13,
                 from ../src/binding.cc:11:
../../nan/nan.h: In function 'void Nan::AsyncQueueWorker(Nan::AsyncWorker*)':
../../nan/nan.h:2298:62: warning: cast between incompatible function types from 'void (*)(uv_work_t*)' {aka 'void (*)(uv_work_s*)'} to 'uv_after_work_cb' {aka 'void (*)(uv_work_s*, int)'} [-Wcast-function-type]
     , reinterpret_cast<uv_after_work_cb>(AsyncExecuteComplete)
                                                              ^
In file included from /usr/src/app/node_modules/node-rdkafka/src/binding.h:21,
                 from ../src/binding.cc:11:
/usr/src/app/node_modules/node-rdkafka/src/producer.h: At global scope:
/usr/src/app/node_modules/node-rdkafka/src/producer.h:70:14: error: 'RdKafka::Headers' has not been declared
     RdKafka::Headers* headers);
              ^~~~~~~
/usr/src/app/node_modules/node-rdkafka/src/producer.h:76:14: error: 'RdKafka::Headers' has not been declared
     RdKafka::Headers* headers);
              ^~~~~~~
In file included from ../../nan/nan.h:54,
                 from /usr/src/app/node_modules/node-rdkafka/src/binding.h:13,
                 from ../src/binding.cc:11:
/root/.node-gyp/10.16.0/include/node/node.h:573:43: warning: cast between incompatible function types from 'void (*)(v8::Local<v8::Object>, v8::Local<v8::Object>)' to 'node::addon_register_func' {aka 'void (*)(v8::Local<v8::Object>, v8::Local<v8::Value>, void*)'} [-Wcast-function-type]
       (node::addon_register_func) (regfunc),                          \
                                           ^
/root/.node-gyp/10.16.0/include/node/node.h:607:3: note: in expansion of macro 'NODE_MODULE_X'
   NODE_MODULE_X(modname, regfunc, NULL, 0)  // NOLINT (readability/null_usage)
   ^~~~~~~~~~~~~
../src/binding.cc:166:1: note: in expansion of macro 'NODE_MODULE'
 NODE_MODULE(kafka, Init)
 ^~~~~~~~~~~
In file included from /root/.node-gyp/10.16.0/include/node/node.h:63,
                 from ../../nan/nan.h:54,
                 from /usr/src/app/node_modules/node-rdkafka/src/binding.h:13,
                 from ../src/binding.cc:11:
/root/.node-gyp/10.16.0/include/node/v8.h: In instantiation of 'void v8::PersistentBase<T>::SetWeak(P*, typename v8::WeakCallbackInfo<P>::Callback, v8::WeakCallbackType) [with P = node::ObjectWrap; T = v8::Object; typename v8::WeakCallbackInfo<P>::Callback = void (*)(const v8::WeakCallbackInfo<node::ObjectWrap>&)]':
/root/.node-gyp/10.16.0/include/node/node_object_wrap.h:84:78:   required from here
/root/.node-gyp/10.16.0/include/node/v8.h:9502:16: warning: cast between incompatible function types from 'v8::WeakCallbackInfo<node::ObjectWrap>::Callback' {aka 'void (*)(const v8::WeakCallbackInfo<node::ObjectWrap>&)'} to 'Callback' {aka 'void (*)(const v8::WeakCallbackInfo<void>&)'} [-Wcast-function-type]
                reinterpret_cast<Callback>(callback), type);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/.node-gyp/10.16.0/include/node/v8.h: In instantiation of 'void v8::PersistentBase<T>::SetWeak(P*, typename v8::WeakCallbackInfo<P>::Callback, v8::WeakCallbackType) [with P = Nan::ObjectWrap; T = v8::Object; typename v8::WeakCallbackInfo<P>::Callback = void (*)(const v8::WeakCallbackInfo<Nan::ObjectWrap>&)]':
../../nan/nan_object_wrap.h:65:61:   required from here
/root/.node-gyp/10.16.0/include/node/v8.h:9502:16: warning: cast between incompatible function types from 'v8::WeakCallbackInfo<Nan::ObjectWrap>::Callback' {aka 'void (*)(const v8::WeakCallbackInfo<Nan::ObjectWrap>&)'} to 'Callback' {aka 'void (*)(const v8::WeakCallbackInfo<void>&)'} [-Wcast-function-type]
make: *** [node-librdkafka.target.mk:119: Release/obj.target/node-librdkafka/src/binding.o] Error 1
make: Leaving directory '/usr/src/app/node_modules/node-rdkafka/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:198:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
gyp ERR! System Linux 4.15.0-55-generic
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /usr/src/app/node_modules/node-rdkafka
gyp ERR! node -v v10.16.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! node-rdkafka@v2.7.2 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 

Are there any ideas how to fix it?

[nervgh]

@MidnightP I’ve found a solution to cache node-rdkafka library as a standalone docker image.

1. I use that reference to build a docker image per specific Node.js major version with included node-rdkafka library:
2. After I build my docker image using the image from previous stage ☝️ :

# The image from Docker Hub
FROM node:10.16-alpine

# Install build of node-rdkafka
RUN apk --no-cache add ca-certificates cyrus-sasl-dev lz4-dev musl-dev openssl-dev
COPY --from=${YOUR_REGISTRY}/node-rdkafka-builds:node-10 /node_modules /node_modules
ENV NODE_PATH='/node_modules'

# Copy package.json & package-lock.json fiels
# IMPORTANT: package.json should not contain node-rdkafka as dependency
COPY ["package*.json", "./"]

RUN npm ci