node-rdkafka fails to install/build on Windows

Environment Information

• Windows 10 Pro, 64 bit
• Node Version 8.12.0
• NPM Version 6.4.1
• C++ Toolchain: msbuild engine version 12.0.40629.0
• node-rdkafka version commit e82bb2e (latest master as of writing)
• nuget: 4.6.2.5055
• node-gyp 3.8.0

Steps to Reproduce

I followed the steps as per this comment on issue 273 which are

git clone https://github.com/Blizzard/node-rdkafka.git node-rdkafka --recursive cd node-rdkafka npm install nan node-gyp rebuild --verbose

Fails with error message: “MSBUILD : error MSB1009: Project file does not exist.” originating from file “build\deps\librdkafka.vcxproj”

node-rdkafka Configuration Settings

I don’t know what information you need here.

Additional context

Any suggestions for how to diagnose and resolve this are welcome!

I also tried installing the package via npm. Executing npm i node-rdkafka the output also includes these lines:

Warning: Missing input files: C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\outdir\v120\x64\Release\zlib.dll C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\librdkafka.sln C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\outdir\v120\x64\Release\librdkafkacpp.dll C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\outdir\v120\x64\Release\librdkafka.dll

Followed later by

Copying deps/librdkafka/win32/outdir/v120/x64/Release/librdkafkacpp.dll to …/build/Release\librdkafkacpp.dll The system cannot find the file specified. Copying deps/librdkafka/win32/outdir/v120/x64/Release/librdkafka.dll to …/build/Release\librdkafka.dll The system cannot find the file specified. nuget_restore, and also build_dependencies Unknown command: ‘…..\deps\restore’ NuGet.CommandLine.CommandLineException: Unknown command: ‘…..\deps\restore’

This seems to suggest that something with the node-rdkafka installation script is not quite right.

[ManfredLange]

@skyfloyd and @t-davies Thank you both for your suggestions. Very much appreciated!

Based on your comments and other sources I was able to make it work with the following approach in my own environment as well in other team members’ environments:

A. Installing Prerequisites

1. Uninstall node.js
2. Remove manually everything that looks like remnants of nodejs. In particular check that the cache directory is gone. In my case the cache was located at C:\Users\Manfred\AppData\Roaming\npm-cache. In my case there was/is also a directory named C:\Users\Manfred\AppData\Roaming\npm. Also check C:\Program Files\nodejs. If any of these directories exist after uninstall, remove them manually. In my environment with multiple installs/uninstalls/reinstalls it was messed up at some point.
3. Install node.js, version 8.12.0 or later from here. This also installs npm version 6.4.1 or later.
4. Open a terminal windows, e.g. PowerShell
5. Install node-gyp using npm install -g node-gyp@3.8.0 (or newer version). Important: Install globally with flag -g.
6. Install VC++ / Visual Studio 2010 build tools. Note that it is not necessary to install the entire Visual Studio 2010, just the build tools. Get the installer from the URL mentioned here. As of writing the link is this. When installing accept all defaults. If you already have these tools installed, first uninstall them, then re-install.
7. Add the path for the 12.0-version VC++ build tools to your Path environment variable. In my case (all defaults) this was located at C:\Program Files (x86)\MSBuild\12.0\Bin. Important: Do not use any of the other versions that you may have. The path must contain 12.0 otherwise you will be using the wrong version. I tested version 14.0 which resulted in compiler or linker errors.
8. Install Python 2.7.15 (or 2.7.14) from python.org. Either pick the 32bit installer or the 64bit installer. Important: Versions other than 2.7.x, e.g. 3.x or later will not work.
9. Add the path for Python 2.7 to the Path environment variable. Also create a new environment variable named “Python” that contains the path including the full name of the executable. In my case this was C:\bin\python27\python.exe.
10. Download nuget.exe version 4.6.2 or later from here. Version 4.7.1 is recommended as of writing.
11. Add the path for nuget to the Path environment variable
12. Download Win32_OpenSSL_1.0.2p.exe and install using all defaults. If this link is broken, Locate the latest 1.0.2 version for Win32 that does not say “Light” on it on this web site. Note that I didn’t test any of the other versions. They may work but during my research I found that everybody was referring to version 1.0.2o or version 1.0.2p.

B. Pre-Build Checks

Here are some checks you can run to determine your chances of a successful install. If any of these tests fail, the install is unlikely to succeed.

1. In your terminal window run the command node -v and confirm it is version 8.12.0 or later
2. Run command npm -v and confim it is version 6.4.1 or later
3. Run command nuget and confirm it is version 4.6.2 or later
4. Run command msbuild and confirm it is version 12.0.x.y (but nothing newer/older!!)
5. Run command python and confirm it is version 2.7.x (3.x or later will not work)

C. Build

Execute npm install node-rdkafka@2.3.4.

Assuming I haven’t missed an important step this should now work. It did in our environments, i.e. more than one Windows based development machine.

[RezaRahmati]

@ManfredLange I wasn’t able to install msbuild 12 from the link provided, instead downloading from here works