[package] openssl/1.1.1g: Cannot make HTTPS requests using Boost.Beast

Package and Environment Details (include every applicable attribute)

• Package Name/Version: openssl/1.1.1g and boost/1.74
• Operating System+version: Linux Ubuntu 20.04 (docker)
• Compiler+version: GCC 10.0.1
• Docker image: ubuntu/20.04
• Conan version: Conan version 1.28.1
• Python version: Python 3.8.2

Conan profile (output of conan profile show default or conan profile show <profile> if custom profile is in use)

Configuration for profile gcc10:

[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
CC=/usr/bin/gcc-10
CXX=/usr/bin/g++-10

Steps to reproduce (Include if Applicable)

Have a conan file with this content:

[requires]
boost/1.74.0
openssl/1.1.1g

[generators]
cmake

And such CMake file:

cmake_minimum_required(VERSION 3.17)
project(beast-test)

include("${PROJECT_BINARY_DIR}/conanbuildinfo.cmake")
conan_basic_setup(TARGETS)

set(CMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY ON)
set(CMAKE_FIND_USE_CMAKE_SYSTEM_PATH OFF)
set(CMAKE_FIND_PACKAGE_PREFER_CONFIG ON)
set(CMAKE_FIND_USE_CMAKE_PATH OFF)
set(CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH OFF)

add_executable(http-beast-ssl main.cpp)
target_link_libraries(http-beast-ssl PRIVATE CONAN_PKG::boost CONAN_PKG::openssl)

The main file looks like this:

#include <boost/asio/ssl.hpp>
#include <boost/lexical_cast.hpp>
#include <openssl/cryptoerr.h>
#include <boost/beast/core.hpp>
#include <boost/beast/http.hpp>
#include <boost/beast/ssl.hpp>
#include <boost/beast/version.hpp>
#include <boost/asio/connect.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/asio/ssl/error.hpp>
#include <boost/asio/ssl/stream.hpp>
#include <boost/beast/version.hpp>
#include <openssl/opensslv.h>
#include <cstdlib>
#include <iostream>
#include <string>

namespace beast = boost::beast; // from <boost/beast.hpp>
namespace http = beast::http;   // from <boost/beast/http.hpp>
namespace net = boost::asio;    // from <boost/asio.hpp>
namespace ssl = net::ssl;       // from <boost/asio/ssl.hpp>
using tcp = net::ip::tcp;       // from <boost/asio/ip/tcp.hpp>

// Performs an HTTP GET and prints the response
int main(int argc, char** argv)
{
    std::cout << OPENSSL_VERSION_TEXT << std::endl;
    try
    {
        // Check command line arguments.
        if(argc != 4 && argc != 5)
        {
            std::cerr <<
                "Usage: http-client-sync-ssl <host> <port> <target> [<HTTP version: 1.0 or 1.1(default)>]\n" <<
                "Example:\n" <<
                "    http-client-sync-ssl www.example.com 443 /\n" <<
                "    http-client-sync-ssl www.example.com 443 / 1.0\n";
            return EXIT_FAILURE;
        }
        auto const host = argv[1];
        auto const port = argv[2];
        auto const target = argv[3];
        int version = argc == 5 && !std::strcmp("1.0", argv[4]) ? 10 : 11;

        // The io_context is required for all I/O
        net::io_context ioc;

        // The SSL context is required, and holds certificates
        ssl::context ctx(ssl::context::tlsv12_client);

        // This holds the root certificate used for verification
		ctx.set_default_verify_paths();

        // Verify the remote server's certificate
        ctx.set_verify_mode(ssl::verify_peer);

            // These objects perform our I/O
        tcp::resolver resolver(ioc);
        beast::ssl_stream<beast::tcp_stream> stream(ioc, ctx);

        // Set SNI Hostname (many hosts need this to handshake successfully)
        if(! SSL_set_tlsext_host_name(stream.native_handle(), host))
        {
            beast::error_code ec{static_cast<int>(::ERR_get_error()), net::error::get_ssl_category()};
            throw beast::system_error{ec};
        }

        // Look up the domain name
        auto const results = resolver.resolve(host, port);

        // Make the connection on the IP address we get from a lookup
        beast::get_lowest_layer(stream).connect(results);

        // Perform the SSL handshake
        stream.handshake(ssl::stream_base::client);

        // Set up an HTTP GET request message
        http::request<http::string_body> req{http::verb::get, target, version};
        req.set(http::field::host, host);
        req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);

        // Send the HTTP request to the remote host
        http::write(stream, req);

        // This buffer is used for reading and must be persisted
        beast::flat_buffer buffer;

        // Declare a container to hold the response
        http::response<http::dynamic_body> res;

        // Receive the HTTP response
        http::read(stream, buffer, res);

        // Write the message to standard out
        std::cout << res << std::endl;

        // Gracefully close the stream
        beast::error_code ec;
        stream.shutdown(ec);
        if(ec == net::error::eof)
        {
            // Rationale:
            // http://stackoverflow.com/questions/25587403/boost-asio-ssl-async-shutdown-always-finishes-with-an-error
            ec = {};
        }
        if(ec)
            throw beast::system_error{ec};

        // If we get here then the connection is closed gracefully
    }
    catch(boost::system::system_error const& e)
    {
        auto const& error = e.code();
        std::string err = error.message();
        if (error.category() == boost::asio::error::get_ssl_category()) {
            err = std::string(" (")
                +boost::lexical_cast<std::string>(ERR_GET_LIB(error.value()))+","
                +boost::lexical_cast<std::string>(ERR_GET_FUNC(error.value()))+","
                +boost::lexical_cast<std::string>(ERR_GET_REASON(error.value()))+") ";
    
            //ERR_PACK /* crypto/err/err.h */
            char buf[128];
            ::ERR_error_string_n(error.value(), buf, sizeof(buf));
            err += buf;
            std::cerr << err << std::endl;
        }
        return EXIT_FAILURE;
    }
    return EXIT_SUCCESS;
    
}

Running this:

./bin/http-beast-ssl example.com 443 /

Output this error:

OpenSSL 1.1.1g  21 Apr 2020
(20,367,134) error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

Important notice

This error don’t happen when using system’s openssl package. The issue I’m reporting here is the results from the findings of boostorg/beast#2119

I’m reporting this issue here because it should either:

• Work out of the box, just like using system’s openssl
• Have a documented way to make it work

Unfortunately, it seem neither of these points are satisfied.

For it to work, it seems openssl should be allowed to read from /etc/ssl/certs, which I suspect that openssl is not configured that way.

[rdeterre]

No problem, I’m be happy to make a pull request 😃

Given that the issue is not only with Boost but with any package that uses OpenSSL, I’m thinking it would make sense to change the default value of “openssldir” to “/etc/ssl” directly there. I’m not sure if I’m doing this the right way but I have a patch that works on my systems. I’ll push this right away.

[prince-chrismc]

@gracicot Je m’excuse if I came across as rude or dismissive, I should have ajoute un thinking emoji to better convey what I meant, let me try again!

💭 perhaps this is just a missing option from the recipe?

⛵ not everyone has the same use case, my applications tend to run in private clouds with no public access to perform CRL or OSCP checks… This is probably sparing me nightmare-ish 🐛 reports.

Lastly, we are hiring! If you want to send me your CV, my team gets all the bleeding edge IP work (in C++ hehe). If you are up for a challenge that is… 😉

@madebr I never tried it, I am too familiar with this issue, I believed it the second I saw the issue.

For me, based on my experience, the big impediment is that one ca bundle is never enough to cover all the Root CA providers. The OS already does this for us… compare the wiki list to the ones you have installed. How many are missing? Have you ever found one?

What scares me the most, there’s an issue in CI, some bad bundle gets published? Then what? How do we recall that?

You’re solution is 100% viable and it can for sure work. My personal thought would be to code the path in like libressl and avoid the hassle.