TypeError: Descriptors cannot not be created directly.

System information

• OS: Windows 11 Pro:
• Modin version: 0.15.1
• Python version: 3.9.7
• Pandas version: 1.4.2
• Code to reproduce the error:

import numpy as np
import modin.pandas as pd

frame_data = np.random.randint(0, 100, size=(2**10, 2**5))
df = pd.DataFrame(frame_data)

Describe the problem

I heard great things about Modin so I decided to give it a try. But I can’t even start. I installed Modin with pip install modin[all]. Any idea what went wrong?

Source code / logs

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\miniconda3\lib\site-packages\modin\config\pubsub.py:253, in Parameter.get(cls)
    252 try:
--> 253     raw = cls._get_raw_from_config()
    254 except KeyError:

File ~\miniconda3\lib\site-packages\modin\config\envvars.py:46, in EnvironmentVariable._get_raw_from_config(cls)
     33 """
     34 Read the value from environment variable.
     35 
   (...)
     44     If value is absent.
     45 """
---> 46 return os.environ[cls.varname]

File ~\miniconda3\lib\os.py:679, in _Environ.__getitem__(self, key)
    677 except KeyError:
    678     # raise KeyError with the original key value
--> 679     raise KeyError(key) from None
    680 return self.decodevalue(value)

KeyError: 'MODIN_ENGINE'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
Input In [2], in <cell line: 5>()
      2 import modin.pandas as pd
      4 frame_data = np.random.randint(0, 100, size=(2**10, 2**5))
----> 5 df = pd.DataFrame(frame_data)

File ~\miniconda3\lib\site-packages\modin\logging\logger_metaclass.py:68, in logger_class_wrapper.<locals>.log_wrap(*args, **kwargs)
     66     logger.info(f"END::PANDAS-API::{class_name}.{method_name}")
     67     return result
---> 68 return method(*args, **kwargs)

File ~\miniconda3\lib\site-packages\modin\pandas\dataframe.py:114, in DataFrame.__init__(self, data, index, columns, dtype, copy, query_compiler)
    102 def __init__(
    103     self,
    104     data=None,
   (...)
    111     # Siblings are other dataframes that share the same query compiler. We
    112     # use this list to update inplace when there is a shallow copy.
    113     self._siblings = []
--> 114     Engine.subscribe(_update_engine)
    115     if isinstance(data, (DataFrame, Series)):
    116         self._query_compiler = data._query_compiler.copy()

File ~\miniconda3\lib\site-packages\modin\config\pubsub.py:213, in Parameter.subscribe(cls, callback)
    204 """
    205 Add `callback` to the `_subs` list and then execute it.
    206 
   (...)
    210     Callable to execute.
    211 """
    212 cls._subs.append(callback)
--> 213 callback(cls)

File ~\miniconda3\lib\site-packages\modin\pandas\__init__.py:114, in _update_engine(publisher)
    111     publisher.put("Native")
    112     IsExperimental.put(True)
    113 if (
--> 114     publisher.get() == "Native"
    115     and StorageFormat.get_value_source() == ValueSource.DEFAULT
    116 ):
    117     StorageFormat.put("Omnisci")
    118     IsExperimental.put(True)

File ~\miniconda3\lib\site-packages\modin\config\pubsub.py:255, in Parameter.get(cls)
    253     raw = cls._get_raw_from_config()
    254 except KeyError:
--> 255     cls._value = cls._get_default()
    256     cls._value_source = ValueSource.DEFAULT
    257 else:

File ~\miniconda3\lib\site-packages\modin\config\envvars.py:93, in Engine._get_default(cls)
     91     return "Python"
     92 try:
---> 93     import ray
     95 except ImportError:
     96     pass

File ~\miniconda3\lib\site-packages\ray\__init__.py:108, in <module>
    105 __commit__ = "4863e33856b54ccf8add5cbe75e41558850a1b75"
    106 __version__ = "1.12.1"
--> 108 import ray._raylet  # noqa: E402
    110 from ray._raylet import (  # noqa: E402
    111     ActorClassID,
    112     ActorID,
   (...)
    123     PlacementGroupID,
    124 )
    126 _config = _Config()

File python\ray\_raylet.pyx:115, in init ray._raylet()

File ~\miniconda3\lib\site-packages\ray\exceptions.py:7, in <module>
      4 from typing import Union
      6 import ray.cloudpickle as pickle
----> 7 from ray.core.generated.common_pb2 import RayException, Language, PYTHON
      8 from ray.core.generated.common_pb2 import Address, ActorDiedErrorContext
      9 import ray.ray_constants as ray_constants

File ~\miniconda3\lib\site-packages\ray\core\generated\common_pb2.py:15, in <module>
     10 # @@protoc_insertion_point(imports)
     12 _sym_db = _symbol_database.Default()
---> 15 from . import runtime_env_common_pb2 as src_dot_ray_dot_protobuf_dot_runtime__env__common__pb2
     18 DESCRIPTOR = _descriptor.FileDescriptor(
     19   name='src/ray/protobuf/common.proto',
     20   package='ray.rpc',
   (...)
     25   ,
     26   dependencies=[src_dot_ray_dot_protobuf_dot_runtime__env__common__pb2.DESCRIPTOR,])
     28 _LANGUAGE = _descriptor.EnumDescriptor(
     29   name='Language',
     30   full_name='ray.rpc.Language',
   (...)
     54   serialized_end=9262,
     55 )

File ~\miniconda3\lib\site-packages\ray\core\generated\runtime_env_common_pb2.py:36, in <module>
     11 _sym_db = _symbol_database.Default()
     16 DESCRIPTOR = _descriptor.FileDescriptor(
     17   name='src/ray/protobuf/runtime_env_common.proto',
     18   package='ray.rpc',
   (...)
     22   serialized_pb=b'\n)src/ray/protobuf/runtime_env_common.proto\x12\x07ray.rpc\"\xeb\x01\n\rPipRuntimeEnv\x12\x37\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1d.ray.rpc.PipRuntimeEnv.ConfigH\x00R\x06\x63onfig\x12*\n\x10virtual_env_name\x18\x02 \x01(\tH\x00R\x0evirtualEnvName\x1a\x62\n\x06\x43onfig\x12\x1a\n\x08packages\x18\x01 \x03(\tR\x08packages\x12\x1b\n\tpip_check\x18\x02 \x01(\x08R\x08pipCheck\x12\x1f\n\x0bpip_version\x18\x03 \x01(\tR\npipVersionB\x11\n\x0fpip_runtime_env\"h\n\x0f\x43ondaRuntimeEnv\x12\x18\n\x06\x63onfig\x18\x01 \x01(\tH\x00R\x06\x63onfig\x12&\n\x0e\x63onda_env_name\x18\x02 \x01(\tH\x00R\x0c\x63ondaEnvNameB\x13\n\x11\x63onda_runtime_env\"m\n\x13\x43ontainerRuntimeEnv\x12\x14\n\x05image\x18\x01 \x01(\tR\x05image\x12\x1f\n\x0bworker_path\x18\x02 \x01(\tR\nworkerPath\x12\x1f\n\x0brun_options\x18\x03 \x03(\tR\nrunOptions\"\x8f\x01\n\x10PluginRuntimeEnv\x12:\n\x07plugins\x18\x01 \x03(\x0b\x32 .ray.rpc.PluginRuntimeEnv.PluginR\x07plugins\x1a?\n\x06Plugin\x12\x1d\n\nclass_path\x18\x01 \x01(\tR\tclassPath\x12\x16\n\x06\x63onfig\x18\x02 \x01(\tR\x06\x63onfig\"\xf0\x02\n\x10PythonRuntimeEnv\x12\x1d\n\npy_modules\x18\x01 \x03(\tR\tpyModules\x12@\n\x0fpip_runtime_env\x18\x02 \x01(\x0b\x32\x16.ray.rpc.PipRuntimeEnvH\x00R\rpipRuntimeEnv\x12\x46\n\x11\x63onda_runtime_env\x18\x03 \x01(\x0b\x32\x18.ray.rpc.CondaRuntimeEnvH\x00R\x0f\x63ondaRuntimeEnv\x12R\n\x15\x63ontainer_runtime_env\x18\x04 \x01(\x0b\x32\x1c.ray.rpc.ContainerRuntimeEnvH\x00R\x13\x63ontainerRuntimeEnv\x12I\n\x12plugin_runtime_env\x18\x05 \x01(\x0b\x32\x19.ray.rpc.PluginRuntimeEnvH\x00R\x10pluginRuntimeEnvB\x14\n\x12python_runtime_env\"\x11\n\x0fMavenRuntimeEnv\"\xf0\x02\n\x0eJavaRuntimeEnv\x12%\n\x0e\x64\x65pendent_jars\x18\x01 \x03(\tR\rdependentJars\x12\x1f\n\x0bjvm_options\x18\x02 \x03(\tR\njvmOptions\x12\x1b\n\tjava_home\x18\x03 \x01(\tR\x08javaHome\x12\x46\n\x11maven_runtime_env\x18\x04 \x01(\x0b\x32\x18.ray.rpc.MavenRuntimeEnvH\x00R\x0fmavenRuntimeEnv\x12R\n\x15\x63ontainer_runtime_env\x18\x05 \x01(\x0b\x32\x1c.ray.rpc.ContainerRuntimeEnvH\x00R\x13\x63ontainerRuntimeEnv\x12I\n\x12plugin_runtime_env\x18\x06 \x01(\x0b\x32\x19.ray.rpc.PluginRuntimeEnvH\x00R\x10pluginRuntimeEnvB\x12\n\x10java_runtime_env\"\xf2\x01\n\rCppRuntimeEnv\x12/\n\x13\x64\x65pendent_libraries\x18\x01 \x03(\tR\x12\x64\x65pendentLibraries\x12R\n\x15\x63ontainer_runtime_env\x18\x02 \x01(\x0b\x32\x1c.ray.rpc.ContainerRuntimeEnvH\x00R\x13\x63ontainerRuntimeEnv\x12I\n\x12plugin_runtime_env\x18\x03 \x01(\x0b\x32\x19.ray.rpc.PluginRuntimeEnvH\x00R\x10pluginRuntimeEnvB\x11\n\x0f\x63pp_runtime_env\"\xb7\x01\n\x0eRuntimeEnvUris\x12&\n\x0fworking_dir_uri\x18\x01 \x01(\tR\rworkingDirUri\x12&\n\x0fpy_modules_uris\x18\x02 \x03(\tR\rpyModulesUris\x12\x1b\n\tconda_uri\x18\x03 \x01(\tR\x08\x63ondaUri\x12\x17\n\x07pip_uri\x18\x04 \x01(\tR\x06pipUri\x12\x1f\n\x0bplugin_uris\x18\x05 \x03(\tR\npluginUris\"\xf2\x04\n\nRuntimeEnv\x12\x1f\n\x0bworking_dir\x18\x01 \x01(\tR\nworkingDir\x12;\n\x08\x65nv_vars\x18\x02 \x03(\x0b\x32 .ray.rpc.RuntimeEnv.EnvVarsEntryR\x07\x65nvVars\x12\x43\n\nextensions\x18\x03 \x03(\x0b\x32#.ray.rpc.RuntimeEnv.ExtensionsEntryR\nextensions\x12L\n\x12python_runtime_env\x18\x04 \x01(\x0b\x32\x19.ray.rpc.PythonRuntimeEnvH\x00R\x10pythonRuntimeEnv\x88\x01\x01\x12\x46\n\x10java_runtime_env\x18\x05 \x01(\x0b\x32\x17.ray.rpc.JavaRuntimeEnvH\x01R\x0ejavaRuntimeEnv\x88\x01\x01\x12\x43\n\x0f\x63pp_runtime_env\x18\x06 \x01(\x0b\x32\x16.ray.rpc.CppRuntimeEnvH\x02R\rcppRuntimeEnv\x88\x01\x01\x12+\n\x04uris\x18\x07 \x01(\x0b\x32\x17.ray.rpc.RuntimeEnvUrisR\x04uris\x1a:\n\x0c\x45nvVarsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1a=\n\x0f\x45xtensionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x42\x15\n\x13_python_runtime_envB\x13\n\x11_java_runtime_envB\x12\n\x10_cpp_runtime_env\"F\n\x10RuntimeEnvConfig\x12\x32\n\x15setup_timeout_seconds\x18\x01 \x01(\x05R\x13setupTimeoutSeconds\"\xde\x01\n\x0eRuntimeEnvInfo\x12\x34\n\x16serialized_runtime_env\x18\x01 \x01(\tR\x14serializedRuntimeEnv\x12\x12\n\x04uris\x18\x02 \x03(\tR\x04uris\x12\x39\n\x19runtime_env_eager_install\x18\x03 \x01(\x08R\x16runtimeEnvEagerInstall\x12G\n\x12runtime_env_config\x18\x04 \x01(\x0b\x32\x19.ray.rpc.RuntimeEnvConfigR\x10runtimeEnvConfigB\x1d\n\x18io.ray.runtime.generated\xf8\x01\x01\x62\x06proto3'
     23 )
     28 _PIPRUNTIMEENV_CONFIG = _descriptor.Descriptor(
     29   name='Config',
     30   full_name='ray.rpc.PipRuntimeEnv.Config',
     31   filename=None,
     32   file=DESCRIPTOR,
     33   containing_type=None,
     34   create_key=_descriptor._internal_create_key,
     35   fields=[
---> 36     _descriptor.FieldDescriptor(
     37       name='packages', full_name='ray.rpc.PipRuntimeEnv.Config.packages', index=0,
     38       number=1, type=9, cpp_type=9, label=3,
     39       has_default_value=False, default_value=[],
     40       message_type=None, enum_type=None, containing_type=None,
     41       is_extension=False, extension_scope=None,
     42       serialized_options=None, json_name='packages', file=DESCRIPTOR,  create_key=_descriptor._internal_create_key),
     43     _descriptor.FieldDescriptor(
     44       name='pip_check', full_name='ray.rpc.PipRuntimeEnv.Config.pip_check', index=1,
     45       number=2, type=8, cpp_type=7, label=1,
     46       has_default_value=False, default_value=False,
     47       message_type=None, enum_type=None, containing_type=None,
     48       is_extension=False, extension_scope=None,
     49       serialized_options=None, json_name='pipCheck', file=DESCRIPTOR,  create_key=_descriptor._internal_create_key),
     50     _descriptor.FieldDescriptor(
     51       name='pip_version', full_name='ray.rpc.PipRuntimeEnv.Config.pip_version', index=2,
     52       number=3, type=9, cpp_type=9, label=1,
     53       has_default_value=False, default_value=b"".decode('utf-8'),
     54       message_type=None, enum_type=None, containing_type=None,
     55       is_extension=False, extension_scope=None,
     56       serialized_options=None, json_name='pipVersion', file=DESCRIPTOR,  create_key=_descriptor._internal_create_key),
     57   ],
     58   extensions=[
     59   ],
     60   nested_types=[],
     61   enum_types=[
     62   ],
     63   serialized_options=None,
     64   is_extendable=False,
     65   syntax='proto3',
     66   extension_ranges=[],
     67   oneofs=[
     68   ],
     69   serialized_start=173,
     70   serialized_end=271,
     71 )
     73 _PIPRUNTIMEENV = _descriptor.Descriptor(
     74   name='PipRuntimeEnv',
     75   full_name='ray.rpc.PipRuntimeEnv',
   (...)
    113   serialized_end=290,
    114 )
    117 _CONDARUNTIMEENV = _descriptor.Descriptor(
    118   name='CondaRuntimeEnv',
    119   full_name='ray.rpc.CondaRuntimeEnv',
   (...)
    157   serialized_end=396,
    158 )

File ~\miniconda3\lib\site-packages\google\protobuf\descriptor.py:560, in FieldDescriptor.__new__(cls, name, full_name, index, number, type, cpp_type, label, default_value, message_type, enum_type, containing_type, is_extension, extension_scope, options, serialized_options, has_default_value, containing_oneof, json_name, file, create_key)
    554 def __new__(cls, name, full_name, index, number, type, cpp_type, label,
    555             default_value, message_type, enum_type, containing_type,
    556             is_extension, extension_scope, options=None,
    557             serialized_options=None,
    558             has_default_value=True, containing_oneof=None, json_name=None,
    559             file=None, create_key=None):  # pylint: disable=redefined-builtin
--> 560   _message.Message._CheckCalledFromGeneratedFile()
    561   if is_extension:
    562     return _message.default_pool.FindExtensionByName(full_name)

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

[mvashishtha]

@Scoodood this bug is #4589. The latest version of Modin includes a fix for that bug, but the fix has not been released yet. You should be able to install a working version of Modin with pip install git+https://github.com/modin-project/modin per the instructions here.

In the Modin slack, I have proposed creating a Modin release just for that fix, so that other users don’t face the same bug.

I’ll close this bug as a duplicate of #4589 for now. Please let me know if you have any more concerns.

[Scoodood]

@mvashishtha Thanks, it works now.