issue with ThreadedMotoServer

hi,

I try to do the following: my python app uses pyspark and regular python.

• pyspark calls spark.read.csv and spark.read.load to read data from s3 - so I have to mock s3 access
• regular python downloads some files from s3 and uses regular python to process the data - so I have to mock s3 access
• for mock_pyspark I need to start ThreadedMotoServer which runs on port 5000 according to: https://github.com/spulec/moto/issues/1543

I run into the following error:

ImportError while importing test module '/Users/mike/foo-app/tests/test_predict.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/gstanje/.pyenv/versions/3.8.9/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_predict.py:14: in <module>
    from moto.server import ThreadedMotoServer
.venv/lib/python3.8/site-packages/moto/server.py:7: in <module>
    from moto.moto_server.werkzeug_app import (
.venv/lib/python3.8/site-packages/moto/moto_server/werkzeug_app.py:7: in <module>
    from flask_cors import CORS
E   ModuleNotFoundError: No module named 'flask_cors'

here is how I installed Moto:

poetry add moto@latest
Using version ^4.0.3 for moto

Updating dependencies
Resolving dependencies... (12.1s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating grpcio (1.48.0 -> 1.48.1)

check installation:

poetry show moto       
 name         : moto                                                                        
 version      : 4.0.3                                                                       
 description  : A library that allows your python tests to easily mock out the boto library 

dependencies
 - boto3 >=1.9.201
 - botocore >=1.12.201
 - cryptography >=3.3.1
 - Jinja2 >=2.10.1
 - MarkupSafe !=2.0.0a1
 - python-dateutil >=2.1,<3.0.0
 - pytz *
 - requests >=2.5
 - responses >=0.13.0
 - werkzeug >=0.5,<2.2.0
 - xmltodict *

I also installed it in the following way - same issue:

poetry add "moto[ec,s3,all]"@latest
Using version ^4.0.3 for moto

Updating dependencies
Resolving dependencies... (311.5s)

Writing lock file

Package operations: 18 installs, 1 update, 1 removal

  • Removing pkgutil-resolve-name (1.3.10)
  • Updating jsonschema (4.9.1 -> 3.2.0)
  • Installing jsonpickle (2.2.0)
  • Installing jsonpointer (2.3)
  • Installing pbr (5.10.0)
  • Installing aws-sam-translator (1.50.0)
  • Installing ecdsa (0.18.0)
  • Installing jschema-to-python (1.2.3)
  • Installing jsonpatch (1.32)
  • Installing junit-xml (1.9)
  • Installing networkx (2.8.6)
  • Installing openapi-schema-validator (0.2.3)
  • Installing sarif-om (1.0.4)
  • Installing aws-xray-sdk (2.10.0)
  • Installing cfn-lint (0.64.1)
  • Installing graphql-core (3.2.1)
  • Installing jsondiff (2.0.0)
  • Installing openapi-spec-validator (0.4.0)
  • Installing python-jose (3.3.0)
  • Installing sshpubkeys (3.3.1)

here the code I try to run:

from unittest import TestCase
from moto import mock_s3
from mypy_boto3_s3.client import S3Client
from pathlib import Path
import os
import glob
import logging
from chardet import detect
from io import BytesIO
from pyspark.sql import SparkSession
import boto3
import os
import pandas as pd
from moto.server import ThreadedMotoServer

AWS_PROFILE = "testing"

class TestPredict(TestCase):
    s3_bucket = "foo-bucket"
    mock_s3 = mock_s3()
    s3 = S3()

    def setUp(self):
        # set aws credentials: https://stackoverflow.com/questions/59535311/moto-does-not-appear-to-be-mocking-aws-interactions-in-a-pytest
        moto_credentials_file_path = Path(__file__).parent.absolute() / 'fixtures' / 'dummy_aws_credentials'
        os.environ['AWS_SHARED_CREDENTIALS_FILE'] = str(moto_credentials_file_path)

        s3 = S3()
        s3.set_credentials(AWS_PROFILE)
        s3_session = s3.get_session()
        self.setUpS3()

        self.server = ThreadedMotoServer()
        self.server.start()

    def tearDown(self):
        self.mock_s3.stop()
        self.server.stop()

    def setUpS3(self):
        self.mock_s3.start()
        self.s3.create_bucket(self.s3_bucket)

    # https://github.com/spulec/moto/issues/1543
    def mock_pyspark(self):
        os.environ["PYSPARK_SUBMIT_ARGS"] = (
            '--packages "org.apache.hadoop:hadoop-aws:3.3.1" pyspark-shell'
        )
        spark = SparkSession.builder.getOrCreate()

        # Setup spark to use s3, and point it to the moto server.
        hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
        hadoop_conf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
        hadoop_conf.set("fs.s3a.access.key", "mock")
        hadoop_conf.set("fs.s3a.secret.key", "mock")
        hadoop_conf.set("fs.s3a.endpoint", "http://127.0.0.1:5000")

        conn = boto3.resource("s3", endpoint_url="http://127.0.0.1:5000")
        conn.create_bucket(Bucket="bucket")

        data = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
        with BytesIO(data.to_csv(index=False).encode()) as buffer:
            conn.Bucket("bucket").put_object(Key="test/test.csv", Body=buffer)

        df = spark.read.csv("s3://bucket/test/test.csv", header=True)
        logging.info("%v", df)

    # to implement
    def upload_tfdf_model(self):
        return

    def upload_s3_data(self):
        self.upload_tfdf_model()
        return

    def test_predict(self):
        self.upload_s3_data()
        self.mock_pyspark()

        return

[bblommers]

I was on the fence whether this was a bug or not, considering the documentation didn’t specify anything. If the consensus is that moto[server] is acceptable, as long as it is documented, I’ll revert the earlier change before the next release. Thanks for the feedback all!

[Arnold1]

I don’t see it as a bug - I just failed to install the right package: moto[server]