Memory restrictions caused failure TLJH install failure on AWS EC2

I have tried multiple times to install TLJH on an AWS EC2 instance, failing every time. The process has thrown a number of different errors, but the most common appears to be with conda installation.

Error logs

ubuntu@ip-172-31-91-142:~$ curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py | sudo -E python3 - --admin cfarr
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6354  100  6354    0     0  45385      0 --:--:-- --:--:-- --:--:-- 45385
Checking if TLJH is already installed...
Setting up hub environment
Installed python & virtual environment
Set up hub virtual environment
Setting up TLJH installer...
Setup tljh package
Starting TLJH installer...
Setting up admin users
Granting passwordless sudo to JupyterHub admins...
Setting up user environment...
Downloading & setting up user environment...
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 508, in <module>
    main()
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 491, in main
    ensure_user_environment(args.user_requirements_txt_url)
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 272, in ensure_user_environment
    'conda==' + conda_version
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/conda.py", line 109, in ensure_conda_packages
    ] + packages).decode()
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/opt/tljh/user/bin/python', '-m', 'conda', 'install', '-c', 'conda-forge', '--json', '--prefix', '/opt/tljh/user', 'conda==4.8.1']' died with <Signals.SIGKILL: 9>.

I was following the guide to install on your own server and had run the following:

sudo -E apt install python3 python3-dev git curl

which failed, Ubuntu suggest I run the following:

sudo apt-get update

which succeeded, so I re-ran the following:

sudo -E apt install python3 python3-dev git curl

which succeeded, so I ran the TLJH install:

curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py | sudo -E python3 - --admin cfarr

which threw the above error.

Any and all help is appreciated!

[twrobinson]

Hi, the read the docs states this “The AWS free tier is fully capable of running a minimal littlest Jupyterhub for testing purposes.”, but to me it seems that the 1 GB free tier instance on AWS (t2.micro) is not sufficient, or is at best “barely sufficient”, to install tljh, and the installation is likely to fail in unpredictable ways.

Installing from User Data results in a hang in “Setting up user environment…”.

Installing from a terminal with "curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py | sudo -E python3 - --admin "

resulted in

Downloading & setting up user environment...
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/exceptions.py", line 1062, in __call__
    return func(*args, **kwargs)
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/cli/main.py", line 84, in _main
    exit_code = do_call(args, p)
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 82, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/cli/main_install.py", line 20, in execute
    install(args, parser, 'install')
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/cli/install.py", line 308, in install
    handle_txn(unlink_link_transaction, prefix, args, newenv)
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/cli/install.py", line 333, in handle_txn
    unlink_link_transaction.download_and_extract()
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/core/link.py", line 191, in download_and_extract
    self._pfe.execute()
  File "/opt/tljh/user/lib/python3.7/site-packages/conda/core/package_cache_data.py", line 661, in execute
    raise CondaMultiError(exceptions)
conda.CondaMultiError: Error with archive /opt/tljh/user/pkgs/conda-4.8.1-py37_0.conda.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:

Zstd decompression failed: Allocation error : not enough memory (errno=-1, retcode=-30, archive_p=94800679832192)

with subsequent attempts getting further:

Downloading traefik 1.7.18...


Ran /opt/tljh/user/bin/jupyter lab build --minimize=False --dev-build=False with exit code 1
[LabBuildApp] JupyterLab 1.2.7
[LabBuildApp] Building in /opt/tljh/user/share/jupyter/lab
[LabBuildApp] Building jupyterlab assets (build:prod)
An error occured.
RuntimeError: JupyterLab failed to build
See the log file for details:  /tmp/jupyterlab-debug-fykwpamn.log

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 508, in <module>
    main()
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 496, in main
    ensure_jupyterlab_extensions()
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 194, in ensure_jupyterlab_extensions
    ] + build_options)
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/utils.py", line 32, in run_subprocess
    raise subprocess.CalledProcessError(cmd=cmd, returncode=proc.returncode)
subprocess.CalledProcessError: Command '['/opt/tljh/user/bin/jupyter', 'lab', 'build', '--minimize=False', '--dev-build=False']' returned non-zero exit status 1.

where the log file ends in:

[LabBuildApp] yarn run v1.15.2
$ ensure-max-old-space webpack --config webpack.prod.config.js
child_process.js:650
    throw err;
    ^

Error: Command failed: /opt/tljh/user/share/jupyter/lab/staging/node_modules/.bin/webpack --config webpack.prod.config.js
    at checkExecSyncError (child_process.js:629:11)
    at Object.execFileSync (child_process.js:647:13)
    at Object.<anonymous> (/opt/tljh/user/share/jupyter/lab/staging/node_modules/@jupyterlab/buildutils/lib/ensure-max-old-space.js:38:17)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[LabBuildApp] JupyterLab failed to build
[LabBuildApp] Traceback (most recent call last):

[LabBuildApp]   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterlab/debuglog.py", line 47, in debug_logging
    yield

[LabBuildApp]   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterlab/labapp.py", line 98, in start
    command=command, app_options=app_options)

[LabBuildApp]   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterlab/commands.py", line 459, in build
    command=command, clean_staging=clean_staging)

[LabBuildApp]   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterlab/commands.py", line 669, in build
    raise RuntimeError(msg)

[LabBuildApp] RuntimeError: JupyterLab failed to build

[LabBuildApp] Exiting application: JupyterLab

and further attempts eventually succeeded, after long hang times with “top” showing kswapd0 dominating the CPU, and “free” showing only 60 MB free.

Cheers,

Tim

[consideRatio]

We have recently updated the memory requirements described in the documentation, as well as made updates that reduces the memory footprint. Since this issue relates to memory it seems, I’ll go for a close.