Promnesia on MacOS

Привет, буду писать по русски, чтобы более точно выразить мысль.

На MacOS поставить backend не удается, постоянно при запуске pip3 пытается запуститься pip от второй версии. Что приводит к ошибке:

~
✦ at [ 15:17:18 ]  ❯ pip3 install --user promnesia
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting promnesia
  Downloading promnesia-0.11.20200510.tar.gz (344 kB)
     |████████████████████████████████| 344 kB 1.6 MB/s
    ERROR: Command errored out with exit status 1:
     command: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/ss/g0zcdcj13p54s6fj2353vlxh0000gn/T/pip-install-ks_GI5/promnesia/setup.py'"'"'; __file__='"'"'/private/var/folders/ss/g0zcdcj13p54s6fj2353vlxh0000gn/T/pip-install-ks_GI5/promnesia/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/ss/g0zcdcj13p54s6fj2353vlxh0000gn/T/pip-pip-egg-info-dOQXj3
         cwd: /private/var/folders/ss/g0zcdcj13p54s6fj2353vlxh0000gn/T/pip-install-ks_GI5/promnesia/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/ss/g0zcdcj13p54s6fj2353vlxh0000gn/T/pip-install-ks_GI5/promnesia/setup.py", line 41
        *DEPS_INDEXER,
        ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Установив python@3.8 через brew и установив promnesia через явный вызов полного пути до pip3, удалось поставить promnesia, но затем при попытке использования получил массу ошибок при запуске, по всей видимости дополнительные пакеты не были обнаружены.

Для чистоты эксперимента создал docker-версию Promnesia: juev/promnesia-docker. При запуске команды:

docker-compose up --build

собирается и запускается контейнер. При запуске используется конфигурационный файл из примеров. Указывается использование директории /data для хранения базы и файлов. И именно эта директория монтируется к хосту. Но при запуске файлы не создаются и при использовании расширения получаю ошибку:

✦ at [ 16:03:21 ]  ❯ docker-compose up --build
Building promnesia
Step 1/6 : FROM python:3
 ---> 659f826fabf4
Step 2/6 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 0ae916acc38b
Step 3/6 : COPY requirements.txt ./
 ---> Using cache
 ---> 0432b193d16d
Step 4/6 : COPY config.py /
 ---> 4f699ac7631b
Step 5/6 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Running in 855c787f61b2
Collecting promnesia
  Downloading promnesia-0.11.20200521-py3-none-any.whl (53 kB)
Collecting logzero
  Downloading logzero-1.5.0-py2.py3-none-any.whl (14 kB)
Collecting pytz
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting sqlalchemy
  Downloading SQLAlchemy-1.3.17-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting hug
  Downloading hug-2.6.1-py2.py3-none-any.whl (75 kB)
Collecting cachew
  Downloading cachew-0.6.3-py3-none-any.whl (12 kB)
Collecting urlextract
  Downloading urlextract-0.14.0-py3-none-any.whl (18 kB)
Collecting more-itertools
  Downloading more_itertools-8.3.0-py3-none-any.whl (44 kB)
Collecting python-magic
  Downloading python_magic-0.4.18-py2.py3-none-any.whl (8.6 kB)
Collecting tzlocal
  Downloading tzlocal-2.1-py2.py3-none-any.whl (16 kB)
Collecting falcon==2.0.0
  Downloading falcon-2.0.0-py2.py3-none-any.whl (163 kB)
Collecting requests
  Downloading requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting uritools
  Downloading uritools-3.0.0-py3-none-any.whl (12 kB)
Collecting appdirs
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting idna
  Downloading idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Installing collected packages: pytz, sqlalchemy, falcon, idna, certifi, chardet, urllib3, requests, hug, cachew, uritools, appdirs, urlextract, more-itertools, python-magic, tzlocal, promnesia, logzero
Successfully installed appdirs-1.4.4 cachew-0.6.3 certifi-2020.4.5.1 chardet-3.0.4 falcon-2.0.0 hug-2.6.1 idna-2.9 logzero-1.5.0 more-itertools-8.3.0 promnesia-0.11.20200521 python-magic-0.4.18 pytz-2020.1 requests-2.23.0 sqlalchemy-1.3.17 tzlocal-2.1 uritools-3.0.0 urlextract-0.14.0 urllib3-1.25.9
Removing intermediate container 855c787f61b2
 ---> f7f4a3fc55d6
Step 6/6 : CMD [ "promnesia", "serve", "--db", "/config.py" ]
 ---> Running in 6372776476d7
Removing intermediate container 6372776476d7
 ---> 3aecd1bca750

Successfully built 3aecd1bca750
Successfully tagged promnesia-docker_promnesia:latest
Recreating promnesia ... done
Attaching to promnesia
promnesia    | [INFO    2020-05-23 13:06:11,996 promnesia server.py:295] Running server: ['python3', '-m', 'hug', '-p', '13131', '-f', '/usr/local/lib/python3.8/site-packages/promnesia/server.py']
promnesia    | [INFO    2020-05-23 13:06:21,741 promnesia server.py:123] url: https://www.gwern.net/
promnesia    | [INFO    2020-05-23 13:06:21,743 promnesia server.py:126] normalised url: gwern.net
promnesia    | [INFO    2020-05-23 13:06:21,745 promnesia server.py:99] Reloading DB: PathWithMtime(path=PosixPath('/config.py'), mtime=1590239156.0)
promnesia    | [INFO    2020-05-23 13:06:21,765 promnesia server.py:132] query: SELECT visits.norm_url, visits.orig_url, visits.dt, visits.locator_title, visits.locator_href, visits.src, visits.context, visits.duration
promnesia    |     FROM visits
promnesia    |     WHERE visits.norm_url = ? OR visits.context IS NOT NULL AND (visits.norm_url LIKE ? || '%' ESCAPE '/')
promnesia    | Traceback (most recent call last):
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1283, in _execute_context
promnesia    |     self.dialect.do_execute(
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
promnesia    |     cursor.execute(statement, parameters)
promnesia    | sqlite3.DatabaseError: file is not a database
promnesia    |
promnesia    | The above exception was the direct cause of the following exception:
promnesia    |
promnesia    | Traceback (most recent call last):
promnesia    |   File "/usr/local/lib/python3.8/wsgiref/handlers.py", line 137, in run
promnesia    |     self.result = application(self.environ, self.start_response)
promnesia    |   File "/usr/local/lib/python3.8/site-packages/falcon/api.py", line 269, in __call__
promnesia    |     responder(req, resp, **params)
promnesia    |   File "/usr/local/lib/python3.8/site-packages/hug/interface.py", line 947, in __call__
promnesia    |     raise exception
promnesia    |   File "/usr/local/lib/python3.8/site-packages/hug/interface.py", line 918, in __call__
promnesia    |     self.call_function(input_parameters), context, request, response, **kwargs
promnesia    |   File "/usr/local/lib/python3.8/site-packages/hug/interface.py", line 840, in call_function
promnesia    |     return self.interface(**parameters)
promnesia    |   File "/usr/local/lib/python3.8/site-packages/hug/interface.py", line 129, in __call__
promnesia    |     return __hug_internal_self._function(*args, **kwargs)
promnesia    |   File "/usr/local/lib/python3.8/site-packages/promnesia/server.py", line 191, in visits
promnesia    |     return search_common(
promnesia    |   File "/usr/local/lib/python3.8/site-packages/promnesia/server.py", line 135, in search_common
promnesia    |     visits = [binder.from_row(row) for row in conn.execute(query)]
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
promnesia    |     return meth(self, multiparams, params)
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
promnesia    |     return connection._execute_clauseelement(self, multiparams, params)
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
promnesia    |     ret = self._execute_context(
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1323, in _execute_context
promnesia    |     self._handle_dbapi_exception(
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1517, in _handle_dbapi_exception
promnesia    |     util.raise_(
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
promnesia    |     raise exception
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1283, in _execute_context
promnesia    |     self.dialect.do_execute(
promnesia    |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
promnesia    |     cursor.execute(statement, parameters)
promnesia    | sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) file is not a database
promnesia    | [SQL: SELECT visits.norm_url, visits.orig_url, visits.dt, visits.locator_title, visits.locator_href, visits.src, visits.context, visits.duration
promnesia    | FROM visits
promnesia    | WHERE visits.norm_url = ? OR visits.context IS NOT NULL AND (visits.norm_url LIKE ? || '%' ESCAPE '/')]
promnesia    | [parameters: ('gwern.net', 'gwern.net')]
promnesia    | (Background on this error at: http://sqlalche.me/e/4xp6)
^CGracefully stopping... (press Ctrl+C again to force)
Stopping promnesia   ... done

То есть с использованием docker удалось более или менее завести backend, но с конфигурацией пока проблема и в результате база не создается.

Как понял проблема в том, что не запускался index, пересоздал файл compose, добавив вызов создания индексов (second commit), но при запуске индексирования возникает ошибка:

Attaching to promnesia-docker_promnesia-index_1, promnesia
promnesia-index_1  | Traceback (most recent call last):
promnesia-index_1  |   File "/usr/local/bin/promnesia", line 8, in <module>
promnesia-index_1  |     sys.exit(main())
promnesia-index_1  |   File "/usr/local/lib/python3.8/site-packages/promnesia/__main__.py", line 169, in main
promnesia-index_1  |     do_index(config_file=args.config)
promnesia-index_1  |   File "/usr/local/lib/python3.8/site-packages/promnesia/__main__.py", line 60, in do_index
promnesia-index_1  |     config.load_from(config_file)
promnesia-index_1  |   File "/usr/local/lib/python3.8/site-packages/promnesia/config.py", line 56, in load_from
promnesia-index_1  |     instance = import_config(config_file)
promnesia-index_1  |   File "/usr/local/lib/python3.8/site-packages/promnesia/config.py", line 72, in import_config
promnesia-index_1  |     spec.loader.exec_module(mod) # type: ignore
promnesia-index_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
promnesia-index_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
promnesia-index_1  |   File "/config.py", line 11, in <module>
promnesia-index_1  |     from promnesia.sources import instapaper, pocket, twitter, roamresearch, hypothesis
promnesia-index_1  |   File "/usr/local/lib/python3.8/site-packages/promnesia/sources/instapaper.py", line 6, in <module>
promnesia-index_1  |     import my.instapaper as ip
promnesia-index_1  | ModuleNotFoundError: No module named 'my'
promnesia-docker_promnesia-index_1 exited with code 1

потратил немало времени на то, чтобы запустить promnesia на macos. Но пока безрезультатно.

[karlicoss]

Добавил еще чутка документации в HPI, надеюсь может некоторые моменты прояснить 😃

“Data flow”

[juev]

https://github.com/karlicoss/promnesia/issues/98#issuecomment-633060994

имя файла копировал из README:

run the server: promnesia serve --db OUTPUT_DIR/promnesia.sqite.