Bug: CHROME_USER_DATA_DIR not working for login

Describe the bug

Hi

I wish to use ArchiveBox to save content from subscriptions behind paywalls.

I have exported cookies in Netscape format (for WGET I understand - COOKIES_FILE) and shared Chromium paths with Archivebox (CHROME_USER_DATA_DIR).

Neither have resulted in logins (evident in screenshots and denied access).

The latter Chromium user data folder process has been particularly problematic - rejecting the path provided (no “Default” profile found where it was there) before suddenly accepting it.

To ensure a usable cookie, I created a VNC session on the server, browsed via Chromium to the site, logged in and then tried using archivebox and providing the path to the User Data folder.

Great project - but the login feature is critical for me to archive content behind paywalls.

Regards

Marc

Steps to reproduce

[SERVER_CONFIG]
SECRET_KEY = XXXXXXXXXXXXX

[ARCHIVE_METHOD_OPTIONS]
COOKIES_FILE = /home/XX/cookies.txt
CHROME_USER_DATA_DIR = /home/XX/.config/chromium

[ARCHIVE_METHOD_TOGGLES]
SAVE_ARCHIVE_DOT_ORG = False

Screenshots or log output

Could not find profile "Default" in CHROME_USER_DATA_DIR.
archivebox_1  |     /home/XX/.config/chromium
archivebox_1  |     Make sure you set it to a Chrome user data directory containing a Default profile folder.

ArchiveBox version

 ArchiveBox v0.6.3: archivebox server

ArchiveBox Dev docker image running on Debian Testing

[ga-it]

I have since found synchronizing the chromium version in the docker file and on the host to be a nightmare.

If they are not exactly synced, the profiles become incompatible.

The chromium version in the current dev image is 90.0.4430.212. When downloading this via https://chromium.cypress.io/ it actually downloads 90.0.4430.0 resulting in an inability to reaccess the profile after use by the docker image.

To prevent this, I have followed the following workaround:

1. install desired chromium version in directory shared as a Docker volume (used “data” for ease)
2. run vncserver as archivebox user and run chromium in vnc session to generate cookies
3. Close chromium in vncserver session
4. chmod -R ugo+rwx /opt/archivebox/.config/chromium
5. mount /opt/archivebox/.config/chromium as docker volume /data/chromium
6. set CHROME_USER_DATA_DIR = /data/chromium
7. set CHROME_BINARY = /data/chrome-linux/chrome (installed version of chrome now common between host VNC session and docker container)
8. chown -R archivebox:archivebox /opt/archivebox/

I found each step to be crucial especially permissions.

Now profile is generated and used by same instance of chrome on docker host and container.

[ga-it]

Thanks @pirate

I have now successfully got Archivebox to use my Chromium profile.

Resolution:

1. Correct mount in Docker and reference in config
2. Exactly the same version of Chromium outside the Docker image and within the Docker image
3. Chromium outside Docker closed when running Archivebox
4. Read/write permissions on Chromium profile

ArchiveBox.conf

[SERVER_CONFIG] SECRET_KEY = XX

[ARCHIVE_METHOD_OPTIONS] CHROME_USER_DATA_DIR = /data/chromium

[ARCHIVE_METHOD_TOGGLES] SAVE_ARCHIVE_DOT_ORG = False

docker-compose.yml

services: archivebox: XX ports: XX environment: XX volumes: XX - /XX/.config/chromium:/data/chromium