Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Handling intermediate pages after form submit

See original GitHub issue

Version of the custom_component

6.0.0

Configuration

multiscrape:
  - resource: "https://www.amaysim.com.au/my-account/my-amaysim/products"
    name: Amaysim
    scan_interval: 30
    log_response: true
    method: GET
    form_submit:
      submit_once: true
      resubmit_on_error: false
      resource: "https://accounts.amaysim.com.au/identity/login"
      select: "#new_session"
      input:
        username: !secret amaysim_username
        password: !secret amaysim_password
    sensor:
      - select: "#outer_wrap > div.inner-wrap > div.page-container > div:nth-child(2) > div.row.margin-bottom > div.small-12.medium-6.columns > div > div > div:nth-child(2) > div:nth-child(2)"
        name: amaysim_remaining_data
        value_template: "{{ value }}"

Describe the bug

I can see from the logs that after successfully submitting the form, it says that it is getting the data from the resource url, with a response code of 200.

I have pasted the contents from the log_response file page_soup.txt below

<html><body><p>/**/('OK')</p></body></html>

Below is the content from the form_submit_response_body.txt

<html><body>You are being <a href="https://accounts.amaysim.com.au/identity">redirected</a>.</body></html>

It seems that after submitting the form, the sensor is scraping data from the intermediate page and not from the resource url.

Debug log


2022-02-01 18:51:06 DEBUG (MainThread) [custom_components.multiscrape] Amaysim # Setting up multiscrape with config:
OrderedDict([('resource', 'https://www.amaysim.com.au/my-account/my-amaysim/products'), ('name', 'Amaysim'), ('scan_interval', datetime.timedelta(seconds=30)), ('log_response', True), ('method', 'GET'), ('form_submit', OrderedDict([('submit_once', True), ('resubmit_on_error', False), ('resource', 'https://accounts.amaysim.com.au/identity/login'), ('select', '#new_session'), ('input', OrderedDict([('username', 'xxxx'), ('password', 'yyyy')]))])), ('sensor', [OrderedDict([('select', Template("#outer_wrap > div.inner-wrap > div.page-container > div:nth-child(2) > div.row.margin-bottom > div.small-12.medium-6.columns > div > div > div:nth-child(2) > div:nth-child(2)")), ('name', 'amaysim_remaining_data'), ('value_template', Template("{{ value }}")), ('force_update', False)])]), ('parser', 'lxml'), ('timeout', 10), ('verify_ssl', True)])
2022-02-01 18:51:06 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Initializing scraper
2022-02-01 18:51:06 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Found form-submit config
2022-02-01 18:51:07 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Refresh triggered
2022-02-01 18:51:07 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Continue with form-submit
2022-02-01 18:51:07 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Requesting page with form from: https://accounts.amaysim.com.au/identity/login
2022-02-01 18:51:07 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Executing form_page-request with a GET to url: https://accounts.amaysim.com.au/identity/login.
2022-02-01 18:51:17 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response status code received: 200
2022-02-01 18:51:17 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response headers written to file: /config/multiscrape/amaysim/form_page_response_headers.txt
2022-02-01 18:51:17 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response headers written to file: /config/multiscrape/amaysim/form_page_response_body.txt
2022-02-01 18:51:17 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Start trying to capture the form in the page
2022-02-01 18:51:17 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Parse HTML with BeautifulSoup parser lxml
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # The page with the form parsed by BeautifulSoup has been written to file: /config/multiscrape/amaysim/form_page_soup.txt
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Try to find form with selector #new_session
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Found the form, now finding all input fields
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Found the following fields: {'utf8': '✓', 'authenticity_token': 'abc123', 'continue': None, 'ga_client_id': '', 'username': None, 'password': ''}
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Found form action /identity/sessions and method post
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Determined the url to submit the form to: https://accounts.amaysim.com.au/identity/sessions
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Merged input fields with input data in config. Result: {'utf8': '✓', 'authenticity_token': 'abc123', 'continue': None, 'ga_client_id': '', 'username': 'xxxx', 'password': 'yyyy'}
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Going now to submit the form
2022-02-01 18:51:18 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Executing form_submit-request with a post to url: https://accounts.amaysim.com.au/identity/sessions.
2022-02-01 18:51:19 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response status code received: 302
2022-02-01 18:51:19 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response headers written to file: /config/multiscrape/amaysim/form_submit_response_headers.txt
2022-02-01 18:51:19 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response headers written to file: /config/multiscrape/amaysim/form_submit_response_body.txt
2022-02-01 18:51:19 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Form seems to be submitted succesfully! Now continuing to update data for sensors
2022-02-01 18:51:19 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Updating data from https://www.amaysim.com.au/my-account/my-amaysim/products
2022-02-01 18:51:19 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Executing page-request with a get to url: https://www.amaysim.com.au/my-account/my-amaysim/products.
2022-02-01 18:51:20 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response status code received: 200
2022-02-01 18:51:20 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response headers written to file: /config/multiscrape/amaysim/page_response_headers.txt
2022-02-01 18:51:20 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response headers written to file: /config/multiscrape/amaysim/page_response_body.txt
2022-02-01 18:51:20 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Data succesfully refreshed. Sensors will now start scraping to update.
2022-02-01 18:51:20 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Start loading the response in BeautifulSoup.
2022-02-01 18:51:20 DEBUG (MainThread) [custom_components.multiscrape.scraper] Amaysim # Response headers written to file: /config/multiscrape/amaysim/page_soup.txt

Issue Analytics

State:
Created 2 years ago
Comments:29 (12 by maintainers)

Top GitHub Comments

1reaction

danieldotnlcommented, Apr 28, 2022

I think it’s related to this issue: https://github.com/home-assistant/core/pull/61293 I’ll prepare a fix.

0reactions

kaizersojecommented, Nov 11, 2022

Hi @danieldotnl, my original issue is still not working. I removed select, submit_once and resource from within form_submit as well. It still does not work after that.