How to wait for response content to load and change the DOM content?

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 1.18.1
• Platform / OS version: Windows 10
• Node.js version: 8.11.4

What steps will reproduce the problem? I am selecting an option from dropdown and this sends a network request. As soon as there is response of this request, it updates the table content. But as I applied the scraping code to wait for the response ( using waitForResponse(url) ), but it scrapes before the content of the table is changed. I want it to wait for a while till the table content is changed.

I have tried,

1. Waiting for some time after the response using ( waitFor (1000) )

2. Waiting for some other existing DOM element. So that it waits for a little for DOM to update.

3. Reloading the page

But nothing seems to work.

The code below is inside an async function which is called later and the result is printed.

await page.waitForSelector('#ctl00_ContentPlaceHolder1_SavedSearchesDropDownList');
await Promise.all([
         page.select('#ctl00_ContentPlaceHolder1_SavedSearchesDropDownList','option2'),
			 
         page.waitForResponse('http://www.someurl.com/Pages/A/B.aspx')
]);
const result = await page.evaluate(() => {
    var result_arr = [];
let table_rows = document.querySelectorAll('#ctl00_ContentPlaceHolder1_DomainsListView_itemContainer tr');

		for(let row of table_rows){
			let value1 = row.querySelector('td:nth-of-type(2)').innerText;
			let value2 = row.querySelector('td:nth-of-type(4)').innerText;

			 result_arr.push({value1,value2});
		};
		return result_arr;
	});

	return result;

What is the expected result? I expect this to wait till the dom is changed by the network response. something like await page.waitFor({waitUntill:'domcontentchanged})

What happens instead? This scrapes instantly as there is a response

[richardnassar-external]

await page.waitForNavigation() , works but what if there is no navigation ?

[Cado21]

i have problem with this, the problem is kind of similar where my case is: the network already idle after navigation and there’s a search field where we can input our search value there then the dom will change accordingly to what we search

example

await page.$eval('#searchfield', el => el.value = 'test');
await page.screenshot({path: 'ss.png}); 

the screenshot result is not what I expecting,

the result of the screenshot only shows changes on searchfield but there’s no changes on search result (even tho there’s no backend call to filter, and it was filtered by javascript)

any idea how do I wait for the dom changes? i tried several ways like networkidle2 and also manually sleep for several second but it’s not working.

please help

---

Found the Solution I change the eval by this:

await page.type('#searchfield', 'test')

hope this helps