Clean way to make pandas DataFrame from search results?

I would like to create a DataFrame from Elasticsearch Query DSL results. Try as I might, I cannot figure out how to create a DataFrame just from the results. E.g. I have tried:

# Create a basic ES client
client = Elasticsearch(['url'])

# Search
search = Search(using=client)

results = search.execute()

search_dict = results.hits.hits

results_df = pd.DataFrame(search_dict)

However, my DataFrame looks like this:

How can I get the values of the _source column to populate a pandas DataFrame?

[honzakral]

s = Search()
# add any filters/queries....
s = s.query(...)

results_df = pd.DataFrame((d.to_dict() for d in s.scan()))

should be exactly what you need. Alternatively omit the .scan() if you only want the top 10 results (useful for testing).

Hope this helps

[honzakral]

@abhimanyu3 I am sorry, but I cannot help you with any details that are related to pandas or python in general. This issue is for elasticsearch_dsl library. You did get all of the data out of elasticsearch as dicts, to trasnform them into a data frame just pass them into pd.DataFrame as seen in previous examples on this very same ticket. Otherwise please seek help in a forum dedicated to pandas directly on how to transform an interator of dictionaries into a data frame.

Thank you!