flatten and line oriented tweets

The twarc2 flatten command will make any expansions/fields inline to a tweet and also make the output file line oriented json – where each line is a tweet.

It would be useful for plugin development if twarc.expansions contained a function (e.g. jsonl) that returned a list where every element was a tweet dictionary whether or not the object it was passed had a data key that was a list or an object.

I find myself repeating the same conditional logic in places where I want to process a file of tweets.

Does that make sense? Or maybe flatten() should just do it?

[edsu]

Yes @SamHames I think I’m finally coming around to your idea that twarc2 should always write whatever the API gives it. Maybe I needed the experience of using --flatten in the wild to recognize what you suspected early on.

Part of the difficulty here are the semantics of flattened. twarc.expansions.flatten moves the includes in a response inline to the objects that reference them. It does not ensure that response['data'] is a list as the streaming API returns response['data'] as a dictionary object. This inconsistency was introduced by the Twitter API and not by us, and it means that any code that processes the data will need to peek at the type of response['data'] in order to know what to do. It’s not a big deal but it’s bothersome.

But maybe we can still put the genie back in the bottle? I propose that we:

1. remove --flatten from all twarc2 sub-commands, but still offer twarc2 flatten, mostly for pipeline use.
2. update twarc.expansions.flatten so that it always returns a list of dictionary objects where each object is a tweet e.g. {"id": "12345", "text": "Just setting up my twttr", "author": {"username": "jack" ...
3. add twarc.expansions.ensure_flattened that checks if data is flattened, and flattens if needed.
4. Plugins and other code that want to process flattened data can use twarc.expansions.ensure_flattened directly.

Please vote with your thumbs: 👍 👎 or leave comments to propose changes to this plan.

[edsu]

Thanks for catching that BIG mistake @igorbrigadir … I would have thought that the unit tests would have caught it but they were only making sure that the id for a referenced tweet was defined, instead of a tweet property that would have actually been added by flattening. Whoops.

In cfa3d55f6bb1a87cece21fb5231806b339f0c1d4 I updated the test to test correctly for flattened referenced tweets to see if it failed (it did) and then I added back these lines and it passed.