question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Video Related Json's are not good

See original GitHub issue

There are video-related posts, that video is none.

   {
      "post_id":"1908691579309019",
      "text":"Справа Тетяни Чорновол – це атака на Україну і її досягнення після Революції Гідності – Артур Герасимов👉 https://bit.ly/3lbW6Wb\n\n📌 Друзі, підписуйтеся на офіційні канали партії!\nTelegram👉 bit.ly/2KOngQh\nInstagram👉 bit.ly/34ZDz6H\nTwitter👉 bit.ly/33Uld4F\nYouTube👉 bit.ly/2GZgFTE",
      "post_text":"Справа Тетяни Чорновол – це атака на Україну і її досягнення після Революції Гідності – Артур Герасимов👉 https://bit.ly/3lbW6Wb\n\n📌 Друзі, підписуйтеся на офіційні канали партії!\nTelegram👉 bit.ly/2KOngQh\nInstagram👉 bit.ly/34ZDz6H\nTwitter👉 bit.ly/33Uld4F\nYouTube👉 bit.ly/2GZgFTE",
      "shared_text":"",
      "time":"2021-07-30 08:45:29",
      "image":"None",
      "image_lowquality":"https://z-m-scontent.fnak4-1.fna.fbcdn.net/v/t15.5256-10/cp0/e15/q65/s350x350/210219254_1908702292641281_2802185200179206338_n.jpg?_nc_cat=102&ccb=1-3&_nc_sid=ccf8b3&_nc_ohc=IrIY3iNbeoYAX8tHpWC&_nc_ad=z-m&_nc_cid=1277&_nc_eh=780f26274c2e3dd2a4b3010ade0469c2&_nc_rml=0&_nc_ht=z-m-scontent.fnak4-1.fna&oh=a321cef17612c99106c0658dc756212d&oe=610CF058",
      "images":[
         
      ],
      "images_description":[
         
      ],
      "images_lowquality":[
         "https://z-m-scontent.fnak4-1.fna.fbcdn.net/v/t15.5256-10/cp0/e15/q65/s350x350/210219254_1908702292641281_2802185200179206338_n.jpg?_nc_cat=102&ccb=1-3&_nc_sid=ccf8b3&_nc_ohc=IrIY3iNbeoYAX8tHpWC&_nc_ad=z-m&_nc_cid=1277&_nc_eh=780f26274c2e3dd2a4b3010ade0469c2&_nc_rml=0&_nc_ht=z-m-scontent.fnak4-1.fna&oh=a321cef17612c99106c0658dc756212d&oe=610CF058"
      ],
      "images_lowquality_description":[
         "None"
      ],
      "video":"None",
      "video_duration_seconds":456,
      "video_height":720,
      "video_id":"1908691579309019",
      "video_quality":"720p",
      "video_size_MB":"None",
      "video_thumbnail":"None",
      "video_watches":7886,
      "video_width":1280,
      "likes":734,
      "comments":72,
      "shares":109,
      "post_url":"https://facebook.com/832922343451368/posts/1908691579309019",
      "link":"https://bit.ly/3lbW6Wb",
      "user_id":"832922343451368",
      "username":"Європейська Солідарність",
      "user_url":"https://facebook.com/EuropeanSolidarity.official/?__tn__=C-R",
      "is_live":false,
      "factcheck":"None",
      "shared_post_id":"None",
      "shared_time":"None",
      "shared_user_id":"None",
      "shared_username":"None",
      "shared_post_url":"None",
      "available":true,
      "comments_full":"None",
      "reactors":"None",
      "w3_fb_url":"None",
      "reactions":"None",
      "reaction_count":"None",
      "image_id":"None",
      "image_ids":[
         
      ],
      "fetched_time":"2021-08-02 07:31:38"
   }

If we specifically visit this post anonymously also the json result is not good

>>> from facebook_scraper import *
>>> proxy = "http://usernamename:password@ip:port"
>>> set_proxy(proxy)
>>> posts = list(get_posts(post_urls=[521405578978039], timeout=90))
>>> posts
[{'original_request_url': 521405578978039, 'post_url': 'https://m.facebook.com/521405578978039'}]

There are several posts has video key: None

534809420905208
517593769498197

@neon-ninja please check

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

3reactions
neon-ninjacommented, Sep 17, 2021

@neon-ninja Hi!.. When i scrape a live video fb posts, the script returns the below JSON, which is incomplete. Any idea regarding this issue or any solution to get the complete JSON

posts = list(get_posts(post_urls=[408850460829674], timeout=90, cookies=cooki_path, options={'comments': True, 'reactors': True}))
>>> posts
[{'original_request_url': 408850460829674, 'post_url': 'https://m.facebook.com/408850460829674'}]

The scraper doesn’t support URLs that resolve to https://m.facebook.com/watch/?ref=watch_permalink. You need a URL like https://m.facebook.com/story.php?story_fbid=4577884442318686&id=174053246035183&m_entstream_source=video_home&player_suborigin=entry_point&player_format=permalink. https://github.com/kevinzg/facebook-scraper/commit/d95de50318d86b4375701edbfc36a003b9a712a3 should help with crafting these links. With this commit:

set_cookies("cookies.json")
pprint(next(get_posts(post_urls=[408850460829674])))

outputs:

{'available': True,
 'comments': 4,
 'comments_full': None,
 'factcheck': None,
 'image': None,
 'image_id': None,
 'image_ids': [],
 'image_lowquality': 'https://scontent.fakl8-1.fna.fbcdn.net/v/t15.5256-10/cp0/e15/q65/p600x600/241688203_408858344162219_7987192019845083941_n.jpg?_nc_cat=100&ccb=1-5&_nc_sid=ccf8b3&efg=eyJpIjoidCJ9&_nc_ohc=3CREk9lzq_oAX_D-Il9&_nc_ht=scontent.fakl8-1.fna&oh=1a4e1f1df671392680643c15c9e259f5&oe=614802E2',
 'images': [],
 'images_description': [],
 'images_lowquality': ['https://scontent.fakl8-1.fna.fbcdn.net/v/t15.5256-10/cp0/e15/q65/p600x600/241688203_408858344162219_7987192019845083941_n.jpg?_nc_cat=100&ccb=1-5&_nc_sid=ccf8b3&efg=eyJpIjoidCJ9&_nc_ohc=3CREk9lzq_oAX_D-Il9&_nc_ht=scontent.fakl8-1.fna&oh=1a4e1f1df671392680643c15c9e259f5&oe=614802E2'],
 'images_lowquality_description': [None],
 'is_live': False,
 'likes': 7,
 'link': None,
 'links': [],
 'original_request_url': 408850460829674,
 'post_id': '408850460829674',
 'post_text': 'La sensación de movimiento es habitualmente de balanceo, '
              'giratorio o de desplazamiento, propio o de la base de '
              'sustentación (como inclinación del suelo o precipitación en el '
              'vacío) pero, en realidad, esas modificaciones en el sujeto o '
              'entorno, no ocurren.\n'
              '\n'
              'El mareo es una alteración del equilibrio también en el que '
              'englobamos sensaciones desagradables de vacío en la cabeza, '
              'inseguridad e inestabilidad, sensación de malestar y de desmayo '
              'inminente, acompañado todo ello de náusea y vómito, sudoración '
              'fría y palidez.\n'
              '\n'
              'El vértigo hace referencia siempre a una alteración del sistema '
              'vestibular (alojado en el oído interno y cuyo centro '
              'neurológico está en el tronco del encéfalo), mientras que el '
              'mareo es un trastorno del equilibrio no siempre relacionado con '
              'éste.',
 'post_url': 'https://facebook.com/story.php?story_fbid=408850460829674&id=174053246035183',
 'reaction_count': None,
 'reactions': None,
 'reactors': None,
 'shared_post_id': None,
 'shared_post_url': None,
 'shared_text': '',
 'shared_time': None,
 'shared_user_id': None,
 'shared_username': None,
 'shares': 0,
 'text': 'La sensación de movimiento es habitualmente de balanceo, giratorio o '
         'de desplazamiento, propio o de la base de sustentación (como '
         'inclinación del suelo o precipitación en el vacío) pero, en '
         'realidad, esas modificaciones en el sujeto o entorno, no ocurren.\n'
         '\n'
         'El mareo es una alteración del equilibrio también en el que '
         'englobamos sensaciones desagradables de vacío en la cabeza, '
         'inseguridad e inestabilidad, sensación de malestar y de desmayo '
         'inminente, acompañado todo ello de náusea y vómito, sudoración fría '
         'y palidez.\n'
         '\n'
         'El vértigo hace referencia siempre a una alteración del sistema '
         'vestibular (alojado en el oído interno y cuyo centro neurológico '
         'está en el tronco del encéfalo), mientras que el mareo es un '
         'trastorno del equilibrio no siempre relacionado con éste.',
 'time': datetime.datetime(2021, 9, 15, 6, 22, 3),
 'timestamp': None,
 'user_id': '174053246035183',
 'user_url': 'https://facebook.com/TelepacificoTV/?refid=52&__tn__=C-R',
 'username': 'Telepacífico',
 'video': 'https://video.fakl8-1.fna.fbcdn.net/v/t42.1790-2/10000000_227892262630430_340897050108360632_n.mp4?_nc_cat=109&vs=749210a2348fe401&_nc_vs=HBksFQAYJEdJQ1dtQUFlZEZ4TlJNOEFBTGozcjVnUEhMc0VidjRHQUFBRhUAAsgBABUAGCRHSUNXbUFCSGh1Yjd6bndDQUo5VmF6OUE0T0E0YnY0R0FBQUYVAgLIAQAoRC1pICclcycgLWZiX3VzZV90ZmR0X3N0YXJ0dGltZSAxIC1pICclcycgLWMgY29weSAtbW92ZmxhZ3MgZmFzdHN0YXJ0KwaIEnByb2dyZXNzaXZlX3JlY2lwZQExIG1lYXN1cmVfb3JpZ2luYWxfcmVzb2x1dGlvbl9zc2ltAChjb21wdXRlX3NzaW1fb25seV9hdF9vcmlnaW5hbF9yZXNvbHV0aW9uABFkaXNhYmxlX3Bvc3RfcHZxcwANc3Vic2FtcGxlX2ZwcwAQdm1hZl9lbmFibGVfbnN1YgAVACUAHAAAJq74vpGd9rkBFQIoAkMzGANhdjEcF0C5Wm8an753GBlkYXNoX2xpdmVfbWRfZnJhZ18yX3ZpZGVvEgAYGHZpZGVvcy52dHMuY2FsbGJhY2sucHJvZDgSVklERU9fVklFV19SRVFVRVNUGwiIFW9lbV90YXJnZXRfZW5jb2RlX3RhZwZvZXBfc2QTb2VtX3JlcXVlc3RfdGltZV9tcw0xNjMxODM2MDUxOTc2DG9lbV9jZmdfcnVsZQp3YXNsaXZlX3NkE29lbV9yb2lfcmVhY2hfY291bnQGMzEwMDU4DG9lbV92aWRlb19pZA80MDg4NTA0NjA4Mjk2NzQSb2VtX3ZpZGVvX2Fzc2V0X2lkDzQwODg1MDQ0MDgyOTY3NhVvZW1fdmlkZW9fcmVzb3VyY2VfaWQPNDA4ODUwNDM3NDk2MzQzHG9lbV9zb3VyY2VfdmlkZW9fZW5jb2RpbmdfaWQPMzgxMDM4OTA2ODIzMzQ3JQQcACW%2BARsHiAFzBDI0MzQCY2QKMjAyMS0wOS0xNANyY2IGMzEwMDAwA2FwcBxDb250ZW50IFRhYiBvZiBhIFBhZ2Ugb24gd3d3AmN0CUJST0FEQ0FTVBNvcmlnaW5hbF9kdXJhdGlvbl9zCDY0OTAuNDM0AnRzD29lcF9wcm9ncmVzc2l2ZQA%3D&ccb=1-5&_nc_sid=cb20ac&efg=eyJ2ZW5jb2RlX3RhZyI6Im9lcF9zZCJ9&_nc_ohc=9R5M7Crcqc0AX-WiC5c&_nc_rml=0&_nc_ht=video.fakl8-1.fna&oh=a4ad7f7dc89b3f96ae98c0cb40e53357&oe=6143F758&_nc_rid=99e54c1676bc400&_nc_vts_prog=1&vts=1&_nc_vts_internal=1',
 'video_duration_seconds': 6491,
 'video_height': 720,
 'video_id': '408850460829674',
 'video_quality': '720p',
 'video_size_MB': None,
 'video_thumbnail': 'https://scontent.fakl8-1.fna.fbcdn.net/v/t15.5256-10/cp0/e15/q65/p600x600/241688203_408858344162219_7987192019845083941_n.jpg?_nc_cat=100&ccb=1-5&_nc_sid=ccf8b3&efg=eyJpIjoidCJ9&_nc_ohc=3CREk9lzq_oAX_D-Il9&_nc_ht=scontent.fakl8-1.fna&oh=1a4e1f1df671392680643c15c9e259f5&oe=614802E2',
 'video_watches': 369,
 'video_width': 1280,
 'w3_fb_url': None,
 'with': None}

can we have an identifier on the sample JSON, that the post is a live video post

This already exists, it’s the is_live key. If you mean a video that “was live”, https://github.com/kevinzg/facebook-scraper/commit/c1671dde0ddd1d5ce2f9facfeb88f8590f15228b adds that as was_live

0reactions
fashan7commented, Sep 16, 2021

@neon-ninja when we scrape using the forum method, we can get some basic details of live videos. can we have an identifier on the sample JSON, that the post is a live video post Live video

set_get_json_as_temp = []
for post in get_posts("TelepacificoTV", pages=5, timeout=45, options={'reactors': True}):
	set_get_json_as_temp.append(post)

{
      "post_id":"1039385906890308",
      "text":"Expertos advierten que la cepa Mu del Covid-19 podría afectar más fuerte a los niños no vacunados. Telepacífico Noticias ¡Tenemos muchas noticias para usted!",
      "post_text":"Expertos advierten que la cepa Mu del Covid-19 podría afectar más fuerte a los niños no vacunados. Telepacífico Noticias ¡Tenemos muchas noticias para usted!",
      "shared_text":"",
      "time":"2021-09-15 11:26:27",
      "timestamp":1631707490,
      "image":"None",
      "image_lowquality":"https://scontent-sin6-2.xx.fbcdn.net/v/t15.5256-10/cp0/e15/q65/p600x600/241752920_1039403706888528_4117109431631879057_n.jpg?_nc_cat=109&ccb=1-5&_nc_sid=ccf8b3&_nc_ohc=Yn7KQz1Tts8AX-75yH7&_nc_ht=scontent-sin6-2.xx&oh=728ba01397b16b883a7c8d8349b8ae44&oe=61485D60",
      "images":[
         
      ],
      "images_description":[
         
      ],
      "images_lowquality":[
         "https://scontent-sin6-2.xx.fbcdn.net/v/t15.5256-10/cp0/e15/q65/p600x600/241752920_1039403706888528_4117109431631879057_n.jpg?_nc_cat=109&ccb=1-5&_nc_sid=ccf8b3&_nc_ohc=Yn7KQz1Tts8AX-75yH7&_nc_ht=scontent-sin6-2.xx&oh=728ba01397b16b883a7c8d8349b8ae44&oe=61485D60"
      ],
      "images_lowquality_description":[
         "None"
      ],
      "video":"https://video-sin6-3.xx.fbcdn.net/v/t42.1790-2/10000000_410123077345243_4495363304045055649_n.mp4?_nc_cat=104&vs=aa0b372a97c88b2&_nc_vs=HBksFQAYJEdJQ1dtQURiMzFZNEFYVUJBS0dPN29rYnZXSV9idjRHQUFBRhUAAsgBABUAGCRHSUNXbUFBVWpiMWdSMjRCQVBKNE41ekpsY0JkYnY0R0FBQUYVAgLIAQAoRC1pICclcycgLWZiX3VzZV90ZmR0X3N0YXJ0dGltZSAxIC1pICclcycgLWMgY29weSAtbW92ZmxhZ3MgZmFzdHN0YXJ0KwaIEnByb2dyZXNzaXZlX3JlY2lwZQExIG1lYXN1cmVfb3JpZ2luYWxfcmVzb2x1dGlvbl9zc2ltAChjb21wdXRlX3NzaW1fb25seV9hdF9vcmlnaW5hbF9yZXNvbHV0aW9uABFkaXNhYmxlX3Bvc3RfcHZxcwANc3Vic2FtcGxlX2ZwcwAQdm1hZl9lbmFibGVfbnN1YgAVACUAHAAAJqSD6Z%2BY1NgDFQIoAkMzGANhdjEcF0Cb4yfvnbItGBlkYXNoX2xpdmVfbWRfZnJhZ18yX3ZpZGVvEgAYGHZpZGVvcy52dHMuY2FsbGJhY2sucHJvZDgSVklERU9fVklFV19SRVFVRVNUGwiIFW9lbV90YXJnZXRfZW5jb2RlX3RhZwZvZXBfc2QTb2VtX3JlcXVlc3RfdGltZV9tcw0xNjMxNzkyNDAwMzMwDG9lbV9jZmdfcnVsZQp3YXNsaXZlX3NkE29lbV9yb2lfcmVhY2hfY291bnQGMzA5OTc5DG9lbV92aWRlb19pZBAxMDM5Mzg1OTA2ODkwMzA4Em9lbV92aWRlb19hc3NldF9pZBAxMDM5Mzg1MzQzNTU3MDMxFW9lbV92aWRlb19yZXNvdXJjZV9pZBAxMDM5Mzg1MzQwMjIzNjk4HG9lbV9zb3VyY2VfdmlkZW9fZW5jb2RpbmdfaWQPMzExNDY0MTAwNzUwNzY1JQQcACW%2BARsHiAFzAzU2OQJjZAoyMDIxLTA5LTE1A3JjYgYzMDk5MDADYXBwHENvbnRlbnQgVGFiIG9mIGEgUGFnZSBvbiB3d3cCY3QJQlJPQURDQVNUE29yaWdpbmFsX2R1cmF0aW9uX3MHMTc4NC43OQJ0cw9vZXBfcHJvZ3Jlc3NpdmUA&ccb=1-5&_nc_sid=a92414&efg=eyJ2ZW5jb2RlX3RhZyI6Im9lcF9zZCJ9&_nc_ohc=3z7dpcAgup4AX-KkRqv&_nc_rml=0&_nc_ht=video-sin6-3.xx&oh=286b3d0c92e32ceeb9e5295f6cb15848&oe=61434D2F&_nc_rid=c2438480a19841e&_nc_vts_prog=1&vts=1&_nc_vts_internal=1",
      "video_duration_seconds":1785,
      "video_height":720,
      "video_id":"1039385906890308",
      "video_quality":"720p",
      "video_size_MB":"None",
      "video_thumbnail":"https://scontent-sin6-2.xx.fbcdn.net/v/t15.5256-10/cp0/e15/q65/p600x600/241752920_1039403706888528_4117109431631879057_n.jpg?_nc_cat=109&ccb=1-5&_nc_sid=ccf8b3&_nc_ohc=Yn7KQz1Tts8AX-75yH7&_nc_ht=scontent-sin6-2.xx&oh=728ba01397b16b883a7c8d8349b8ae44&oe=61485D60",
      "video_watches":3806,
      "video_width":1280,
      "likes":34,
      "comments":2,
      "shares":0,
      "post_url":"https://facebook.com/174053246035183/posts/1039385906890308",
      "link":"None",
      "links":[
         {
            "link":"/TelepacificoNoticias/?__tn__=%2As-R",
            "text":"Telepacífico Noticias"
         }
      ],
      "user_id":"174053246035183",
      "username":"Telepacífico",
      "user_url":"https://facebook.com/TelepacificoTV/?__tn__=C-R",
      "is_live":false,
      "factcheck":"None",
      "shared_post_id":"None",
      "shared_time":"None",
      "shared_user_id":"None",
      "shared_username":"None",
      "shared_post_url":"None",
      "available":true,
      "comments_full":"None",
      "reactors":"None",
      "w3_fb_url":"None",
      "reactions":"None",
      "reaction_count":"None",
      "with":"None",
      "image_id":"None",
      "image_ids":[
         
      ],
      "fetched_time":"2021-09-16 11:40:27"
   },

because if we set cookies and scrape, we cant get the proper JSON from the program

Read more comments on GitHub >

github_iconTop Results From Across the Web

Power BI Tutorial | Working with JSON data - YouTube
Working with JSON data in Power BI Desktop is one of the best -k... ... try restarting your device. Your browser can't play...
Read more >
Google Chrome video not playing- Json response with html ...
In the developer tools in XHR, I can see the following: Request sent for JSON file; Response returns the Json response which contains...
Read more >
An Introduction to JSON | DigitalOcean
JSON is also readable, lightweight, offers a good alternative to XML, ... in mind that though they appear similar, a JSON object is...
Read more >
Working With JSON Data in Python
Watch Now This tutorial has a related video course created by the Real Python ... Keep in mind, JSON isn't the only format...
Read more >
Working with JSON - Learn web development | MDN
This is not a big issue — JavaScript provides a global JSON object that ... We then do a very similar operation with...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found