More accurately record time spent consuming video media

The Problem

One of the larger time-sinks today is video. Be that through streaming services like Netflix, Amazon Prime Video, or HBO. Media sites like YouTube, Twitch, or Vimeo. Or from downloaded or streamed media on players like VLC or MPV.

Activitywatch, unfortunately, fails to effectively record the time spent on these activities as it relies on mouse and keyboard input to determine activity. This means when watching a video, Activitywatch will mark the time as afk after a short while. Even though the user is present, and spending time on an activity at their device.

This was brought up in https://github.com/ActivityWatch/activitywatch/issues/186 which was marked as wontfix. I believe that this something that CAN be solved, and should be seriously considered given the amount of time that can be spent consuming media.

Afk time can be disabled, but this then pollutes the data. Users may go afk for a variety of times in a variety of applications or websites throughout the day, which could pollute the afk time to the point of video-specific time may no longer be useful.

Possible solutions

Note: Not all problems/disadvantages/pitfalls are meant to be solvable. I am including them for devils advocates sake and to foster a more robust discussion.

Application/Site Tagging

Compile a list of common media applications and websites. When the use goes afk on this site or application, mark the time as non-afk. This isn’t technically tagging, but it could be setup to work in a tag-like way, which would make this into a very extendable solution for more than just videos.

Advantages

• Easy to maintain
  • Add sites/applications as you see fit, or as the community points them out
• Simple

Disadvantages/Pitfalls

• Initial list will be a chore to create
  • Can always ask the community for assistance. You create the functionality, and get the community to fill it in with you curating.
• Is not quite perfect.
• What if the user is legitimately afk on a youtube page?
• What if the user is afk on the homepage of a media site, and not on an actual video?
• What if the user is afk on a video, and it’s paused?

Enhancements

These are here to try and solve for some of the problems presented under disadvantages.

• User-defined or user changeable lists and filters
  • The repo still maintains a list, but users have the ability to add items to theirs without waiting for a pull request to merge and a new release.
  • There are a variety of flavorful solutions to this that I will not go into
• Combine with monitoring hardware to detect if a video might be paused

Monitoring Hardware

Monitor audio output to see if a video is playing.

Advantages

• Simple. May not be easy to implement, but is a simple solution to the problem

Disadvantages/Pitfalls

• What if the user is listening to music, like spotify, and is afk on a media player?
• What if the user is playing music from a media site like YouTube?
• Difficult to implement?
• Monitoring multiple audio devices necessary?

Enhancements

• Combine with site/application tagging

General Enhancements

These are enhancements that could apply to any solution, to increase accuracy and to enable the user to correct mismatches and errors.

User-Defined Lists & Filters

Let the user create/modify the list of sites/applications, and/or the patterns used to match them.

• Takes burden off of developer to maintain accurate lists and patterns to meet everyone’s specific needs
  • I would still expect than an “official” or pre-defined list of items and patterns would be bundled or downloadable
• Some passionate users can open issues to merge items and patterns back into the repo with the dev curating them, or just to share them.

Tagging and Pattern Matching

This goes above and beyond, but would really turn this into a much more powerful tool.

Instead of just solving the video problem. Create an extendable solution that encompass the general problem category that the video problem is part of. This would be in the form of tagging, being able to automatically tag domains & applications with predefined or user defined tags. This can be facilitated with pattern matching lists, and depending on the data’s schema/format could be applied to already existing data greatly enhancing it’s utility.

As an example, time in VLC, YouTube, or Netflix could be tagged as video, which gives users the power to filter this time separately, combine it in reports, or to more easily correct collection errors.

This of course could be setup to be user-manageable, with points listed in the previous section.

Conclusion

I believe the ability to capture time spent consuming video-based media will have an impact on the future usability of this project as these sorts of services continue to expand and bring in more and more people. Solving this problem can not only provide a solution to this problem, but could also greatly enhance the utility and power of this application.

What are your thoughts? (please to not be automarking this as closed, this took some time and effort to create).

[ErikBjare]

I just merged https://github.com/ActivityWatch/aw-webui/pull/262 which implements the “audible-as-active” feature. It makes it so that if your browser is the active app, and the active browser tab is audible (playing sound), then it will not count that time as AFK (and therefore make it show on your Activity view).

It requires that you’re running the web watcher for your browser.

This vastly improves the situation when you watch a video in your web browser, but it is not a complete solution, so I’ll leave the issue open for now.

[Aziks0]

@8Dion8 It has been implemented, see ActivityWatch/aw-webui#375.

Just need to wait for the next release now 👀.