Problems with grr_api_client & grr_api_shell

Hello all,

My goal is to have the ability to automate processes for GRR, like removing dead clients, with scripts. We are using a single sign-on for authentication with GRR, and that can’t be removed. Because of the single sign-on, I cannot use the grr_api_client or the grr_api_shell, as it receives the following error because it’s being blocked by single sign-on: grr_api_client.errors.AccessForbiddenError: No username header found (I can send the full error if needed)

Is there an undocumented way to get a list of all clients, sort them by “dead” clients, and then give the option to delete all “dead” clients, without the use of grr_api_client or grr_api_shell?

Is there undocumented commands that might be able to help me in the right direction?

Or maybe, is there a work around for single sign-on?

If that isn’t possible, is it possible to write a script that initializes the grr_console, authenticates with a password, and then executes commands line by line.

For example, could I execute this

In [1]: client = o("C.0a2d3036f8e0c4be")
In [2]: print client.Get(client.Schema.HOSTNAME)
example.host.com
In [3]: aff4.FACTORY.Delete("C.0a2d3036f8e0c4bf")

But in a script? From my attempts, I’m not confident it’s possible, but I thought I’d ask regardless.

Thank you in advance for any help!

[pidydx]

Have you looked at the GRR cron jobs designed for keeping the datastore clean? The CleanInactive* cron jobs have a configurable TTL for how old each object type needs to be before it will delete it.

[scudette]

Thanks Misha, I didn’t know about the new raw API client. I will have to try it next week and see how it compares to just the console.

On Thu, 18 Oct 2018, 2:48 AM mbushkov [e-mail] wrote:

Caleb, quick question - why do you need to remove dead clients? Do they obstruct search results? Do you use GRR clients database for inventory purposes?

A few points regarding GRR API.

Technically it’s possible to integrate GRR API client library with custom single sign on logic. To do this you need to create a new virtualenv, install grr-api-client PIP package and then follow these instructions < https://github.com/google/grr/tree/master/api_client/python#initializing-the-grr-api-object

.

However: If you have direct ssh access to the GRR server, you can simply run grr_api_shell_raw_access binary (shipped with the server). It will give you exactly the same functionality as grr_api_shell, but will write directly to the database and will effectively bypass authentication checks (so you’ll have no single sign on issues).

@Michael: grr_api_shell_raw_access is also much faster than grr_api_shell, since nothing has to be converted to JSON or be passed through HTTP. Code that runs through *grr_api_shell_raw_access *works pretty much as fast as code running through grr_console.

Unfortunately, GRR API doesn’t support deleting clients (old clients disappear from search index 6 months after the last ping, though). You can use grr_api_shell_raw_access to find clients that haven’t checked in in a while (please see this example < https://github.com/google/grr/tree/master/api_client/python#example-collect-client-ids-for-a-given-hostname

) and then delete them through grr_console.

A few points regarding using grr_console:

• grr_console accepts --comand_file [path] argument. When this argument is provided, grr_console will execute Python code from a given file instead of starting up an IPython shell.
• Code like aff4.FACTORY.Delete(“C.0a2d3036f8e0c4bf”) should work, but will likely stop working quite soon as we’re overhauling our whole datastore implementation at the moment. As a rule of thumb grr_api_shell_raw_access should always be preferred to grr_console, unless you need to do something that doesn’t have a corresponding GRR API call at all.

Cheers, Misha

On Wed, Oct 17, 2018 at 1:06 AM Michael Cohen [e-mail] wrote:

Note that the grr console is just an ipython shell so you can write your script and then in the shell just type run -i myscript.py and it will run. You can also just run python scripts (importing the console module directly). In that case you ssh to the server and run your script so you do not need auth at all (well auth is handled via the ssh).

In most of our GRR work we dont really need the user authentication features of the GRR API and we find the API slows things down significantly so we just write console scripts and run the on the server with SSH instead.

Hope this helps, Michael.

On 16/10/2018, Caleb [e-mail] wrote:

Hello all,

My goal is to have the ability to automate processes for GRR, like removing dead clients, with scripts. We are using a single sign-on for authentication with GRR, and that can’t be removed. Because of the single sign-on, I cannot use the grr_api_client or the grr_api_shell, as it receives the following error because it’s being blocked by single sign-on: grr_api_client.errors.AccessForbiddenError: No username header found (I can send the full error if needed)

Is there an undocumented way to get a list of all clients, sort them by “dead” clients, and then give the option to delete all “dead” clients, without the use of grr_api_client or grr_api_shell?

Is there undocumented commands that might be able to help me in the right direction?

Or maybe, is there a work around for single sign-on?

If that isn’t possible, is it possible to write a script that initializes the grr_console, authenticates with a password, and then executes commands line by line.

For example, could I execute this

In [1]: client = o("C.0a2d3036f8e0c4be")
In [2]: print client.Get(client.Schema.HOSTNAME)
example.host.com
In [3]: aff4.FACTORY.Delete("C.0a2d3036f8e0c4bf")

But in a script? From my attempts, I’m not confident it’s possible, but I thought I’d ask regardless.

Thank you in advance for any help!

– You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/google/grr/issues/633

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/grr/issues/633#issuecomment-430417986, or mute the thread < https://github.com/notifications/unsubscribe-auth/AA3PFb4ucF58fSCqKfqKoLsRBJiu2RtFks5ullhjgaJpZM4XeXzB

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/google/grr/issues/633#issuecomment-430682730, or mute the thread https://github.com/notifications/unsubscribe-auth/ADrYolXlJzUQLtfwLAR-0DQnBmB847qfks5ul1FAgaJpZM4XeXzB .