Allow actions at (x, y) to return objectID on success

I’m using actions that accept x, y positions as inputs (e.g. event = controller.step('OpenObject', x=0.2, y=0.5)). Once it’s executed, I’m trying to figure out what object was interacted with.

1. Is there any way to do it with the currently exposed methods?

2. Alternately, is there a way to just get the object that the raycast would hit?

public bool ScreenToWorldTarget(float x, float y, ref SimObjPhysics target, bool requireWithinViewportRange)

does this and populates target, but this isn’t an exposed method. Could this be made a callable easily?

Is there an easy way to do what I’m after? I thought I could just modify the interaction functions to return the target as a feedback parameter: e.g.actionFinished(true) --> actionFinished(true, feedback) like some methods that return info in actionReturn, but I’m finding it hard to parse recent code. PutObject has an actionFinished(true), but OpenObject does not, so I got confused.

[mattdeitke]

I don’t necessarily think returning the objectId will work in every case, since some actions (i.e., TouchThenApplyForce) already return information to the actionReturn.

What I may add is a query action to the environment for the object at (x, y):

controller.step('GetObject', x=x, y=y, forceAction=True/False)

where forceAction specifies if visibility/interactability distance to the object should be ignored.

If an object is there, the objectId is returned inside of actionReturn. If not, the action return will be either null or empty. Then, if an objectId is returned, you can guarantee that calling:

controller.step('OpenObject', x=x, y=y, forceAction=True/False)

targets an object, or can equivalently pass in the returned objectId, which is marginally faster:

objectId = controller.step('GetObject', x=x, y=y, forceAction=True/False).metadata['actionReturn']
if objectId:
    controller.step('OpenObject', objectId=objectId, forceAction=True/False)

Is this of interest?

[mattdeitke]

This functionality is now supported in v2.7.4. with the new GetObjectInFrame query action:

forceAction = True
query = controller.step('GetObjectInFrame', x=x, y=y, forceAction=forceAction)
if query:
    controller.step(
        action='OpenObject',
        objectId=query.metadata['actionReturn'],
        forceAction=forceAction
    )

Documentation will be updated shortly. But, briefly, forceAction=True lets the targeted object be further away than the initialized visibility/insteractable distance of 1.5 meters.

Also note that bool(ai2thor.server.Event) (i.e., bool(query)) is now an alias to query.metadata['lastActionSuccess'].

---

If you’re using Python 3.8+ you can also use the new walrus operator:

forceAction = True
if (query :=  = controller.step('GetObjectInFrame', x=x, y=y, forceAction=forceAction):
    controller.step(
        action='OpenObject',
        objectId=query.metadata['actionReturn'],
        forceAction=forceAction
    )

which merges the query declaration inside of the conditional.