503 Service Unavailable (IBKR/Gateway issue)

Describe the bug IBKR/Gateway sometimes runs into returning a 503 Service Unavailable error. In this issue we try to understand what can be done to overcome it.

Environment IBeam-independent, it is the Gateway’s issue

Additional context So far it has been observed when calling:

• /iserver/account/{accountId}/orders

---

In https://github.com/Voyz/ibeam/issues/7#issuecomment-945684378 @Wenuka outlines:

when I tried to place an order (using https://localhost:{port_num}/v1/api/iserver/account/{accountId}/orders API) I am getting a 503 response as below,

{'_content': b'', '_content_consumed': True, '_next': None, 'status_code': 503, 'headers': {'x-response-time': '10000ms', 'X-Cnection': 'close', 'Expires': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Pragma': 'no-cache', 'Date': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Connection': 'close', 'Server-Timing': 'cdn-cache; desc=MISS, edge; dur=4, origin; dur=10019', 'Vary': 'Origin', 'Transfer-Encoding': 'chunked'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f15b168d520>, 'url': 'https://localhost:5000/v1/api/iserver/account/XXXXXX/orders', 'encoding': None, 'history': [], 'reason': 'Service Unavailable', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(seconds=10, microseconds=68901), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f15b166aeb0>}

Then I killed the docker session and re-run using the changed proxyRemoteHost value and it worked as expected.

---

In https://github.com/Voyz/ibeam/issues/19#issuecomment-946766857 @isque03 outlines:

Gateway logs show this:

GET /v1/api/tickle,200|61ms 02:43:34.032 DEBUG vert.x-eventloop-thread-0 ClientPortalService : tickle > {“session”:“asdasdadasd”,“ssoExpires”:469642,“collission”:false,“userId”:123456,“iserver”:{“authStatus”:{“authenticated”:true,“competing”:true,“connected”:true,“message”:“Another session with the same user name has disconnected your session.”,“MAC”:“00:00:00:00:00:00”}}}

The only workaround I have found so far is to manually log into IBKR on the mobile app (or I suppose web/desktop). It will say “another session is in use, disconnect?” I click “yes” to disconnect, then I explicitly click the logout in the mobile app. After this I’m able to get a properly working brokerage session from the logout/reauthenticate flow.

My suspicion is that it’s the call for brokerage session ssodh/init that the gateway makes. See this document again: https://www.tradersinsight.news/ibkr-quant-news/tutorial-web-api-connect-to-brokerage-session/

Specifically the compete param in here:

`SSODH Init Request

POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init

The body, of type x-www-form-urlencoded, must have the following parameters:

      machineId (String)       8 char alphanumberic string e.g. CCCCCC01-99
      mac (String)                   six 2-char alphanumerical pairs separated by a hyphen
      **compete (Boolean)      whether or not the session should compete, usually set to false**
      locale (String)                set to “en_US
      username (String)        set to dash “-“`

The compete param here recommends setting it to false. What I think this does is trigger any other brokerage session to be invalidated, and allow a new one to be created. Setting compete “true” I beleive means that it will not kick out the other session. (Even though that seems sort of backwards from what you’d expect) The gateway passes complete:true. (From gateway logs) post={“username”:“XXXX”,“machineId”:“12349”,“compete”:true}

Programmatic workarounds here? I think the only choices are:

1. File a ticket with IBKR and see if they’ll add the ability to control the compete param in the gateway
2. Have iBeam actually implement the ssodh/init /complete flows to explicitly trigger brokerage session creatation (this would replace the call to /reauthenticate). I have not proven this compete true|false theory, so I wouldn’t recommend going down this path yet.

[DanielHTpg]

@Voyz The issue with no accounts being returned was one of the biggest problems I had after switching the proxyRemoteHost. Sometimes it only went away the next day. Since using docker-compose I haven’t this issue.

[alex1999Tra]

I am using voyz/ibeam:0.4.0-rc1. I observed that the gateway is running and authenticated. But the /iserver/accounts returns an empty account with 200 status. And iserver/marketdata/snapshot returns 500. I am not sure if this is also related to the issue. Any suggestions?

Ibeam log 2021-11-11 17:27:31,747|I| Gateway running and authenticated 2021-11-11 17:28:31,751|I| Gateway running and authenticated 2021-11-11 17:29:31,721|I| Gateway running and authenticated 2021-11-11 17:30:31,715|I| Gateway running and authenticated

IBKR Gateway log 11-11-2021 17:28:02 UTC: DEBUG : _make_request : https://localhost:5000 "GET /v1/api/iserver/accounts HTTP/1.1" 200 None 11-11-2021 17:28:02 UTC: DEBUG : _make_request : https://localhost:5000 "GET /v1/api/iserver/marketdata/snapshot?conids=199169591&fields=31%2C55%2C70%2C71%2C74 HTTP/1.1" 500 None 11-11-2021 17:28:02 UTC: ERROR : invest : There are errors when investing 500 Server Error: Internal Server Error for url: https://localhost:5000/v1/api/iserver/marketdata/snapshot?conids=199169591&fields=31%2C55%2C70%2C71%2C74