Interactive Brokers TWS API — and yet it works!

Frankie
8 min readFeb 23, 2024

--

If you glance over the forums, it looks like this software was forged in the fires of hell. Follow me in a journey to the depths.

This article was originally published at https://wasteofserver.com

A friend asked if I could help as he was having a hard time integrating IKBR TWS API. Apparently, he’s not alone:

Even after all the above pain in my day job, I’m still yet to come across an offering as crap as the TWS API. — DanWritesCode

I have been developing code for about 15 years, professionally, and have never seen a piece of garbage like the TWS Api. — puzzled_orc

I put IBKR TWS not just at the bottom of the list, it’s twice as bad as the most crappy system out there. — Causal-Capital

After reviewing TWS offer, it’s definitely legacy code.

The API grew to support a multitude of requests in a non-standard and non-organized fashion. But legacy code, is also code that has been running for ages, most likely providing mission-critical functionality, and there’s certainly value in that.

When you publish an API you’re creating a contract with the world. As soon as it’s out there, you can’t really change it without creating ripples across the entire ecosystem. If requirements take an unexpected curve, as they generally do, it’s easy to find yourself cornered and forced to provide a less than ideal interface.

What you can definitely do, is improve the documentation and provide clear code samples with what should be best practices. While I can’t really point a finger at the API as I have no idea of the challenges involved at the time, the documentation could improve.

TWS Connection

Let’s see how you’re supposed to integrate the API connection. Notice that per the API instructions “it is important that the main EReader object is not created until after a connection has been established”, which reads like misplaced behaviour.

Erapper wrapper = new EWrapperImpl();
EReaderSignal signal = new EJavaSignal();
EClientSocket client = new EClientSocket(wrapper, signal);

client.eConnect("host", PORT, CONNECTION_ID);

Per the docs, this is a standard way to start TWS API

We can see that:

  • wrapper implements the interface where you'll get callbacks (i.e. receive things from the API, market data, fills, etc)
  • client allows you to interact with TWS (i.e. send things to the API, orders, etc)

And then we have both signal and reader classes.

  • signal is used to signal the reader thread that there's data to process
  • reader processes incoming messages

According to IBKR documentation, you should then start reader and a launch a new thread to process the queue:

final EReader reader = new EReader(client, signal);   
reader.start();

// IBKR thread created solely to empty the messaging queue
new Thread(() -> {
while (client.isConnected()) {
m_signal.waitForSignal();
try {
reader.processMsgs();
} catch (Exception e) {
// ...
}
}
}).start();finl EReader reader = new EReader(client, signal); reader.start(); // IBKR thread created solely to empty the messaging queue new Thread(() -> { while (client.isConnected()) { m_signal.waitForSignal(); try { reader.processMsgs(); } catch (Exception e) { // ... } } }).start();

I didn’t dig up TWS code to see how it behaves internally, but from a birds-eye perspective, both signal and reader are misplaced and would be better encapsulated within the client.

The user just needs a way to pass data to TWS ( client) and a way to get data from TWS ( wrapper). All other low level implementation details would better belong inside the implementation.

A legacy system connected to the future, by Microsoft Copilot

Broken API socket connection

One of the places where my friend was having issues was in the re-connecting code. You should design to accommodate failure, much more so if you’re developing an automated trading strategy.

The socket connection between your code and TWS will eventually break, so that’s a good place to start. Glancing over TWS API we have that:

// creates socket connection to TWS/IBG.
client.eConnect("host", PORT, CONNECTION_ID);

// closes the socket connection and terminates its thread.
eDisconnect (bool resetState=true)// creates socket connection to TWS/IBG. client.eConnect("host", PORT, CONNECTION_ID); // closes the socket connection and terminates its thread. eDisconnect (bool resetState=true)

If we break the socket at the network level, like IKBR states, we get a callback in the wrapper implementation:

@Override
public void error(int id, int errorCode, String errorMsg, String advancedOrderRejectJson) {
log.warn("error: id={}, errorCode={}, errorMsg={}, advancedOrderRejectJson={}",
id, errorCode, errorMsg, advancedOrderRejectJson);
}@Override public void error(int id, int errorCode, String errorMsg, String advancedOrderRejectJson) { log.warn("error: id={}, errorCode={}, errorMsg={}, advancedOrderRejectJson={}", id, errorCode, errorMsg, advancedOrderRejectJson); }

On a broken socket (network disconnect), the above will log something like:

error: id=-1
errorCode=502
errorMsg=Couldn't connect to TWS. Confirm that "Enable ActiveX and Socket
Clients" is enabled and connection port is the same as "Socket Port" on the
TWS "Edit->Global Configuration...->API->Settings" menu. Live Trading ports:
TWS: 7496; IB Gateway: 4001. Simulated Trading ports for new installations
of version 954.1 or newer: TWS: 7497; IB Gateway: 4002,
advancedOrderRejectJson=nullerror: id=-1 errorCode=502 errorMsg=Couldn't connect to TWS. Confirm that "Enable ActiveX and Socket Clients" is enabled and connection port is the same as "Socket Port" on the TWS "Edit->Global Configuration...->API->Settings" menu. Live Trading ports: TWS: 7496; IB Gateway: 4001. Simulated Trading ports for new installations of version 954.1 or newer: TWS: 7497; IB Gateway: 4002, advancedOrderRejectJson=null

Notice that at this point we’re not connected to TWS, but client.isConnected() still returns true. While not ideal, it's documented. You must issue a eDisconnect().

// pseudo code for broken socket behaviour

client.isConnected() -> returns true
// here we break the socket
client.isConnected() -> returns true
client.eDisconnect();
wrapper.connectionClosed(); // we get a callback here
client.isConnected() -> returns false// pseudo code for broken socket behaviour client.isConnected() -> returns true // here we break the socket client.isConnected() -> returns true client.eDisconnect(); wrapper.connectionClosed(); // we get a callback here client.isConnected() -> returns false

Given the above, it would look as though we could issue a connect() to re-establish connection.

client.eConnect("host", PORT, CONNECTION_ID);
wrapper.connectAck(); // we get a callback here
wrapper.connectionClosed(); // and then we get a callback here

If we manually disconnect, we get a callback on connectionClosed() after re-connecting

What’s happening here?

Remember the thread that we created to empty the queue? Let's revisit that:

new Thread(() -> {
while (client.isConnected()) { // when client disconnets it should exit
signal.waitForSignal(); // but it's waiting on signal!
try {
reader.processMsgs();
} catch (Exception e) {
// ...
}
}
}).start();

This code, which is provided as an example on how to empty the queue, has a massive red herring.

Even though client.isConnected() returns false, we're stopped on waitForSignal() so the while loop won't exit!

As soon as the above thread gets a new signal, client is already connected and the above thread will not stop! This was creating havoc in my friend's code! As previously stated, all this behaviour could (should?) have been encapsulated upstream but, given that it's not, we can do it ourselves.

Within the mess, we can isolate and encapsulate behaviour.

A proper start, stop & reconnect sequence

Per everything above, there is a meticulous sequence that you must follow for TWS to properly connect. You must create some objects, but some can only be created after specific callbacks have been received, etc.

When things start getting involved and into a spaghetti-like sequence, I recommend that you stick with the cleanest possible approach. In this case, it’s a scenario where you create all required variables and another one where you reset them.

Properly encapsulate Reader

Making sure reader is properly initialized and terminated is a big part of the original issue, so we’re isolating it to reduce clutter.

/**
* Responsible for reading messages from the TWS socket.
* <p>
* Should be stopped when the connection to TWS is lost.
*/
public class Reader implements Runnable {
private final AtomicBoolean running = new AtomicBoolean(true);

private final EReaderSignal readerSignal;
private final EReader reader;

public Reader(EClientSocket clientSocket, EReaderSignal readerSignal) {
this.readerSignal = readerSignal;
reader = new EReader(clientSocket, readerSignal);
reader.start();
}

@Override
public void run() {
while (running.get()) {
readerSignal.waitForSignal();
try {
reader.processMsgs();
} catch (Exception e) {
// dispatch to error mechanism
}
}
}

public void stop() {
running.set(false);
readerSignal.issueSignal();
reader.interrupt();
}
}

We can now control in a clean fashion when reader starts and stops.

And then, the calling class may look something like this:

public class StartTwsApi {
private static final Logger log = LoggerFactory.getLogger(Start.class);
EWrapperImpl wrapper;
EJavaSignal readerSignal;
EClientSocket clientSocket;
Executor executor = Executors.newSingleThreadExecutor();
Reader reader;

StartTwsApi() {
initializeWrapperAndSocket();
connectToTWS();
}

private void initializeWrapperAndSocket() {
wrapper = new EWrapperImpl(this, this);
readerSignal = new EJavaSignal();
clientSocket = new EClientSocket(wrapper, readerSignal);
}

private void connectToTWS() {
log.info("Connecting to TWS...");
clientSocket.eConnect("localhost", 4005, 2);

// safeguard to make sure we do connect / keep trying
ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);
executorService.schedule(() -> {
if (!clientSocket.isConnected()) {
log.warn("Still not connected (10s). Retrying connect...");
onTwsConnectionFailure();
}
}, 10, TimeUnit.SECONDS);
}

private void startReader() {
executor.execute(reader);
}

private void stopReader() {
if (reader != null) {
reader.stop();
}
}

/**
* You'll call this method when you receive a callback from TWS
* stating that the connection is lost.
* <p>
* It stops the reader thread and re-connects to TWS.
*/
public void onTwsConnectionFailure() {
stopReader();
log.info("Stopped reader. Will re-connect in 5 seconds...");
Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS);
connectToTWS();
}

/**
* Call this method when you get a connectAck() on TWS
* wrapper callback implementation
*/
public void connectAck() {
log.info("connectAck received. Safe to start reader thread...");
reader = new Reader(clientSocket, readerSignal);
startReader();
}

}

And even though it’s still convoluted, now works as expected.

My recommendation is that you abstract all this behaviour under a specific TWS module, and then pass a cleaner interface back to your actual code.

There is much more that we’ve delved on. Historical data, for instance, is an area where there were pretty interesting inconsistencies:

  • public void historicalTicksBidAsk(int reqId, List<HistoricalTickBidAsk> ticks, boolean done)
    tickData is received in a single callback as a List of Objects (max 1000)
  • public void historicalData(int reqId, Bar bar)
    barData is received as a series of multiple callbacks with a single Bar object; when all bars arrive, you get a callback on historicalDataEnd(int reqId, String startDateStr, String endDateStr).

Last thoughts:

Designing an API is hard. Designing an API to stand the test of time is even harder. While I can identify antipatterns and a lack of behaviour guidelines, I may only assume that business needs and decisions dictated the curved approach we now see. The very same decisions that made IBKR into a 45 billion company.

I don’t know if there’s an audience for TWS API articles. If you want more, let me know in the comments below.

— — — — —

My latest acquisition were the Arctis Nova 7 headphones. Sound quality is excellent. So much so that I immediately noticed the sound difference between Spotify native app and Spotify web. The native app is vastly superior! Microphone quality is also excellent. If you’re looking into a new headset, definitely check this one out.

Originally published at https://wasteofserver.com on February 23, 2024.

--

--