How and why I wrote a parser for the site Tradingview. Automate DIY trade

Problem: At the moment, the well-known TSLAB is the most convenient and complete software (hereinafter referred to as software) for automating trade on the Russian stock market.

Despite the undoubted advantages in the form of a convenient visual editor for writing trading scripts, which allows you to write robots even without knowledge of programming languages, there are a number of shortcomings that make using this software for me is extremely un practical. And I think not only for me, given that the average size of the bill on the Moscow Exchange does not usually exceed 500 thousand rubles.

1. Cost: Subscription fee 4500r. / Month + virtual server rental (1000 p. / Month)
This fixed cost imposes a very strong burden on the financial result of my trading. Therefore, having an account size of 500 thousand rubles. and hoping to get at least 20% per annum from it, with the existing costs, you need to earn about 32-35% to reach the planned yield.

2. Instability of work: Despite the fact that my algorithms work mainly with market bids (type of bids, which implies 100% execution), my positions often doubled or were not executed at all.

Task: Write software for trade automation to minimize the fixed costs with a user-friendly interface for creating trade scripts that allows you to write trading robots without deep knowledge of programming.

The architecture of the entire project with the current and functional and planned modifications is shown in the figure below.

The most important link in the program is undoubtedly the website Tradingview (hereinafter referred to as TV). It just provides us with convenient functionality for writing our trading scripts due to the built-in language Pine_Script.

The language by the way does not require specific knowledge and is basically similar to the Easy Language language of the Metastock package, and the availability of online help in Russian makes writing code as pleasant as possible.

An example of the strategy of breaking the moving average (literally three lines of code):

mov_average=sma(close,x) strategy.entry("My Long Entry Id", strategy.long,when=close>mov_average) strategy.entry("My Short Entry Id", strategy.short,when=close<mov_average) 

Now having a convenient interface for writing trading scripts, it remains to establish the process of sending orders from TV directly to the trading system (in my case it is the Quik program), or directly to the broker's server. The only problem is that TV has no open API for implementing this functionality.

In an attempt to solve this problem, the first thing that occurred to me was to use the plug-in for testing WebSelenium and, by searching the XPath locators, find the elements we need that are responsible for the buy-sell signals.

The signals themselves are displayed in the table and problems like should not have arisen. But to search for the most extreme signal, the table required scrolling, but I could not find the scrolling element (see the figure below).

So I had to find another solution.

Visually, the TV signals are displayed in the Canvas element. The color of the signal can be changed if necessary (ex: red-sale, green-buy).

Colors set in TV we set in our application. The application itself is written in Java, the GUI is implemented using the Swing library.
Next, in the program itself, we need to highlight the canvas area (or simply the scanned area) in which we will look for reference colors.

The figure below shows the tradingview site with three selected instruments, for each of them the color of the trading signal is set. These colors are duplicated in my program Parse_Signal.

.

After setting the scan area and setting it in the settings of the type of the instrument being traded (by the way, the program settings take 5 minutes and are saved to a file with the extension .txt). Next, press the "START" button and the program starts to work.

It works in two streams.

1 first thread:

• Scans the selection (in this case, canvas).
  Scan do classically using the functionality of the class Robot:

  
  

   BufferedImage buf= robot.createScreenCapture(new Rectangle(selection.x, selection.y, selection.width, selection.height)) 

  
  

• Then it splits the resulting scan into an array of pixels:

  
  

   int[] pixels = copyFromBufferedImage(buf); //,     ,        : public int[] copyFromBufferedImage(BufferedImage bi) { int[] pict = new int[bi.getHeight() * bi.getWidth()]; for (int i = 0; i < bi.getWidth(); i++) for (int j = 0; j < bi.getHeight(); j++) pict[i * bi.getHeight() + j] = bi.getRGB(i, j) & 0xFFFFFF; // 0xFFFFFF:   3   RGB return pict; 

  
  

• Looks for control colors of trading signals in an array of pixels. Search is carried out from left to right. Those. It is the color of the rightmost pixel that is relevant for the program:

  
  

   for(int i=0;i<pixels.length;i++ ) { if (pixels[i] == (buy.getBackground().getRGB() & 0xFFFFFF)) { position = 1; //System.out.println("  ")} else if (pixels[i] == (sell.getBackground().getRGB() & 0xFFFFFF)) { position = -1; //System.out.println("  ");} else if (pixels[i] == (hold.getBackground().getRGB() & 0xFFFFFF)) { position = 0; ...................... ...................... 

  
  

  

  
  

• Template recording of a trade transaction (to a file with the .tri extension), depending on the color found. Here, in fact, everything is simple in the Quik trading terminal there is the ability to automatically read transactions from a file. It is enough for us to register them according to a certain pattern. Quik, however, when a new record appears, sends a request to the broker's server. File reads occur every 500 ms. Information about trading signals can be optionally sent either to the mail or phone, or to the trading system (three parameters can be selected at the same time). 1 thread runs at 500 ms.

  
  

 if (position==1&&status!=1&&b1==1) { if(dialog.isSend_phone()==true) { new SMS().sendSms(dialog.getPhone(), "TS_1:    "+ (String)dialog.cbFirst.getSelectedItem()+" "+price+" "+new Date(), "TEST-SMS", dialog.getLogin(), dialog.getPassword());} if(dialog.isSend_trade()==true){ tr.Order_Buy();} if(dialog.isSend_mail()==true){ test.sendSignal("BUY","TS_1: Buy in signal at price "+ (String)dialog.cbFirst.getSelectedItem()+" "+price+" "+new Date());} status = 1;} ...................... ...................... 

2, the program stream makes a request for the price of the instrument being traded by parsing the html pages of the Finam site. Used plugin JSOUP. Here, everything just unloads the html pages and searches for the code of the trading instrument I need (ex: Si, Sber, etc.).

 public void run() { while (true) { Document doc = null; Document doc_2 = null; try { doc = Jsoup.connect("https://www.finam.ru/quotes/futures/moex/").get(); doc_2 = Jsoup.connect("https://www.finam.ru/profile/mosbirzha-fyuchersy/sbrf").get();} catch (IOException e) { e.printStackTrace(); continue;} StringBuffer buffer = new StringBuffer(doc.text()); StringBuffer buffer_2 = new StringBuffer(doc_2.text()); Map<String, String> map = new HashMap<>() try {map.put(elements[1], buffer.substring(buffer.indexOf("Si "), buffer.indexOf("Si ") + 8).split("Si ")[1]); map.put(elements[2], buffer.substring(buffer.indexOf("RTS "), buffer.indexOf("RTS ") + 10).split("RTS ")[1]); map.put(elements[3], buffer.substring(buffer.indexOf("LKOH "), buffer.indexOf("LKOH ") + 10).split("LKOH ")[1]); map.put(elements[4], buffer.substring(buffer.indexOf("BR "), buffer.indexOf("BR ") +8).split("BR ")[1]); map.put(elements[5], buffer.substring(buffer.indexOf("GAZP "), buffer.indexOf("GAZP ") + 10).split("GAZP ")[1]); map.put(elements[6], buffer.substring(buffer.indexOf("GOLD "), buffer.indexOf("GOLD ") + 11).split("GOLD ")[1]); map.put(elements[7], buffer.substring(buffer.indexOf("MOEX "), buffer.indexOf("MOEX ") + 10).split("MOEX ")[1]); map.put(elements[8], buffer.substring(buffer.indexOf("MIX "), buffer.indexOf("MIX ") + 10).split("MIX ")[1]); map.put(elements[9], "0"); map.put(elements[10], buffer_2.substring(buffer_2.indexOf("  "), buffer_2.indexOf("  ") + 23).split(" ")[2] + buffer_2.substring(buffer_2.indexOf("  "), buffer_2.indexOf("  ") + 23).split(" ")[3]);} catch (Exception e) { System.out.println(e); text.setText("     "); continue;} price = String.valueOf((int) Double.parseDouble(map.get((String)