We made a news aggregator with a focus on cryptocurrencies - intwt.com

Hi Habr!

The cryptocurrency market is growing every day, and with it the amount of information is growing.

Therefore, we decided to launch the project intwt.com - this is an aggregator of news and posts from social networks for traders and all who are interested in cryptocurrency.
')


A simple, convenient and understandable service interface is designed to make it a really effective tool for monitoring important information.

At the moment we are analyzing more than 3 thousand news sources, in English, Russian and Chinese, as a result we receive about 3 thousand new materials daily.

Each material is analyzed by the system on the mention of cryptocurrency and popularity in social networks.

Using the news filter you can set up your individual tape, save it in your account and if you want to include broadcasting in your telegram channel.

We constantly monitor important indicators for 2716 cryptocurrencies and monitor the emergence of new currencies in the market.

Using a special interface to view the list of cryptocurrencies you can see the leaders of growth and decline in the market.

For each currency, you can see the latest news and all indicators on a separate page, for example, such as Price, Capitalization, etc., as well as the price chart for the entire period of the currency existence in the market.

In your account you can create a cryptocurrency portfolio and track its dynamics on a chart.

At the moment, we do not think about monetization, since the service is quite young and is gaining an audience, but most likely it will be advertising and paid access by subscription to the PRO functions.

Some technical details

Service can be divided into two parts

1. The frontend is a SPA application written in Vue and a backend written in Go, which distributes minimal HTML with content for search engines and code to launch the SPA application. This approach allows you to avoid server rendering and make friends with search engines. Although Yandex immediately blocked us, as doorway.
2. The parser is allocated to a separate service with its base and admin panel, so that it can be easily transferred to a separate server. Here we used Go, PostgreSQL, Beanstalkd for parsing queuing and Rotating TOR proxy which avoids blocking by IP. To parse some sites, you have to use browserless chrome to bypass the defense mechanisms. The admin for the parser is made on Laravel.

All services are running inside Docker, currently 19 containers are running. All this is through GitLab CI. To monitor the system, we use Prometheus and Grafana, and for the Sentry error log.

What is planned next?

Developing a mobile application for ios and android, creating a platform for specialists with the ability to post author articles, videos and reviews on cryptocurrency. Subscribe to the author. And of course, an automated regression analysis of the impact of news on the price movement of a currency.

We will be glad to hear criticism or ideas for the development of the project.

PS real author of the post Dmitry , all questions to him.