High-Load FinTech System
We developed an automated transaction system designed for a large volume of financial data.
Client
The client’s company operates globally and engages in money transfers. The client approached us in 2023 with a very specific task: to automate financial transactions. The product was intended to optimize the working time of the company’s employees.
Despite the client’s business growth and increasing transaction flow, they continued to use Excel spreadsheets for accounting. Without automation, scaling the business is difficult, and the client understood this perfectly. They already had three or four candidates for the developer role. We won based on our estimate and cooperation terms: it was critically important for the client to stay within budget.
Challenges
A third-party company had already performed analytics, working on user stories and task decomposition in Jira. However, the analytics had little in common with the client’s real business processes. The Stories turned out to be fragmented. Building an entire project on them would have taken longer than starting anew.
We took the initiative to define the functional perimeter ourselves. This allowed us to base our work on the real state of affairs, provide the client with a working solution faster, and avoid inflating the budget.
The main focus should be on working with large volumes of data.
1. Stability & Speed
The service must be designed for high loads (thousands of transactions per day) and demonstrate uninterrupted operation.
2. Availability
Data types are continually evolving, and this variability must be considered. The client should have a way to independently, without involving a developer, set connection rules for new file sources.
High-Load System Design
Data Transformation
We decided to use an ETL approach. Raw data is aggregated from various sources, transformed, and then loaded into a single target system.
ETL is demanding on system resources. However, we thoroughly studied the client’s IT landscape at the start and confirmed that there would be sufficient capacity.
Data Integration
We connected an external system for building analytical reports, Qlik. It is a tool based on AI and ML for modern data architectures and advanced analytics with extended capabilities.
Connection Builder
We created a connection constructor in the form of a JSON structure. It describes the data type and validation rules in the ETL database: where and what data should go. All SQL queries to sources are generated dynamically based on the file type. The client does not need to involve developers to connect new file types.
Horizontal Scaling
Data is divided into several segments and distributed among servers. The scale of each segment can be increased or decreased autonomously.
Information Security
Financial companies are frequent targets of cyberattacks. We understand the nature of cyber threats in the industry, which allows us to effectively reduce risks. The developed product complies with data security standards for stored and transmitted data.
Technologies
Server Side
Go (Echo)
Server Side
Kafka
Interface
React
RDBMS
PostgreSQL
RDBMS
ETL
Infrastructure
Ansible
Infrastructure
Keycloak
Result
Today, the client’s company processes one million transactions per day. The number of routine operations has been reduced to entering a few data processing rules. Hundreds of Excel tables have been reduced to zero. The client plans to make the solution a boxed product and monetize it.
System Сapabilities:
- Different roles are assigned: clients, acquirers, and providers, as well as adding contracts and pricing under which they operate.
- Monetary data can be created manually and automatically.
- Matching transactions between the collector and the bank (when the acquiring bank confirms that the payment has indeed been processed).
- Automatic uploading of transaction data via API.
- Reconciling exchange rate differences.
- Summary reports and notifications of all system events via email and messenger.
automated the client’s business.
transactions are processed in the system per day.
What happens next:
Having received and processed your request, we will reach you shortly to detail your project needs.
After examining requirements, our analysts and developers devise a project proposal with the scope of works, team size, time and cost estimates.
We arrange a meeting with you to discuss the offer and come to an agreement.
We sign a contract and start working on your project as quickly as possible.