High-Load FinTech System
We сreated a financial transaction processing system built for high-load environments.
Client
The client is a global money transfer company operating across multiple markets. They came to us in 2023 with a clear goal: to automate financial transaction processing. The system was intended to manage and control transaction workflows within the company.
As the business grew, transaction volumes increased, but operations were still managed in Excel spreadsheets. Scaling under these conditions was not feasible, and the client understood that well. They had already considered several development vendors before choosing us.
We won the project by offering a solution that could handle the complexity while staying within a tight budget.
Challenges
A third-party team had already conducted initial analysis, working on user stories and task breakdowns in Jira. However, this analysis did not reflect the client’s actual business processes and could not be used as a reliable foundation for development. The user stories were fragmented, and relying on them would have slowed down development and increased costs.
We defined the functional scope ourselves. This allowed us to work from the actual state of the business, deliver a working solution faster, and avoid unnecessary costs.
This meant the main focus had to be on handling large volumes of data.
1. Stability & Performance
The system had to reliably process millions of transactions per day without data loss or performance degradation.
2. Flexibility
Data formats vary frequently, so the system would need to handle this variability. The client needed a way to define connection rules for new data sources without developer involvement, without involving developers.
High-Load System Design
Data Transformation
We used an ETL approach to process incoming data. Raw data from multiple sources is aggregated, transformed, and loaded into a unified target system.
ETL processes introduce additional load to the infrastructure, so we analyzed the client’s IT landscape upfront and made sure the infrastructure could handle it.
Data Integration
We integrated Qlik, a third-party analytics platform, to enable reporting and data visualization.
Connection Builder
We built a connection builder based on a JSON schema. It defines data schemas and rules for processing and validation within the ETL pipeline, including how data should be mapped and routed.
SQL queries to data sources are generated dynamically based on the file type. This allows the client to configure new data sources without involving developers.
Horizontal Scaling
Data is partitioned into segments and distributed across multiple servers. Each segment can be scaled independently based on load.
Information Security
Financial companies are frequent targets of cyberattacks. We accounted for this when designing the system and implemented data protection and access control mechanisms at the architectural level. The solution meets security requirements for both stored and transmitted data.
Technologies
Backend
Go (Echo)
Streaming
Kafka
Frontend
React
Database
PostgreSQL
Data Processing
ETL
Infrastructure
Ansible
IAM
Keycloak
Result
We built a system that covers the full transaction processing cycle from data ingestion to reconciliation, analytics, and reporting.
The system supports processing up to 1 million transactions per day. Manual work has been reduced to configuring processing rules, eliminating the need for hundreds of Excel spreadsheets.
The solution has been validated in real-world use by the client’s operational team. The client is now considering offering it as a standalone product.
System Сapabilities:
- Role-based access for clients, acquirers, and providers, including contract and pricing configuration.
- Creation of transaction records, both manual and automated.
- Transaction reconciliation between payment processors and banks (when the acquiring bank confirms the transaction has been processed).
- Automated export of transaction data via API.
- Exchange rate reconciliation.
- Summary reports and system-wide notifications via email and messaging platforms.
Visual materials were modified to comply with NDA requirements, while the system logic and workflows are preserved.
of transaction processing automated.
transactions processed daily.
What happens next:
Having received and processed your request, we will reach you shortly to detail your project needs.
After examining requirements, our analysts and developers devise a project proposal with the scope of works, team size, time and cost estimates.
We arrange a meeting with you to discuss the offer and come to an agreement.
We sign a contract and start working on your project as quickly as possible.