Microservices

Why GraphQL Excels for Querying Large, Complex Datasets: A Real-World Trading API Case Study

Handling large, complex datasets efficiently is a critical challenge in modern APIs. In a real-world project, I built a .NET GraphQL API integrated with SignalR to retrieve and stream thousands of trades from a database in real time. The Trades class, with approximately 100 properties (e.g., trade ID, price, volume, timestamp, counterparty details), represents a rich dataset where different domains (e.g., analytics, reporting, trading desks) require specific subsets of data. Previously, I illustrated GraphQL’s benefits with a simplified task management API using a WorkTasks table. Here, I’ll show how those benefits apply to this real-world trading API, enabling precise, scalable, and flexible querying of trade data.

In a microservices architecture, ensuring each service communicates reliably with databases like SQL Server and Oracle is critical but challenging due to restricted permissions. Health checks monitor service health, and in this post, I focus on advanced checks that execute custom queries to verify table access, beyond standard NuGet packages. I’ve learned to anticipate issues like permission errors, ensuring robust systems for global teams.

I developed a MultiDatabase health check to test specific queries across SQL Server and Oracle, addressing the challenge of limited database access rights that require DBA intervention (e.g., user creation, encrypted connection strings). This check integrates with CI/CD pipelines, catching issues before deployment, complementing my 95% test coverage in projects like RunningTracker.

I helped migrate a .NET monolith to microservices, processing 4-5 million daily operations for financial advisors. Using Kafka and Kubernetes, I streamlined complex commission rules. My 20+ years in procurement enabled me to align teams and deliver tough updates. Despite delays, my refactoring ensured cloud readiness.

The application was built using .NET and was running on a monolithic architecture on a Virtual Machine.

We decided to use Kafka as our message broker to decouple the services and allow for asynchronous communication between them.