Insight into Cardmarket: Developments in IT

With this column we want to give our users a deeper insight into what is going on at Cardmarket. Every month you can expect one of our team to lift the curtain and allow you an exclusive peek behind the scenes. This time we welcome guest author Tommi Böttge, head of our IT department!


tommi
Tommi Böttge, the head of our IT department, got his very own card in the WoW TCG (illustration above)

This article is also available in German.
Diesen Artikel gibt es auch auf Deutsch.


Robert let the cat out of the bag: As you can read in his first "Insight into Cardmarket" article, almost the entire IT team is intensely working on the reprogramming of Cardmarket. The events that led to this groundbreaking decision are now well known. And it's out of question that those events were not particularly pleasing—especially not for our IT team. But what exactly does "IT" and "IT team" mean at Cardmarket?

The History of Cardmarket's IT

The codebase of our unique marketplace was initially written by our founder all by himself, and he was also the only one maintaining, expanding, and modernizing it through the first seven years. When I began working at Cardmarket in October 2011, my first task was to draft a redesign of Cardmarket's software and to plan its complete reprogramming. Meanwhile we hired our first developer specialized in graphic design and front end in Carlos. The first big graphical revamp of Cardmarket started with him. After almost two years my plans were complete, yielding a document spanning more than a thousand pages. We realized quite quickly that such a plan was impossible to realize with only two developers. Thus we put "Cardmarket 2.0"(1) on the back burner.

From now on I had to delve even deeper into Cardmarket's existing code. About 40% of my time was reserved for refactoring, the modernizing of outdated code, and many process-oriented aproaches made way for object-oriented solutions. 20% of my work went into a variety of little features to aid the tasks of the still small support and marketing teams of that time. The rest was put into the development of the Cardmarket API 1.0 (rather 2.0beta). The first version of the API documentation's starting page dates back to September 24, 2013—essentially marking its birth. And this API was my baby from now on. At its core, it was similar to realizing the shelved Cardmarket 2.0 concept—just as a "RESTful API" and not as a website.

Because of Cardmarket's code structure at the time, the API had always acted like its own application within the Cardmarket cosmos. Its code was only loosely coupled to the one of the website. For example it sometimes retrieved data from the database differently than the website did. As a result, further development of the API required further refactoring of the codebase. We were in need of new developers!

In about 2015 Sijing—today our "jack of all trades"—joined the team, first as a student, from 2017 on as a permanent employee. In Ole we also got a second front-end developer that year. The following two years were characterized by a lot of invisible improvements:

  • routing and MVC (how the website is accessed and delivered),
  • ORM and ODM (how access to the databases is abstracted),
  • localization (how multiple languages are managed),
  • notifications, tasks, and mail,
  • and a lot more administration tools for support and marketing

But also visible to users:

  • new games,
  • Cardmarket Series,
  • Cardmarket Insight and data tables,
  • the second big revamp including responsive design,
  • little features within the shipping process,
  • and countless additional small improvements.

In addition, various legal requirements change frequently, and we need to implement them in the software. And we're never done with this aspect when running a marketplace like Cardmarket.

Cardmarket's software development had basically been a chain of either truly necessary or fancy little hobby projects until then. It quickly became apparent that four full-time developers needed more structure. So we created the first processes to define minimum requirements, a "Definition of Done," set criteria for running tests, and so on. This was essential to satisfy the demands of our users and the other departments. With DJ, Sebastian, Rafael, Sedat, and Konstantinos as back-end, Marius and Guy as front-end developers, as well as Wadi as a tester (and first non-developer) we tripled the team size in 2019.

Subsequently we finished the reconstruction of Insight—from a premium finance blog maintained by a single contributor into a freely available program where a whole host of authors covered a full spectrum of topics. We expanded the Cardmarket Series pages step by step, integrating for example a comprehensive trial management system. We finished our plans for the Series coverage website and began writing code for it, but then Covid hit us and plans had to be shelved once more. We released the completely rebuilt Shopping Wizard. We also experimented with a card scanner to assist our sellers in listing cards. The pages for spoilers and altered-art cards were redesigned. Changeover of more and more pages from the old to the new responsive design ran alongside.

In the second half of 2020 the problems started to pile up. It began with small hiccups—database requests took too long and never completed. The architecture wasn't able to handle too many concurrent requests, especially at peak times. Whether it was the release of a new set when all presales had to change their status or ban list announcements that drove way more than the usual number of users to change the prices of their offers in panic, such events brought our system to its knees more than once. Even though—or precisely because—the API never kept pace with the website's development over the previous one and a half years, it proved an additional factor that exacerbated the system's problems. API access was barely limited at all, not to mention we never set clear rules for its usage. Our users were able to adjust the prices of their offers quickly and based on prices from their direct competitors. Yes, a lot of working implementations against our API existed, even if strong voices maintain the contrary.

This perpetually accelerating loop of mutual price adjustments paired with highly increased access numbers to the website finally brought the marketplace down. We had created a monster and lost the balance between reducing technical debt and releasing innovative additions to our core product. So we took the API down, likewise the Shopping Wizard, and also many small beloved features that made your life easier on Cardmarket in all imaginable situations. The website literally stripped naked. At least we're happy to say: We brought back the most important features, and even if only a handful of users have API access now, it's open to our big sellers again.

We're sitting on a monolithic application that outgrew itself over the course of the past almost fifteen years. We created our very own framework for Cardmarket, and all the code is built on that. We always refrained from using established frameworks and rather re-invented many wheels multiple times over. Only the API is built on top of "Zend Framework 2"—but this added even more of burden. The current Cardmarket is almost entirely based on PHP (95%) as the programming language of choice, with MySQL and MongoDB as our database solutions. Everything is intricately interwoven with everything else and impossible to unravel.

The Future of Cardmarket's IT

But if the system malfunctions between October 2020 and March 2021 had one positive, it was this: We had to concentrate on doing a lot of homework afterward and to decide where Cardmarket would be heading in the future. We had a full team of highly motivated developers. We had introduced Scrum as the process framework for agile software development. With Ajda we hired a product owner and our first female IT employee. Marcell got on board shortly after and took over system administration, which our founder had dealt with almost all by himself until then. We introduced Jira as our project management tool and Miro as an endless whiteboard for product visualization to support us with our daily work. Git superseded the "old" SVN as our software versioning system. Around Easter 2021 our team was boosted by the consultant Thomas. He developed a very concrete plan for "The Cloud" and nowadays is preparing the ground for our developers and helping us with the technical implementation.

"We're going into the cloud!"—This has become such a buzzword whenever an IT team decides to modernize their product. But "going into the cloud" doesn't mean the same for everyone. Often it "only" describes the separation of the actual software into independent parts, and those parts running individually in the cloud, communicating and exchanging information via defined processes. This is of course a great approach with multiple advantages. First, all those parts run independently and distributed. This prevents the whole product from crashing at the same time. Likewise all those parts can be exchanged, expanded, and scaled individually—according to the ever-changing requirements in the lifetime of software.

For us, however, "going into the cloud" means something more radical. We do want these individual parts, these independent processes, and this defined exchange of information between them—but the existing software, the existing codebase can't be divided. We have to write everything anew. What helps us in this giant task is called "Domain Driven Design." We've split Cardmarket into small packages and defined their responsibilities. For example the "User Domain" is responsible for everything related to user registration and management, while the "Product Domain" cares about our product catalog, and the "Shipment Domain" concerns itself with everything related to the shipping process. We identify all events, processes, actors, data, and more within each of these domains, and all these processes run apart and independently from each other, passing information at the end. All parts can consume such information if needed and act on them accordingly.

Let's take a simple example: A buyer is confirming the arrival of an order. The "Shipment Domain" now cares about the business logic of the shipment itself. It changes the status of the order to "Arrived" and opens the possibility of an evaluation, while it passes the information about the arrived shipment onto a "Message Bus." Every other part that's interested can now consume this information and do its thing. A dwarf (2) from the "Wallet Domain"—responsible for the money (flow) on Cardmarket—is already waiting to consume exactly this information, checks if the seller was paid for the order, and cashes out the funds if not done yet.

The best thing: This business logic runs fully behind API endpoints. The new Cardmarket is being developed as a RESTful API and our very own website also "only" communicates through this API to the business logic. For everyone exclusively using our website: Absolutely nothing changes—except for some improvements to the pages themselves. For everyone who wants to implement their own powerful tool for Cardmarket: The approach of writing an API-based web application means it is in our very own interest to keep all API endpoints fresh, complete, and free of bugs. Standardization is key, and we'll have a comprehensive documentation with Swagger/Open API Specification from the beginning.

This clear separation between the business logic in the background, the website as the view logic on top, and the endpoints in the middle as well-defined contracts between both also allows us to make a clean distinction between the back- and front-end teams. For our main back-end technologies we've chosen TypeScript within AWS Lambda functions and libraries connected to AWS Aurora as our database solution via TypeORM, while the front-end team is developing based on React. In addition we use many other modern technologies and tools to complement our tech stack. It is a huge challenge for the complete team: Everyone had to learn new technology, get familiar with a new infrastructure, and understand a completely different architecture—from one day to the next. But we already left the first phase behind us and are in the middle of the implementation of Cardmarket 3.0. At the beginning of this year our front-end team was also reinforced by Sonny and Ioannis, while our back-end team welcomed Tamas and Guillermo.

We still have to cover the greater part of the distance. But we already gained the speed and momentum for us to be optimistic to finish this marathon in under two hours, 30 minutes (meaning: at the beginning of 2023).


(1) For anyone wondering, this also explains the version jump when we talk about "currently working on Cardmarket 3.0."

(2) Cardmarket employs an indefinite number of little dwarves, who putter around in the depths of our code and take care of anything from recurring processes to repair work on corrupt data. Well, we originally meant to go for brownies, but they wouldn't have been reliable enough.



0 Comments

To leave your comment please log into your Cardmarket account or create a new account.

cardPreview