On 15 March when we became aware of the problem, we were supposed to be able to handle 3 or 4 times the peak load, so we took a reasonably relaxed view of Sunday.
But this was a mistake. Requests to your stores exploded up to 20 times!
That weekend we were able to successfully process many transactions.
Fortunately, the “internet” has slowed down access to our servers a bit due to a general overload.
However, more traffic doesn’t necessarily mean more transactions (i.e. actions that you and we benefit from). On average, doubling the number of transactions increases system load by about 3.5 times.
As machinists in the engine room, we rebooted servers 24h/24, renewed and reinforced them, and we were surprised by the bottlenecks that were constantly appearing (line too small, exhausted RAM memory, full hard disks, too many connections to this or that service, mail server blockages due to heavy traffic, exhausted budget at this or that service provider…), interesting errors that only occur under heavy load.
In the following days, we wanted to make the whole system “flexible”, i.e. to be able to switch on sufficient safe resources (computing power, memory, network capacity) as required. Unfortunately, this cannot be achieved simply by “more computers”, incoming requests must be coordinated, sessions must be managed and shared caches (intermediate storage) must be efficient and fast. Your management systems (PCG) must also have fast access to data.
This purpose has now been achieved. Although we (and our colleagues in the data processing centre) were able to monitor the load just after the weekend, we made a few mistakes here and there due to the large number of connections, resulting in sporadic order cancellations, PCG bugs or missing images. Thank you for your patience! Currently, the system has been running very stably for a few weeks now, regardless of the total load. We have seen almost 2000 transactions per hour with very fast shops.
We are now in the process of organizing the new ideas that have emerged during this intensive period – not only ideas from the machine room, but especially requests from your users for better management of new customers and tours. The first functionalities, such as tour-specific waiting lists, are in progress and were put online after Easter.