April 10 was perhaps the most eventful day on the Bitcoin markets in nearly two years. The price started the day at roughly $200, briefly spiked up to an all-time high of $266, and then precipitously crashed, dropping down to nearly $100 in a matter of hours, and perhaps signaling the end of the four-month-long rally since the currency first started to rise from its plateau of $13 in January this year. Trade volume on all major exchanges, even if denominated in bitcoins, hit an all-time high, including a total of over $45 million USD (550,000 BTC) traded on MtGox on April 12 alone, and over $30 million on April 12 – altogether earning the exchange significantly over $100,000 in revenue over the three days. Public attention on Bitcoin hit an all-time high, with the Bitcoin subreddit briefly becoming the 14th most viewed Reddit community, even surpassing front-page categories such as “movies” and “technology“. For the next two days, the carnage continued, and the price fell to a low of $54.20 before partially recovering roughly to the April 1 level of $100, as the Bitcoin community is left wondering what is going to happen next.
The major events of the crash are as follows. At 12:00 GMT on Wednesday, March 10, the Bitcoin price struck its new high of $266, and the direction the price was going slowly began to turn down. After collapsing to $240 the price briefly rebounded to $258, but then quickly resumed its downward path, breaking below $225 in four hours, $200 in five hours and striking a low of $105 at 19:20 GMT, before once again temporarily rebounding to a maximum of $203 for several minutes. During the downswing, the trading ending lag on MtGox – that is, the amount of time between a user making an order on the exchange and the order being processed, itself reached a new record high: over seventy minutes.
Seeking an explanation for the crash, many quickly came up with one possible answer: DDoS attacks. A distributed denial-of-service, or DDoS, attack, consists of an individual or organization with a large computer network attempting to deliberately overwhelm a target with requests so that the server is too busy to handle legitimate users. Such attacks had happened to MtGox before in the past few months, the likely intent having been to trigger a price crash which the perpetrators can profit from by shorting or buying at the bottom, and so naturally many were wondering if the same had happened on April 10. But MtGox’s Twitter account, which would become the main outlet for public relations messages from the exchange throughout the entire period, soon confirmed that this was not the case; “it is not DDOSed, it is lag due to high volume trades,” the tweet explained. A few hours later, trading somewhat subsided, and latency decreased, but soon enough another problem appeared: some users were having problems reaching MtGox’s website at all. Fortunately, the issue was not too serious; “Network maintenance, don’t freak out!”, a second tweet explained.
Soon enough, however, the DDoSers did finally join the scene. Access to the site continued to be sluggish, and less than than twenty minutes after the tweet about the maintennance, MtGox released yet another tweet: “Maintenance Over, however we are now under a DDoS attack.” Also, users investigating MtGox’s public order logs soon discovered that another kind of DDoS was taking place at the same time: MtGox’s database was being flooded with thousands of small orders of very low value, which many believe could have no possible purpose other than to slow MtGox down.
A few hours after that, MtGox responded to the situation with a drastic move that had not been done for nearly two years: for two hours, the exchange shut down trades entirely, claiming that its staff needed the time to upgrade its servers to handle the increased load. After the two hour maintenance, however, the exchange had further technical issues, and remained essentially inaccessible for eight hours further. The Bitcoin economy was already in a state of shock, and loss of the largest exchange in the Bitcoin economy, on which roughly 60 of all trades normally take place, put the markets into a state of panic. Other exchanges, many of which were also intermittently down due to the combination of DDOS and trading activity, disagreed on exactly what the price was supposed to be, but a general consensus soon formed: it was going further down. BitStamp dropped down to about $60, and when MtGox finally re-opened it quickly dropped from its pre-shutdown level of $120 to $75, briefly rose back to $130, and then once agan plunged, ultimately hitting about $60 as well. On April 12, MtGox struck a low of $54.25, but the panic finally subsided, and the price has since rebounded to over $100.
What is most interesting about the price crash, however, is not the change to the price itself – rather, it’s what we learned from it. Ultimately, the crash has served as a sudden, massive stress test to all sides of the Bitcoin ecosystem: the infrastructure’s ability to hold together, the market’s ability to withstand and recover from the sudden drop in price, and the Bitcoin community’s ability to retain its confidence. And the test proved to be very revealing. We now know exactly what are the strengths of the Bitcoin economy, where the performance is acceptable and where we seriously need to improve.
The main weakness of the Bitcoin economy is now obvious: the exchanges, and specifically the singular dominant position of MtGox. After MtGox was first created in 2010, it quickly became by far the largest exchange in the Bitcoin economy, at its peak in June 2011 reaching a market share of over 97. Since then, the relative size of MtGox has decreased somewhat as other exchanges have slowly gained the community’s trust, but the market remains an oligopoly, with MtGox still retaining a market share of about 65. The sheer volume of bitcoins traded on the exchange, combined with the prime importance of exchanges to the Bitcoin economy, gives MtGox the power to, whether through malice, incompetence or simple human error, significantly manipulate the Bitcoin price in either direction unilaterally.
On April 10, it was MtGox’s poor preparation that proved to be the deciding factor. The trading lag problem that MtGox faced on the first day of the crash was not new; in fact, it had been present during nearly every minor flash crash on the way up to $260 since the price first broke through its 2011 all-time high of $32, with trading engine lag frequently reaching twenty to forty minutes at its peaks. Despite this advance warning, and despite the massive amount of resources that the company had at its disposal following three months of rapidly increasing profits from trade commissions, the exchange did nothing to improve its systems – until it was already too late.
Other exchanges tended to perform better than MtGox throughout the three days of the crash, but they too were not unscathed. BitStamp shut down entirely for about one hour on April 11, but remained relatively responsive for the rest of the duration. Bitfloor had three brief shutdowns lasting one to two hours each, and BTC-E saw about three hours of downtime on April 11.
Thus, the technical issues alone are relatively excusable; other Bitcoin exchanges, including BitFloor, BitStamp and Coinbase, are also struggling to adapt to the Bitcoin community’s rapid growth, although some are managing better than others. MtGox’s main failure, rather, is how it dealt with them. Soon after its tweet confirming that there was no DDoS, the exchange released a longer message(Pastebin) explaining what had happened in more detail. The message started with the following, now infamous, phrase: “we would like to reassure you [that] no we were not last night victim of a DDoS but instead victim of our own success!” It then followed:
Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the number of trade made a huge impact on the overall system that started to lag. As expected in such situation people started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze the trade engine!
“To give you an idea of how impressive things were here are some numbers that we would love to share with you guys:
The number of trades executed triple in the last 24hrs.
The number of new account opened went from 60k for March alone to 75k new account created for the first few days of April! We now have roughly 20,000 new accounts created each day.
In th eyes of many Bitcoin users, perhaps grieving at the sudden disappearance at their hope of instant riches, but also legitimately angry at the exchange’s failure to perform up to the Bitcoin community’s expectations, the message amounted to little more than rubbing salt on their wounds, gloating that while the rest of the Bitcoin economy, and even MtGox’s actual service, was in disarray the exchange’s usage volumes and profits were higher than ever.
The second major error that MtGox made was shutting down the exchange. Upgrading one’s systems is a laudable goal, and even if it took a twelve-hour downtime it had to have been done eventually, but MtGox missed completely on timing. At a time when the Bitcoin market was in the middle of its greatest phase of instability in nearly two years, and what Bitcoin users wanted most of all was simply confidence that the infrastructure still worked, almost without warning the exchange responsible for nearly two thirds of all of the world’s Bitcoin trade simply shut down. To be fair, there was some warning; at the end of their message denying a DDoS attack MtGox wrote that they would have to “close the exchange for two hours in the next 12 to 24hrs to add several new servers to [their] system,” but few received the message, and moreover there is a large difference between the two hours that were promised and the ten for which the exchange was down in reality.
Since then MtGox has made an honest effort at damage control, and took two major steps to regain the public trust. The first, unfortuantely, proved to be counterproductive. Seeking to make up for their poor performance in the eyes of the Bitcoin community, MtGox temporarily lifted all trading fees on the exchange. At first glance, this appears to be an act of niceness, but an understanding of basic economics would have quickly revealed the folly of the scheme: once fees were taken off, trade volume spiked up massively, and trading lag once again hit over 45 minutes on April 13, requiring MtGox to shut down once again for five minutes.
The other measure that MtGox took, however, was quite laudable, and undoubtedly had a positive effect on maintaining confidence. MtGox CEO Mark Karpeles opened thread on Reddit saying “We are MtGox. Ask [us] anything.” Some quotes from MtGox’s answers include:
We absolutely understand this. The fact is that we are programmers and engineers, not PR guys, and we are still building out our capabilities beyond technology and into servicing our customers better. So, yes, we’re moving on this now and have secured help … Please keep in mind that we are not native English speakers, and that was a poor choice of words. The “success” is in the quick uptake of bitcoin most of all.
NO. Everything is accounted for (BTC and money). Fractional reserve is absolutely against our principles. In fact 90~95 of BTC are held in cold storage.
No. We have a company policy that forbids employees (contractually) from trading on the exchange
Our system was designed to handle 2~3x our normal load, but now we’re experiencing 10x the amount, which was difficult to prepare for (it takes weeks) with the sudden new accounts. We have two problems: the DDos and volume related to new accounts. The trade engine is capable of accepting much more of a load. Within 2~3 weeks we will completely rewrite the trade engine, in the meantime we shut down the system today and installed a new server with the current trade engine. Of course, if we didn’t have DDos everything would be fine, so now we’re dealing with two issues at once.
MtGox now much more frequently releases updates on Twitter, and posts longer messages on issues like outages and denial of service attacks when needed. As a result, the exchange has arguably shifted from being one of the most closed companies in the Bitcoin community to one of the more open.
Nevertheless, what the events of the past three days have shown is two things. First, the current state of Bitcoin exchanges as a whole is woefully inadequate to handle the kind of load that would come with being a mainstream part of the global financial system. The root of the problem is not the lack of a sufficient number of servers; rather, it’s architecture. Aside from the brief maintennance and DDOS on April 11, MtGox’s network-facing servers generally performed well, and it was clear that processing orders was the bottleneck. From the information that is publicly available, many have concluded that MtGox’s database architecture can currently only handle orders in series (ie. one at a time) – a setup that works decently for small numbers of transactions, but breaks down completely at a large scale.
“A modern stock exchange today really isn’t an application, it a group of systems that pass messages. Stock trading systems are designed that way so that when you place an order it wil be executed in a predictable and constant amount of time,” security expect Andreas Antonoupolos explains on The Daily Bitcoin Show. The exchanges we have today, on the other hand are largely designed more like traditional web applications, an industry in which order of operations is not particularly important because the activity of most accounts is independent. This is understandable; most Bitcoin exchanges were originally designed by web developers, who may be experienced in their own fields but had no experience implementing financial systems before they came to Bitcoin. However, it nevertheless means that the performance of nearly all exchanges is currently highly suboptimal.
Fortunately, Bitcoin is now seeing more and more attention from mainstream financial institutions, and it is quite likely that from this attention much more high-quality exchanges will emerge. There are already specially designed high-volume exchanges appearing specifically to serve the needs of high-volume institutional investors; Tradehill’s Prime and the Malta-based Exante’s Bitcoin hedge fund may well be only the first two of many more to come, and now that the recent FINCEN ruling has established guidelines describing the legal status of Bitcoin exchanges it would not be surprising if a mainstream financial firm was already working on a professionally designed Bitcoin exchange for ordinary users as well.
Second, the Bitcoin economy is now simply too large for one exchange to handle, no matter how efficient and scalable its architecture may be. Both this crash and many previous crashes, large and small, were ultimately caused by some kind of lag or glitch on MtGox, and the fact that the actions of a single company are essentially the deciding factor for whether each individual flash crash turns out to be minor or catastrophic bodes ill for the future stability of the Bitcoin markets. There are alternatives; BitStamp has expanded greatly in the past few months, as have Bitfloor and BTC-e, and very soon they will have the opportunity to expand much further. The reason is this: one of MtGox’s main advantages over exchanges like BitStamp to date has been the fact that it allows users to place orders even if they do not have enough money in the right currency to fill them – a very useful feature, as it allows a trader to, for example, put a buy threshold at $110 and a sell at $140 and set himself to profit from a movement in either direction if he believes that the price will ultimately stabilize in between. Starting April 17, however, this feature will be gone, making MtGox no longer any better than BitStamp or anyone else.
There are also several other options are on the horizon. One developer in San Francisco has started ButterCoin, an open source project to create an exchange platform that can handle a high level of load. Another “exchange” worth watching is Ripple, a new decentralized digital payment network that intends to allow users to store and handle any currency, including fiat currencies, bitcoins and potentially even precious metals, using Bitcoin-like wallets, addresses and transactions. Ripple includes a built-in “decentralized exchange” functionality that allows users to trade one currency in the Ripple network for another directly – that is, without involving any third parties to complete the transaction. The system is not perfect; just like Bitcoin, ultimately every server must verify every transaction, but unlike MtGox the architecture was designed to be highly parallelizable from the start, allowing servers to scale to any number of transactions simply by adding more cores.
But there is also an aspect of Bitcoin that has held its own extremely well the last three days: its reputation. Even after Bitcoin crashed, articles on The Economist, Time and TechCrunch rushed out to look “beyond the bubble” and defend it, the Winklevoss twins, known for their very early involvement in Faceboook, caught the media’s eye for their support of Bitcoin, and the Business Insider ran a piece by Dan Kaminsky positively discussing one of Bitcoin’s greatest strengths: its security. “It is a fairly open secret that almost all systems can be hacked, somehow,” Kaminsky wrote, “… by all extant metrics in security system review, this system should have failed instantaneously, at every possible layer. And, to be fair, it has failed at other layers – BitCoin thefts have occurred, in the meta-code that surrounds the core technology itself. But the core technology actually works, and has continued to work, to a degree not everyone predicted.” After the crash, CNBC added a Bitcoin price ticker to their webpage – an indicator that at least that organization believes that Bitcoin is something worth watching for many months to come. In June 2011, the response was the opposite; media attention quickly turned grim, and over the next five months mainstream websites wrote a number or articles describing Bitcoin’s ignominous fall. This time, the extent to which the usually fickle mainstream media is willing to continue believing in Bitcoin should shock everyone. To an extent that many still do not realize, this is a hugely positive sign for Bitcoin going forward.
From here, it is hard to tell what will happen. The last two major crashes, one in June 2011 and the other in August 2012 had very different aftermaths; after the crash from $32 to about $15 June, the price continued dropping, almost hitting pre-bubble levels with its November low of $1.994. After August, the price recovered quickly, bouncing between $10 and $13 for four months before finally starting its present rise. This time, anything could happen. We could always see further drops, with one capitulation following another and the price perhaps even dropping below $30, we could stabilize at $100 to $180 or we could see $500 within two months; at this point it is simply too hard to tell. Even if Bitcoin’s meteoric rise does not continue, however, all in all in two years’ time we will have gained greatly from the ordeal. Out of the bubble of 2011 came companies such as BitPay, WalletBit (now BIPS), BitStamp and BitInstant, all of which are now mainstays of the Bitcoin economy. We have already seen a number of businesses come out of 2013; the BitcoinStore, Prime and Ripple are three great examples, and there are many other projects brewing behind the scenes, and if Bitcoin continues to grow it will be these that set the stage for the bubble of 2015 – or the bubble of May 2013. Until it comes, we’ll never know.