Cloud Auctions

This past week I attended a networking/cloud event to learn some new information about cloud computing.  Before the event began I met Laurent Raufaste who was an IT administrator at the start-up company Percolate.   The event that we were attending was about using the clouds for quantitative analysis of financial instruments (yeah I know sounds really exciting). As we started chatting we realized that neither one of us were quants but we had a common interest in cloud computing.  As we talked Laurent mentioned to me a truly innovative way of using cloud computing.

The technology had to do with  cloud auctions.  Percolate  is a social media based company that specializes in tracking social media sites and helping users analyze trends so they can create better content to put out to their information consumers.  What that means is they have to look at massive streams of social content and then choose what will get the most attention to their readership.  In order to do that the needed to create an algorithm that could look through mountains of data and then provide a recommendation.  Essentially they are looking at a "Big Data" solution.

As a start-up Percolate needed to find a way to do these calculations in the quickest time possible but as inexpensively as possible. One or the solutions that the uncovered was cloud auctions  The way a cloud auction works is cloud providers have excess capacity that if they don't have in use becomes wasted capacity.  Instead of leaving the capacity unused they open up an auction for the unused capacity to users to bid on.  The bidders that are above the spot price at the time will get the amount of capacity they bid on.

So let's say a cloud vendor has a big customer leave and now they have unused capacity in their data center they can accept bids from all those companies that are looking for inexpensive compute resources.  The spot price of the computing resources would change periodically based on supply and demand, and customers whose bid exceeds it gain access to the available instances. If the price you bid exceeds the current spot price for the resources then you get access to the instance.  As long as you are above the spot price the instance is yours. This essentially could give you access to cloud resources for penny’s.

There are a couple of big caveats for this type of resource allocation.  First it takes a while for the instances to be set up so if the spot price moves you may not get access to the instance.  Second once your bid price is below the spot price or compute capacity is needed by the vendor, you're instance will immediately be shut down.  For most commercial software packages this is not acceptable but if applications are written with that knowledge in mind then they can deal with losing their environment.

Percolate for example uses this compute technology to do MapReduce calculations on their data. It is not a huge deal to them when the instances go down because the master nodes are located on non-volatile compute resources so when there is no response back from the node then another node will pick up the computation of the node that just went down.  This ability to get access to resources in this manner has saved a lot of money for an early start-up.  Very few vendors offer this technology right now but it wouldn't surprise me if a company popped up that took this concept and was able to lay it down across multiple cloud vendors (or even non-cloud vendors).  We shall see what the future will bring.