Last Saturday we made some changes to our own infrastructure. The planned downtime was 60 minutes tops. Unfortunately and due to a problem with network latency that downtime extended to more than two hours. More on the reasons below. First a little recap of what we did.

Upgraded Infrastructure

We split up the infrastructure Scalarium runs on across more instances. We're managing a good number of instances by now and started to see where we needed to add more capacity or break out processing power to bigger EC2 instances. We're now running on a cluster of of medium and large EC2 instances.

On the way, we migrated some of our components to newer versions, namely CouchDB and Redis.

• CouchDB has been upgrade to the recent 1.0 release. While we were on that matter we also made sure that disk access speed (which CouchDB heavily relies on) was improved drastically by moving it to a RAID 0 of four EBS volumes. On the way we also set up another replica constantly streaming changes from the main database, all thanks to the continuous replication introduced in CouchDB 0.10. If you haven't checked it out yet, CouchDB's replication really is its killer feature. We easily streamed the data from the old database server, while the new replica already streamed the changes down from the new server as they came in.

  The good news for you is that support for RAID across EBS volumes is halfway there for Scalarium. There's been excessive benchmarking around it, so be sure to have a look at them, as it will help with the decision on which scheme to pick.

• Redis has been upgraded to the latest release candidate of the 2.0 release line, and has been given a lot more memory to work with. Redis has become an important part of our infrastructure, and it's only fair it gets a good amount of memory to work with. The 2.0 line also brings some interesting new features that we'll be moving some of our use cases to. Migrating data to the new Redis release over the network went just as smoothly as with CouchDB. We told the new server to connect to the old instance as a slave to stream down the full dataset. Then we unset the slave status, and the job was done.

Overall the move itself went smoothly and as planned, and we started pre-warming CouchDB's views on the new server and bringing back up the site.

The Extended Downtime

When we started browsing through the sites for some immediate tests we realized that it was incredibly slow. We profiled the queries sent to CouchDB from our application servers and found that they had huge latency issues, as they took about six times longer than the ones we ran from other new instances set up in the maintenance window. We profiled other access times and found that connection times were the road block. Once connections were set up things went at normal speeds. These things happen on EC2 from time to time, and the recommendation is to go get a beer and wait for them to resolve themselves. You won't find that recommendation in the manuals, but it's not unheard of that latency issues in fact stop after some time.

Waiting for them to resolve themselves was not good enough for us, so we shut down the new instances and brought up new ones. One the way, we added more security groups to them so that it's ensured that the whole cluster is having at least one group in common. Sure enough, the new instances resolved the problem, and we saw the latency times we expected.

In the end, the downtime took a lot longer than expected, and for that we apologize. No instances were harmed during the downtime. The good news is that with the upgraded infrastructure we can put our focus back on new features. We already rolled out a good stash of new features in July that haven't been talked about yet here. We'll showcase some of them over the next week or so. Stay tuned!

This week we finally rolled out some noteworthy changes, and we thought it's a good time to give you a tour of what's new. The most visible change is the overhauled user interface. We simplified workflows, tried to gather more information of interest where possible and useful, and generally cleaned up lots of smaller things. The result is a cleaner interface, making it easier to navigate through and work with Scalarium.

When you log in, the first thing you'll notice is the new dashboard, now showing all the important information in one place, clouds, applications, events and the latest deployments. You can also see how many instances are running in any one cloud without having to go to the details page.

We also reworked the details pages for clouds and applications, moving less frequently used actions into sub menus.

You'll also notice that we sprinkled in some friendly reminders here and there, in case there's something you may have forgotten. After all, getting you application up and running as fast as possible is what the cloud is all about.

When an instance failed during its initial setup, it wasn't easy to figure out what exactly went wrong. Now we immediately bring the error to your attention, including a link to the logs for easier investigation of what caused the problem.

As mentioned in an earlier post, you can now start clouds in any of the available EC2 regions, therefore being able to easily deploy your apps across the globe using Scalarium.

Last week we sent out a big pile of beta invites. If you didn't get yours or haven't signed up for the beta yet, feel free to get in touch or sign up on our beta site.

We'll also be at a couple of conferences in June, namely RailsWayCon, Berlin Buzzwords and IT-Profits in Berlin and MongoUK in London. We'll always have invite codes with us, so come talk to us.

We just finished putting the finishing touches on a nice update for Scalarium: Support for all EC2 regions, including the new Asia Pacific region in Singapore, which was just opened to the public earlier this week. What that means is that you can create clouds in either the two US regions, Asia or in Europe, allowing you to create instances in their respective availability zones.

So far, Scalarium has focussed on supporting the EU region, simply because that's our home turf, but right from the start it was built to be agnostic to the region a cloud is running on, and we always intended on opening up to all the regions, allowing our customers to use Scalarium to run their applications close to their customers.

The change in Scalarium itself is rather subtle, but here's a peek, also giving you a preview of the UI changes we're currently working on:

We'll be rolling out the changes within a week or two, including the new interface, so stay tuned!

It's been a long time coming, but we finally published the prices for using Scalarium, including our support package. Long story short: You're paying for what you use. Only want to fire up an instance for an hour? Then in the true spirit of the cloud, we'll only charge you for an hour. No minimum usage, no baseline fee. Simple like that, after all, that's what Amazon does too, right? Scalarium usage costs are on top of Amazon's EC2 fees by the way. We're managing your instances, and we're using your keys. The instances are yours to keep. That way it's easy for you to buy Reserved Instances from Amazon, and reduce your cloud costs by a whole lot.

Anyway, what took us so long you ask? It sure isn't easy to find a pricing model that's cloud-ready, as in: pay only for what you use. However, that usually implies that you either have to somehow get the metering data from Amazon or simply meter the instance usage yourself. At first we refrained from doing the latter, as that'd imply quite some work on our part, and there's a tiny margin of error.

Of course it'd be awesome to be able to access Amazon's metering data through an API. In the sense of Amazon's Web Services, it'd only be logical. No such luck, amigo. Plus, we'd have to fetch them regularly, not just once per month, to always be in the loop about the full usage.

We've considered other options before we ended up deciding on the thing that'd be the hardest to work with: Charge per running instance per month, charge per average number of running instances, charge a monthly base fee on top, that'd include a couple of instances already. We were excited about this payment model at first, but there's problems with it.

It's simply not in the spirit of the cloud. Charging a full month for an instance you've only used for two hours? That just didn't feel right. So we dropped it and thought again, and the result was something that we, in the end, came up with independently. It'd be the only way we could fairly charge based on usage, and that'd enable even small companies to give Scalarium a spin without spending big money from the get go.

So that's where we ended up: No monthly baseline charges, you pay for what you use. Simple like that. After all, there's only work to do on our part when you have instances running. It's the fairest way to pay for a Scalarium's cloud management.

We'll probably add paid add-ons in the future that'll still be charged on a monthly basis, and we'll keep improving Scalarium constantly. If you find something's not to your liking, or you're missing a certain feature, please get in touch.

Over the last couple of days we started collecting material for a knowledge base, but had yet to set up something to collect the texts in. Apart from that we needed a common place to collect input, questions and problems from users, where our support staff could be at the ready to reply. Fret no more, we've got you (and ourselves, for that matter) covered. You can reach our support system at support.scalarium.com. We already added in the most important questions on Scalarium and how parts of it work. Let us know if there's things that you feel should be in there.