For the past several months, we’ve been working on improving the SproutCore internals on a project codenamed Amber. I’ve talked about it some in various venues, but I wanted to take the opportunity to give a full status update of where we are and where we’re going with it.

I also plan on talking more about the work we’re doing going forward. Not all of it will be immediately actionable, but there’s been a lack of information lately about what’s happening on master that I hope to remedy.

What is Amber?

Today, SproutCore has amazing tools for building up large applications using JavaScript. Those tools have been honed by companies working on some of the most complex client-side apps on the Internet, and SproutCore has years of experience working with the problems of complex client-side data.

However, much like Rails 2.3, those tools are provided in a very monolithic way. If you want to use SproutCore, you’re opting into the whole shebang, which can add up to many hundreds of kilobytes.

While that does not present much of an issue for desktop applications, and has served applications quite well, it does have a number of challenges.

1. The approach doesn’t work very well for applications that start small and grow into the full feature-set as needed. This means that people often pass over SproutCore at the beginning even if it would be a good fit by the time their project is done.
2. The approach doesn’t work very well for existing applications. Unlike jQuery, it’s hard to drop SproutCore into an existing app and start using its tools without making a significant investment.
3. Importantly, this approach has drawbacks for mobile applications. Mobile devices pay a large price for downloading and parsing code, even if unused, so SproutCore’s inability to start small, and carefully grow can present real difficulties.

Amber aims to solve these problems without fundamentally altering the core of SproutCore.

A Saving Grace

This problem is almost entirely a problem of properly factoring the code. SproutCore’s build tools can handle more granular dependencies, but SproutCore’s team has long treated the Foundation framework as a place to put almost anything that could be categorized as “miscellaneous”.

This means that the smallest possible SproutCore build (Runtime + Foundation) ends up including all kinds of miscellaneous junk that isn’t necessary for many applications, and certainly not the kinds of small applications that are a good fit for SproutCore except for this problem.

The Road Ahead

The solution, then, is to more granularly factor SproutCore into a smaller core and a number of optional components that you can use as you need them. I’ve already done a bunch of initial work to streamline the size of the minimal SproutCore installation.

I’m doing all of the refactoring work on master (as we did for Rails), so you can keep an eye on the progress there. Initially, my goal is to make it possible to run an Amber application using the normal SproutCore build tools, and I was able to get a Hello World application running through them yesterday. With the work so far, the size is down over 50% from the previous minimal build (somewhere around 65k + jQuery), and boot time of hello world is down to 800ms, from a previous boot time of over 2 seconds.

Incidentally, in case you haven’t looked at the Webkit Inspector lately, and are thinking about boot in terms of server response time, the vast majority of a typical page load is in the client, not on the server. In comparison, popular sites like New Twitter or Facebook load in multiple seconds. That doesn’t mean that a large site written in Amber would load in 800ms, of course, but it should give you some perspective about the cost of loading any large application into memory. If you’re interested, time the boot time of native applications.

There is still quite a bit of work ahead to further streamline things (my personal goal is to get Amber down to 40k + jQuery and get boot time below 3-400ms). Additionally, we will need to spend some time documenting use of Amber as a standalone framework.

The Amber framework itself will ship as part of the SproutCore 1.5 release, but it will still be in a fairly early state and subject to changes before the subsequent release. Because it is fundamentally a refactoring effort, it is possible to keep everything green and existing applications running while the process is ongoing. At Strobe, we plan to start using Amber internally for new applications relatively soon, and I’ll keep everyone up to date on when it is appropriate to start playing with it for real apps.

FAQ

• What is Amber? Amber is the new core of the SproutCore controller and view layers. Its goal is to reduce the size and load time of new SproutCore applications.
• How does Amber affect existing apps? At this time, Amber is an internal refactoring effort, and the Foundation framework, the current core, depends on Amber. As a result, existing applications should not see any breaking changes from the Amber effort.
• When will Amber be ready to experiment with? We are very close to using Amber for projects internal to Strobe. That said, we expect the size and scope of Amber to continue to shrink over the next few months, and we expect the overall starting experience to improve for Amber-specific applications. Keep an eye on this blog for more specifics.
• When will Amber be ready for production? I personally expect Amber to be an important part of the release following SproutCore 1.5. The team hasn’t really discussed the roadmap for that release yet, so it’s hard to say what form it will take exactly, but it is the most important thing I am personally working on, so expect it to get some love (both technical and documentary).

Some Technical Details

If you’re not interested in the technical nitty-gritty, you can feel free to stop here.

In SproutCore 1.4, SproutCore’s “core” was broken up into two pieces: Runtime and Foundation.

• Runtime: The browser-agnostic part of SproutCore. It defines the object model, the binding and observer system, the run loop, and other miscellaneous tools (notably the Enumerable interface).
• Foundation: The core of the browser support in SproutCore. It defines the controller and view layers, a number of browser-support features like efficient event handling, selection support and a whole grab bag of miscellaneous features.

The initial work I did was extremely low-hanging: move some obvious Runtime mismatches into Foundation. For instance, 1.4′s Runtime had cookie support, which clearly doesn’t match the purpose of Runtime.

Next, I created a new Amber framework inside of SproutCore and made Foundation depend on it. SproutCore is internally structured as a series of frameworks with dependencies between each other. By making Foundation depend on Amber, I avoided causing any backwards-compatible breakage in applications that depend on Foundation explicitly.

Next, I started pulling in core elements of Foundation into Amber. The goal was to split up Foundation into the absolutely essential elements and the grab-bag elements which weren’t really core. Here’s the process I used:

1. Move a file (like views/view.js) into the Amber framework from Foundation
2. Move its associated tests into the Amber framework
3. Attempt to run Amber tests
4. When they inevitably failed, try to figure out what files in Foundation I didn’t move over, and start again from 1 until tests pass
5. Start again from 1 with a new feature

One reason this process was so fragile is that SproutCore does not require every file to list its dependencies. Instead, SproutCore includes every file from a framework, and uses optional requires to control the compilation order. This was almost certainly a design mistake, because it makes it impossible to create a sane dependency graph, and one that we will correct moving forward (in much the same way that ActiveSupport in Rails 3 took the time to properly indicate requires in each file).

The initial work was pretty straight-forward, but I ended up pulling in entire unnecessary files just to get things working. To illustrate, I ended up pulling in all of control.js just for the SC.MIXED_STATE constant which was used by pane.js. So after the initial work, I started to look at individual files to find additional reductions.

Another good illustration is the String utilities. In addition to some common inflections (like camelize), the file contained a table for removing diacritics. While useful, this table is quite large and not very amenable to gzipping, so I moved it back to Foundation.

The next step is to take a serious look at huge files like view.js. That file alone amounts for about 12k of the final minified and gzipped distribution. In order to achieve this, I have needed to take a serious look at how SproutCore classes are put together. While JavaScript has the tools necessary to allow breaking up a class across multiple files, SproutCore’s object model didn’t expose that very well.

In the past couple of days, I have added a new experimental reopen method to SproutCore’s object model that makes it easier to split the implementation of a class across a number of files, which will make it easier for us to further refactor large files like SC.View and start to understand what their constituent parts really are.

At the moment, the Amber framework has:

• Controllers: Array and Object
• Support: Selection (used with ArrayController) and Delegate
• View System: SC.View, SC.Pane and core Theming
• Detection: Browser and Feature
• Browser: Device Orientation, Event Normalization, Event Delegation
• Localization: Core
• Misc: jQuery extensions, JSON support, Sparse Array, String utilities

Runtime has:

• Object model (classes, mixins)
• Array extensions
• Protocols: Enumerable, Comparable, Copyable, Freezable
• Observers: Observable and Bindings
• Run Loop: Baked in batched DOM updates
• Useful Utilities: Range Observer, Set, Logger

These lists aren’t exhaustive, but they should give you a general sense for what the core frameworks are shaping up to look like.

As I said before, keep an eye on this space for more information moving forward.

Tweet