It’s possible to mix a Module into a second Module which can then be mixed into a class. The result is that an entire Module chain is moved into the a classes inheritance chain with one call to include.

Although this sort of mixin chaining is generally a bad practice which can lead to overly complex inheritance chains, it’s important in understanding how include works. rb_include_module is a complex function. As a first pass at it we’ll only look at the basic include loop…

The variable c here is used to store the current point of insertion. The first time through the loop c points to the klass passed into the function.
In the Web example above c is set to Web and module is TwoStrand before entering the while loop. An include class is created for TwoStrand with a super pointer to Web’s super, Object. Web’s super is set to TwoStrand. Since any additional Modules to be inserted should be inserted above TwoStrand, the point of insertion, c, is set to the newly created TwoStrand include class. On the next line module is set to TwoStrand’s super, OneStrand. The next time through the loop a new include class is created for TwoStrand and it’s super is set to the super of the OneStrand include class, Object. The OneStrand include class’s super is updated to point to the new TwoStrand include class and once again the point of insertion is moved. Eventually super returns a null pointer and the loop exits.

An interesting property of the include class architecture is that an include class does not keep track of changes to the inheritance chain of the Module it “wraps”.

Although Strand’s super now points to Tangled, the Include class created for Strand when it was mixed into Web is unaffected by the change. This is known as the Dynamic Module Include Problem. This makes sense if one bears in mind that the Strand Module in Web’s inheritance chain is actually an Include class. Inclusion of Tangled within the Strand Module means that the the super pointer of Strand now points to an Include class for Tangled. Any classes or Modules that includes Strand from here out will pull a new Include class that points to Strand’s Tangled Include class into their inheritance chain when rb_include_module. Existing Include classes for Strand however remain blissfully unaware that their “real” Module now has an altered inheritance chain.

Part of the code that we did not look at in our first pass through rb_include_module is responsible for ensuring a Module is only included once in a classes inheritance chain.

Note that OneStrand was not inserted twice and did not change it’s position in the inheritance hierarchy. Let’s add some code back into rb_include_module to see how this is implemented.

A for loop has been introduced inside of the while loop. For each Module in module’s super chain the inheritance chain of klass is analyzed looking for a class whose method table is the same as the current Module’s method table. This method table comparison is a slightly non-intuitive way of checking if the current inheritance chain includes either the actual Module being considered, or an include class mapping to that Module. In either case the method tables would be the same, and would indicate that this Module is already included. If so, the include class creation is skipped and the while loop continues to the next Module being considered for inclusion. Note that this means that while a Module may only be included once, that does not mean that a re-include will not have side effects. Specifically, new additions to the Modules inheritance chain will be picked up by a re-include.

The TwoStrand Module is skipped over because its method table pointer is the same as the existing include class in Web’s super chain. When super is called on TwoStrand this time around, Tangled is returned. Since Tangled isn’t currently part of Web’s inheritance chain, a new include class is created for it and inserted immediately above TwoStrand.

It is possible for a Module to be included into an inheritance chain twice if it included into a super class after it has been included into a subclass.

Blacksmith’s super points to a Doppelganger Include class which points to Craftsman at the time Doppelganger is included into Craftsman. Craftsman has no knowledge of what classes may point to it and gamely allows its super to be set to a new Include class for Doppelganger. It’s hard to imagine a situation where you would want this behavior, but understanding how it can happen may help debug a bizarre situation.

The next (and last post) on Modules will cover extend and class methods. As always, thanks for reading.

A module which defines code that will be shared by other modules or classes is known as a Mixin. Mixing a module into a class is generally done with a call to the include method.

A call to the ancestors of NeedALittleHelp will confirm that Useful has been inserted into NeedALittleHelp’s inheritance chain

The inheritance chain might be drawn as…

An included module will always be inserted into the inheritance chain below a declared super class.

It’s worth pointing out that a single inheritance chain has been preserved. The include method has simply manipulated Klass’s inheritance chain by inserting the MixinModule module between the class and it’s super…

It’s possible to mix multiple modules into a class as below.

Once again the Klass’s inheritance chain has been manipulated so that the super of Klass now points to AnotherMixin whose super points to MixinModule…

This raises the question of what occurs when these modules are included into a new class in reverse order…

Here the super of MixinModule is pointing to AnotherMixin. This is odd behavior…

It would seem that the super pointer of a module can point to different places depending on context. Clearly include cannot be altering the super pointer of the actual module it acts on, as the order in which a class includes the module can result in differing inheritance chains. At the same time, it’s convenient for there to be only once actual instance of a module to allow for minimal overhead in updating the module’s method table. Ruby solves this problem with something called an Include Class. When a module is included into a class (or module), a new class is created on the fly. The method table of this class is a pointer to the included modules method table. This ensures that each class that includes a module has a “copy” of the module who’s super pointer is free for manipulation. Since the method table is a pointer to the one true method table for the module, updates to the modules methods are shared between all “copies” of the module. This module “copy” is an Include Class.

The code below is a simplified version of the C function for creating a new include class.

As described above a new object is created of type T_ICLASS which is the C struct for storing include classes. A pointer to the method table of of the module is copied to the include class. On the next line the super for the include class is set to the super class passed into the function as a parameter. The class pointer for the new class is set to the module which has been copied. This reference to the actual module is is useful for the case, as in the ancestor method call above, where the interpreter needs to know the module associated with the include class. This does mean that, unlike other classes, the class of an include class is not a metaclass. We’ll discuss the implications of this when looking at including class methods. A better diagram of the NeedALittleHelp inheritance chain might look like this…

The next post will continue to explore modules and includes class. We’ll dig deeper into implementation and cover some of the nuances of inheritance chain manipulation. As always thanks for reading and feel free to leave comments with questions, corrections, or haiku.

There are many different types of DSL’s. The broadest classification, coined I believe by Martin Fowler, distinguishes between internal and external DSL’s. External DSL’s are languages with a language specific compiler or interpreter. If you are using YACC or ANTLR to define the grammar of your DSL, you’re writing an external DSL. Nathan Sobo presented an interesting tool at RubyConf07 for defining language parsers and interpreters. Treetop allows you to create an external DSL using a grammar definition DSL and then easily define the interpreter in Ruby. Here is a simple phone number parser…

The really interesting bit is the phone_number method defined on the phone_number rule. The ability to add methods to nodes allows for a lot of exciting interpreter possibilities.

The advantage of external DSL’s is unlimited expressiveness. As long as you can define a parser for your language, you can express your intent in whatever syntax maps most closely to your domain. An interesting example of an external DSL is ANSI SQL. I’m not referring to the general purpose procedural monstrosities implemented by various vendors, but rather to the small language specific to the domain of set manipulation. Another example is Make for the domain of software builds. All that expressiveness comes at a high cost. The big disadvantage of external DSL’s is that you have to define the grammar, build the parser, and build the interpreter/compiler. If the author doesn’t define conditional statements, the language doesn’t have conditional statements, etc…

Internal DSL’s are languages whose definitions mandate syntactically correct programs for a “hosting” general purpose language. The hosting language’s interpreter or compiler is then responsible for program execution. The advantages and disadvantages of internal DSL’s are an exact mirror to those of external DSL’s. On the up side, because an internal DSL is defined within the context of an existing general purpose language, the language designer picks up a lot of functionality for free. Someone has already written a parser and interpreter/compiler for the host language, and provided all the common features that one expects from a modern language. This leaves the language designer free to focus on a compelling domain syntax for his or her little language. There is of course a catch, the host language puts immutable restrictions on the expressiveness of an internal DSL. This can be more or less of a problem depending on the natural expressiveness of the host language. Ruby is a fairly expressive. Syntactic features like optional braces, optional parenthesis, do end blocks and inline conditionals allow for creative authors to declare a fairly intuitive DSL syntax without violating Ruby grammar.

Within the field of internal DSL’s Ruby allows for a variety of styles. Jamis Buck has written an excellent introduction to these styles. We’ll focus here on the DSL’s style Jamis calls Class Macros.

The most prominent example of a class macro based DSL in Ruby is ActiveRecord…

Some would argue that this sort of thing isn’t really a DSL at all. The definition of “DSL” is very slippery, and subject to a debate that is as useless as it is endless. I’ll use Jamis’s inclusion of it into the pantheon as my argumentum ad verecundiam.

This style of DSL is used to alter subclasses in an intuitive and domain expressive way. In some cases this becomes nothing more than an extremely fancy implementation of the Template Method pattern, where the template methods are defined based on instructions communicated via the DSL. In the extreme case, subclasses can be reduced to configuration files…

Note that HttpGettable is defined as a module not a base class. In general, in Ruby, if you think you need an Abstract Base Class, it pays to consider if the relationship is really an is a, or can be better expressed as a can. If it truly is an is a relationship, go ahead and create a Base class. There has been a lot of discussion as to how one can force a class to be abstract in Ruby. My rule of thumb is, unless someone might die if the class is instantiated, I’ll depend on a reasonable name (like AbstractGetter) to communicate the usage to other developers. If you are developing pacemakers, you might want to define initialize to raise an exception.

HttpGettable could also be implemented using the configuration to define what Kent Beck calls Constant Methods…

This implementation trades some of the metaprogramming nesting complexity in favor of a well known module include idiom. It provides for an aesthetically pleasing syntax, but at the cost of some complexity. One could also give up on DSL’s and use plain old constants…

So which option is best? That really depends on circumstances, aesthetic taste, and the teams ruby-fu as a whole. Skipping the whole DSL option in favor of a less expressive but much simpler implementation is certainly a valid option. Metaprogramming for metaprogramming’s sake is a bad idea that adds cost in complexity without compensation. The Gettable example is intentionally trivial to clearly demonstrate the various options. In the real world I probably would rely on constants for something so straightforward. When trying to express multiple complex predicates I lean towards the DSL style…

ActiveRecord is another, more robust, example of where this style is clearly warranted.

Thanks to Jay Fields, Jake Scruggs, and John Hume for their input on this topic. I’d like to recommend a new shortcut by my friend and colleague Philippe Hanrigou. Troubleshooting Ruby Processes: Leveraging System Tools when the Usual Ruby Tricks Stop Working is an excellent resource for learning the tools and techniques for resolving those incredibly frustrating “why does my app just sit there doing nothing?” defects. Hint: It probably isn’t a mongrel problem.

As always thanks for reading.

Whether it’s called OnceAndOnlyOnce or DRY good developers should agree that in the overwhelming majority cases duplication in software is a bad thing. Duplication comes in many forms. The simplest to identify (and remove) is actual text duplication. More difficult to find (and remove) is duplication of concerns. Ruby class methods can help with the middle ground between these to extremes. Take accessors. Writing Ruby like this is grounds for the revocation of one’s programming license…

So what’s the responsibility of the weapon method? How about the climbing_instrument method? Both methods basically have the same concern, exposing access to instance data. Arguments about the wisdom of breaking encapsulation in this way aside, the Ninja class is breaking the DRY principle. This concern, exposing instance data, is so common in fact that Ruby defines a class method on Module whose purpose is to dynamically define instance methods which expose instance data.

I call the sort of duplication eliminated by attr_reader structural duplication. When otherwise well factored code features heavy structural duplication, one sees methods where the words are different, but the whitespace is exactly the same. Class methods, particularly when combined with convention and metaprogramming, are a clean way to remove this sort of duplication.

Take for example an initialized value…

This sort of initialized value is prone to a classic bug. The ninja can never be disarmed…

Useful as this may be to snake_eyes, it’s clearly not the intent of the disarm method. The fix is straight forward…

The weapon method redefines itself to be a simple attr_reader the first time it is accessed, it then initializes and returns @weapon.

Now assume that Ninja needs climbing instrument initialized in the same way…

Once again we can see a clear violation of DRY in the ninja class. The logic for defining initialized attributes is duplicated across the weapon and climbing_instrument methods. This logic can usefully be factored to a class method…

There are a couple points worth discussing here. One is the implicit trade off made by this sort of class method. A certain degree of readability has undeniably been lost in the pursuit of removing duplication. Readability is one of the most important aspects of code quality, and changes that reduce readability should not be entered into lightly. My inclination is to err towards readability for a very small amount of local duplication, and quickly switch towards slightly less readable better factored code when duplication surpasses minimal levels.

The other item worth exploring is where the attr_initialized method should live. The interpreter must see the method definition before the class method can be used. An alternative to cluttering up the class definition with the class method is to squirrel it away in a module. This results in a very clean Ninja class…

This touches on the concept of simplexity. The Ninja class hasn’t actually been made simpler, the complexity has simply been hidden off somewhere out of site. That said, once can certainly make the case that housing the attr_initialized method with the Ninja class is a severe violation of the Single Responsibility Principle. In general, I recommend moving these class methods out to a separate module, and leaving the class definition free to express the domain intent.

Thanks to John Hume and Jay Fields for their input and thoughts on structural duplication and class methods. For more examples of of this class method usage, see this post by Farooq Ali. As always, thanks for reading.

The term annotation here refers to a Ruby technique that belongs to the same family as Java annotations, C# attributes, and C compiler directives. These language features are of course not identical, and the class method derived Ruby version is its own beast altogether. Nevertheless, there are family resemblances. Ruby class methods can be used to issue what might be called interpreter directives. They can also be used to allow for an Aspect Oriented Programming (AOP) like abstraction of cross cutting concerns.

Take for an example a valuable but long running test that should only be run as part of an acceptance build. In the (hopefully) rare case when the tests fails in the build environment, the developers responsible for the failed check-in can run the test locally to resolve the issue, but as a general rule developers don’t want to take the performance hit as part of their local test suite execution.

A simple interpreter directive might be implemented as…

On some systems, the incidence of this sort of test might climb to include a handful of “acceptance build only” tests. At this point, it makes good sense to implement a custom interpreter directive. There are three implementation options here…

Annotating Keyword…

Annotated Block…

Annotated Method Declaration…

Let’s take a look at the implementation of an Annotating Keyword…

CalculatorTest extends the TestDirective Module, so methods defined in TestDirective will be included as class methods in CalculatorTest. When the interpreter encounters the acceptance_only keyword, it’s recognized as a call to the acceptance_only class method. This method simply sets a class level instance variable indicating wether or not the current build is an acceptance build. The real work is done by method_added, a hook method provided by the Module class. When a class (or module) is defined in Ruby, the actual class or module is created as soon as the interpreter encounters the opening declaration (e.g. “class Foo”). As the interpreter encounters method definitions, it adds methods to the method table of the class. After a new method is added, the method_added class method is called with the method name passed as a parameter. In the Annotating Keyword implementation, method_added checks if the new method is valid based on state set by calls to acceptance_only. If the method is not valid in the current context, method_added removes the method from the class definition. As a final step, state is reset back to “normal”. Note that we could choose to not reset state, making method_added work very much like method accessibility keywords (public/protected/private).

The implementation of an Annotated Block…

The greatest strength of an Annotated Block is its exceedingly straight forward implementation. The method declaration itself is a parameter to the acceptance_only class method. acceptance_only then chooses wether of not to evaluate the method definition based on on a check of the environment variable.

Lastly, here is the implementation of Annotated Method Declaration…

Conceptually, Annotated Method Declaration is very similar to Annotated Block. The difference is that instead of passing a method declaration to the annotating class method, the method body and method name are passed, and the annotating method takes care of the plumbing of actually defining the method.

The CalculatorTest example demonstrates an interpreter directive style use of annotations. These same techniques are useful for more AOPish situations. Take the canonical AOP example of logging. Logging is a cross cutting concern. That is to say, logging code can be found throughout an application, tangled up with the actual work of a method. Take an example of a requirement that certain method calls must logged…

An Annotating Keyword implementation would allow for a better separation of concerns …

The Loggable implementation demonstrates the primary weaknesses of Annotating Keyword, the complexity of it’s implementation…

Briefly, method_added aliases the method to be logged and redefines the accessor to return a method which logs relevant information before calling the original method.

So which style is appropriate where? Annotating Keyword is certainly the most aesthetically pleasant to use, but it suffer from what Anders Hejlsberg calls simplexity. The simple usage syntax hides a complex implementation. Truly simple solutions are simple from top to bottom. This complexity increases when multiple annotations come into play. I’m inclined to use it for interpreter directive style annotations, or where I need to chain AOP style annotations.

Annotated Block is the middle-ground option. It avoids the use of the method_added hook, leaving it free for other uses (some of which may not be fully within the developers control). Like Annotating Keyword, it’s chain-able, albeit only through some unfortunately indented code. Simplicity is a double edged sword here. Because an Annotating Block know so little about the methods it wraps, it is limited in how it can interact with those methods. In particular, it does not have access to the arguments of the methods it wraps. The only way it can supplement its weak position is to maintain some sort of state that interacts with method_added, adding the complexity of Annotating Keyword. I prefer it for simple interpreter directive usages limited to including or excluding the definition of methods or execution of a code block.

Annotated Method Declaration is the only option that allows for code execution before the annotated method is defined. This is a powerful property when coupled with its access to the parameters passed to the called method. The major drawback of Annotated Method Declaration is that it cannot be chained. I prefer to use it for AOP style annotations when an inability to add multiple annotations is acceptable.

Martin Fowler has an entry on RubyAnnotations on his bliki. Be sure to read Jay Field’s recent write up on class method declaration syntax. Thanks to Paul Gross for pointing out that a post on annotations should be part of this series.

Thanks for reading. As always your comments are appreciated.

This post is about Ruby class methods and not Ruby static methods. There is a good reason for this. There are no static methods in Ruby.

In Java and C# “static method” and “class method” are synonyms, with “static” indicating that the methods are statically bound. What does it mean for a method to be statically bound? Consider a normal (non-static) instance method in Java. Client code that consumes the method doesn’t know at compile time what method body will be associated with that usage. It may be the instance method defined on the class, but it may just as well be an overridden version of the method defined in a subclass. The method implementation must be “dynamically bound” to the invocation at runtime. The compiler doesn’t have this issue with class methods, as in Java and C# class methods cannot be overridden. Classes methods in Java and C# are not invoked on class instance, but are rather scoped to the class name. As class method references can only resolve to one location, the compiler can “bind” the reference once at compile time “statically”. Classes methods in Ruby are normal instance methods stored on the class’s Metaclass. Even if Ruby were a compiled language, there would be no way to know at compile time which implementation of a method would eventually be responding to a message passed to a class object at runtime.

In any OO language, the most common use of class methods is for object creation. Every Ruby programmer is familiar with the primary object creation class method, new. Most programmers have also had cause to define alternative class methods for instantiating an object. In “Effective Java”, Joshua Bloch enumerates the advantages of this sort of factory method. These advantages are wholly applicable to Ruby.

Class methods have names

Java requires all method signatures to be unique. Suppose a Java programmer wants to allow the instantiation of an object in two different ways. Both instantiation methods rely on a boolean and an int parameter. One way to solve this is to define two constructors…

This is valid Java, but it’s just a wee bit hackish to be overloading methods based on the order of their parameters. Consumers of the class must infer the difference between the two constructors from the parameter names or a reading of the source. In Ruby this problem is more acute, firstly because of duck typing, and secondly because Ruby has no facilities for method overloading. For example, consider a Ruby class defined as…

The second definition of initialize completely replaces the first definition. Hacked up solutions to this type of problem aren’t relegated to the Java world. In fact, duck typing allows for some horribly creative solutions in Ruby. WARNING! Don’t do this…

or this…

or this…

Instead strongly prefer…

For an in the wild example of a factory class method in Ruby, look no further than ActiveRecord:Base.create

Class methods don’t have to return a new object

Sending the new message to a class will always cause the interpreter to create a new instance of the class. This occurs in rb_class_new_instance, the function bound to the new method of Object.

rb_class_new_instance calls rb_obj_alloc to allocate a new object which is handed off to the initialize method by rb_obj_call_init. As an interesting side note, notice that you can avoid having the initialize methods called by instantiating objects by calling SomeClass.allocate which returns the return value of rb_obj_alloc.

Sometimes a new object is not required, but rather a reference to an instance from a pool of existing objects. Database connections are one common example. Class methods are an excellent solution to the problem of how to provide access to the instance pool. When the available pool of instances is limited to one object, this solution is an implementation of the Singleton Pattern. “Singleton” here refers to the OO pattern documented in Design Patterns by Gang-Of-Four, not the Ruby specific Singleton class.

The Ruby Standard library includes a Singleton module to take the work out of implementing this pattern. The source includes extensive documentation on the implementation, so I won’t cover it here. Suffice to say that the module adds a class method, instance to the including class.

Why have this module at all? The Ruby language already provides a facility for creating objects with only one instance, classes. Why not define a singleton as a class…

The interpreter guarantees that there will only ever be one instance of Foo’s Metaclass, the Foo class itself. The existence of metaclasses means this question is applicable to Smalltalk as well. In “The Design Patterns Smalltalk Companion” by Sherman R. Alpert, Kyle Brown, and Bobby Wolf, two cases are made against implementing the Singleton Pattern with class methods. The first is conceptual. Classes should not be domain objects, their role is to allow for accessing and instantiating domain objects. Defining domain logic directly in classes is a perversion of this intent. The second argument is pragmatic. If a requirement is introduced that requires the usage of multiple instances of the formally singleton object, the refactoring away from class method calls is more laborious then a traditional Singleton pattern implementation. Although ‘The Design Patterns Smalltalk Companion” is a valuable read for Ruby programmers, in this case I don’t find the arguments particularly compelling . Class methods probably are the simplest thing that could possible work when Singleton Pattern like functionality is needed.

Class methods can return subclasses

A classic usage of factory methods is to return a subclass of the class receiving the message based on an input parameter. This is good encapsulation, as the consumer of the class does not need to know the details of which subclasses exist and which is appropriate for it’s needs.
If one is fortunate enough to have useful conventions in place, a bit a metaprogramming can make this usage particularly powerful. For example instead of …

one could implement by_nationality as…

Not only is this implementation more succinct, it extensible to handle any Pirate nationality that follows the convention. Keep in mind that classes aren’t the only place to store behavior. Class methods might return a new instance of the class with modules mixed in…

Perhaps even more powerful, a class method might alter the Singleton class of an instance before retuning the object. This is essentially subclassing on the fly. Suppose Pirate’s have fighting behavior. A class method could be created to return a pirate with a customized fighting style…

Even further out on the metaprogramming ledge, the consumer could define the fighting style directly…

This last example, while interesting, doesn’t strike me as good style. The consumer knowing both that it needs a Pirate to fight, and the implementation details of that fighting style seems like an awful lot for the consumer to know.

That covers using class methods for object creation. I’ll be covering class methods usage for annotations, DSL’s, and removal of structural duplication in the next post(s). As always thanks for reading. I’m always happy to get your feedback, inquiries, corrections, and criticism.

Thanks to John Hume for contributing his insight and expertise to this post.

JRuby is a Java implementation of the Ruby interpreter. That is to say, that one can write plain ol’ Ruby files and hand them off to the JRuby interpreter for execution in the same way that they can be executed via Matz’s Ruby interpreter (MRI). From a Ruby object model perspective, JRuby isn’t doing anything all that different from MRI. Specifically, a Ruby class Pirate, when interpreted by JRuby ,is NOT transformed into a Java Pirate class. Rather, Pirate is represented by an instance of RubyClass in much the same way that MRI uses structs.

JRuby itself is very important, and gaining a familiarity with it’s implementation is a good idea, independent of MRI. That said, the parallels between the two interpreters are a boon for any developer in acquiring an advanced understanding of Ruby internals. Inspection of JRuby source can give insight and perspective into how MRI works. This is particularly useful to developers more comfortable in Java than C.

In JRuby, Ruby objects are represented by instances of the RubyObject class…

The data members on RubyObject should look very familiar, they map one to one to RObject and RBasic in MRI. There are a few interesting differences.

The Ruby class of an object, stored as klass in MRI, is stored in a data member named metaClass. This can be very confusing. The value stored in metaClass is not necessarily a Metaclass or even a Singleton class in the Ruby sense. It is simply the class of the object. If the object is an instance of Class, than metaClass does indeed store a Metaclass, if it is a “normal” object, it stores either a “normal” Ruby class or a Singleton class, depending on whether or not instance specific behavior has been defined.

flags also work a bit differently in JRuby. They are used to store interesting state for an object (is it frozen? tainted?) as in MRI, but certain information that would be stored in flags in MRI is simply defined as methods in JRuby. JRuby has the advantage of being able to subclass RubyObject, making it possible to represent this information differently by overriding the methods in subclasses. For example…

Like MRI, JRuby hides the actual class of an object by returning a “real class”. The differences are just OO implementation details. Rather than inspect the class itself, RubyObject asks it’s class for it’s “real class”…

If the the object’s metaClass is storing a Singleton class, represented by the JRuby MetaClass class, the class delegates to it’s super…

Eventually the interpreter hits upon a “real class” represented by an instance of RubyClass, which returns itself…

In both flavors of Ruby it’s convenient to say, “the Singleton class is created when instance specific behavior is added to an object”. A more accurate statement is, “a Singleton class is created the first time an attempt is made to access the Singleton class of an object”. In practice the two are often equivalent. Once again JRuby’s behavior parallels MRI…

Here, the interpreter is checking the Java object stored in metaClass to see if it is a Singleton class associated with the object. If so it returns the object stored there, otherwise it generates a new Singleton class by calling makeMetaClass. As in MRI, getSingletonClass passes it’s current class in for use as the superclass of the new Singleton class. The second parameter, is used for lexical scoping and does not play directly into the object model. For clarity I’ve omitted some of the makeMetaClass source …

makeMetaClass parallels MRI exactly. The JRuby interpreter creates a new class and assigns it to the metaClass replacing the current class of the object.

In JRuby, classes are created in the static newClass method. As all classes inherit directly from another class or the Object class, the interpreter calls newSubClass on the “superclass to be” to actually generate the class…

Importantly, makeMetaClass is called immediately after the new class’s instantiation to create the Metaclass of the new class. As this method is being called from the new class’s super class, the first parameter passed to makeMetaClass, getMetaClass refers to the class of the new class’s super class. In other words, the “The superclass of the Metaclass is the Metaclass of the superclass.”

I encourage you to crack open the JRuby source yourself and take a look around. This post is accurate as of JRuby 1.1. The JRuby folks have lots of big and exciting changes coming down the pipeline with regards to the internal representation of classes. Thanks to Ola Bini for being good enough to answer some of my JRuby questions and to the entire JRuby team for gifting us with their work.

Before getting into Metaclass details it might be useful to clear up a bit of terminology. Unfortunately, terminology around Ruby’s object model is not used consistently in existing literature, MRI source, JRuby source, or throughout the community. A prime example of this is the rb_make_metaclass function in MRI, which is used to make both Metaclasses and Singleton classes. That said, the following glossary is the one I stick with, and recommend to others.

• Singleton Class: Hidden Ruby class for storing instance specific behavior
• Metaclass: The Singleton class of a Class object. Differs in some ways from a “normal” Singleton class. Occasionally inappropriately used as a synonym for Singleton class.
• Eigenclass: Alternative name for a Singleton Class. Eigenclass (roughly “own class” in German) has the advantage of not having a well known OO design pattern named after it. It’s preferred by some of the Ruby elite, but has never really caught on with the community at large.
• Virtual Class: Alternative name for a Singleton Class. Best avoided. In the current MRI implementation, there is nothing particularly virtual about these classes and virtual is already a very loaded term in the OO community.

Take a look a page 382 of PickAxe, Second edition for some additional background on Metaclass terminology.

With diction out of the way, focus can return to the question, “What is it that Ruby wants that forces Metaclasses to exist?” This can be answered in a number of ways, but one simple answer is “Ruby wants class methods.”

Take this simple factory method…

Understanding what happens when the create_by_country message is sent to Pirate is not difficult if one bares in mind Ruby’s “everything is an object” ethos. If Ruby is going to treat the Pirate class as an object, then the class should have a klass pointer and flags collection.

Looking at the definition of the RClass struct confirms that the Pirate class is doing it’s part…

RClass references an RBasic struct which includes a klass pointer and flags collection. Note that with the inclusion of an iv_tbl, the data members of RClass are a superset of RObject’s members. Put another way, anything represented by a RClass struct fulfills Ruby requirements for “objectness”. Once it is understood that the Pirate class is itself an object, understanding how the create_by_country message is handled becomes a matter of understanding method dispatch. When the Pirate class receives the create_by_country message, the interpreter follows Pirate’s klass pointer and inspects the m_tbl it finds there for a method named create_by_country.

Pirate is an instance of Class, and so at first glance one would think that the class returned by Pirate’s klass pointer should be the Class class. The problem is Ruby can’t store create_by_country in Class’ m_tbl, or all instance of Class, that is to say every class in the Ruby process, would have access to the method. This problem should sound familiar. create_by_country is actually instance specific behavior for the Pirate instance of Class. Since instance specific behavior is stored in Singleton classes, Ruby does what it knows, and creates a Singleton class for Pirate to store the create_by_country method body. This special Singleton class, found by following the klass pointer of a class instance is called a Metaclass.

If that were all there is to Metaclasses, The Pirate class object model would look something like this…

It turns out that Ruby needs a bit more from classes. Since classes are objects, Ruby wants them to play nicely in an Object Oriented playground. All the other children in that playground know a game called polymorphism, and so classes must too. Assume that the Pirate class inherits from the Human class. Polymorphism (and Liskov’s Substitution Principle) say that any instance of Human can be replaced with an instance of Pirate. That is to say a Pirate is a Human, and so we can expect it to do all the things a human does; eat, sleep, etc…

Ruby wants this principle to be extended to classes. So if Pirate extends Human, then the Pirate class is a Human class, and any class methods exposed by Human should be available as class methods on Pirate. “Normal” Singleton class creation doesn’t support Ruby’s desire on this. Remember that for a non-class Ruby object, the Singleton class replaces the “real” class, which stays in the dispatch loop via the super pointer on the Singleton class. If Metaclasses worked the same way, then the Human/Pirate object model would look something like this…

A message sent to Pirate would never be dispatched to ‘Human’s m_tbl. If Ruby wants classes to be polymorphic, the interpreter needs a way to move from the Metaclass of Pirate to the Metaclass of Human when dispatching class methods. The sanest way the interpreter can do this, is to use ‘Pirate’s super pointer, and so that is what it does.

The creation of “normal” Singleton classes and the creation of Metaclasses is handled by two different lines of code. A quick look at this code makes this all a bit more concrete…

The second parameter to rb_make_metaclass will be used as the super pointer for the Singleton class that will be created. Note that when making a “normal” Singleton class, this parameter is the current class of the object under manipulation. When the object is itself a class however, as in rb_class_initialize, the second parameter is the class found by following the klass pointer of the object’s super pointer. This is a bit convoluted, in fact it leads to one of the most confusing sentences in Ruby, “The superclass of the Metaclass is the Metaclass of the superclass”. A corrected diagram clears this up…

With the object model arranged this way, the interpreter is free to use standard method dispatch to resolve messages sent to a class object. Note an additional bit of weirdness in this diagram. An instances of Human is an instance of Object. This means that the Human class has an is a relationship with the Object class and should have access to any class methods that might be stored on Object. Standard Metaclass rules apply, the super pointer of the Metaclass of Human must point to the Metaclass of Human’s superclass, Object.

Creating this parallel inheritance hierarchy as new classes are defined is one thing, but it would be painful in the extreme to have to re-situate the super pointers of all subclasses the first time a class defined a class method. Ruby avoids this pain in an interesting way. Notice that the code example above, to create the Metaclass, was extracted from a function named rb_class_initialize. As this implies, Metaclasses are created at the time of a classes initialization. This is markedly different form “normal” Singleton classes which are instantiated “on demand” at the time of first access.

As always, I hope you find this useful. If you are looking for more in depth coverage of the Metaclass be sure to check out the translated fourth chapter of The Ruby Hackers Guide. Comments, questions, and corrections are always appreciated.

In my last post I explained how the code above forces the interpreter to assign a newly created Singleton class to the pirate object. If that’s true, how is it possible that the class of a new Object is equal to the class of pirate? It’s not possible, the Ruby interpreter is more or less lying.

rb_obj_class in object.c is the the underlying C function that is accessed when the class message is sent to an object. It’s definition…

rb_obj_class calls the function rb_class_real on the return value of CLASS_OF the recipient of the class message. CLASS_OF is a macro which expands into a call to rb_class_of. I’ll skip the details of rb_class_of, in most cases it simply returns the RBASIC klass pointer, i.e. the struct representing the class of the object.

In the case of the pirate object, the ‘pirate Singleton class is passed to rb_class_real. From there on out, Ruby’s deception is straight forward…

rb_class_real uses a couple of macros to inspect the RBASIC flags of the class, checking if it is a Singleton class or Include class (T_ICLASS). Include classes are special proxy classes that are created when a module is included. I’ll cover them in greater detail in a later post. The relevant thing here, is that Include classes are similar to Singleton classes in that the interpreter considers them an implementation detail which should not be exposed to language consumers (this means you). If the class under inspection is not a Singleton or Include class, it is returned, otherwise the super pointer is followed and the checks are run again. In this way, the interpreter loops through the class hierarchy of the object looking for a “normal” class. For the pirate object, the super pointer for ‘pirate points to Object, and so Object is returned from rb_real_class.

Knowing what the interpreter does, doesn’t answer the question of why it does it. Wouldn’t it be simpler (and more honest) to return the Singleton ‘pirate class when the class message is sent to pirate? There are a couple reasons why Ruby works this way.

Firstly, Matz views Singleton and Include classes as implementation details. Yes, Ruby let’s you add instance specific behavior and mixin modules, but how it does it is not really your business. It’s not an accident that the language does not have a built-in singleton_class method. While this is a reasonable view to have, it’s worth bearing in mind that the existence of class << self as a language construct, more or less releases the Singleton class from Pandora’s box of encapsulation.

There is another more subtle reason for the class method to work this way. As far as I know this was first identified by Kent Beck. Kent wrote a two part article in 1993 for “Smalltalk Report”, “Instance-Specific Behavior: How and Why” and Instance-Specific Behavior: Digitalk Implementation and the Deeper Meaning of it All”. In these articles Kent detailed a pattern for adding instance specific behavior in Smalltalk which is nearly identical to what would later be known as the Singleton class in Ruby. Kent’s pattern notabley does not include anything like rb_real_class.

Two years later, in 1995, Kent published an article for “Smalltalk Report” regrettably entitled, “What, What happened to Garbage Collection?”, in which he pointed out a problem with the Instance-Specific Behavior pattern. Assume that equality for a Point class is defined as below…

Without the existence of something like rb_real_class, two points which share an x and y coordinate will not be equal if one of them has defined a Singleton class.

This is an interesting problem. From a purist’s view, it’s fair to say that they are no longer in fact equal as they are no longer interchangeable. One of the Points is now capable of differing or additional behavior. From a pragmatist’s view however, having equality semantics change because of added instance specific behavior could create nightmarish situations as various frameworks and libraries relying on the original semantic begin to breakdown in subtle and difficult to deduce ways.

Kent’s pragmatic solution is to introduce a realClass method. Singleton classes (actually an instance of Behavior in Smalltalk), return the realClass of their super. Classes return themselves. Object>>class is redefined to send a realClass message to it’s associated class.

This is exactly the same behavior as rb_real_class, albeit implemented in an OO fashion. In fact looking at JRuby, an OO implementation of the Ruby interpreter, we find exactly the same pattern. In JRuby, Singleton classes are represented by instances of MetaClass…

Thanks to my colleague and friend John Hume for his input on this post.

A good way to look at singleton classes is to ask the question, “What is it that Ruby wants which forces the existence of a singleton class?” The answer is, “Ruby wants instance specific behavior”.

An example…

Here the speak method has been added to an instance of Object called pirate. The pirate object will respond to the speak method, but other objects know nothing of it. So how does Ruby make this happen? One option would be to store the behavior right along with the object (a la prototyped languages like JavaScript), but a quick look at the C struct that represents an object show that this is not the option Matz took.

An object has only two data members, a hash to store instance variables called *iv_tbl and a reference to another struct of type RBasic called basic. RBasic is the fundamental struct in Ruby. It holds the data all Ruby “things” need for the interpreter to work with them. It’s definition is…

flags are used to store metadata about an object. An example would be a bit to indicate an object is frozen or tainted. The only other data member is the class associated with the object, stored in klass.

Note that there is nowhere in RBasic or RObject to tuck away a method body. That leaves the class associated with the klass pointer as the only place left to store the instance specific behavior. This makes a lot of sense. The entire purpose of classes in Ruby is to serve as repositories for behavior. Here Ruby finds itself in a tough spot. If the interpreter follows the klass pointer of the pirate object, it will find the Object class. Any method’s added there would be accessible to any instance of Object.

Faced with a dilemma, Ruby does something sneaky. When an attempt is made to define instance specific behavior, Ruby generates an instance of a brand new class and changes the object’s class pointer to point to this new class. It then adds the homeless instance specific behavior to the new class. Now, when the message for instance specific behavior is sent to the object, standard method dispatch kicks in, and the method body is found. This special, created on the fly, instance specific class, is called the Singleton class.

There is one more trick involved. It wouldn’t do for the object to lose all the behavior it picked up from it’s class of instantiation, so Ruby sets the super pointer of the Singleton class to point to the original class of the object. Once again standard method dispatch works just fine when searching for methods.

Below are a couple of diagrams showing the pirate object’s underlying structure before and after the speak method is defined. The prefixed apostrophe on ‘pirate can be read as “The Singleton class of pirate.”

Before Definition of Instance Specific Behavior

After Definition of Instance Specific Behavior

All of this class inheritance chain manipulation occurs in the rb_make_metaclass() function. I’ve omitted some code for clarity.

If you find all this interesting and useful, I strongly encourage you to take a look at the Ruby Hackers Guide. It’s a guide to Ruby internals written by Minero Aoki. Unfortunately, translation is an on going project that has possibly stalled out. Still, the chapters which have been translated are a wealth of Ruby internals information.