Writing custom native Ruby extension in C from scratch

You are probably reading this because you have an idea for a useful library for Ruby and you did not find it already implemented on RubyGems.org.

I was in the same boat, implemented it and here is how:

1. Quick initialization of a Ruby gem
2. Creating a Ruby module and a Ruby class in C
3. Creating instance methods in C for the Ruby class
4. Publishing the gem to RubyGems.org
5. Adding more C files to the compilation

But first, let’s set our goal here

Ok, you have your goal set, and we have to do the same here. We will build a simple extension with one module, one class and two instance methods. The constructor will accept an array as an argument and store it into an instance variable. The instance method #mean will calculate the mean value and return it.

So the API in Ruby would look like this:

module MyExtension
  class Model
    # Initializes model with given data
    #
    # @param values [Array<Numeric>] an array of numbers
    def initialize(values)
      # implemented in C
    end

    # Instance method calculating mean model's data
    #
    # @return [Float] mean calculated from model's data
    def mean
      # implemented in C
    end
  end
end

1. Quick initialization of a Ruby gem

Fortunately, there is a really easy way to generate almost everything we need with bundle gem <name-of-your-gem> --coc --ext --mit --test.

See bundler documentation

Make sure the name you picked for your extension is not already taken on RubyGems.org if you were going to publish it as an open source library.

Let’s open a terminal and run

bundle gem my_extension --coc --ext --mit --test

It creates a folder called my_extension/ and fills it with the necessary files.

$ bundle gem my_extension --coc --ext --mit --test
Creating gem 'my_extension'...
MIT License enabled in config
Code of conduct enabled in config
      create  my_extension/Gemfile
      create  my_extension/lib/my_extension.rb
      create  my_extension/lib/my_extension/version.rb
      create  my_extension/my_extension.gemspec
      create  my_extension/Rakefile
      create  my_extension/README.md
      create  my_extension/bin/console
      create  my_extension/bin/setup
      create  my_extension/.gitignore
      create  my_extension/.travis.yml
      create  my_extension/.rspec
      create  my_extension/spec/spec_helper.rb
      create  my_extension/spec/my_extension_spec.rb
      create  my_extension/LICENSE.txt
      create  my_extension/CODE_OF_CONDUCT.md
      create  my_extension/ext/my_extension/extconf.rb
      create  my_extension/ext/my_extension/my_extension.h
      create  my_extension/ext/my_extension/my_extension.c
Initializing git repo in /Users/admin/domcermak.cz/my_extension
Gem 'my_extension' was successfully created. For more information on making a RubyGem visit https://bundler.io/guides/creating_gem.html

Then we enter the gem’s directory and run bin/setup. It’s going to fail, because there are some todos in my_extension.gemspec.

$ bin/setup 

bundle install
+ bundle install
You have one or more invalid gemspecs that need to be fixed.
The gemspec at /Users/admin/domcermak.cz/my_extension/my_extension.gemspec is not
valid. Please fix this gemspec.
The validation error was '"FIXME" or "TODO" is not a description'

We can fill them, or comment them out e.g.

# ...
Gem::Specification.new do |spec|
  spec.name          = "my_extension"
  spec.version       = MyExtension::VERSION
  spec.authors       = ["Dominik Čermák"]
  spec.email         = ["dominik.cermak@enerfis.cz"]

  spec.summary       = "This is example Ruby gem."
  # spec.description   = %q{TODO: Write a longer description or delete this line.}
  # spec.homepage      = "TODO: Put your gem's website or public repo URL here."
  spec.license       = "MIT"
# ...

The next run of bin/setup succeeds and installs dependencies.

2. Creating a Ruby module and a Ruby class in C

All our C files are located in ext/my_extension/. Currently, there are just my_extension.c and my_extension.h. Open up ext/my_extension/my_extension.c and you can see the following code.

#include "my_extension.h"

VALUE rb_mMyExtension;

void
Init_my_extension(void)
{
  rb_mMyExtension = rb_define_module("MyExtension");
}

with already defined module MyExtension. As you may notice, the same module is already defined in lib/my_extension.rb.

According to our goal, we should define a class Model within MyExtension. We can do that by writing

#include "my_extension.h"

void Init_my_extension(void) {
  // create module
  VALUE rb_mMyExtension = rb_define_module("MyExtension");
  
  // create class under the module
  VALUE rb_cModel = rb_define_class_under( rb_mMyExtension, "Model", rb_cObject );
}

There is no need to create the model globally, so let’s keep it in the init function.

To install the gem onto your local machine, run bundle exec rake install. It compiles the C code into a tmp/ folder and bundles the gem from the compiled binaries into a pkg/. All files contained in these folders are listed in the .gitignore file.

Everything should go just fine.

3. Creating instance methods in C for the Ruby class

3.1. TDD first

We will implement a constructor and a mean instance method, but first, let’s create a test in spec/my_extension_spec.rb

The file already contains

RSpec.describe MyExtension do
  it "has a version number" do
    expect(MyExtension::VERSION).not_to be nil
  end

  it "does something useful" do
    expect(false).to eq(true)
  end
end

so we change the second test to

it "calculates mean from values" do
  values = [1, 1, 1, 3, 3, 3]
  model = MyExtension::Model.new values

  expect(model.mean).to eq(2)
end

After running bundle exec rake spec we can see, that the Model class is defined, but we are trying to use an implicit constructor.

MyExtension
  has a version number
  calculates mean from values (FAILED - 1)

Failures:

  1) MyExtension calculates mean from values
     Failure/Error: model = MyExtension::Model.new values
     
     ArgumentError:
       wrong number of arguments (given 1, expected 0)
     # ./spec/my_extension_spec.rb:8:in `initialize'
     # ./spec/my_extension_spec.rb:8:in `new'
     # ./spec/my_extension_spec.rb:8:in `block (2 levels) in <top (required)>'

So let’s define new one!

3.2. Implementation of an initialize method

Our constructor must be defined on the Model, must accept a Ruby Array of numbers and store it to an instance variable.

So, open ext/my_extension/my_extension.c and define a function representing constructor, which accepts one argument and stores it into a variable.

// function is defined as static, because we don't need to see it from outside this file.
static VALUE t_init(VALUE self, VALUE values) {
    rb_iv_set( self, "@values", values );
    
    return self;
}

then use it as our Ruby constructor

void Init_my_extension(void) {
  VALUE rb_mMyExtension = rb_define_module("MyExtension");
  VALUE rb_cModel = rb_define_class_under( rb_mMyExtension, "Model", rb_cObject );

  // 1. method is defined on `Model`
  // 2. method is called "initialize"
  // 3. when "initialize" is called in Ruby, `t_init` is called in C
  // 4. method accepts one argument (it's instance method, so `self` does not count)
  rb_define_method( rb_cModel, "initialize", t_init, 1 );
}

Hit bundle exec rake install to compile our code, bundle exec rake spec to run tests, and we can see the tests still fail, but on the missing mean method now.

MyExtension
  has a version number
  calculates mean from values (FAILED - 1)

Failures:

  1) MyExtension calculates mean from values
     Failure/Error: expect(model.mean).to eq(2)
     
     NoMethodError:
       undefined method `mean' for #<MyExtension::Model:0x00007fe7b5a2f108>
     # ./spec/my_extension_spec.rb:10:in `block (2 levels) in <top (required)>'

3.3. Implementation of a mean method

This is little tricky, because we need to convert Ruby Array to C double *, calculate the mean of values and then convert a C double to the Ruby Float.

// convert Ruby `Array<Numeric>` to C `double *`
static double * t_parse_dbl_ary( VALUE ary, long * size ) {
    // checks that `ary` argument is really Ruby Array
    // otherwise raises exception 
    Check_Type(ary, T_ARRAY);
    long len = RARRAY_LEN( ary );
    VALUE * values = RARRAY_PTR( ary );
    double * d_data = ALLOC_N( double, len );

    for ( int i = 0 ; i < len ; ++ i ) {
        // converts Ruby `Numeric` to C `double`
        // raises `TypeError` if not convertable
    	d_data[i] = NUM2DBL( values[i] );
    }

    *size = len;

    return d_data;
}

static VALUE t_mean(VALUE self) {
    long size;
    VALUE values = rb_iv_get( self, "@values" );
    double * dbl_values = t_parse_dbl_ary(values, &size);
    double sum = .0;

    for (long i = 0; i < size; ++ i ) {
        sum += dbl_values[i];
    }
    double mean = sum / size;

    // convert C `double` to Ruby `Float`
    return rb_float_new(mean);
}

then use the t_mean function in

void Init_my_extension(void) {
  VALUE rb_mMyExtension = rb_define_module("MyExtension");
  VALUE rb_cModel = rb_define_class_under( rb_mMyExtension, "Model", rb_cObject );

  rb_define_method( rb_cModel, "initialize", t_init, 1 );
  
  // 1. method is defined on `Model`
  // 2. method is called "mean"
  // 3. when "mean" is called in Ruby, `t_mean` is called in C
  // 4. Ruby method "mean" accepts no arguments
  rb_define_method( rb_cModel, "mean", t_mean, 0 );
}

The implementation should be done now! So, compile and run the tests using bundle exec rake install && bundle exec rake spec

MyExtension
  has a version number
  calculates mean from values

Finished in 0.00289 seconds (files took 0.12574 seconds to load)
2 examples, 0 failures

Tests passed, we are good to go for release to RubyGems.org.

4. Publishing a gem to RubyGems.org

Make sure to create account on RubyGems.org.

1. Download gem credentials to push the gem to RubyGems.org

curl -u <your_gem_account_name> https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials

2. Release the gem!

bundle exec rake release

5. Adding more C files to compilation

For those who want to create larger gem in C, there might be a problem with multiple C files compilation.

Let’s say we want to change

static VALUE t_mean(VALUE self) {
    long size;
    VALUE values = rb_iv_get( self, "@values" );
    double * dbl_values = t_parse_dbl_ary(values, &size);
    double sum = .0;

    for (long i = 0; i < size; ++ i ) {
        sum += dbl_values[i];
    }
    double mean = sum / size;
    xfree(dbl_values);

    return rb_float_new(mean);
}

to

static VALUE t_mean(VALUE self) {
    long size;
    VALUE values = rb_iv_get( self, "@values" );
    double * dbl_values = t_parse_dbl_ary(values, &size);
    double mean = sum(dbl_values, size) / size;
    xfree(dbl_values);

    return rb_float_new(mean);
}

and extract sum function to another file.

So we create ext/my_extension/sum.h

#ifndef SUM_H
#define SUM_H 1

double sum( double *, long );

#endif /* SUM_H */

and ext/my_extension/sum.c

#include "sum.h"

double sum( double * ary, long size ) {
    double s = .0;
    
    for ( long i = 0; i < size; ++ i ) {
        s += ary[i];
    }
    
    return s;
}

To use the sum function in the t_mean, we have to include sum.h into ext/my_extension/my_extension.h

#ifndef MY_EXTENSION_H
#define MY_EXTENSION_H 1

#include "ruby.h"
#include "sum.h"

#endif /* MY_EXTENSION_H */

It should be ok now, but running bundle exec rake install fails and suggests us to run gem install /Users/admin/domcermak.cz/my_extension/pkg/my_extension-0.1.0.gem, which results in

In file included from my_extension.c:1:
./my_extension.h:5:10: fatal error: 'sum.h' file not found

This is caused by wrong compilation scope defined in my_extension.gemspec. All we need to do is add

spec.files = Dir["ext/**/*.{rb,c,h}", "lib/**/*.rb"]

Running bundle exec rake install then passes correctly.

Documentation

Ruby API for C extensions is referenced here. Don’t be afraid to use it.

Common problems & their solutions

• dyld: lazy symbol binding failed: Symbol not found: <symbol>
  • Delete tmp/ and run bundle exec rake install again. This error is from linker caused by missing object file.
• fatal error: '<header-file-name>.h' file not found
  • Extend scope for compilation in <gem-name>.gemspec. More described in the section 5. Adding more C files to compilation.
• The C implementation does not seem to be called.
  • Make sure you ran bundle exec rake install after the change in C code.
  • Make sure the same methods and constants are not defined also in Ruby because then the Ruby code takes precedence over the C code. So if you created a method on class A in C and you need to comment the method in Ruby, then don’t do this

    class A
      # Comment to the method
      def method(a) # this overrides the method of the same name implemented in C
      end
    end
    

    but do this

    class A
      # @!parse [ruby]
      #
      #   # Comment to the method
      #   def method(a)
      #   end
    end
    

References

• RubyGuides: Write Ruby C extension
• RubyDoc: Extending ruby
• TenderLoveMaking: Writing Ruby C extensions part 1
• Chris Lalancette: Writing Ruby extensions in C part 5