What attributes RubyGems’ Marshal file really contain?

RubyGems started with a single marshaled file called Marshal.4.8.gz containing the array of every Gem::Specification object for every gem that has been uploaded to RubyGems directory. Since then we have new indexes for RubyGems to speed things up, but good old Marshal.4.8.gz is still around carrying important information about gems from RubyGems.org. Are you interested to know what is and what is not there and finally understand why gem specification rails -r does not give you the information on licensing even though it’s part of rails’ gemspec file?

When I started building gem-compare for comparing .gem files gem-compare would always download the .gem file and compare againts these downloaded .gem files, but since a lot can be accomplished by just downloading specifications, I wanted gem-compare to download the full .gem files only if it compares something that is not included in the available gemspec. So to say gem-compare needed to be aware of Gem::Specification attributes that are available while quering RubyGems.

Here is a little Ruby script that tries to collect all Gem::Specification attributes from Marshal.4.8.gz that has at least one valid value which means they are present:

And the results it gives:

That means quering againts the not included specification attributes won’t give you the right results:

Of course you may notice that Gem::SpecFetcher which would be used in this example will now actually ask on /specs.4.8.gz index and then download separate gemspec files from /quick/Marshal.4.8/GEM-VERSION.gemspec.rz, but if you look carefully you find out that these gemspec files obviously contain the same info as the old big Marshal.4.8 which is easier to go through at once.

That is why gem compare rails 3.0.0 4.0.0 --runtime won’t download full .gem files, but when you run gem compare rails 3.0.0 4.0.0 -p 'license' gem-compare needs to download .gem files to give you the accurate information.

This hepled me to understand why quering RubyGems didn’t show me the information I would expect and I hope that cleared some things for you too.

Leave a comment

Your email address will not be published. Required fields are marked *