Mad Marmot: Tag rails

Use pagination to save memory when iterating over a lot of ActiveRecord objects.

Luke Ludwig — Sat, 16 Feb 2008 22:57:00 -0600

It is very easy to consume a ton of memory when using ActiveRecord's find(:all) method. When transitioning the Team Sport Tech rails app from using file column to attachment_fu I wrote a migration to convert all of the photos we had to the new database format and the new location on disk. Without thinking I wrote code like this:

    Photo.find(:all).each do |photo|
      # conversion commands
      .
      .
    end

I didn't notice any issues executing this on my laptop which has 2 GB of RAM, but when I went to run this migration on our staging server over at Engine Yard I had problems. Our staging server only has 640 MB of RAM. Our database has over 100,000 photos. That is an array with 100,000 ActiveRecord objects in memory all at once. As the migration executed on the staging server I could see that the rake task was using all available RAM and it was paging like crazy, utilizing 600 MB of virtual memory from disk. The cpu fluctuated between 0 and 1 percent of utilization due to the paging. If adequate memory was available this migration would take around 10 minutes and the cpu would be working like crazy. I did a few other things for awhile and came back to this migration 2 hours later. Still working and it probably had a long ways yet to go.

While the memory limitation on the staging server isn't that important, the real question was how the production server with 1024 MB of RAM would handle the migration since the live app has to be down for maintenance when we roll this out. Instead of wondering I decided to fix the real problem, which was the stupid code I wrote to load all of the photos in memory for iteration. I quickly realized that the problem was already solved for me in the form of pagination. Pagination is usually thought of as a way to display N number of items to the user on a web page and providing links to move to the "Next" or "Previous" pages. When doing this the pagination code smartly loads only the ActiveRecord objects that are needed for the current page in memory, which is exactly the behavior I needed.

We recently switched from using the classic rails pagination to the newer will_paginate plugin. Since I knew I would need to use will_paginate in this manner in other migrations, and possibly in actual application code, I wrote this iterator for reuse:

    def apply_to_all(klass, per_page)
      objects = klass.paginate :page => 1, :per_page => per_page
      num_pages = objects.page_count

total_entries = objects.total_entries

      for page in 1 .. num_pages
        objects = klass.paginate(:page => page, :per_page => per_page, :total_entries => total_entries)
        objects.each do |object|
          yield object
        end
      end
    end

So I changed my migration code to look like this:

    apply_to_all(Photo, 500) do |photo|
      # conversion commands
      .
      .
    end

A key thing to note is the use of total_entries. Each time the paginate method is called, 2 sql queries are performed. One to get the records, and one to count how many records there are in total. This counting query can be very expensive when there are a lot of records, and if you provide the total_entries option to the paginate method the counting query will be skipped. While I chose to reuse the will_paginate plugin here, it wouldn't be very difficult to rewrite the above code without it by using LIMIT with an offset, which is how the will_paginate plugin works. Additionally it may be useful to add a third parameter to the apply_to_all method called options, which is simply passed onto the paginate method allowing such things as conditions or includes to be execute in the paginate sql query.

So now the migration code loads 500 Photo objects in memory at once. I ran this migration on the staging server and it barely used any RAM and executed in the expected 10 minutes. This short piece of code is a prime example of why I enjoy programming in ruby.

RMagick has memory problems. MiniMagick with Attachment_Fu is slowwww. Go ImageScience.

Luke Ludwig — Sat, 16 Feb 2008 18:35:00 -0600

Most Rails applications have to deal with resizing uploaded images for the creation of thumbnails. The main choices include RMagick, MiniMagick, or ImageScience, all of which come packaged as gems. Alternatively you can write your own which really isn't that difficult. So which one should you use? First I would recommend not writing your own, because it is really nice to take advantage of one of the very fine attachment plugins that are available. File column is the old standby rails attachment plugin, but it uses RMagick. Attachment_Fu is more flexible since you have the choice of using RMagick, MiniMagick, or ImageScience and can switch between them easily. At TeamSportTech where I work I recently transitioned our relatively large rails application from using file column and RMagick to using attachment_fu and ImageScience. Originally I was planning on using MiniMagick instead of ImageScience, but it turns out that MiniMagick is quite slow. The following timing results are on my Mac Book when running in development mode, and include uploading a single 3.4 MB jpeg which is resized down to 4 different sizes. MiniMagick consistently took 18 seconds to accomplish this, RMagick 7 seconds, and ImageScience 6 seconds. This doesn't accurately represent a production environment, but I do believe it is a fair comparison. Note that the time to upload is not a factor since this was done entirely on my local computer.

So why is MiniMagick with attachment_fu so slow? And why not use RMagick? RMagick and MiniMagick use the well known ImageMagick C libraries. RMagick works by providing API ruby bindings to the ImageMagick libraries, which means that RMagick operates within the ruby process. It is well documented that RMagick consumes a lot of memory and has memory leaks as well. See Craig Ambrose's article and a Mephisto article. It appears like RMagick2 provides better memory management.

Even if you have memory to spare, the more memory your Mongrels utilize the more garbage collection that will have to be done, which eventually will bog down the Mongrels. MiniMagick solves this memory problem by utilizing the shell commands provided by ImageMagick, which means all image manipulations are done in separate processes outside the ruby process' memory space. ImageScience is a different beast altogether since it uses the FreeImage library instead of ImageMagick, and it is more similar to RMagick in that it runs inside the ruby process. MiniMagick has extra overhead to start up a separate process for each ImageMagick command it executes. It also has extra disk overhead since it must operate on images on the filesystem instead of images loaded in memory.

I put a print out in the MiniMagick source to track all the ImageMagick commands that MiniMagick runs to accomplish this single upload with 4 resizing operations. Here is the output with timing in the order that they were executed:

1.421884 seconds:     identify "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"
0.594993 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"
0.592467 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"

0.602928 seconds:     identify "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-1"
0.594078 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-1"
0.584066 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-1"
1.444892 seconds:     mogrify -resize "240x240" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-1"
0.024799 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-1"
0.019388 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-1"

0.593593 seconds:     identify "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
0.580201 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
0.572714 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
0.590181 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
0.600532 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
1.396459 seconds:     mogrify -shave "648x0" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
0.973819 seconds:     mogrify -resize "75x75!" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
0.016706 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"
0.016919 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-2"

0.594606 seconds:     identify "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-3"
0.690731 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-3"
0.592638 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-3"
1.513243 seconds:     mogrify -resize "550x550" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-3"
0.028199 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-3"
0.029103 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-3"

0.610143 seconds:     identify "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"
0.593422 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"
0.638836 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"
1.777569 seconds:     mogrify -resize "1024x1024" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"
0.089828 seconds:     identify -format "%w\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"
0.057866 seconds:     identify -format "%h\\n" "/var/folders/UE/UE7FB01vFkmqEPkTei2ABk+++TI/-Tmp-/minimagick15344-0"

So as you can see, MiniMagick executed 30 separate shell commands. You can also see the temporary files on disk that it had to create to do these operations. To be fair, it is more correct to say that Attachment Fu executed 30 MiniMagick commands. I was curious why so many identify commands were needed. Each of the identify -format commands returns either the width (%w) or height (%h) of the image, which Attachment Fu stores in the database. This is very wasteful, because the initial identify command done for each resizing operation does include the width and height in the output, which should be parsed to save time. And for some reason notice that there are 10 identify commands to retrieve the width and 10 for the height, even though there are only 5 images involved (the original and 4 resized images). I don't know why this would be necessary and haven't checked the source to find out. Instead of pursuing these weird findings of how Attachment Fu uses MiniMagick, I decided to switch to Image Science. It is fast and is not a hog on memory.

Running ar_sendmail with monit

Luke Ludwig — Wed, 05 Dec 2007 19:35:00 -0600

Sending email from a web application, especially blast emails to a lot of people, can take a lot of time. Generally you don't want the user to wait until all the emails have been handed off to the smtp server. You also probably don't want to tie up an entire mongrel with sending mail. The ar_mailer gem solves this problem in excellent fashion, by saving pending emails to the database and having a separate ruby daemon process periodically check the database and send emails. I recently set up one of our rails apps at work to use ar_mailer. Configuring it to use ar_mailer was incredibly easy, but it was tricky to get the ar_sendmail ruby daemon process to run under monit. On our production servers which we have hosted at Engine Yard, we want every process that our application depends on to be monitored by monit.

The primary feature that ar_sendmail lacks to play nice with monit is the ability to leave a pid file after it starts up and to remove it when the process exits. This has already been pointed out on rubyforge as a feature request. Here is what I did to get ar_sendmail working under monit: (ar_mailer 1.3.1)

0) Find the file ar_sendmail.rb on your system and open it for editing.

For me this was at /usr/local/lib/ruby/gems/1.8/gems/ar_mailer-1.3.1/lib/action_mailer/. Another common location is /usr/lib/ruby/gems/1.8/gems/ar_mailer-1.3.1/lib/action_mailer/

1) Create a class variable to store the pid file path in and a method to remove the pid file.

I put this just below "attr_accessor :failed_auth_count"

  @@pid_file = nil

  def self.remove_pid_file
    FileUtils.rm(@@pid_file) if @@pid_file
  end

2) Modify the self.run method in ar_sendmail.rb to create a pid file, and to not start up if an ar_sendmail process is already running

    if options[:Daemon] then 
      require 'webrick/server'       
      @@pid_file = "#{options[:Chdir]}/log/ar_mailer.pid"            
      if File.exists? @@pid_file             
        # check to see if process is actually running
        pid = ''
        File.open(@@pid_file, 'r') {|f| pid = f.read.chomp }             
        if system("ps -p #{pid} | grep #{pid}") # returns true if process is running, o.w. false
          $stderr.puts "Warning: The pid file #{@@pid_file} exists and ar_sendmail is running. Shutting down."         
          exit
        else
          # not running, so remove existing pid file and continue
          self.remove_pid_file
          log "ar_sendmail is not running. Removing existing pid file and starting up..."          
        end
      end
      WEBrick::Daemon.start  
      File.open(@@pid_file, 'w') {|f| f.write("#{Process.pid}\n")}    
    end

If the pid file already exists, this code will check to see if the process is actually running. If it is then it will exit, otherwise it will remove the file and continue. This is useful for when the process dies ungracefully somehow (server crashes, killed with -9, .etc), in which case it will leave a pid file behind. Note that this differs from the code suggested on the rubyforge feature request, not only in that it checks the existence of the pid file, but it references options[:Chdir] instead of Dir.pwd in order to be compatible with the -c and --chdir ar_sendmail option.

3) Modify the do_exit method in ar_sendmail.rb to remove the pid file

  def do_exit
    log "caught signal, shutting down and removing pid file"
    self.remove_pid_file
    exit
  end

4) Create the monit file /etc/monit.d/ar_sendmail.teamsport.monitrc

check process ar_sendmail_teamsport
  with pidfile /data/teamsport/current/log/ar_mailer.pid
  start program = "/usr/bin/ar_sendmail -d -e production -c /data/teamsport/current/" as uid teamsport and gid teamsport
  stop program = "/usr/local/bin/stop_ar_sendmail" as uid teamsport and gid teamsport
  if totalmem is greater than 65.0 MB for 2 cycles then restart      # eating up memory?
  if loadavg(5min) greater than 10 for 8 cycles then restart          # bad, bad, bad
  if 20 restarts within 20 cycles then timeout                        # something is wrong, call the sys-admin
  group ar_sendmail

Note that I am using a simple shell script script called stop_ar_sendmail to stop the ar_sendmail process. ar_sendmail has signal handlers for SIGINT and SIGTERM so we should use these signals to kill it, which will cause the do_exit method to be triggered. The stop_ar_sendmail script looks like this:

#!/bin/bash
kill -2 `cat /data/teamsport/current/log/ar_mailer.pid`

Originally I tried putting the contents of this shell script in the .montrc file like this:

stop program = "/bin/kill -2 `cat /data/teamsport/current/log/ar_mailer.pid`" as uid teamsport and gid teamsport

This however does not work, since apparently monit doesn't know what to do with the backticks. Alternatively you could use some sort of grep and kill script, such as pkill, to stop the ar_sendmail process. Ideally in the future, ar_sendmail will support some sort of stop command in the same manner as mongrel so that you could run "ar_sendmail stop" to stop it.

Once you have this all setup you can control ar_sendmail via monit. When creating the ar_sendmail.teamsport.monitrc file, make sure you change "teamsport" to the user that you want to run the process as. Then do a "sudo monit reload" and monit should see that ar_sendmail is not running and will start it for you. To make sure everything is working correctly try "sudo monit stop ar_sendmail_teamsport" and "sudo monit start ar_sendmail_teamsport" (replacing "teamsport" with the appropriate user name).

Other than not working well with monit out of the box, the only other issue I have with ar_sendmail is the memory footprint, which is dependent to some degree on your rails app. The ar_sendmail process for my app runs at around 50 MB, and just to send mail! I assume this is due to the fact that ar_sendmail loads the rails app's environment.rb file. The environment.rb file runs the boot.rb file, which bootstraps and initializes the entire rails app. Additionally, our environment.rb has several other plugins required inside of it. I think the environment.rb file is loaded primarily just to get at the ActiveMailer smtp_settings, which is a slick way to allow for easy integration of ar_mailer with minimal changes to your existing rails app. Many people wouldn't think twice about 50 MB, but rails hosts charge quite a bit for RAM. I can definitely envision a slimmed down ar_sendmail that doesn't load the rails app's environment.rb file, but it seems almost impossible to do this without making integration with existing rails apps more difficult.

How to add a new column to Rails' sessions

Luke Ludwig — Mon, 26 Nov 2007 17:59:00 -0600

For a rails app at work we store user access privileges in the session. This is done as an optimization to avoid an extra sql query that would need to be done for every page view to determine if the user has edit privileges. Security-minded people may see this as a security hole. For this application I don't see this as a big deal. Its not like our rails app is controlling the launch of nuclear missiles.

The problem is that when an admin goes to modify the user access privileges for someone, the changes won't take affect until the user next logs in since the user access privileges are stored in the session. So if the user is already logged in this is a problem. They will have to log out and log back in for the changes to take affect. The solution to this problem is to modify the session of the user who's access privileges were modified.

To do this we need to add a user_id column to the sessions table. This can be done like any other migration. The tricky part is accessing the user_id column. We will want to set the user_id of the session when someone logs in. This of course will not work:

session.user_id = @user.id

The session we are so familiar with is not like other ActiveRecord objects. If you are using ActiveRecordStore for your session storage (the rails built-in database session storage mechanism), this is easy. For ActiveRecordStore, the session has a model attribute allowing you to access the ActiveRecord session object. First you need to make a session model class which extends ActiveRecord. Mine looked lke this:

class Session < ActiveRecord::Base

end

Then inside your "login" action you can do:

session.model.user_id = @user.id

Then just rely on the automatic session saving built into rails to save the user_id to the database.

If you are using SqlSessionStore, you don't have access to the model attribute. Instead you can do this to set the user_id:

my_session = Session.find_by_session_id(session.session_id)
my_session.update_attribute(:user_id, current_user.id)

Now presumably at some point in your application you will need to access and make use of the user_id column. For me this is when the user access privileges are modified. Since we created a session.rb model class, we can use the built in rails find method to find all sessions for the user.

sessions = Session.find(:all, :conditions => "user_id = #{user.id}", :select => "session_id")

To take advantage of the built-in session data marshaling in rails, we need to access the session like this:

user_session = CGI::Session::ActiveRecordStore::Session.find_by_session_id(sess.session_id)

Then we can set a session value using the data attribute and save the session off. Here is the full code for updating the sessions:

        sessions.each do |sess|
        user_session = CGI::Session::ActiveRecordStore::Session.find_by_session_id(sess.session_id)
        user_session.data[:access_privileges] = new_access_privileges
        user_session.save
        end