Posted by Luke Ludwig
Sun, 17 Feb 2008 04:57:00 GMT
It is very easy to consume a ton of memory when using ActiveRecord's find(:all) method. When transitioning the
Team Sport Tech rails app from using file column to attachment_fu I wrote a migration to convert all of the photos we had to the new database format and the new location on disk. Without thinking I wrote code like this:
Photo.find(:all).each do |photo|
# conversion commands
.
.
end I didn't notice any issues executing this on my laptop which has 2 GB of RAM, but when I went to run this migration on our staging server over at
Engine Yard I had problems. Our staging server only has 640 MB of RAM. Our database has over 100,000 photos. That is an array with 100,000 ActiveRecord objects in memory all at once. As the migration executed on the staging server I could see that the rake task was using all available RAM and it was paging like crazy, utilizing 600 MB of virtual memory from disk. The cpu fluctuated between 0 and 1 percent of utilization due to the paging. If adequate memory was available this migration would take around 10 minutes and the cpu would be working like crazy. I did a few other things for awhile and came back to this migration 2 hours later. Still working and it probably had a long ways yet to go.
Read more...
Posted in Programming | Tags ActiveRecord, apply_to_all, attachment_fu, rails, ruby, will_paginate | no comments | no trackbacks
Posted by Luke Ludwig
Sun, 17 Feb 2008 00:35:00 GMT
Most Rails applications have to deal with resizing uploaded images for the creation of thumbnails. The main choices include RMagick, MiniMagick, or ImageScience, all of which come packaged as gems. Alternatively you can write your own which really isn't that difficult. So which one should you use? First I would recommend not writing your own, because it is really nice to take advantage of one of the very fine attachment plugins that are available. File column is the old standby rails attachment plugin, but it uses RMagick. Attachment_Fu is more flexible since you have the choice of using RMagick, MiniMagick, or ImageScience and can switch between them easily. At TeamSportTech where I work I recently transitioned our relatively large rails application from using file column and RMagick to using attachment_fu and ImageScience. Originally I was planning on using MiniMagick instead of ImageScience, but it turns out that MiniMagick is quite slow. The following timing results are on my Mac Book when running in development mode, and include uploading a single 3.4 MB jpeg which is resized down to 4 different sizes. MiniMagick consistently took 18 seconds to accomplish this, RMagick 7 seconds, and ImageScience 6 seconds. This doesn't accurately represent a production environment, but I do believe it is a fair comparison. Note that the time to upload is not a factor since this was done entirely on my local computer.
So why is MiniMagick with attachment_fu so slow? And why not use RMagick? RMagick and MiniMagick use the well known ImageMagick C libraries. RMagick works by providing API ruby bindings to the ImageMagick libraries, which means that RMagick operates within the ruby process. It is well documented that RMagick consumes a lot of memory and has memory leaks as well. See Craig Ambrose's article and a Mephisto article. It appears like RMagick2 provides better memory management.
Read more...
Posted in Programming | Tags attachment_fu, file column, FreeImage, ImageMagick, ImageScience, MiniMagick, rails, RMagick, ruby | no comments | no trackbacks