Kete Kete

Comments and changes to this ticket

• 

  Walter McGinnis January 23rd, 2009 @ 12:27 PM

  To clarify, the "numerous issues" seem to be tests erroring out or
  failing after a certain point of running either the entire test suit
  (i.e. rake test) or running the integration test (i.e. rake
  test:integration). These tests are getting computationally intensive
  and seem to be eating up resources ("too many open files") and then
  the tests fail.

  As James has mentioned, there is likely a configuration change to make
  sure that our machines don't get swamped and cause all tests to error
  out or fail.

  On Jan 23, 2009, at 12:15 PM, Lighthouse wrote:

• 

  Kieran P February 16th, 2009 @ 01:08 PM

  To clarify, the "too many open files" issue is not the same issue that occurred in unit/functionals. This one is because too many errors writing/readings occurred (webrat dumps info to a file in tmp/).

  So I looked for errors. I went through each test and ran them individually. They all pass fine. So I ran the full suite, and this error came up again.

  My best guess is that memcache is running out of memory quickly, and gives the "no connection to server" error I get occasionally in development mode. Each test after that keeps failing for the same reason and eventually writes enough error fails...

  Possibly, the easiest solution is to increase your memcache memory limit to 256mb or so while running the tests, possibly even 512mb if you can. A CI server dedicated to this would be very helpful.

  Might be helpful to find a way to clear/clean up memcache from within the tests.

• 

  Andy Robertson February 24th, 2009 @ 11:11 AM

  This could maybe help. I needed to do this on the kete server here as they all share the same memcache server. It was spawning way too many connections to the the memcache server at one point I saw about 2048 (A nice power of 2) connections and kete's stopped serving pages. It occcured to me that they could all be stepping on each other. They were. The current config we use defaults ActionController::Base.session_options[:cache] to 172.0.0.1:11211 with no memcache namespace. It is used only for the sessions right. Session info is also in cookies.

  A cleaner fix would be to use the cache_fu plugin but heres a little hack that I did from what I could find on the web.

  What was required was a separate namespace for memcache for each of the kete. I found an example on the internet and I've put this into /config/initializers for each of my kete so they get a different memcache namespace

  
  memcache.rb
  
  memcache_options = {
            :namespace => 'keteham',
  }
  memcache_servers = [ '127.0.0.1:11211' ]
  cache_params = *([memcache_servers, memcache_options].flatten)
  CACHE = MemCache.new *cache_params
  ActionController::Base.session_options[:cache] = CACHE
  
  

  Where keteham is replaced with a short name I've given for each of my kete, like ketenp, ketewaimak, ketetest...

  Seems to have kept the number of connections down nicely. I couldn't find anything equiv for making backgroundrb to use namespaces however.

• 

  Kieran P March 4th, 2009 @ 05:24 PM

  • → Assigned user changed from “Kete” to “Kieran P”
  • → Milestone cleared.
• 

  Kieran P June 8th, 2009 @ 05:44 PM

  • → State changed from “new” to “open”

  The solution Andy detailed in the post above doesn't appear to work in the test suite because the tests run on the same instance of kete, so namespacing doesn't appear to do anything. Increasing memcache memory size from 62mb to 124mb and even 256mb has no effect either.

  The error (one of many I got during a full test suite test) can be seen here: http://gist.github.com/125626

  A few tests before and after this error worked fine, so Rails connection details must be correct.

  The error, which appears to originate from vendor/rails/activesupport/lib/active_support/vendor/memcache-client-1.5.0/memcache.rb:663:in request_setup' , which is triggering because server.socket.nil? is true. It has already passed a check for active? (testing if memcache is online).

  The server.socket method appears to use multi threading so (the following is an unconfirmed hypothesis) it may be possible that Rails/test suite is trying to request a page while a memcache connection is being initialized, thus can't establish its own connections, and triggers the error.

  After consulting with members on the Ruby on Rails IRC channel, the general consensus there is that memcache shouldn't be used for session storage and that either the current method (cookie store) or previous method (file store) would be more appropriate and less likely to error.

  The memcache connection errors themselves do not cause the test suite to fail, and is not caused by the writing of webrat error files I mentioned in my last comment. In fact, it appears no webrat files were written to disk. A possible cause may be the continuous opening of public/500.html, given that it is ruby that is opening these, and not apache serving them, so some file restrictions may apply.

  Looking at details from Google, a similar issue looks to come from the Kernel, when it has no more 'file handles' (http://www.patoche.org/LTT/kernel/00000128.html). That link details a possible fix (untested). However, given that we can't expect all users to edit their system, I'm still looking round for a better fix.

  Based on that and similar articles about the "too many files open" error, Ruby on Rails may not be releasing its file pointer when it opens and reads from 500.html . I'll dig into Rails source code tomorrow an see if anything can be done.

• 

  Kieran P November 2nd, 2009 @ 11:12 AM

  • → State changed from “open” to “hold”