#177 ✓resolved
Kieran P

RSS feeds increasingly slower as more content is added

Reported by Kieran P | March 25th, 2009 @ 02:00 PM | in 1.2

As the amount of content (thus more parsing) on a site increases, so does the load time for the rss feeds. Sometimes as much as 30 seconds. Because most non search feeds aren't cached, this can cause performance issues very quickly. To fix, walter suggests:

  • Convert from XML::Builder to Nokogiri::XML::Builder (my initial benchmarks, 7 records, show to to 35,000x faster rendering, and 127.28x faster overall).

  • Cache the output to avoid unnecessary Nokogiri parsing. Ensure the caches are cleared out properly.

Comments and changes to this ticket

  • Kieran P

    Kieran P March 26th, 2009 @ 04:44 PM

    The migration to Nokogiri is complete and most of the feeds have gone from ~15s page loads to just 1-2s. So big improvements. I'll have more details performance data soon.

    While working on this though, I came across the following quirks to look out for:

    • .rxml and .xml.builder don't work with Nokogiri. Instead, you have to use the format .xml.erb. The end result doesn't look different though.

    • There cannot be whitespace at the beginning of the document or it throws XML errors. So be sure to use -%> and no blank lines.

    • If you're handling a Time object, make sure it is properly transformed to a string. Sometimes it throws errors, so .to_s is all it takes.

    • The code needs the latest unreleased Nokogiri, which contains fixes for block scope. The maintainer has informed me the next release is several weeks off, so for no, we have to clone and build the gem ourselves. (tasks to do this automatically are built into 'rake manage_gems:required:install).

    And the new syntax isn't too different from the old one. Here is a shortend snippet from the tags feed. Note the handy xml.cdata method for wrapping user content in safe xml tags.

    Nokogiri::XML::Builder.new(:encoding => 'UTF-8') { |xml|
    xml.rss(:version => '2.0') {
      xml.channel {
        xml.title "Tags"
        xml.description "Tags in this basket"
        xml.link "http://link/site/tags/rss.xml"
        xml.language 'en-nz'
        for tag in @tags
          xml.item {
            xml.title {
              xml.cdata tag[:name]
            }
            @tag_search = basket_all_tagged_url
            xml.link @tag_search
            xml.guid @tag_search
          }
        end
      }
    }
    }.to_xml
    
    
  • Kieran P

    Kieran P March 26th, 2009 @ 05:29 PM

    • State changed from “new” to “resolved”

    Here are the stats. Rough figures, but should clearly show the benefit of Nokogiri. The winner info is measure by (winner rps / loser rpm = x times better):

    TR = Total Request
    TT = Total Time
    LR = Logest Request
    RPS = requests per second (mean)
    
    

    Tags Feed

    Builder

    TR: 2
    TT: 36.322 s
    RPS: 0.06
    LR: 18386 ms
    
    

    Nokogiri

    TR: 10
    TT: 1.827 s
    RPS: 5.47
    LR: 279 ms
    
    

    Winner - Nokogiri

    91.16 x
    
    

    Basket Feed

    Builder

    TR: 100
    TT: 129.157 s
    RPS: 0.77
    LR: 1895 ms
    
    

    Nokogiri

    TR: 100
    TT: 1.852 s
    RPS: 54.01
    LR: 188 ms
    
    

    Winner - Nokogiri

    70.14 x
    
    

    Moderation Feed

    Builder

    TR: 100
    TT: 9.760 s
    RPS: 10.25
    LR: 3662 ms
    
    

    Nokogiri

    TR: 100
    TT: 0.675 s
    RPS: 148.2
    LR: 156 ms
    
    

    Winner - Nokogiri

    14.4x
    
    

    Members Feed

    Builder

    TR: 10
    TT: 7.003 s
    RPS: 1.43
    LR: 6915 ms
    
    

    Nokogiri

    TR: 10
    TT: 0.108 s
    RPS: 92.55
    LR: 18 ms
    
    

    Winner - Nokogiri

    64.72 x
    
    

    Search Feed

    Builder Cached

    TR: 100
    TT: 24.327 s
    RPS: 4.11
    LR: 22840 ms
    
    

    Nokogiri Cached

    TR: 100
    TT: 3.832 s
    RPS: 26.10
    LR: 2527 ms
    
    

    Winner - Nokogiri

    6.35 x
    
    

    Biggest one to benefit was the tags. Search would have been, but it was cached already.

    So those are the statistics. The work has been polished, tested, and merged into master. Resolving ticket.

  • Steve Richert

    Steve Richert March 11th, 2010 @ 10:30 AM

    Are you able to cache only a portion of a Nokogiri builder document, as you are with the regular builder?

  • Kieran P

    Kieran P March 11th, 2010 @ 11:00 AM

    Hey Steve,

    I haven't tried. We cache the entire feed with a unique cache key that takes permissions into account.

    Regards
    Kieran

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

Kete was developed by Horowhenua Library Trust and Katipo Communications Ltd. to build a digital library of Horowhenua material.

People watching this ticket

Pages