
RSS feeds increasingly slower as more content is added
Reported by Kieran P | March 25th, 2009 @ 02:00 PM | in 1.2
As the amount of content (thus more parsing) on a site increases, so does the load time for the rss feeds. Sometimes as much as 30 seconds. Because most non search feeds aren't cached, this can cause performance issues very quickly. To fix, walter suggests:
-
Convert from XML::Builder to Nokogiri::XML::Builder (my initial benchmarks, 7 records, show to to 35,000x faster rendering, and 127.28x faster overall).
-
Cache the output to avoid unnecessary Nokogiri parsing. Ensure the caches are cleared out properly.
Comments and changes to this ticket
-
Kieran P March 26th, 2009 @ 04:44 PM
The migration to Nokogiri is complete and most of the feeds have gone from ~15s page loads to just 1-2s. So big improvements. I'll have more details performance data soon.
While working on this though, I came across the following quirks to look out for:
-
.rxml and .xml.builder don't work with Nokogiri. Instead, you have to use the format .xml.erb. The end result doesn't look different though.
-
There cannot be whitespace at the beginning of the document or it throws XML errors. So be sure to use -%> and no blank lines.
-
If you're handling a Time object, make sure it is properly transformed to a string. Sometimes it throws errors, so .to_s is all it takes.
-
The code needs the latest unreleased Nokogiri, which contains fixes for block scope. The maintainer has informed me the next release is several weeks off, so for no, we have to clone and build the gem ourselves. (tasks to do this automatically are built into 'rake manage_gems:required:install).
And the new syntax isn't too different from the old one. Here is a shortend snippet from the tags feed. Note the handy xml.cdata method for wrapping user content in safe xml tags.
Nokogiri::XML::Builder.new(:encoding => 'UTF-8') { |xml| xml.rss(:version => '2.0') { xml.channel { xml.title "Tags" xml.description "Tags in this basket" xml.link "http://link/site/tags/rss.xml" xml.language 'en-nz' for tag in @tags xml.item { xml.title { xml.cdata tag[:name] } @tag_search = basket_all_tagged_url xml.link @tag_search xml.guid @tag_search } end } } }.to_xml
-
-
Kieran P March 26th, 2009 @ 05:29 PM
- State changed from new to resolved
Here are the stats. Rough figures, but should clearly show the benefit of Nokogiri. The winner info is measure by (winner rps / loser rpm = x times better):
TR = Total Request TT = Total Time LR = Logest Request RPS = requests per second (mean)
Tags Feed
Builder
TR: 2 TT: 36.322 s RPS: 0.06 LR: 18386 ms
Nokogiri
TR: 10 TT: 1.827 s RPS: 5.47 LR: 279 ms
Winner - Nokogiri
91.16 x
Basket Feed
Builder
TR: 100 TT: 129.157 s RPS: 0.77 LR: 1895 ms
Nokogiri
TR: 100 TT: 1.852 s RPS: 54.01 LR: 188 ms
Winner - Nokogiri
70.14 x
Moderation Feed
Builder
TR: 100 TT: 9.760 s RPS: 10.25 LR: 3662 ms
Nokogiri
TR: 100 TT: 0.675 s RPS: 148.2 LR: 156 ms
Winner - Nokogiri
14.4x
Members Feed
Builder
TR: 10 TT: 7.003 s RPS: 1.43 LR: 6915 ms
Nokogiri
TR: 10 TT: 0.108 s RPS: 92.55 LR: 18 ms
Winner - Nokogiri
64.72 x
Search Feed
Builder Cached
TR: 100 TT: 24.327 s RPS: 4.11 LR: 22840 ms
Nokogiri Cached
TR: 100 TT: 3.832 s RPS: 26.10 LR: 2527 ms
Winner - Nokogiri
6.35 x
Biggest one to benefit was the tags. Search would have been, but it was cached already.
So those are the statistics. The work has been polished, tested, and merged into master. Resolving ticket.
-
Steve Richert March 11th, 2010 @ 10:30 AM
Are you able to cache only a portion of a Nokogiri builder document, as you are with the regular builder?
-
Kieran P March 11th, 2010 @ 11:00 AM
Hey Steve,
I haven't tried. We cache the entire feed with a unique cache key that takes permissions into account.
Regards
Kieran
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
Kete was developed by Horowhenua Library Trust and Katipo Communications Ltd. to build a digital library of Horowhenua material.
People watching this ticket
- Nobody is watching this ticket.