It is currently March 28th, 2024, 8:11 pm

Example using YQL "universal" feed reader

Tips and Tricks from the Rainmeter Community
User avatar
moshi
Posts: 1740
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

Post by moshi »

win :)

try this in the YQL console, five latest news from BBC News

SELECT entry.title,entry.link,entry.updated FROM feednormalizer WHERE url='http://feeds.bbci.co.uk/news/rss.xml' AND output='atom_1.0' | sort(field='entry.updated',descending='true') | truncate(count=5)

http://developer.yahoo.com/yql/console/
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

Post by jsmorley »

moshi wrote:win :)

try this in the YQL console, five latest news from BBC News

SELECT entry.title,entry.link,entry.updated FROM feednormalizer WHERE url='http://feeds.bbci.co.uk/news/rss.xml' AND output='atom_1.0' | sort(field='entry.updated',descending='true') | truncate(count=5)

http://developer.yahoo.com/yql/console/
Nice. The sorting is a real plus with some feeds like that bbc one and google.news.
User avatar
moshi
Posts: 1740
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

Post by moshi »

well depends, it's good if the feed is permanently visible.

when i use my Evi skin to search for news on a topic, using the q= parameter, i prefer them to be sorted by relevance.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

Post by jsmorley »

Anyone who already downloaded this skin from the first post should get this version. The desirability of sorting by pubdate / descending is pretty much a given for a "universal" feed reader.
You do not have the required permissions to view the files attached to this post.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

Post by jsmorley »

That sorting parameter seems to break http://feeds.bbci.co.uk/news/rss.xml, as each and every item is duplicated. So there are two of each. I'm not getting that on other feeds like google or breaking news.

I think there is a "unique" option that might be needed. I'll look into it.

It's always something...

Edit: This fixes that:

Code: Select all

http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true") | unique(field="entry.title")
But... it breaks the Rainmeter forum feed, where repeating "titles" are very common.

Second edit:

This seems to handle it better... Updated the example in the first post.

Code: Select all

http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true") | unique(field="entry.title", field="entry.updated")
User avatar
moshi
Posts: 1740
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

Post by moshi »

the entry.link field is probably the only truely unique one.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

Post by jsmorley »

moshi wrote:the entry.link field is probably the only truely unique one.
Yeah, perhaps so.

Edit: No, with

Code: Select all

URL=http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true") | unique(field="entry.link")
I still get duplicates on that bbc feed.

However, with

unique(field="entry.title", field="entry.updated")

I don't...

Not sure, but I think it might actually be struggling with doing unique with "entry.link", maybe due to how a URL is formatted/encoded in the feed.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

Post by jsmorley »

One thing we have to be careful of is that YQL feednormalizer is VERY sensitive to URL-encoding, much more so than your browser is.

For instance:

Code: Select all

http://backend.deviantart.com/rss.xml?q=in%3Acustomization%2Fskins%2Fsysmonitor%2Frainmeter+sort%3Atime&type=deviation
Which works fine in your browser, or even in Rainmeter as a straight URL= with WebParser, will just blow up when used with this YQL method.

What you need to do, any time that a URL for a feed seems to be giving you trouble, is to go to:

http://meyerweb.com/eric/tools/dencoder/ (there are no doubt hundreds of these...)

Paste in everything after http:// from the URL you want to use, and click "Encode".

Replace everything after http:// with the results from there, and it should be fine.

Code: Select all

http://backend.deviantart.com/rss.xml?q=in%3Acustomization%2Fskins%2Fsysmonitor%2Frainmeter+sort%3Atime&type=deviation

becomes:

http://backend.deviantart.com%2Frss.xml%3Fq%3Din%3Acustomization%2Fskins%2Fsysmonitor%2Frainmeter%2Bsort%3Atime%26type%3Ddeviation
And that works...

I really think it hates the ?q= as that seems to be confused somehow with the existing ?q= on the overall YQL URL, but rather than hand-encoding things, it is easier to just use that site above and encode the entire feed URL.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

Post by jsmorley »

P.S. While I have had a 100% success rate with English language feed sites, there are a few of these that just seem to fail:

German: http://diepresse.com/rss/home (works)
Greek: http://www.avgi.gr/RSSStartPage.action (fail)
Hebrew: http://www.haaretz.co.il/cmlink/1.1617539 (fail)
Devanagari: http://rss.jagran.com/rss/news/national.xml (works)
Malayalam: http://www.manoramaonline.com/rss/news/ (works)
Tamil: http://www.dinakaran.com/rss_Latest.asp (fail)
Japanese Kanji: http://sankei.jp.msn.com/rss/news/points.xml (works)
Cyrillic: http://izvestia.ru/xml/rss/all.xml (works)
Hungarian: http://nol.hu.feedsportal.com/c/33248/f/557942/index.rss (works)

Now it is possible that some of those sites that (fail) have a redirect going on or some malformed XML, or something else that is throwing YQL, not sure. It is illustrative that the ones that fail all seem to end in some specific file like .action / .1617539 / .asp rather than just a folder or .rss, .xml etc.
User avatar
moshi
Posts: 1740
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

Post by moshi »

i think the Greek feed has a similar issue. if you don't select top-level fields, it works just fine.

the Hebrew feed might be something completely different, maybe related to the right-to-left writing.