It is currently November 15th, 2019, 3:57 am

Example using YQL "universal" feed reader

Tips and Tricks from the Rainmeter Community
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 3:36 pm

win :)

try this in the YQL console, five latest news from BBC News

SELECT entry.title,entry.link,entry.updated FROM feednormalizer WHERE url='http://feeds.bbci.co.uk/news/rss.xml' AND output='atom_1.0' | sort(field='entry.updated',descending='true') | truncate(count=5)

http://developer.yahoo.com/yql/console/
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 3:38 pm

moshi wrote:win :)

try this in the YQL console, five latest news from BBC News

SELECT entry.title,entry.link,entry.updated FROM feednormalizer WHERE url='http://feeds.bbci.co.uk/news/rss.xml' AND output='atom_1.0' | sort(field='entry.updated',descending='true') | truncate(count=5)

http://developer.yahoo.com/yql/console/
Nice. The sorting is a real plus with some feeds like that bbc one and google.news.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 3:41 pm

well depends, it's good if the feed is permanently visible.

when i use my Evi skin to search for news on a topic, using the q= parameter, i prefer them to be sorted by relevance.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 3:43 pm

Anyone who already downloaded this skin from the first post should get this version. The desirability of sorting by pubdate / descending is pretty much a given for a "universal" feed reader.
You do not have the required permissions to view the files attached to this post.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 3:58 pm

That sorting parameter seems to break http://feeds.bbci.co.uk/news/rss.xml, as each and every item is duplicated. So there are two of each. I'm not getting that on other feeds like google or breaking news.

I think there is a "unique" option that might be needed. I'll look into it.

It's always something...

Edit: This fixes that:

Code: Select all

http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true") | unique(field="entry.title")
But... it breaks the Rainmeter forum feed, where repeating "titles" are very common.

Second edit:

This seems to handle it better... Updated the example in the first post.

Code: Select all

http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true") | unique(field="entry.title", field="entry.updated")
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 4:19 pm

the entry.link field is probably the only truely unique one.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 4:22 pm

moshi wrote:the entry.link field is probably the only truely unique one.
Yeah, perhaps so.

Edit: No, with

Code: Select all

URL=http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true") | unique(field="entry.link")
I still get duplicates on that bbc feed.

However, with

unique(field="entry.title", field="entry.updated")

I don't...

Not sure, but I think it might actually be struggling with doing unique with "entry.link", maybe due to how a URL is formatted/encoded in the feed.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 5:08 pm

One thing we have to be careful of is that YQL feednormalizer is VERY sensitive to URL-encoding, much more so than your browser is.

For instance:

Code: Select all

http://backend.deviantart.com/rss.xml?q=in%3Acustomization%2Fskins%2Fsysmonitor%2Frainmeter+sort%3Atime&type=deviation
Which works fine in your browser, or even in Rainmeter as a straight URL= with WebParser, will just blow up when used with this YQL method.

What you need to do, any time that a URL for a feed seems to be giving you trouble, is to go to:

http://meyerweb.com/eric/tools/dencoder/ (there are no doubt hundreds of these...)

Paste in everything after http:// from the URL you want to use, and click "Encode".

Replace everything after http:// with the results from there, and it should be fine.

Code: Select all

http://backend.deviantart.com/rss.xml?q=in%3Acustomization%2Fskins%2Fsysmonitor%2Frainmeter+sort%3Atime&type=deviation

becomes:

http://backend.deviantart.com%2Frss.xml%3Fq%3Din%3Acustomization%2Fskins%2Fsysmonitor%2Frainmeter%2Bsort%3Atime%26type%3Ddeviation
And that works...

I really think it hates the ?q= as that seems to be confused somehow with the existing ?q= on the overall YQL URL, but rather than hand-encoding things, it is easier to just use that site above and encode the entire feed URL.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 6:02 pm

P.S. While I have had a 100% success rate with English language feed sites, there are a few of these that just seem to fail:

German: http://diepresse.com/rss/home (works)
Greek: http://www.avgi.gr/RSSStartPage.action (fail)
Hebrew: http://www.haaretz.co.il/cmlink/1.1617539 (fail)
Devanagari: http://rss.jagran.com/rss/news/national.xml (works)
Malayalam: http://www.manoramaonline.com/rss/news/ (works)
Tamil: http://www.dinakaran.com/rss_Latest.asp (fail)
Japanese Kanji: http://sankei.jp.msn.com/rss/news/points.xml (works)
Cyrillic: http://izvestia.ru/xml/rss/all.xml (works)
Hungarian: http://nol.hu.feedsportal.com/c/33248/f/557942/index.rss (works)

Now it is possible that some of those sites that (fail) have a redirect going on or some malformed XML, or something else that is throwing YQL, not sure. It is illustrative that the ones that fail all seem to end in some specific file like .action / .1617539 / .asp rather than just a folder or .rss, .xml etc.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 7:17 pm

i think the Greek feed has a similar issue. if you don't select top-level fields, it works just fine.

the Hebrew feed might be something completely different, maybe related to the right-to-left writing.