It is currently November 15th, 2019, 3:49 am

Example using YQL "universal" feed reader

Tips and Tricks from the Rainmeter Community
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 3:41 am

There was a thread started today by Moshi, who had stumbled on a nice tool that could perhaps be used (at least until something better comes along) to replace the ability of Google Reader to provide "universal" feed capabilities for skins.

Now that Google Reader is a fond memory, I think this is an approach that is worth looking at.

The key to it is a public API provided by Yahoo! that will take as input on the URL any feed in RSS/ATOM/JSON format and provided a standard, "normalized" feed in any of those formats as output.

So for instance to access the Rainmeter Forum feed here, and return an XML formatted ATOM feed you can easily parse, you might use:

URL="http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://rainmeter.net/forum/feed.php" and output="atom_1.0"

For the same thing with for instance Google News, you might use:

URL="http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://news.google.com/?output=rss" and output="atom_1.0"

In any case, no matter what feed you access this way, a standard, reliable, feed in the ATOM format will be returned in the WebParser measure for you to parse.

A HUGE bonus with this is that the "dates" returned in the feed <pubdate> / <updated> etc. are also normalized to a standard date format of 2013-12-07T01:23:32Z and always in UTC (Universal Coordinated Time / GMT). What this means is that with just a handful of Lua code, you can easily compare the timestamp of any feed item to the current time, and flag "new" items as you like with a different color or an image, or whatever you like.

The example skin can be found here:

http://rainmeter.net/forum/viewtopic.php?f=27&t=17224
1.jpg
2.jpg
3.jpg
You do not have the required permissions to view the files attached to this post.
User avatar
Mordasius
Posts: 1063
Joined: January 22nd, 2011, 4:23 pm
Location: GMT +8

Re: Example using YQL "universal" feed reader

Mordasius » December 7th, 2013, 5:34 am

This works just fine on all the feeds I like to keep track of. :)

However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.

Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 5:38 am

Mordasius wrote:This works just fine on all the feeds I like to keep track of. :)

However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.

Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.
Yeah, Google News is pretty psychotic about how it arranges things in the feed as well.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 9:12 am

Mordasius wrote:This works just fine on all the feeds I like to keep track of. :)

However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.

Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.
you can do sorting with the right YQL syntax. after all the whole thing is a kind of database. you know, all this creating tables in Lua thing can be done in the Webparser Url= option.


example for merging two RSS feeds and sorting them by pubdate:

Code: Select all

URL="http://query.yahooapis.com/v1/public/yql?q=select title,link,pubDate from rss where url in ("http://rt.com/news/today/rss/","http://english.ruvr.ru/rss/export_all.xml") | sort(field="pubDate",descending="true") | truncate(count=8)"
now from rss is a different beast than from feednormalizer, you would need to figure out the right syntax to select the needed fields under the <entry> tags, not from toplevel.

i can't test this weekend, but YQL makes things a lot easier here.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 11:53 am

jsmorley wrote:
Google News
have a look at the &scoring=n parameter.
http://blog.slashpoundbang.com/post/12975232033/google-news-search-parameters-the-missing-manual
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 1:35 pm

Not sure google news rss supports that any more.

http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://news.google.com/?output=rss%26scoring=d" and output="atom_1.0"

Still seems to sort by "relevance" for me.

P.S. I found that I had to URL-encode the & or YQL really hated it.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 2:04 pm

moshi wrote: example for merging two RSS feeds and sorting them by pubdate:

Code: Select all

URL="http://query.yahooapis.com/v1/public/yql?q=select title,link,pubDate from rss where url in ("http://rt.com/news/today/rss/","http://english.ruvr.ru/rss/export_all.xml") | sort(field="pubDate",descending="true") | truncate(count=8)"
now from rss is a different beast than from feednormalizer, you would need to figure out the right syntax to select the needed fields under the <entry> tags, not from toplevel.
Unless I miss my guess, that is going to make having a skin that is "universal" hard to do. Since we are setting feednormalizer to ATOM or RSS or some other single output format, having to sort on the presumably unknown input format fields would be a problem.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 2:50 pm

jsmorley wrote: Not sure google news rss supports that any more.

Still seems to sort by "relevance" for me.

P.S. I found that I had to URL-encode the & or YQL really hated it.

yeah, seems like it. for a web search the current parameter is &tbs=sbd:1, but that doesn't do anything on a feed.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: Example using YQL "universal" feed reader

moshi » December 7th, 2013, 3:04 pm

jsmorley wrote: Unless I miss my guess, that is going to make having a skin that is "universal" hard to do. Since we are setting feednormalizer to ATOM or RSS or some other single output format, having to sort on the presumably unknown input format fields would be a problem.
no, i don't think so. the select happens after the normalizing. the only problem is to get the desired fields.
the last example here: http://www.yqlblog.net/blog/2013/03/07/yql-feednormalizer-table/ uses entry.title, not item.title although an RSS feed is transformed to an ATOM feed.
i guess "input" means something different here and is rather related to the database operation than the used feed.

something like this:

Code: Select all

select entry.title,entry.link,entry.updated from feednormalizer from url="http://feeds.bbci.co.uk/news/rss.xml" and output="atom_1.0" | sort("entry.updated",descending="true")
the result will no longer be an ATOM feed though i guess and need a new RegExp, more on monday, this kind of stuff isn't really fun on an iPad
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Example using YQL "universal" feed reader

jsmorley » December 7th, 2013, 3:26 pm

Yeah, we can play more on Monday.

Your example "almost" worked, but there was a small syntax error. It should be:

Code: Select all

http://query.yahooapis.com/v1/public/yql?q=select entry.title,entry.link,entry.updated from feednormalizer where url="https://news.google.com/news/feeds?output=rss" AND output="atom_1.0" | sort(field="entry.updated", descending="true")
If you use the descending="true", then you also need to use the "=" format for the sort entity, as in field="entry.updated".

The output seems to be valid ATOM formatting ok as far as I can tell. Certainly better than having to jump out to Lua, where sorting table entries and keeping all the table dimensions sync'd up can be a tad complicated.

The only issue is that we lose the overall "feed" info fields. I'll have to play with that. Think we need to extend the "select" to include top level fields outside of the <entry>'s.

Edit: Think I got it:

Code: Select all

URL=http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true")