It is currently April 26th, 2024, 7:27 pm

universal "feed reader" with YQL

Tips and Tricks from the Rainmeter Community
User avatar
jsmorley
Developer
Posts: 22631
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: universal "feed reader" with YQL

Post by jsmorley »

I was able to get it working ok, although the solution is a little more "brute force" than I would like:

All you need is an @Include of:

Code: Select all

[Variables]
Decoder=""":"''",""":"''","'":"'","&apos;":"'","&":"&","&":"&","<":"<","<":"<",">":">",">":">","&160;":" ","nbsp;":" ","&#161;":"¡","&iexcl;":"¡","&#162;":"¢","&cent;":"¢","&#163;":"£","&pound;":"£","&#164;":"¤","&curren;":"¤","&#165;":"¥","&yen;":"¥","&#166;":"¦","&brvbar;":"¦","&#167;":"§","&sect;":"§","&#168;":"¨","&uml;":"¨","&#169;":"©","&copy;":"©","&#170;":"ª","&ordf;":"ª","&#171;":"«","&laquo;":"«","&#172;":"¬","&not;":"¬","&#174;":"®","&reg;":"®","&#175;":"¯","&macr;":"¯","&#176;":"°","&deg;":"°","&#177;":"±","&plusmn;":"±","&#178;":"²","&sup2;":"²","&#179;":"³","&sup3;":"³","&#180;":"´","&acute;":"´","&#181;":"µ","&micro;":"µ","&#182;":"¶","&para;":"¶","&#183;":"·","&middot;":"·","&#184;":"¸","&cedil;":"¸","&#185;":"¹","&sup1;":"¹","&#186;":"º","&ordm;":"º","&#187;":"»","&raquo;":"»","&#188;":"¼","&frac14;":"¼","&#189;":"½","&frac12;":"½","&#190;":"¾","&frac34;":"¾","&#191;":"¿","&iquest;":"¿","&#215;":"×","&times;":"×","&#247;":"÷","&divide;":"÷","&#192;":"À","&Agrave;":"À","&#193;":"Á","&Aacute;":"Á","&#194;":"Â","&Acirc;":"Â","&#195;":"Ã","&Atilde;":"Ã","&#196;":"Ä","&Auml;":"Ä","&#197;":"Å","&Aring;":"Å","&#198;":"Æ","&AElig;":"Æ","&#199;":"Ç","&Ccedil;":"Ç","&#200;":"È","&Egrave;":"È","&#201;":"É","&Eacute;":"É","&#202;":"Ê","&Ecirc;":"Ê","&#203;":"Ë","&Euml;":"Ë","&#204;":"Ì","&Igrave;":"Ì","&#205;":"Í","&Iacute;":"Í","&#206;":"Î","&Icirc;":"Î","&#207;":"Ï","&Iuml;":"Ï","&#208;":"Ð","&ETH;":"Ð","&#209;":"Ñ","&Ntilde;":"Ñ","&#210;":"Ò","&Ograve;":"Ò","&#211;":"Ó","&Oacute;":"Ó","&#212;":"Ô","&Ocirc;":"Ô","&#213;":"Õ","&Otilde;":"Õ","&#214;":"Ö","&Ouml;":"Ö","&#216;":"Ø","&Oslash;":"Ø","&#217;":"Ù","&Ugrave;":"Ù","&#218;":"Ú","&Uacute;":"Ú","&#219;":"Û","&Ucirc;":"Û","&#220;":"Ü","&Uuml;":"Ü","&#221;":"Ý","&Yacute;":"Ý","&#222;":"Þ","&THORN;":"Þ","&#223;":"","&szlig;":"ß","&#224;":"à","&agrave;":"à","&#225;":"á","&aacute;":"á","&#226;":"â","&acirc;":"â","&#227;":"ã","&atilde;":"ã","&#228;":"ä","&auml;":"ä","&#229;":"å","&aring;":"å","&#230;":"æ","&aelig;":"æ","&#231;":"ç","&ccedil;":"ç","&#232;":"è","&egrave;":"è","&#233;":"é","&eacute;":"é","&#234;":"ê","&ecirc;":"ê","&#235;":"ë","&euml;":"ë","&#236;":"ì","&igrave;":"ì","&#237;":"í","&iacute;":"í","&#238;":"î","&icirc;":"î","&#239;":"ï","&iuml;":"ï","&#240;":"ð","&eth;":"ð","&#241;":"ñ","&ntilde;":"ñ","&#242;":"ò","&ograve;":"ò","&#243;":"ó","&oacute;":"ó","&#244;":"ô","&ocirc;":"ô","&#245;":"õ","&otilde;":"õ","&#246;":"ö","&ouml;":"ö","&#248;":"ø","&oslash;":"ø","&#249;":"ù","&ugrave;":"ù","&#250;":"ú","&uacute;":"ú","&#251;":"û","&ucirc;":"û","&#252;":"ü","&uuml;":"ü","&#253;":"ý","&yacute;":"ý","&#254;":"þ","&thorn;":"þ","&#255;":"ÿ","&yuml;":"ÿ"
Then the skin:

Code: Select all

[Rainmeter]
Update=1000
AccurateText=1
DynamicWindowSize=1

[Metadata]
Name=YQLFeed
Author=JSMorley
Information=Demonstration of using the YQL syntax to obtain feeds produced in any format
Version=Dec 6, 2013
License=Creative Commons Attribution-Non-Commercial-Share Alike 3.0

[Variables]
@Include=#@#Decoder.inc
FeedURL=http://rainmeter.net/forum/feed.php
FeedInfo=<title.*>(.*)</title>.*<link href="(.*)".*<updated>(.*)</updated>
Entry=.*<entry>(.*)</entry>
EntryItems=<title.*>(.*)</title>.*<link href="(.*)".*<updated>(.*)</updated>

[MeasureFeed]
Measure=Plugin
Plugin=WebParser
URL=http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="#FeedURL#" and output="atom_1.0"
RegExp=(?siU)#FeedInfo##Entry##Entry##Entry##Entry##Entry#

; Feed Information

[MeasureFeedTitle]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
Substitute=#Decoder#
StringIndex=1

[MeasureFeedLink]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=2

[MeasureFeedTime]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=3

; Entry 1

[MeasureEntry1Title]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=4
RegExp=(?siU)#EntryItems#
Substitute=#Decoder#
DecodeCharacterReference=1
StringIndex2=1

[MeasureEntry1Link]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=4
RegExp=(?siU)#EntryItems#
StringIndex2=2

[MeasureEntry1Time]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=4
RegExp=(?siU)#EntryItems#
StringIndex2=3

; Entry 2

[MeasureEntry2Title]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=5
RegExp=(?siU)#EntryItems#
Substitute=#Decoder#
DecodeCharacterReference=1
StringIndex2=1

[MeasureEntry2Link]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=5
RegExp=(?siU)#EntryItems#
StringIndex2=2

[MeasureEntry2Time]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=5
RegExp=(?siU)#EntryItems#
StringIndex2=3

; Entry 3

[MeasureEntry3Title]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=6
RegExp=(?siU)#EntryItems#
Substitute=#Decoder#
DecodeCharacterReference=1
StringIndex2=1

[MeasureEntry3Link]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=6
RegExp=(?siU)#EntryItems#
StringIndex2=2

[MeasureEntry3Time]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=6
RegExp=(?siU)#EntryItems#
StringIndex2=3

; Entry 4

[MeasureEntry4Title]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=7
RegExp=(?siU)#EntryItems#
Substitute=#Decoder#
DecodeCharacterReference=1
StringIndex2=1

[MeasureEntry4Link]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=7
RegExp=(?siU)#EntryItems#
StringIndex2=2

[MeasureEntry4Time]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=7
RegExp=(?siU)#EntryItems#
StringIndex2=3

; Entry 5

[MeasureEntry5Title]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=8
RegExp=(?siU)#EntryItems#
Substitute=#Decoder#
DecodeCharacterReference=1
StringIndex2=1

[MeasureEntry5Link]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=8
RegExp=(?siU)#EntryItems#
StringIndex2=2

[MeasureEntry5Time]
Measure=Plugin
Plugin=WebParser
URL=[MeasureFeed]
StringIndex=8
RegExp=(?siU)#EntryItems#
StringIndex2=3

[MeterBackground]
Meter=Image
ImagePath=#@#Images\
ImageName=Background.png
W=300
H=295
ScaleMargins=3,3,3,3

[MeterHeadBackground]
Meter=Image
X=2
Y=2
W=296
H=40
SolidColor=10,10,10,255

[MeterFeedTitle]
Meter=String
MeasureName=MeasureFeedTitle
X=8
Y=7
W=294
H=36
FontFace=Segoe UI
FontSize=15
FontColor=200,200,200,255
StringStyle=Bold
AntiAlias=1
ClipString=1
LeftMouseUpAction=["[MeasureFeedLink]"]

[MeterEntry1Title]
Meter=String
MeasureName=MeasureEntry1Title
X=3
Y=44
W=287
H=44
FontFace=Segoe UI
FontSize=12
FontColor=200,200,200,255
SolidColor=30,30,30,255
Padding=4,0,4,4
AntiAlias=1
ClipString=1
ToolTipText=[MeasureEntry1Title]#CRLF##CRLF#Published: [MeasureEntry1Time]
ToolTipTitle=[MeasureFeedTitle]
ToolTipType=1
ToolTipWidth=300
DynamicVariables=1
LeftMouseUpAction=["[MeasureEntry1Link]"]

[MeterEntry2Title]
Meter=String
MeasureName=MeasureEntry2Title
X=3
Y=2R
W=287
H=44
FontFace=Segoe UI
FontSize=12
FontColor=200,200,200,255
SolidColor=47,47,47,255
Padding=4,0,4,4
AntiAlias=1
ClipString=1
ToolTipText=[MeasureEntry2Title]#CRLF##CRLF#Published: [MeasureEntry2Time]
ToolTipTitle=[MeasureFeedTitle]
ToolTipType=1
ToolTipWidth=300
DynamicVariables=1
LeftMouseUpAction=["[MeasureEntry2Link]"]

[MeterEntry3Title]
Meter=String
MeasureName=MeasureEntry3Title
X=3
Y=2R
W=287
H=44
FontFace=Segoe UI
FontSize=12
FontColor=200,200,200,255
SolidColor=30,30,30,255
Padding=4,0,4,4
AntiAlias=1
ClipString=1
ToolTipText=[MeasureEntry3Title]#CRLF##CRLF#Published: [MeasureEntry3Time]
ToolTipTitle=[MeasureFeedTitle]
ToolTipType=1
ToolTipWidth=300
DynamicVariables=1
LeftMouseUpAction=["[MeasureEntry3Link]"]

[MeterEntry4Title]
Meter=String
MeasureName=MeasureEntry4Title
X=3
Y=2R
W=287
H=44
FontFace=Segoe UI
FontSize=12
FontColor=200,200,200,255
SolidColor=47,47,47,255
Padding=4,0,4,4
AntiAlias=1
ClipString=1
ToolTipText=[MeasureEntry4Title]#CRLF##CRLF#Published: [MeasureEntry4Time]
ToolTipTitle=[MeasureFeedTitle]
ToolTipType=1
ToolTipWidth=300
DynamicVariables=1
LeftMouseUpAction=["[MeasureEntry4Link]"]

[MeterEntry5Title]
Meter=String
MeasureName=MeasureEntry5Title
X=3
Y=2R
W=287
H=44
FontFace=Segoe UI
FontSize=12
FontColor=200,200,200,255
SolidColor=30,30,30,255
Padding=4,0,4,4
AntiAlias=1
ClipString=1
ToolTipText=[MeasureEntry5Title]#CRLF##CRLF#Published: [MeasureEntry5Time]
ToolTipTitle=[MeasureFeedTitle]
ToolTipType=1
ToolTipWidth=300
DynamicVariables=1
LeftMouseUpAction=["[MeasureEntry5Link]"]
2013-12-06_115318.jpg
YQLFeed_1.1.rmskin
The solution is to "double-tap" the character decoding that is done. YQL will encode the data as it sees the need from the source. However, it messes up sometimes, and we get stuff like &amp. I don't see any easy fix for that on the YQL side of things.

However, if you use DecodeCharacterReference=1 on the "child" WebParser measures in the skin that will turn that &amp into just &. Fine as far as that goes, but we don't get another shot at DecodeCharacterReference on that measure, so we ALSO use a Substitute that uses that @Include file above.

As far as I can tell, that produces a clean result in all cases.

Here are some other URL's you can plug into the skin and try:

FeedURL=http://rainmeter.net/forum/feed.php
FeedURL=http://feeds.gawker.com/lifehacker/full.xml
FeedURL=http://news.google.com/?output=rss
FeedURL=http://www.breakingnews.com/feeds/rss/
You do not have the required permissions to view the files attached to this post.
Top
User avatar
moshi
Posts: 1740
Joined: November 13th, 2012, 9:53 pm

Re: universal "feed reader" with YQL

Post by moshi »

it's all cool, and it will work for me, but decoding with Substitute= is.a battle you can not win. you have not even covered all the latin letters, other scripts are missing completely.

in a perfect world everybody would use UTF-8 for their feeds, but until then, a way to decode twice would be really nice.
would it be so hard to implement a DecodeCharacterReference=2 option?
also please think about a !WriteFile bang. right now i use a few nasty tricks with !WriteKeyValue, like breaking the INI format with linebreaks, or batch files that still run when they start with:

Code: Select all

[foo]
=xplanet -source sun ...
that would all be much nicer with a !WriteFile bang, and it would also solve the problem of double-encoded feeds.
Top
User avatar
jsmorley
Developer
Posts: 22631
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: universal "feed reader" with YQL

Post by jsmorley »

moshi wrote:it's all cool, and it will work for me, but decoding with Substitute= is.a battle you can not win. you have not even covered all the latin letters, other scripts are missing completely.

in a perfect world everybody would use UTF-8 for their feeds, but until then, a way to decode twice would be really nice.
would it be so hard to implement a DecodeCharacterReference=2 option?
also please think about a !WriteFile bang. right now i use a few nasty tricks with !WriteKeyValue, like breaking the INI format with linebreaks, or batch files that still run when they start with:

Code: Select all

[foo]
=xplanet -source sun ...
that would all be much nicer with a !WriteFile bang, and it would also solve the problem of double-encoded feeds.
We are considering a new value for DecodeCharacterReference to handle double-encoding. Looking into any implications for the code.

As to the !WriteFile bang, I'll see what the team thinks. There is very robust and easy to do file reading/writing in Lua, and we are hesitant to try and jam all the capabilities available in a scripting language into Rainmeter proper. Not saying I'm against it, but we will have to chew on it a bit.
Top

Return to “Community Tips & Tricks”