It is currently September 16th, 2019, 7:20 am

universal "feed reader" with YQL

Tips and Tricks from the Rainmeter Community
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: universal feed reader with YQL

moshi » December 6th, 2013, 2:27 pm

jsmorley wrote:Have you messed with converting a "known" standard date/time string into a TimeStamp in Rainmeter? It's easy in Lua, but rather not go there if I don't have to.
no, not really, that's way above my head. what i use is a port of the GNU date command:

Code: Select all

[MeasureTimeParse]
Measure=Plugin
Plugin=RunCommand
Program=""#@#Addons\date2\date2.exe""
Parameter="-d "[MeasureTimeParsePre]" +"%A, %H:%M""
State=Hide
OutputType=UTF8
DynamicVariables=1
[MeasureTimeParsePre] looks like 2013-12-25 16:15:00 +0700

for a (UNIX) timestamp i'd just use:

Code: Select all

Parameter="-d "[MeasureTimeParsePre]" +"%s""

also:
http://rainmeter.net/forum/viewtopic.php?f=14&t=13363&hilit=timestamp ;-)
User avatar
jsmorley
Developer
Posts: 19440
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: universal feed reader with YQL

jsmorley » December 6th, 2013, 2:35 pm

moshi wrote: no, not really, that's way above my head. what i use is a port of the GNU date command:

Code: Select all

[MeasureTimeParse]
Measure=Plugin
Plugin=RunCommand
Program=""#@#Addons\date2\date2.exe""
Parameter="-d "[MeasureTimeParsePre]" +"%A, %H:%M""
State=Hide
OutputType=UTF8
DynamicVariables=1
[MeasureTimeParsePre] looks like 2013-12-25 16:15:00 +0700

for a (UNIX) timestamp i'd just use:

Code: Select all

Parameter="-d "[MeasureTimeParsePre]" +"%s""

also:
http://rainmeter.net/forum/viewtopic.php?f=14&t=13363&hilit=timestamp ;-)
Ok, I'm fine with using Lua, as it has all that capability built-in and I don't need an "addon". I just need to work it so that capability is something a user can just "plug-into" the skin in a reasonably clear way, so it is only there if they want to compare a feed date/time with current.

Thanks.
User avatar
jsmorley
Developer
Posts: 19440
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: universal "feed reader" with YQL

jsmorley » December 6th, 2013, 5:30 pm

Moshi, found one problem with this YQL tool.

When I get the Rainmeter feed normally with WebParser, there is a point where it has:

Tips & Tricks

Using DecodeCharacterReference=1 on the measure takes care of that:

Tips & Tricks

With the YQL version of the feed, I get:

Tips & Tricks

Which DecodeCharacterReference=1 turns into:

Tips & Tricks

Since what it does is replace the & with &, leaving... &, or right back where we started...

What I see in the raw feed from rainmeter.net is:

<title type="html"><![CDATA[Tips & Tricks • Re: universal feed reader with YQL]]></title>

YQL is turning that into:

<title type="html">Tips &amp; Tricks • Re: universal feed reader with YQL</title>

That is just stupid. It is trying to encode an already encoded entity, and even then doing a poor job of it. It is apparently just seeing the "&" and saying "Damn, I'd better encode that!!!" and ignoring the fact that it already is "&".

This is going to be a major problem with the approach in Rainmeter, unless there is some parameter to the YQL API that can correct it.

Using a Substitute like Sub="&":"&",""":"''","<":"<",">":">" or something is just not going to be good enough. There are many, many possible encoded characters in a feed. You would need the Substitute from hell to deal with all of them.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: universal "feed reader" with YQL

moshi » December 6th, 2013, 7:39 pm

i can't test this weekend, but how about downloading it with DecodeCharacterReference=1 and then parsing the downloaded file with again with DecodeCharacterReference=1 ?
User avatar
jsmorley
Developer
Posts: 19440
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: universal "feed reader" with YQL

jsmorley » December 6th, 2013, 7:49 pm

moshi wrote:i can't test this weekend, but how about downloading it with DecodeCharacterReference=1 and then parsing the downloaded file with again with DecodeCharacterReference=1 ?
I dunno... Let me call Rube Goldberg and see what he thinks. ;-)

Edit:

Doesn't seem to help:

Code: Select all

[Metadata]
Name=YQLFeed
Author=JSMorley
Information=Demonstration of using the YQL syntax to obtain feeds produced in any format
Version=Dec 6, 2013
License=Creative Commons Attribution-Non-Commercial-Share Alike 3.0

[Variables]
FeedURL=http://rainmeter.net/forum/feed.php
FeedInfo=<title.*>(.*)</title>.*<link href="(.*)".*<updated>(.*)</updated>
Entry=.*<entry>(.*)</entry>
EntryItems=<title.*>(.*)</title>.*<link href="(.*)".*<updated>(.*)</updated>
Sub="&":"&",""":"''","<":"<",">":">"

[MeasureGrandparent]
Measure=Plugin
Plugin=WebParser
URL=http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="#FeedURL#" and output="atom_1.0"
RegExp=(?siU)^(.*)$
DecodeCharacterReference=1

[MeasureFeed]
Measure=Plugin
Plugin=WebParser
URL=[MeasureGrandparent]
RegExp=(?siU)#FeedInfo##Entry##Entry##Entry##Entry##Entry#
DecodeCharacterReference=1
2013-12-06_115318.jpg
You do not have the required permissions to view the files attached to this post.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: universal "feed reader" with YQL

moshi » December 6th, 2013, 8:25 pm

i was thinking about Download=1 and DownloadFile=something.xml, with either DecodeCharacterReference=1 or Substitute="&":"&" to have a "physical", "first-step-decoded" file to parse.
User avatar
jsmorley
Developer
Posts: 19440
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: universal "feed reader" with YQL

jsmorley » December 6th, 2013, 8:29 pm

moshi wrote:i was thinking about Download=1 and DownloadFile=something.xml, with either DecodeCharacterReference=1 or Substitute="&":"&" to have a "physical", "first-step-decoded" file to parse.
Not sure there is any difference with what I did with the "grandparent", but I'll play with that in a while.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: universal "feed reader" with YQL

moshi » December 6th, 2013, 8:54 pm

if that fails, you could use FinishAction=!WriteKeyValue to a second file. then all decode and substitute options will work, of course you would have to substitute all line breaks, so everything can be overwritten on the next update.
User avatar
jsmorley
Developer
Posts: 19440
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: universal "feed reader" with YQL

jsmorley » December 6th, 2013, 9:05 pm

There is only so far I am willing to go with something that is supposed to be "generic" and easy to use. Only so many bags I will hang on the side of the machine.
User avatar
moshi
Posts: 1851
Joined: November 13th, 2012, 9:53 pm

Re: universal "feed reader" with YQL

moshi » December 6th, 2013, 9:32 pm

i think this is pretty much the same as i did in the example skin in the first post.
it only works for &amp; though, i don't think that'll work for things like &#39; though.

i'd love to have a !WriteFile bang btw., that would generate/overwrite a file. that would not only be cool for double-encoded feeds but also for dynamically writing scripts. i know i can do this with echo and the RunCommand plugin, but with all the capabilities Rainmeter has now, i think it should be a generic feature.