It is currently March 30th, 2020, 1:43 pm

⭐ weather.com - Parsing the JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
SilverAzide
Posts: 716
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SilverAzide »

jsmorley wrote:
February 12th, 2020, 7:14 pm
I can break this up into separate .inc files for each "day", but all things being equal, I'd just as soon not if I don't really need to. Makes it a bit of a pain to maintain, particularly in these early days, when there is some tweaking going on...

Having said that, if having fewer child measures being stood up makes a demonstrable difference, I will do so.
Per your suspicion, I confirmed that the "stuttering" behavior I was experiencing is NOT caused by the child measures. Both the HTML-based and JSON-based variants of the measures behave identically. Commenting out the Day2-Day6 measures (that's 192 measures, BTW) had only the tiniest of impacts on performance.

BUT!! Just to clarify -- for my own sanity -- the stuttering isn't in my imagination and I can easily reproduce it. Turns out that the ActionTimer in my skin will stutter and lag severely if the Rainmeter About window is open AND the skin is selected (both conditions must be met for the skin painting to lag). So what I was seeing was due to me debugging my skin with the About window open, simple as that. So, I think we can file this one away as a non-issue.

That said, skins that don't have a forecast (i.e., just showing current/today/tonight) won't need those 192 Day2-Day6 measures. So... just a thought.
DeviantArt Gadgets More...
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

SilverAzide wrote:
February 13th, 2020, 1:02 am
Per your suspicion, I confirmed that the "stuttering" behavior I was experiencing is NOT caused by the child measures. Both the HTML-based and JSON-based variants of the measures behave identically. Commenting out the Day2-Day6 measures (that's 192 measures, BTW) had only the tiniest of impacts on performance.

BUT!! Just to clarify -- for my own sanity -- the stuttering isn't in my imagination and I can easily reproduce it. Turns out that the ActionTimer in my skin will stutter and lag severely if the Rainmeter About window is open AND the skin is selected (both conditions must be met for the skin painting to lag). So what I was seeing was due to me debugging my skin with the About window open, simple as that. So, I think we can file this one away as a non-issue.

That said, skins that don't have a forecast (i.e., just showing current/today/tonight) won't need those 192 Day2-Day6 measures. So... just a thought.
Ah, yeah. Having a ton of measures and displaying About / Skins can certainly have an impact. That panel is updated with real-time actual values once a second as well, and it is trying to manage a TON of stuff while displaying. Having About / Skins open is VERY expensive, and you shouldn't judge your performance with it open.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

I have added information for Moon Phases and Moon Set / Rise times.

This is in a new @Include file WeatherComJSONMoon.inc

This will create the following new measures:
[@MoonDay1DayLong] : i.e. Wednesday
[@MoonDay1DayShort] : i.e. Wed
[@MoonDay1PhaseIcon] : Moon Phase Icon Name
[@MoonDay1PhaseName] : Moon Phase Name
[@MoonDay1RiseTime] : Time of Moon Rise
[@MoonDay1SetTime] : Time of Moon Set

...

The pattern will continue for Day2 through Day6
The 128x128 icons I created in #@#Images\MoonPhase are:

New Moon : N.png
Waning Crescent : WNC.png
Last Quarter : LQ.png
Waning Gibbous : WNG.png
Full Moon : F.png
Waxing Gibbous : WXG.png
First Quarter : FQ.png
Waxing Crescent : WXC.png

1.png
2.png

They are not perfect icons, and will likely look best on a dark background...


The data in the JSON feed will look like:
Day1 : "moonIcon":"WNG","moonPhrase":"Waning Gibbous","moonrise":"2020-02-12T21:54:21-0500","moonset":"2020-02-12T09:24:02-0500","dayOfWeek":"Wednesday",
Day2 : "moonIcon":"WNG","moonPhrase":"Waning Gibbous","moonrise":"2020-02-13T23:05:26-0500","moonset":"2020-02-13T09:56:15-0500","dayOfWeek":"Thursday",
Day3 : "moonIcon":"WNG","moonPhrase":"Waning Gibbous","moonrise":"","moonset":"2020-02-14T10:29:08-0500","dayOfWeek":"Friday",
Day4 : "moonIcon":"LQ","moonPhrase":"Last Quarter","moonrise":"2020-02-15T00:15:34-0500","moonset":"2020-02-15T11:03:31-0500","dayOfWeek":"Saturday",
Day5 : "moonIcon":"WNC","moonPhrase":"Waning Crescent","moonrise":"2020-02-16T01:23:31-0500","moonset":"2020-02-16T11:41:32-0500","dayOfWeek":"Sunday",
Day6 : "moonIcon":"WNC","moonPhrase":"Waning Crescent","moonrise":"2020-02-17T02:29:48-0500","moonset":"2020-02-17T12:24:38-0500","dayOfWeek":"Monday",

And I parse it with:

Code: Select all

(?siU)"vt1dailyForecast":\[{"validDate":"(.*)",.*"moonIcon":"(.*)","moonPhrase":"(.*)","moonrise":"(.*)","moonset":"(.*)",.*{"validDate":"(.*)",.*"moonIcon":"(.*)","moonPhrase":"(.*)","moonrise":"(.*)","moonset":"(.*)",.*{"validDate":"(.*)",.*"moonIcon":"(.*)","moonPhrase":"(.*)","moonrise":"(.*)","moonset":"(.*)",.*{"validDate":"(.*)",.*"moonIcon":"(.*)","moonPhrase":"(.*)","moonrise":"(.*)","moonset":"(.*)",.*{"validDate":"(.*)",.*"moonIcon":"(.*)","moonPhrase":"(.*)","moonrise":"(.*)","moonset":"(.*)",.*{"validDate":"(.*)",.*"moonIcon":"(.*)","moonPhrase":"(.*)","moonrise":"(.*)","moonset":"(.*)",

New .rmskin in the first post of this thread
You do not have the required permissions to view the files attached to this post.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

Man, I'm personally steering clear of MoonSet / MoonRise times. They are a challenge to understand.

1) The moon "sets" in the morning sometime, and then "rises" at night. Depending on the time that these happen, I'm not fully confident of the "day" that is returned in the timestamp... "night" can be in the PM of today, or the AM of tomorrow. I'm guessing the "date" part of the timestamp may just be a place-holder. Just use the day of the week name(s). They seem consistent and correct. I think these timestamps reflect a more general sense of "That day's moon", rather than specific dates.

2) I have run into an instance where there is an entirely missing MoonSet / MoonRise timestamp. Instead of the literal null used elsewhere, in this case it is delineated by "" or an empty string. This is a challenge to deal with when using Time measures, as those just hate a TimeStamp option that is an empty string. It can be worked around, but if you are using these times to create some graphical representation of where the moon is in its daily cycle through the sky, this will take some thought indeed when data is missing.

Day3 : "moonIcon":"WNG","moonPhrase":"Waning Gibbous","moonrise":"","moonset":"2020-02-14T10:29:08-0500","dayOfWeek":"Friday",

I can only hope that all data will always be there for at least "today / day1", so you can display the information for that. Doing "forecasts" for the moon might be more tricky.

A better approach, if anyone cares to take it on, is to get far more robust moon information from this site:

https://www.timeanddate.com/astronomy/usa/alexandria
https://www.timeanddate.com/moon/phases/usa/alexandria

But that's a conversation for another thread.

Have fun with it... ;-)
User avatar
Yincognito
Posts: 1072
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

jsmorley wrote:
February 12th, 2020, 10:11 pm
What I discovered is that the JSON contains a field for the "max temperature since 7am" in the "current" section, and that will in fact be the "high" for today once the data switches over from "day" to "night". [...] Then you always have both a "high" and "low" for today.
That field is the only one not included in my skin. I have no need for max/min temperatures for the 24h day, since I've always used the day/night cycles (and their temperatures). It's true that visually, on the main page, there are these "Max" and "Min" temperatures for the 24h day (hence, taken as such by skin authors), but those are actually the day and night cycle's temps internally. From a purely technical point of view, there are no min or max temps (bar the "max temperature since 7am" you noticed, and that is for past tense) in TWC (and wxdata) sources, they just happen to label them as such on the main page since almost always the day cycle is warmer than the night one, everywhere on Earth.
SilverAzide wrote:
February 13th, 2020, 1:02 am
Commenting out the Day2-Day6 measures (that's 192 measures, BTW) [...] That said, skins that don't have a forecast (i.e., just showing current/today/tonight) won't need those 192 Day2-Day6 measures. So... just a thought.
I had a similar number of WebParser measures before rewriting my skin recently. Now I have 57, with only 23 of them for the all the forecast days, capturing all but the field jsmorley discovered above - no alerts and hourly forecast used though. :D
jsmorley wrote:
February 13th, 2020, 1:33 am
Man, I'm personally steering clear of MoonSet / MoonRise times. They are a challenge to understand.
1) The moon "sets" in the morning sometime, and then "rises" at night. Depending on the time that these happen, I'm not fully confident of the "day" that is returned in the timestamp... "night" can be in the PM of today, or the AM of tomorrow. I'm guessing the "date" part of the timestamp may just be a place-holder.
I disagree. It's not a challenge, it's normal - that's how the moon behaves, it's not the sun, after all. The day part of the TWC timestamp is, from my observation, accurate - I just compared the 13 Feb and 20 Feb data for Reykjavik (Iceland) between TWC and timeanddate.com and, bar the rounding of minutes, they are the same (both the time and the date). What you said about the rise/set data could be said regarding the old wxdata, where they were horrendously messy, but I'm actually "over the moon" (so to speak) about the fact that the TWC data seems accurate in that respect. If you can briefly paste here an instance when the TWC data seems inaccurate in that regard, I would be interested to see it.
jsmorley wrote:
February 13th, 2020, 1:33 am
2) I have run into an instance where there is an entirely missing MoonSet / MoonRise timestamp. Instead of the literal null used elsewhere, in this case it is delineated by "" or an empty string. This is a challenge to deal with when using Time measures, as those just hate a TimeStamp option that is an empty string. It can be worked around, but if you are using these times to create some graphical representation of where the moon is in its daily cycle through the sky, this will take some thought indeed when data is missing.
Yes, the "" happens in some cases where you expect null. I'm using my old and improved regex way to deal with these: Substitute="(?i)(?:^$|^N\/A$|^null$)":"whatever unavailability text here". In case of dates, I just replace missing data with a standard neutral date (like 2000-01-01T00:00:00+0000) that is used as default even when connection to TWC cannot be established. For the graphical representation, just use moonIcon string, and not moonset/moonrise data - that's the best you can do, since there are no more "moon days" representation as wxdata used to have. Me, I got rid of my nice moon lit percent estimation as well from the "old" skin, as there is no suitable data for it to use in the "new" system.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

Yincognito wrote:
February 13th, 2020, 2:27 pm
That field is the only one not included in my skin. I have no need for max/min temperatures for the 24h day, since I've always used the day/night cycles (and their temperatures). It's true that visually, on the main page, there are these "Max" and "Min" temperatures for the 24h day (hence, taken as such by skin authors), but those are actually the day and night cycle's temps internally. From a purely technical point of view, there are no min or max temps (bar the "max temperature since 7am" you noticed, and that is for past tense) in TWC (and wxdata) sources, they just happen to label them as such on the main page since almost always the day cycle is warmer than the night one, everywhere on Earth.

Up to you of course. I find that [@CurrentMaxTemperatureSince7AM] works perfectly to maintain the display of today's "high" temperature even after the day has changed from "today" to "tonight". As you say, everywhere in the world, the maximum temperature during the day is going to be higher than at night. Personally, I kinda like having the "forecasted" OR "historical" value for "high" displayed for today, no matter what the hour. It's why they have that in the JSON, and I like to take advantage of it. Of course I don't use it until it actually is what weather.com treats as "night".

And really, it wouldn't matter a bit if for some reason there was a higher temperature at "night' than was ever achieved during the day. I doubt that would happen, but I guess it could, and [@CurrentMaxTemperatureSince7AM] would still reflect the overall day's "high" in that case. It's not constrained by "day" or "night".

What weather.com returns for the forecasted "day" temperature is in fact the "high" for the day, and the forecasted "night" temperature is in fact the "low" for the day. It's just what they mean, and what they can only mean. I just like that there is also a perfectly accurate way to deal with the fact that there can't be any "forecast" for "day" once it is "night". weather.com is one of the few sites or apps that throw away the display of "today's high" once that becomes "historical" instead of "forecasted" at some point in the day. Take a look at how Yahoo Weather shows it.

I disagree. It's not a challenge, it's normal - that's how the moon behaves, it's not the sun, after all. The day part of the TWC timestamp is, from my observation, accurate - I just compared the 13 Feb and 20 Feb data for Reykjavik (Iceland) between TWC and timeanddate.com and, bar the rounding of minutes, they are the same (both the time and the date). What you said about the rise/set data could be said regarding the old wxdata, where they were horrendously messy, but I'm actually "over the moon" (so to speak) about the fact that the TWC data seems accurate in that respect. If you can briefly paste here an instance when the TWC data seems inaccurate in that regard, I would be interested to see it.
Not saying that anything is inaccurate as such, just that the "date" part of the timestamps feels confusing to me. During a given day's cycle, the moon first "sets", generally at some point in the morning, but it might be in the early afternoon. Then the moon "rises", generally at some point in the night of that day, or the early morning of the next day. The "date" part of the timestamps always reflect today's date, when I'm not sure that is entirely, technically true. The moon "rise" time might well actually be tomorrow morning. I may be looking at this all wrong, and to be honest I just barely even care. I don't alter my daily schedule much based on what phase of the moon it is.

As far as the missing timestamp information, it doesn't really matter if they return null or "" as either is going to blow up a Time measure. What I do when this happens is to return the timestamp for the "day" in general, (always there) so the Time measures are happy and aren't hammering the error log, then just ignore it and set the value to "N/A" with the Format option on the Time measure. I gather you don't use Time measures in your stuff, but just parse the timestamp. That's up to you, but I prefer giving the option to the user to format date and time strings any way they like. I like "Thursday, February 13, 2020 at 10:02 am EST" myself, and you just can't get that parsing the timestamp.
User avatar
SilverAzide
Posts: 716
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SilverAzide »

jsmorley wrote:
February 13th, 2020, 2:47 pm
Not saying that anything is inaccurate as such, just that the "date" part of the timestamps feels confusing to me. During a given day's cycle, the moon first "sets", generally at some point in the morning, but it might be in the early afternoon. Then the moon "rises", generally at some point in the night of that day, or the early morning of the next day. The "date" part of the timestamps always reflect today's date, when I'm not sure that is entirely, technically true. The moon "rise" time might well actually be tomorrow morning. I may be looking at this all wrong, and to be honest I just barely even care. I don't alter my daily schedule much based on what phase of the moon it is.
I've been using this moon data to some extent, but it definitely IS confusing. I just ran into a specific case while up late one evening (after midnight), where the moon was still "up" but the moonrise time was later in the day and the moonset was the following day. So, for that particular date, the moonset time (the moon hadn't set yet) was before the moonrise time (in other words, on that particular date, the moon sets before it rises, which is something that never happens with the sun).

So if you are looking at a skin and the clock strikes midnight, do you flip the moon meters to show "today's" values (which could be wrong if the moon is still up, and unlike the sun will be radically different values), or do you come up with some crazy math to figure out which rise-set values to show depending on the current state of the moon?

This is more complicated than my poor brain can handle.
DeviantArt Gadgets More...
User avatar
Yincognito
Posts: 1072
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

jsmorley wrote:
February 13th, 2020, 2:47 pm
Up to you of course. I find that [@CurrentMaxTemperatureSince7AM] works perfectly to maintain the display of today's "high" temperature even after the day has changed from "today" to "tonight".
If I knew you were interested in it, I would have mentioned it from the start. :)
jsmorley wrote:
February 13th, 2020, 2:47 pm
What weather.com returns for the forecasted "day" temperature is in fact the "high" for the day, and the forecasted "night" temperature is in fact the "low" for the day. It's just what they mean, and what they can only mean.
That may well be. Point is, it's much less ambiguous if they are treated as "day" and "night" temperatures, since one doesn't have to worry about changing some temp's label if it's past 1 PM or 2 PM, or deal with the "max" temp data missing in the "day" section. When they are taken as such, the code becomes simpler with minimal impact for the user (everybody can figure out that the "day" / higher temp is the maximum one and the fact that the day temp missing is because a good part of the daytime has passed, after all). That being said, each skin designer has his own preferences and the general habit is to have a "max" and a "min" for every forecast anyway, and I'm not contesting them at all.
jsmorley wrote:
February 13th, 2020, 2:47 pm
Not saying that anything is inaccurate as such, just that the "date" part of the timestamps feels confusing to me. During a given day's cycle, the moon first "sets", generally at some point in the morning, but it might be in the early afternoon. Then the moon "rises", generally at some point in the night of that day, or the early morning of the next day. The "date" part of the timestamps always reflect today's date, when I'm not sure that is entirely, technically true.
I see what you mean. This might have an explanation though. As you may have already noticed, the 24h TWC "day" starts at 7 AM, when the updates and first weather measurements are done. Thus, the fact that the date part of a moon rise/set is still "today" (even though it's past midnight) might have something to do with the fact that the time part of the timestamp has not yet reached 7 AM of the next day (i.e. the start of the next "TWC day")...

Update: Just checked the next 2 weeks moon rise / set times for my location, and when data was missing (e.g. no moonrise on 14 Feb) at TWC, it was missing at timeanddate.com as well. So it appears such instances are normal, and it's probably best for us non-scientists to not bother with too many "why?"-s and "how?"-s on this and just accept that these things happen (and should happen) from an astronomical point of view.
Last edited by Yincognito on February 13th, 2020, 6:40 pm, edited 1 time in total.
User avatar
Yincognito
Posts: 1072
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

SilverAzide wrote:
February 13th, 2020, 5:07 pm
I just ran into a specific case while up late one evening (after midnight), where the moon was still "up" but the moonrise time was later in the day and the moonset was the following day. So, for that particular date, the moonset time (the moon hadn't set yet) was before the moonrise time (in other words, on that particular date, the moon sets before it rises, which is something that never happens with the sun).
I might have poorly understood your case, but if you said the moon "was still up" when you looked at it, shouldn't it have to set before it could rise again? Therefore, set before next rise... :???:
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

Yincognito wrote:
February 13th, 2020, 5:55 pm
Update: Just checked the next 2 weeks moon rise / set times for my location, and when data was missing (e.g. no moonrise on 14 Feb) at TWC, it was missing at timeanddate.com as well. So it appears such instances are normal, and it's probably best for us non-scientists to not bother with too many "why?"-s and "how?"-s on this and just accept that these things happen (and should happen) from an astronomical point of view.
So tomorrow the moon will be taking a break and just not rising?