It is currently March 28th, 2024, 11:18 am

⭐ Weather.com - Parsing the V3 JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

nikko wrote: June 22nd, 2020, 10:13 am tnanx for explanation y and js. one more question can you explain this most complex thing;
wxPhraseLong":\[(?:null,|".*"[,|\]]){2}(null|".*")[,|\]](null|".*")[,|\]].*
I've gone over what most of those characters do here:

https://forum.rainmeter.net/viewtopic.php?f=118&t=34628&start=510#p178928

The difference is that I needed to account for the fact that there could be commas in several of the fields, and just capturing (.*),, or any number of any character followed by a comma, risks having the value be truncated if a comma, like "rain, followed by sun" is in the data.

So for several of the string fields, I search for the null static value, which is not surrounded by quotes, OR ".*", which gets everything between quotes. That is then ended by either a comma or a square bracket.

Regular expression is not particularly for the faint of heart, but it is actually pretty perfect for what it's intent is. It is as "terse" as it can possibly be, to keep the expressions short, while being incredibly powerful, with the tools and logic needed to parse just about anything. I certainly wouldn't want to make a career out of writing it, but it has stood the test of time admirably, with nothing that has ever challenged it.

The only way to understand what a particular expression does is to mentally "walk through it", putting what it is saying into logical phrases in your head. "search for this, followed by this or that, ending with this". Regular expression is probably the most purely "logical" exercise you will ever do in programming.

The first key to is is knowing what the reserved characters, the "vocabulary" of the thing, mean:

https://www.debuggex.com/cheatsheet/regex/pcre
That site loads kinda slow, but be patient, it's one of the best "cheatsheets".
User avatar
Yincognito
Rainmeter Sage
Posts: 7017
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

nikko wrote: June 22nd, 2020, 10:13 am tnanx for explanation y and js. one more question can you explain this most complex thing;
wxPhraseLong":\[(?:null,|".*"[,|\]]){2}(null|".*")[,|\]](null|".*")[,|\]].*
To add to what jsmorley said, besides the all known RegEx101 (which, despite being the most featured, has some things that I don't like regarding the small size of the "window docks" - having them adjustable doesn't compensate for that drawback, in my view), another online regex tester is RegExr, which I use almost exclusively to build and test my regular expressions. Once you set the regex "flavor" (PCRE instead of Javascript in our case, since this is used in Rainmeter) from the top right corner, you're good to go. For example, in the case of the regex you asked about, this is how it would look like (pay attention to what buttons / options / left-side sidebar menus are selected and the text I've written in the middle, as I intentionally left them that way):
RegExr.jpg
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Right, and the last note I would make is to remember that with the non-capture group (?: ...), when you specify an "instance" of the group to "skip to", using {n} as a "quantifier", the number in the {} is "zero-based", in that what you are saying is "skip this many". To get the first instance, you don't want to skip any, so the number is {0}. To get the second instance, you want to skip one, so the number is {1}.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

In any case, it has been stable for 2-3 weeks now, and I'm cautiously hopeful.

I'm just keeping an eye on things, since the "sections" that are supported in the JSON are:

getSunV3LocationPointUrlConfig
getSunWeatherAlertHeadlinesUrlConfig
getSunV3HourlyForecastUrlConfig
getSunV3CurrentObservationsUrlConfig
getSunV3CurrentDateTimeByGeocodeUrlConfig
getSunV3DailyForecastUrlConfig
getSunV3HistoricalOneDayHourlyConditionsUrlConfig
etSunV3GlobalAirQualityUrlConfig
getSunIndexPollenDaypartUrlConfig
getSunV3SnowSkiConditionsUrlConfig
getSunIndexRunWeatherDaypartUrlConfig
getSunV3TidalPredictionsUrlConfig

And I'm curious to see if there is a change to

getSunIndexPollenDaypartUrlConfig

Which isn't titled with the "V3" yet, so I'm holding off on that for a bit to see if that gets an update as well.

Alerts are an entirely different animal, and honestly, I just couldn't care less about their opinion on "running weather", there is no "good" weather for running... but Pollen is of some interest to a lot of folks.
User avatar
Yincognito
Rainmeter Sage
Posts: 7017
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

jsmorley wrote: June 22nd, 2020, 1:01 pmAnd I'm curious to see if there is a change to

getSunIndexPollenDaypartUrlConfig

Which isn't titled with the "V3" yet, so I'm holding off on that for a bit to see if that gets an update as well.
I don't think you need to worry about that (besides the title name change, which is trivial to adapt to), since that section is already structured like all V3 ones (i.e. only one field name occurence and its values in an [array]). You could have worried if the structure was the "old type" (i.e. as much occurences of the field name as there are days / day parts, stuff present in {sections} instead of [arrays], etc).

As I said, these structures are basically matrices, where the V2 enumerated things "column by column", so to speak, and V3 is enumerating them "row by row":

-------------------------Day 1------------------Day 2-----------------Day 3-------
Field 1---------ValueDay1Field1-----ValueDay2Field1-----ValueDay3Field1
Field 2---------ValueDay1Field2-----ValueDay2Field2-----ValueDay3Field2
Field 3---------ValueDay1Field3-----ValueDay2Field3-----ValueDay3Field3
User avatar
Defrilitus
Posts: 1
Joined: February 1st, 2020, 4:17 pm

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Defrilitus »

I have been a Rainmeter user for nearly fifteen years, but must admit that the last weather.com update left me completely nonplussed. I would therefore like, first of all, to thank JS Morley & Co for all their (successful) efforts to make the new JSON weather version accessible to all. I freely admit that, without JS Morley, I would have been completely lost ;-)

Considering the above, I would like to give something back to the community and I remember that, some months ago, there was a question about time zones.

I like to see the interval between the last update and the current local time (pure laziness, of course!). This is, of course, simple within the same time-zone; but becomes more difficult when different time-zones are involved (every so often I like to check the weather in another part of the world).

The alphabetic time-zone abbreviations provided by weather.com (e.g., “EDT”) are absolutely useless, as they can mean anything and everything. I therefore extract the numeric time-zone (e.g., +0200) from each weather.com forecast page that I consult, and use this to calculate the time elapsed between the last update and my (local) system time.

My solution may be somewhat clunky (regular expressions are not my cup of tea), but it works; and I think that showing a location's time-zone is important. By the way, I would be happy to give any explanations needed; in the meantime, please feel free to use/adapt/improve as you wish.

One caveat: I have no idea how the measures/meters below would work with American formatting (e.g., "am").

Code: Select all

;WeatherComJSONMeasures

[@LocationTimeZone]
Measure=WebParser
URL=[@CurrentConditionsParent]
StringIndex=17
RegExpSubstitute=1
Substitute="+0000":"±0000","^.*(......)$":"\1",'"':""

; Measures

[@LocationTimeAdjusted]
; Location time adjusted for calculations
Measure=Time
TimeStamp=[@CurrentObservationTimeStamp]
TimeStampFormat=%Y-%m-%dT%H:%M
DynamicVariables=1

[@LocationZoneCalc]
; Location time-zone formatted for calculations
Measure=Calc
Formula=([@LocationTimeZone] * 0.01)
DynamicVariables=1
Substitute=".15":".25",".3":".5",".45":".75","±0000":"0"

[@LocalTimeAdjusted]
; Local (system) time adjusted for calculations using location time-zone
Measure=Time
TimeZone=[@LocationZoneCalc]
Format=%Y-%m-%dT%H:%M:%S
DynamicVariables=1

[@TimeDifference]
; Difference between location time and local (system) time, calculated using location time-zone for both
Measure=Calc
Formula=([@LocalTimeAdjusted:TimeStamp]-[@LocationTimeAdjusted:TimeStamp])
; Change MeterTimeElapsed text if time difference > 24h
IfAboveValue=86399
IfAboveAction=[!SetOption MeterTimeElapsed Text "> 24h"]
; Otherwise keep things as they are
IfBelowValue=86400
IfBelowAction=[!SetOption MeterTimeElapsed Text "%1"]
DynamicVariables=1

[@TimeElapsed]
; @TimeDifference formatted for display
Measure=Time
TimeStamp=[@TimeDifference]
Format=%#Hh %#Mm
DynamicVariables=1
RegExpSubstitute=1
Substitute="^0h 0m":"< 1m","^0h ":""

; Meters

[SeparatorUpdate]
Meter=IMAGE
MeterStyle=sSeparator

[MeterUpdateString]
Meter=STRING
Text="Last update"

[MeterUpdate]
MeasureName=@CurrentObservationTime
Meter=String
Text=%1

[SeparatorTimeElapsed]
Meter=IMAGE
MeterStyle=sSeparator

[MeterTimeElapsedString]
Meter=STRING
Text="Time elapsed"

[MeterTimeElapsed]
MeasureName=@TimeElapsed
Meter=String

[SeparatorTimeZone]
Meter=IMAGE
MeterStyle=sSeparator

[MeterTimeZoneString]
Meter=STRING
Text="Time zone"

[MeterTimeZone]
MeasureName=@LocationTimeZone
Meter=String
Thanks again for all your help.
Last edited by Defrilitus on July 1st, 2020, 8:22 pm, edited 2 times in total.
EeK
Posts: 11
Joined: January 28th, 2020, 1:26 am

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by EeK »

Guys, did weather skins break again? Mine stopped working yesterday. I even posted here about it, with a screenshot of my desktop, but, apparently, it was removed.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

EeK wrote: July 31st, 2020, 2:38 am Guys, did weather skins break again? Mine stopped working yesterday. I even posted here about it, with a screenshot of my desktop, but, apparently, it was removed.
https://forum.rainmeter.net/viewtopic.php?f=13&t=35773
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Ok, I think I have fully addressed the new issues with the weather.com site.

Please get the .rmskin in the first post of this thread.


ONE IMPORTANT NOTE!


The new approach of using the API for the weather.com JSON requires a change to how you access a particular location.

All of the location "codes", like USVA0944 or bbd510e73cb5870f973cea9253bb1f9bfe239a129dfc2b89589cbb6d2d46b4bc or 22308 and such are NO LONGER SUPPORTED.

The new approach requires that you provide a Latitude and Longitude in the WeatherComJSONVariables.inc file. See that file for details on how you can get that for your location.

Please read WeatherCOMJSONVariables.inc carefully, it HAS changed.

Code: Select all

[Variables]

; NOTE that Location Codes like USVA0944 or other types of codes are no longer supported.

; You must use location Latitude and Longitude.

; You can right click the skin and select "Choose location" to use a litle skin I wrote.
; Simply type any search string into "Search For Location", hit Enter, then click
; the magnifying glass at the bottom left.
; Once you have found the location you want, click on the Latitude,Longitude numbers
; on the right to set that value and refresh the main weather skin.
; Click the Name of the location on the left to see the location on a map in your browser.
; 
; These values can also be found manually for your location at:
; https://nominatim.openstreetmap.org/search.php?q=Your Address Text&format=json
; Where "Your Address Text" can be pretty much anything that can zero in on the location you want.
; https://nominatim.openstreetmap.org/search.php?q=Fort Hunt, Virginia&format=json
; https://nominatim.openstreetmap.org/search.php?q=22308&format=json
; https://nominatim.openstreetmap.org/search.php?q=The White House, Washington D.C.&format=json
; Then copy the lat: and lon: values from that site and paste them below.

Latitude=38.723249
Longitude=-77.069950
;Fort Hunt, Virginia
;Latitude=38.723249
;Longitude=-77.069950
;Auburn, Alabama
;Latitude=32.5971687
;Longitude=-85.5565993

;Latitude=25.7742658
;Longitude=-80.1936589

; Language used for string values returned in the data.
; If another language is desired the correct code can be found at:
; https://admin.rainmeter.net/LanguageCodes.php
; For example, French as used in France would be Language=fr-FR

Language=en-US
;Language=fr-FR
;Language=hu-HU

; Units of measure. This must be one of the following lower case letters:
; e		This is "Imperial", using Fahrenheit for temperatures.
; m		This is "Metric", using Celsius for temperatures.
; h		This is "Hybrid", used primarily in the UK. Celsius for temperatures, but miles for distances.

Units=e
;Units=m
;Units=h

; How often to update the weather information. Default and recommended is 600, which would be 10 minutes.

UpdateRate=600

; Date and time formats you desire.
; See https://docs.rainmeter.net/manual/measures/time/#FormatCodes 
; For formatting codes to use.

DateFormat=%A, %B %#d, %Y
TimeFormat=%#I:%M %p

; Don't change any below here.

APIKey=d522aa97197fd864d36b418f39ebb323

CommonSubstitute='"':"","^null$":"","\\u002F":"/","\\u005C":"\","\\u003C":"<","\\u003E":">"
URLSite=https://api.weather.com/v3/aggcommon/v3-wx-observations-current;v3-wx-forecast-daily-15day;v3alertsHeadlines;v3-location-point?format=json&geocode=#Latitude#,#Longitude#&units=#Units#&language=#Language#&apiKey=#APIKey#
Attachments
@IncludeFiles.zip
(59.25 KiB) Downloaded 326 times
jazde86
Posts: 1
Joined: July 31st, 2020, 9:31 pm

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jazde86 »

Hey jsmorley, thanks for the update. I would prefer if you could replace the LocationName with the City string, as the website does not resolve to the precise place where you live too. Or add it as LocationCity maybe, because it's in the JSON available. More options are always welcome.

Thanks for your effort.
Post Reply