WebParser XMLHttpRequest for weather forecast

[emp00]

Dear Team,

I'd like to use WebParser to call in principle this page and collect/parse specific weather data (hourly UV-index and cloud cover):
https://14-tage-wettervorhersage.de/wetter/paris/stuendlich/623/

In the page source I found this section describing the fetchURL with the required parameters (here for Paris/France as an example):

Code: Select all

function loadNext(pagination) {
	var fetchURL = '/get/get_hourly_next.php';
    const params = new URLSearchParams();
    
    params.append('cityid', '623');
    params.append('city', 'paris');
    params.append('tab', 'stuendlich');
    
    if (pagination !== 0) {
        params.append('s', pagination);
    }

	fetch(fetchURL, {
		method: 'POST',
		headers: {
			'X-Requested-With': 'XMLHttpRequest',
			'Content-Type': 'application/x-www-form-urlencoded',
		},
		body: params
	})
	.then(response => response.text())
  .then(html => {

This XMLHttpRequest obviously pulls the hourly weather data I'm interested in, found in tables towards the end of the page source. I'm focused on the "drawWolken"-datatable: This is the data I would like to parse and display in a Rainmeter gadget ==> Cloud cover in % und the UV-index for the coming 24 hours.

Code: Select all

function drawWolken() {
var dataTable = new google.visualization.DataTable();
dataTable.addColumn('string', 'Stunde');
dataTable.addColumn('number', 'Wolken');
// A column for custom tooltip content
dataTable.addColumn({'type': 'string', 'role': 'tooltip', 'p': {'html': true}});
dataTable.addColumn('number', 'UV-Index');
// A column for custom tooltip content
dataTable.addColumn({'type': 'string', 'role': 'tooltip', 'p': {'html': true}});
dataTable.addRows([
['Do. 22:00',57,'57 %',0,'0 (Niedrig)'],
['Do. 23:00',84,'84 %',0,'0 (Niedrig)'],
['Fr. 00:00',80,'80 %',0,'0 (Niedrig)'],
['Fr. 01:00',76,'76 %',0,'0 (Niedrig)'],
['Fr. 02:00',84,'84 %',0,'0 (Niedrig)'],
['Fr. 03:00',92,'92 %',0,'0 (Niedrig)'],
['Fr. 04:00',100,'100 %',0,'0 (Niedrig)'],
['Fr. 05:00',100,'100 %',0,'0 (Niedrig)'],
['Fr. 06:00',100,'100 %',0,'0 (Niedrig)'],
['Fr. 07:00',100,'100 %',0,'0 (Niedrig)'],
['Fr. 08:00',100,'100 %',0,'0 (Niedrig)'],
['Fr. 09:00',100,'100 %',0,'0 (Niedrig)'],
['Fr. 10:00',100,'100 %',1,'1 (Niedrig)'],
['Fr. 11:00',99,'99 %',1,'1 (Niedrig)'],
['Fr. 12:00',99,'99 %',1,'1 (Niedrig)'],
['Fr. 13:00',94,'94 %',1,'1 (Niedrig)'],
['Fr. 14:00',84,'84 %',1,'1 (Niedrig)'],
['Fr. 15:00',76,'76 %',2,'2 (Niedrig)'],
['Fr. 16:00',76,'76 %',1,'1 (Niedrig)'],
['Fr. 17:00',76,'76 %',1,'1 (Niedrig)'],
['Fr. 18:00',76,'76 %',0,'0 (Niedrig)'],
['Fr. 19:00',37,'37 %',0,'0 (Niedrig)'],
['Fr. 20:00',40,'40 %',0,'0 (Niedrig)'],
['Fr. 21:00',42,'42 %',0,'0 (Niedrig)'],
]);

Now the question: How do I need to construct a WebParser call to pull exactly this datatable?

Help is much appreciated... I'm not a specialist in php unfortunately...

[balala]

Now the question: How do I need to construct a WebParser call to pull exactly this datatable?

Help is much appreciated... I'm not a specialist in php unfortunately...

Me neither, but don't even have to be.
First if you want to get the whole database as one, it's a little bit simpler. But I think getting all those elements (what there are between the brackets) one by one is a much better solution.
To get them this way, first you need a parent WebParser measure, to acquire the online data. Something like this:

Code: Select all

[MeasureRainmeter]
Measure=WebParser
UpdateRate=600
Url=https://14-tage-wettervorhersage.de/wetter/paris/stuendlich/623/
RegExp=(?siU)function drawWolken.*\(\[.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],

Each of the elements in the RegExp option (the .*\[(.*)], elements) return one such data. I'm not sure if the number of those data can vary. If it can, there are additional solutions to ensure the code doesn't fail, but this is another story, for later.
If you have the above parent WebParser measure, you need one for each such element. I post here only three, but if you need more, you obviously can and have to add more such measures:

Code: Select all

[MeasureDataBase1]
Measure=WebParser
Url=[MeasureRainmeter]
StringIndex=1

[MeasureDataBase2]
Measure=WebParser
Url=[MeasureRainmeter]
StringIndex=2

[MeasureDataBase3]
Measure=WebParser
Url=[MeasureRainmeter]
StringIndex=3

Finally you need a meter, to get visible what has been acquired. For instance use this meter:

Code: Select all

[MeterData]
Meter=STRING
MeasureName=MeasureDataBase1
MeasureName2=MeasureDataBase2
MeasureName3=MeasureDataBase3
X=0
Y=0
Padding=15,5,15,5
FontColor=220,220,220
FontEffectColor=0,0,0
StringEffect=Shadow
SolidColor=0,0,0,150
FontSize=8
FontFace=Segoe UI
StringStyle=BOLD
StringAlign=LEFT
AntiAlias=1
Text=1. %1#CRLF#2. %2#CRLF#3. %3
DynamicVariables=1

To get this properly working, you need one more thing: a DynamicWindowSize=1 option onto the [Rainmeter] section. Add it, to be sure the skin is properly resized when the online data has been got.
Just to make sure you can use it properly, here is the whole code. Give it a try please and let me know if it does what you expect:

Code: Select all

[Rainmeter]
DynamicwindowSize=1

[MeasureRainmeter]
Measure=WebParser
UpdateRate=600
Url=https://14-tage-wettervorhersage.de/wetter/paris/stuendlich/623/
RegExp=(?siU)function drawWolken.*\(\[.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],.*\[(.*)],

[MeasureDataBase1]
Measure=WebParser
Url=[MeasureRainmeter]
StringIndex=1

[MeasureDataBase2]
Measure=WebParser
Url=[MeasureRainmeter]
StringIndex=2

[MeasureDataBase3]
Measure=WebParser
Url=[MeasureRainmeter]
StringIndex=3

[MeterData]
Meter=STRING
MeasureName=MeasureDataBase1
MeasureName2=MeasureDataBase2
MeasureName3=MeasureDataBase3
X=0
Y=0
Padding=15,5,15,5
FontColor=220,220,220
FontEffectColor=0,0,0
StringEffect=Shadow
SolidColor=0,0,0,150
FontSize=8
FontFace=Segoe UI
StringStyle=BOLD
StringAlign=LEFT
AntiAlias=1
Text=1. %1#CRLF#2. %2#CRLF#3. %3
DynamicVariables=1

[emp00]

Dear @balala, thank you for the help - your code works - BUT:

- It parses only the initially loaded page and this only contains the coming 24 hours of hourly data
- My target is to grab and display the hourly data of the coming 3 days: this data is actually available for German cities
- Example Frankfurt: https://14-tage-wettervorhersage.de/wetter/frankfurt-am-main/stuendlich/168720/
- Here, in the initially loaded page also only the first 24 hour data is loaded into the page code

- To get the data for the next 24 hours you "only" need to click on the button "vor »", you can click twice (or back = "« zurück")
- This button calls the script-function loadNext(pagination) that you can see in my initial posting
- Here the XMLHttpRequest (see initial posting) pulls the data in question! 1 click = +24 hours!
- My stupid idea was to call this XML-Request directly with WebParser, so that I can automatically pull the weather-data for 3 days ...

I have not found any Rainmeter gadget showing an hourly cloud-cover/UV-index trend... By chance I found this website providing the data in question and therefore I thought we could find a way to parse this into Rainmeter. In case this is too difficult, am I missing any existing gadget which already has the weather information in question? My application is photovoltaics for private purposes -> energy harvesting optimization

[balala]

Dear @balala, thank you for the help - your code works - BUT:

I'm glad.

- It parses only the initially loaded page and this only contains the coming 24 hours of hourly data

This is what I understood you want to get from your initial request.

- My target is to grab and display the hourly data of the coming 3 days: this data is actually available for German cities
- Example Frankfurt: https://14-tage-wettervorhersage.de/wetter/frankfurt-am-main/stuendlich/168720/
- Here, in the initially loaded page also only the first 24 hour data is loaded into the page code

- To get the data for the next 24 hours you "only" need to click on the button "vor »", you can click twice (or back = "« zurück")
- This button calls the script-function loadNext(pagination) that you can see in my initial posting
- Here the XMLHttpRequest (see initial posting) pulls the data in question! 1 click = +24 hours!
- My stupid idea was to call this XML-Request directly with WebParser, so that I can automatically pull the weather-data for 3 days ...

Sorry, I can't find the vor button, looking for it in vain. Where is it? A screenshot would be useful. Maybe just my ignorance, but it seems I really can't identify it. Sorry...

I have not found any Rainmeter gadget showing an hourly cloud-cover/UV-index trend... By chance I found this website providing the data in question and therefore I thought we could find a way to parse this into Rainmeter. In case this is too difficult, am I missing any existing gadget which already has the weather information in question? My application is photovoltaics for private purposes -> energy harvesting optimization

There are for sure skins (not gadgets - this is not a proper term in Rainmeter terminology) showing hourly weather conditions, but most probably they are using other data source. If you want to give it a try, here is an example. If you download and install it, when you activate it click the H button to get hourly conditions:

Screenshot 2024-03-15 230255.png

[emp00]

No problem @balala, thanks again for your reply. The "vor" button (loading +24h data) is indeed not so easy to see, here you go:



If you click on "Grafik+" you get a nice trend graphics -> my goal is to display such an hourly cloud cover & UV-index trend (for the next 3 days) on a Rainmeter skin, similar to what this website does:



I just tried out "Robi's Weather Skin" that you recommended, not bad! Thanks! It pulls data from weather.com, might be good as well, I need to check. However, this skin is "massive" - I prefer minimalistic skins displaying only the information that I need, so no moon phases etcpp... Just my preference.

Actually, I already use a modified variant of the AstroWeather skin! I trimmed it to my "minimalistic" preference, see pic below. So I already have a skin using weather.com data - However, I was not aware, that I can get more than 24 hour of data: AstroWeather does not have this option. For the cloud cover, it displays icons - I need to check what's behind numerically: Is it as "accurate" as the %-data from 14-tage-wettervorhersage.de above? Could I make a cloud trend graphics as well from this data? Also I'm still missing the hourly UV-index, do you know if this is available via the weather.com api as well?

If you have a concrete skin example showing an hourly ckoud+UV trend, similar to the "Grafik+" above and for the next 3 days - this would be perfect. Otherwise I'm willing to make my own skin (preferably by modifying my AstroWeather variant), but I need some help / documentation how to parse (only) the hourly cloud cover & UV-index numerical data. I can work best with skin examples and modify them - starting from scratch is hard for me.

My AstroWeather skin mod variant - here's a link to the .rmskin of course I'm willing to share

[eclectic-tech]

Open-Meteo can return cloud cover, UV-Index, and GHI or other Solar Radiation Variables.

Try this URL to see the JSON returned information (cloud cover and GHI are hourly for 7 days, UV-Index is daily):
https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&hourly=cloud_cover,shortwave_radiation&daily=uv_index_max

Then it can be dissected by Webparser or JSON parser.

EDIT: You can investigate all options here: https://open-meteo.com/en/docs

Hope this help in your project.

[emp00]

Open-Meteo can return cloud cover, UV-Index, and GHI or other Solar Radiation Variables.

Wow, excellent - thank you so much! I have also studied your posting here and I have installed Open-Meteo Weather_1.2024.01.24.rmskin ! Really, I'd like to switch from AstroWeather (weather.com) to your skin and the Open-Meteo api. Many thanks for the api call for "cloud cover and GHI are hourly for 7 days, UV-Index is daily" - this is exactly the data I want to display!

Stupid questions:
- Your skin "only" shows daily weather for 5 days. I noticed that you have created includes such as "OpenMeteoAPIMeasures7Day.inc" - but I have not understood how to change the skin so that it displays 7 days instead of 5 days weather? I read your instructions twice... Sorry!
- Does your skin already have code for hourly (instead of daily) weather display or do I need to set this up myself? I've seen your great documentation of measures - just asking if hourly meters are also already available?
- My target is to display 7 days of hourly cloud cover% and UV-index as a graphical trend: Lines or bars as Y-axis going up/down and X-axis simply day/hour? I really prefer visual trends over numbers and icons. Does anybody already have a skin which can show such hourly trends based on the Open-Meteo data? I'd love to start with something already close to my needs instead of starting from scratch.

(If you prefer, we can continue this discussion in your thread, I'm following this one now)

[eclectic-tech]

Wow, excellent - thank you so much! I have also studied your posting here and I have installed Open-Meteo Weather_1.2024.01.24.rmskin ! Really, I'd like to switch from AstroWeather (weather.com) to your skin and the Open-Meteo api. Many thanks for the api call for "cloud cover and GHI are hourly for 7 days, UV-Index is daily" - this is exactly the data I want to display!

Stupid questions:
- Your skin "only" shows daily weather for 5 days. I noticed that you have created includes such as "OpenMeteoAPIMeasures7Day.inc" - but I have not understood how to change the skin so that it displays 7 days instead of 5 days weather? I read your instructions twice... Sorry!
- Does your skin already have code for hourly (instead of daily) weather display or do I need to set this up myself? I've seen your great documentation of measures - just asking if hourly meters are also already available?
- My target is to display 7 days of hourly cloud cover% and UV-index as a graphical trend: Lines or bars as Y-axis going up/down and X-axis simply day/hour? I really prefer visual trends over numbers and icons. Does anybody already have a skin which can show such hourly trends based on the Open-Meteo data? I'd love to start with something already close to my needs instead of starting from scratch.

(If you prefer, we can continue this discussion in your thread, I'm following this one now)

There is no such thing as a "stupid" question so ask anything, I may not know the answer, but it will help everyone learn.

-=-
The includes are setup to return weather information based on how many days you need. To get 7 days' worth of weather information, comment out the current @include line and uncomment the 7-day line:

Code: Select all

; =====================================

; Include only one of these measures
; @includeForecast=#@#OPENMETEOAPIMeasures3Day.inc
; @includeForecast=#@#OPENMETEOAPIMeasures5Day.inc
@includeForecast=#@#OPENMETEOAPIMeasures7Day.inc
; @includeForecast=#@#OPENMETEOAPIMeasures10Day.inc
; @includeForecast=#@#OPENMETEOAPIMeasures.inc

You need to create the meters to show the additional data; the values are already in the additional measures for days 6 and 7.

-=-
The example skin is set to show hourly data (updated at 15 min intervals) for the current weather. The forecast are set up with daily information. Changing them to hourly is possible but the amount of data would probably choke the webparser so the number of items requested should be very selective (only a few data sets).

-=-
Open-Meteo can return hourly values. You specify which values you want, and at what time interval, in your URL:

api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&hourly=cloud_cover,shortwave_radiation&daily=uv_index_max

After &hourly= you specify what data you desire.

If you look at the documentation at Open-Meteo (the edited link I provided before), you can set what data you want and test right on that page.

If time permits, I will look into a simple example chart skin, but it may be a few days before I can get to that; too many "Honey-Do" projects

[Yincognito]

Besides eclectic-tech's reply concerning open-meteo.com...

Also I'm still missing the hourly UV-index, do you know if this is available via the weather.com api as well?

Yes, see here:
https://forum.rainmeter.net/viewtopic.php?t=34628&start=660#p212374
It can also be aggregated with other data, like:

Code: Select all

https://api.weather.com/v3/aggcommon/v3-wx-observations-current;v3-wx-forecast-hourly-15day;v3-location-point?geocode=LAT,LON&format=json&units=m&language=en-CA&apiKey=APIKEY

If you have a concrete skin example showing an hourly ckoud+UV trend, similar to the "Grafik+" above and for the next 3 days - this would be perfect. Otherwise I'm willing to make my own skin (preferably by modifying my AstroWeather variant), but I need some help / documentation how to parse (only) the hourly cloud cover & UV-index numerical data. I can work best with skin examples and modify them - starting from scratch is hard for me.

Of course everybody would love skin examples that one can just change a few things in and have it all, but the hard way is often the most productive / efficient one, and in some cases, even the easier one (I know, hard to believe, right?). To return what you need from the response in the link above is just about a RegExp=(?siU)\{"cloudCover":\[ARRAYPATTERN\],.*"uvIndex":\[ARRAYPATTERN\], in a WebParser measure, where ARRAYPATTERN would be something like (.*),(.*),(.*) but where the 3 (.*) capture parts would be as many hours there are in 15 days.

You might have to use both StringIndex and StringIndex2 in the WebParser measure when querying the data, so that you can get around the maximum of 99 possible captures in a StringIndex... option, but besides that, it's something quite simple to do (I'm referring to getting the data, the bells and whistles of the graphs are up to you since you know best how you'd like them to look).

I perfectly understand the idea of not reinventing the wheel and just grab the closest skin to what you want if it exists, but personally, I find it much easier to just write some 50 basic lines for a start on this, than looking through tons and tons of skins to identify which one matches my needs. That's just me though...

[eclectic-tech]

...
Of course everybody would love skin examples that one can just change a few things in and have it all, but the hard way is often the most productive / efficient one, and in some cases, even the easier one (I know, hard to believe, right?).
...
I perfectly understand the idea of not reinventing the wheel and just grab the closest skin to what you want if it exists, but personally, I find it much easier to just write some 50 basic lines for a start on this, than looking through tons and tons of skins to identify which one matches my needs. That's just me though...