It is currently November 15th, 2019, 3:54 am

Forcing a User Agent String with cURL

Tips and Tricks from the Rainmeter Community
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Forcing a User Agent String with cURL

jsmorley » May 30th, 2016, 3:34 pm

Edit: June 1, 2016 The functionality to set a custom User Agent String has been added to the WebParser plugin in the latest 4.0 beta version of Rainmeter, and there is no need to use cURL for this purpose any longer. I will leave this Tips & Tricks however, as there may be other reasons you might want to use features of cURL with Rainmeter, and this shows how you might do so.

User Agent String

One thing you may very rarely run into when using WebParser to parse a website, is that the site reacts to the browser or program you are connecting with, and might format the output differently depending on what capabilities it assumes you have. This is done by the site using the User Agent String that is sent as part of the HTTP header when connecting.

WebParser uses a User Agent String of "Rainmeter WebParser plugin" when accessing a site. If you wish to connect to a site in Rainmeter using a different User Agent String, you can do this by using cURL to get the entire site into a file, and then you can still use WebParser to parse this local file and return whatever results you want.

Getting cURL

What you need to do is go to https://curl.haxx.se/download.html, scroll down and find "Win32 - Generic" or "Win64 - Generic" as you desire, and select the latest version as a .zip file.

Unzip this file, and put it somewhere in your skin's @Resource folder where you can access it.

What User Agent String to use
You can decide what User Agent String you want to use by connecting to https://www.whatismybrowser.com/detect/what-is-my-user-agent in your browser and copying the string displayed there, or connect to http://www.useragentstring.com/pages/useragentstring.php if you want to manually choose one of many, many browsers and programs.

The approach

What we will do is use the RunCommand plugin to execute that curl.exe that we put in @Resources. We will pass it a URL and some parameters, and have it output the results to a file. We will then use the WebParser plugin to access this local file and parse it as we normally would.


An Example
This .rmskin includes version 7.48.0 32bit of curl

UserAgentString_JSMorley.rmskin
1.jpg

Code: Select all

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1

[Metadata]
Name=UserAgentString
Author=JSMorley
Version=May 30, 2016
License=Creative Commons Attribution-Non-Commercial-Share Alike 3.0
Information=Demonstrates using cURL.exe (included) to force a particular User Agent String when accessing and downloading a remote resource.

[Variables]
UserAgentString=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36
URL=https://www.whatismybrowser.com/detect/what-is-my-user-agent
OutputFile=#CURRENTPATH#MyAgent.html

[MeasureGetSiteWebParser]
Measure=Plugin
Plugin=WebParser
URL=#URL#
RegExp=(?siU)<h1>What is my User Agent?.*<div class="value">(.*)</div>
StringIndex=1
FinishAction=[!CommandMeasure MeasureGetSiteCURL "Run"]

[MeasureGetSiteCURL]
Measure=Plugin
Plugin=RunCommand
Program=""#@#curl\curl.exe""
Parameter=-s -k -A "#UserAgentString#" "#URL#"
OutputFile=#OutputFile#
OutputType=ANSI
FinishAction=[!CommandMeasure MeasureResultFile "Update"]

[MeasureResultFile]
Measure=Plugin
Plugin=WebParser
URL=file://#OutputFile#
RegExp=(?siU)<h1>What is my User Agent?.*<div class="value">(.*)</div>
StringIndex=1

[MeterResults]
Meter=String
MeasureName=MeasureGetSiteWebParser
MeasureName2=MeasureResultFile
FontSize=10
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
AntiAlias=1
InlineSetting=Weight | 700
InlinePattern=WebParser User Agent String
InlineSetting2=Weight | 700
InlinePattern2=cURL User Agent String
Text=WebParser User Agent String#CRLF#%1#CRLF##CRLF#cURL User Agent String#CRLF#%2 
So this will demonstrate the result with normal WebParser, and with cURL and a defined User Agent String.

The parameters I used on curl.exe are:
-s : Silent (no download progress & statistics)
-k : Allow insecure connection on https
-A : Set User Agent String

Note that I am not advocating using this approach to bypass checks that a site may do to "force" you to connect to it via a known browser, in order to ensure you at least are presented with advertising or other information displayed on the site. While this is very, very rare, it would be my opinion that this should be respected.
You do not have the required permissions to view the files attached to this post.
User avatar
jsmorley
Developer
Posts: 19710
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Forcing a User Agent String with cURL

jsmorley » May 31st, 2016, 3:28 pm

So the very day I used that site to detect your User Agent Sting as an example, they changed it to obfuscate the results from parsing via HTML. ;-)

I have corrected the skin and .rmskin to use a different site.