Vulnerablecode useragent

[Hritik14]

Much of what we're doing now is scraping the webpages for the required information. Doing such activity without even declaring who we are might not be seen as a very good practice.
We need to settle on a useragent choice. Some websites even block regular scraper useragents. There are two solutions I can think of:

• One consistent and custom useragent
• A list of browser user agents

There could exist benefit of either methods but I'm not so sure.

References:
https://forum.mattermost.org/t/mattermost-website-returning-403-when-headers-contain-the-word-python/11412

[pombredanne]

@Hritik14 thanks!
I am split on this one.

• on the one hand, I want to be well behaved and polite with our own agent string
• on the hand hand, condoning web sites that prohibits certain agents like mattermost without a good reason would go entirely against our mission

I think we should have this may:

1. a default of our own defined user agent such as VulnerableCode
2. an alternative way to provide a config with a list of other user agents we could use (and rotate through)

So may be the best way is to have a config TBD that provides a list of user agent strings and would default to VulnerableCode

[Hritik14]

condoning web sites that prohibits certain agents

I think we should contact the providers and have their concent about the scraping (maybe we can even maintain a document as proofs of consents for later use). I'm confident most of them would agree and would not blacklist our own defined user agent.

best way is to have a config TBD that provides a list of user agent strings and would default to VulnerableCode

This sounds good to me too.

[warrior-hub]

Hi @pombredanne 👋
I’d like to work on this issue.

I will start by adding a default User-Agent for all web requests
and make it configurable via environment variables.

Thanks!

[Mahaboobunnisa123]

Hi @pombredanne, I would like to take this issue and work on it. I will analyze everything and comes with a PR ready soon.