My concerns with this project as a whole.

Dear Bugcrowd,

First of all, I would like to thank your team for opensourcing this project and allowing members of the community to contribute to the development process. I have thought long and hard about the core concept of this project and have come to the conclusion that there are some big changes that I would love to see Bugcrowd make in their approach. I will break down my two major concerns and a minor one into individual sections and try my best to always give some suggestions as to how you might be able to resolve these issues.

Associating the security vulnerability class with a score

This is the biggest point by far, because this is related to the actual idea behind this project. The VRT currently associates a vulnerability class as seen on the OWASP Top 10 list with a severity score. In my opinion, and I have said it many times before in the past (https://twitter.com/EdOverflow/status/931538905668702209), programs should not take this approach when evaluating the bounty amount and sadly this is far too common (I have made this mistake in the past too). Researchers are left disappointed when all of a sudden the program sets a lower priority score for something that the VRT told them was a P2, for instance. This has not only caused confusion within the community (I regularly get DMs about this issue), researchers are starting to falsely state what the issue type is hoping that they will get a higher score.

The priority score should be determined by the impact and what the company’s threat model is. Broken Cryptography > Cryptographic Flaw > Incorrect Usage having access to test data “usually” does not have the same impact as the same issue allowing the hacker to access PII. Therefore my suggestion is that you rethink the core concept of this project and consider redesigning it so that the impact has a direct correlation with the score and not the security vulnerability category. On top of that, it could even help if Bugcrowd added some sort of functionality that enables the program to supply the platform with their threat model and ensure that this influences the score.

Your README.md does state the following, but personally I see this as a way to simply unintentaionally cover up the real issue here:

It is important to remember that while the recommended priority, from P1 to P5 might apply without context, it’s possible that application complexity, bounty brief restrictions or unusual impact could result in a different rating.

As stated earlier, context should be foremost when evaluating an issue.

In my opinion CWE, summarize this concern best: “Prioritizing Weaknesses Based Upon Your Organization’s Mission”. ^[1] This brings me to my next section.

Missing Common Weakness Enumeration

After considering the first section, I am hoping that the security categories would be a separate option in the submission process as seen on HackerOne. HackerOne currently uses the Common Weakness Enumeration list in section 2 of the submission form. This has already been suggested to you before in https://github.com/bugcrowd/vulnerability-rating-taxonomy/issues/99, but personally I would like to see this as a separate dropdown option on Bugcrowd, therefore you might need to restructure the current VRT JSON mapping in order to accommodate for this change.

I would like to emphasize the fact from section one again, this option should not directly influence the priority score.

Rewording the CONTRIBUTING.MD file

Currently you state the following:

Please open your feedback as an Issue and label it as either a bug or an enhancement.

This might be confusing for some as only members of the Bugcrowd GitHub organization can label issues. For example, I cannot label the following ticket.

In addition, I would suggest increasing the number of labels so that it is easier to refine one’s search while looking for previously submitted tickets. You may use https://github.com/securitytxt/security-txt as an example of how this can be done.

[barnett]

Hey @EdOverflow,

Sorry for my absence, in this PR since the last comment. As per CVSS vs VRT, I understand your concerns and we will take it into consideration on future CVSS related work. At this point I think the conversation can be closed and we can bring up individual issues within here as they come up. Thanks for taking the time to provide feedback and we look forward to seeing you around the platform!

[barnett]

Associating the security vulnerability class with a score

👍

Missing Common Weakness Enumeration

Admittedly, the Base Score is the only metric required by the National Vulnerability Database (NVD), but research has shown that evaluating the bounty amount based on the technical severity — the Base Score — only, is inaccurate and has a tendency to under evaluate the bounty amount.

While you have pointed to an outlier program, the platform does not set technical severity based on CVSS unlike we do with VRT. So while customers can up/down grade the severity rating (we ask they provide reasoning to the researcher), we see the grand majority using VRT mapped severity to determine payouts. The utility of the CVSS/CWE and other mappings is for the companies’ own internal policies and processes. Due to that we have scoped the calculator to how most of our CVSS-using customers used it, within their internal workflow, only leveraging the base score.

On a side note, I would encourage you take a look at a recent piece that @tomnomnom and I published on evaluating the bounty amount based on all three CVSS metric groups: https://edoverflow.com/2017/the-math-behind-bug-bounties/.

I read it the day it was published 😄 . Definitely an interesting concept but at this time Bugcrowd is committed to using VRT as our de-facto scoring mechanism. We decided against CVSS due to the “opinion” nature of the scoring which the researcher is unable to debate on due to their limited knowledge of the company’s architecture/setup (ie black-box testing). With VRT and our upcoming changes to how we communicate rewards in relation to severity and “context” we are hoping to have a transparent rubric that is used to score a submission and land at a payment amount.

Rewording the CONTRIBUTING.MD file

I can see how question isn’t the best descriptor, I’ll add a needs discussion tag to better filter on similar issues.