fedcode-next: Extract unpublished vulnerabilities from commit histories and trackers

[pombredanne]

We should extract unpublished vulnerabilities from commit histories and issue trackers

• Parse Git commit messages
  • To include fix commits VCIO-next: Add support to track fix commits: Include commits and patches that fix a vulnerability #207
  • To find new or existing vulnerabilities
    • https://github.com/CERTCC/git_vul_driller
    • https://github.com/Ananya-0306/vuln-finder
    • https://github.com/aklyussef/VulnerabilityPatchFinder
    • https://github.com/samaritan/archeogit
    • https://github.com/cve-search/git-vuln-finder

Beyond, that we could:

• Parse issues and trackers such as github issues. This is tracked in Process unstructured data sources, such as issues #251
• Parse CHANGELOGs. See Parse CHANGELOGs to discover new Vulnerabilities #233 and https://github.com/pyupio/changelogs/

These are valuable information and we can search for CVE and security-related keywords and track these in a curation queue. And eventually submit these as NVD CVEs.

See also:

• Extract interesting data from CVE and other vulnerabilities body #551

[pombredanne]

We now can parse Git commit messages for vulnerabilities.

• The main PR is Add support for parsing Git commit messages #1992

This is an initial script to parse Git commit messages that can be easily integrated with our model. The script takes a Git repository as input, parses all commits, and returns the CVEs along with their corresponding fixed commits.

We now have a "collect_fix_commits" importer, and we import fix commits from at least these repositories:

• Add support for parsing Git commit messages #1992 (comment)