Initial Documentation

We're forking from ScancodeIO docs <scancodeio.readthedocs.io/>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Author: Hritik14

README.rst

@@ -23,267 +23,44 @@ ecosystem.
 
 .. image:: README.gif
 
-Why?
-----
-
-The existing solutions are commercial proprietary vulnerability databases, which
-in itself does not make sense because the data is about FOSS (Free and Open
-Source Software).
-
-The National Vulnerability Database which is a primary centralized data source
-for known vulnerabilities is not particularly well suited to address FOSS
-security issues because:
-
-1. It predates the explosion of FOSS software usage
-2. It's data format reflects a commercial vendor-centric point of view in part
-   due to the usage of `CPE <https://nvd.nist.gov/products/cpe>`__ to map
-   vulnerabilities to existing packages.
-3. CPEs are just not designed to map FOSS to vulnerabilities owing to their
-   vendor-product centric semantics. This makes it really hard to answer the
-   fundamental questions "Is package foo vulnerable" and "Is package foo
-   vulnerable to vulnerability bar?"
-
-How
----
-
-VulnerableCode independently aggregates many software vulnerability data sources
-and supports data re-creation in a decentralized fashion. These data sources
-(see complete list `here <./SOURCES.rst>`_) include security advisories
-published by Linux and BSD distributions, application software package managers
-and package repositories, FOSS projects, GitHub and more. Thanks to this
-approach, the data is focused on specific ecosystems yet aggregated in a single
-database that enables querying a richer graph of relations between multiple
-incarnations of a package. Being specific increases the accuracy and validity
-of the data as the same version of an upstream package across different
-ecosystems may or may not be vulnerable to the same vulnerability.
-
-The packages are identified using Package URL `PURL
-<https://github.com/package-url/purl-spec>`__ as primary identifiers rather than
-CPEs. This makes answers to questions such as "Is package foo vulnerable
-to vulnerability bar?"  much more accurate and easy to interpret.
-
-
-The primary access to the data is through a REST API.
-
-In addition, an emerging web interface goal is to support vulnerabilities data
-browsing and search and progressively to enable community curation of the data
-with the addition of new packages and vulnerabilities, and reviewing and
-updating their relationships.
-
-We also plan to mine for vulnerabilities which didn't receive any
-exposure due to various reasons like but not limited to the complicated
-procedure to receive CVE ID or not able to classify a bug as a security
-compromise.
-
-Recent presentations:
-
-- `Open Source Summit 2020 <docs/Why-Is-There-No-Free-Software-Vulnerability-Database-v1.0.pdf>`__
-
-
-
-Setting up VulnerableCode
--------------------------
-
-First clone the source code::
-
-    git clone https://github.com/nexB/vulnerablecode.git
-    cd vulnerablecode
-
-
-Using Docker Compose
-~~~~~~~~~~~~~~~~~~~~~~
-
-Please find the docker documentation in `Docker Installation <https://vulnerablecode.readthedocs.io/en/latest/getting-started/docker_installation.html>`__
+Getting started
+---------------
 
-TL;DR
-""""""
+Run with Docker
+^^^^^^^^^^^^^^^^
 
 .. code-block:: bash
 
     git clone https://github.com/nexB/vulnerablecode.git && cd vulnerablecode
     make envfile
     docker-compose build
     docker-compose up
+    docker-compose run vulnerablecode ./manage.py import --list
 
-Go to http://localhost:8000/ on a web browser to access the web UI.
-
-
-Without Docker Compose
-~~~~~~~~~~~~~~~~~~~~~~
-
-**System requirements**
-
--  Python 3.8+
--  PostgreSQL 9+
--  Compiler toolchain and development files for Python and PostgreSQL
-
-On Debian-based distros, these can be installed with::
-
-    sudo apt-get install python3-venv python3-dev postgresql libpq-dev build-essential
-
-
-**Database configuration**
-
-- Create a user named ``vulnerablecode``. Use ``vulnerablecode`` as password
-  when prompted::
-
-    sudo -u postgres createuser --no-createrole --no-superuser --login \
-        --inherit --createdb --pwprompt vulnerablecode``
-
-- Create a databased named ``vulnerablecode``::
-
-    createdb --encoding=utf-8 --owner=vulnerablecode  --user=vulnerablecode \
-        --password --host=localhost --port=5432 vulnerablecode
-
-
-**Application dependencies**
-
-Create a virtualenv, install dependencies, generate static files and run the database migrations::
-
-    make envfile
-    python3 -m venv venv
-    source venv/bin/activate
-    pip install -r requirements.txt
-    python manage.py collectstatic
-    python manage.py migrate
-
-
-For a production mode, an environment variable named ``SECRET_KEY`` needs to be
-set. The recommended way to generate this key is to use the code Django includes
-for this purpose::
-
-    SECRET_KEY=$(python -c "from django.core.management import utils; print(utils.get_random_secret_key())")
-
-You will also need to setup the `ALLOWED_HOSTS` array inside `vulnerablecode/settings.py` according to
-[django specifications](https://docs.djangoproject.com/en/3.2/ref/settings/#allowed-hosts). One example would be:
-.. code-block:: python
-
-    ALLOWED_HOSTS = ['vulnerablecode.your.domain.example.com']
-
-You can specify several hosts by separating them with a comma (`,`)
-
-Using Nix
-~~~~~~~~~
-
-You can install VulnerableCode with `Nix <https://nixos.org/download.html>`__
-(`Flake <https://nixos.wiki/wiki/Flakes>`__ support is needed)::
+At this point, the VulnerableCode app should be running at port ``8000`` on your Docker host.
 
-    cd etc/nix
-    nix-shell -p nixFlakes --run "nix --print-build-logs flake check " # build & run tests
+Local development installation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-There are several options to use the Nix version::
-
-    # Enter an interactive environment with all dependencies set up.
-    cd etc/nix
-    nix develop
-    > ../../manage.py ... # invoke the local checkout
-    > vulnerablecode-manage.py ... # invoke manage.py as installed in the nix store
-
-    # Test the import prodecure using the Nix version.
-    etc/nix/test-import-using-nix.sh --all # import everything
-    # Test the import using the local checkout.
-    INSTALL_DIR=. etc/nix/test-import-using-nix.sh ruby # import ruby only
-
-
-**Keeping the Nix setup in sync**
-
-The Nix installation uses `mach-nix <https://github.com/DavHau/mach-nix>`__ to
-handle Python dependencies because some dependencies are currently not available
-as Nix packages. All Python dependencies are automatically fetched from
-``./requirements.txt``. If the ``mach-nix``-based installation fails, you might
-need to update ``mach-nix`` itself and the `pypi-deps-db
-<https://github.com/DavHau/pypi-deps-db>`_ version in use (see
-``etc/nix/flake.nix:inputs.machnix`` and ``machnixFor.pypiDataRev``).
-
-Non-Python dependencies are curated in::
-
-    etc/nix/flake.nix:vulnerablecode.propagatedBuildInputs
-
-
-
-Run Tests
----------
-Make sure to install dev dependencies by running ``pip install -r requirements-dev.txt``
-Use these commands to run code style checks and the test suite::
-
-    black -l 100 --check .
-    python -m pytest
-
-
-.. _Data import:
-
-Data import
------------
-
-Some data importers use the GitHub APIs. For this, export the ``GH_TOKEN``
-environment variable with::
-
-    export GH_TOKEN=yourgithubtoken
-
-If you are running behind a proxy, you will need to setup the standard ``https_proxy`` variable.
-
-    export https_proxy=https?://<proxy>:<port>
-
-See `GitHub docs
-<https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token>`_
-for instructions on how to obtain your GitHub token.
-
-To run all data importers use::
-
-    python manage.py import --all
-
-To list available importers use::
-
-    python manage.py import --list
-
-To run specific importers::
-
-    python manage.py import rust npm
-
-
-REST API access
----------------
-
-Start the webserver::
-
-    python manage.py runserver
-
-
-For full documentation about API endpoints use this URL::
-
-    http://127.0.0.1:8000/api/docs
-
-
-
-Continuous periodic Data import
--------------------------------
-
-
-If you want to run the import periodically, you can use a systemd timer::
-
-    $ cat ~/.config/systemd/user/vulnerablecode.service
-
-    [Unit]
-    Description=Update vulnerability database
-
-    [Service]
-    Type=oneshot
-    ExecStart=/path/to/venv/bin/python /path/to/vulnerablecode/manage.py import --all
-
-    $ cat ~/.config/systemd/user/vulnerablecode.timer
+.. code-block:: bash
 
-    [Unit]
-    Description=Periodically update vulnerability database
+    sudo apt-get install \
+        python3-venv python3-dev postgresql libpq-dev build-essential
+    git clone https://github.com/nexB/vulnerablecode.git && cd vulnerablecode
+    make dev envfile postgres
+    make test
+    make run
 
-    [Timer]
-    OnCalendar=daily
+At this point, the VulnerableCode app should be running at port ``8000`` on your machine.
 
-    [Install]
-    WantedBy=multi-user.target
+Populate VulnerableCode data
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+To run all importers and improvers use:
 
-Start this "timer" with::
+.. code-block:: bash
 
-    systemctl --user daemon-reload
-    systemctl --user start vulnerablecode.timer
+   ./manage.py import --all
+   ./manage.py improve --all
 
+Read more about VulnerableCode here: https://vulnerablecode.readthedocs.org/

SOURCES.rst

@@ -1,6 +1,6 @@
 +----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
-|Importer Name:  | Data Source                                                                                          |Ecosystems Covered                                  |
-+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
+|Importer Name   | Data Source                                                                                          |Ecosystems Covered                                  |
++================+======================================================================================================+====================================================+
 |rust            | https://github.com/RustSec/advisory-db                                                               |rust crates                                         |
 +----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
 |alpine          | https://secdb.alpinelinux.org/                                                                       |alpine packages                                     |     
@@ -48,6 +48,6 @@
 |suse_scores     | https://ftp.suse.com/pub/projects/security/yaml/suse-cvss-scores.yaml                                |vulnerability severity scores by SUSE               |
 +----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
 |mozilla         | https://github.com/mozilla/foundation-security-advisories                                            |mozilla                                             |
-+-----------------------------------------------------------------------------------------------------------------------+----------------------------------------------------+
++----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
 |mattermost      | https://mattermost.com/security-updates/                                                             |mattermost server, desktop and mobile apps          |
 +----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+

docs/source/command-line-interface.rst

@@ -0,0 +1,59 @@
+.. _command_line_interface:
+
+Command Line Interface
+======================
+
+The main entry point is Django's :guilabel:`manage.py` management commands.
+
+``$ ./manage.py --help``
+-----------------------
+
+Lists all sub-commands available, including Django built-in commands.
+VulnerableCode's own commands are listed under the ``[vulnerabilities]`` section::
+
+    $ ./mange.py --help
+    ...
+    [vulnerabilities]
+        create_cpe_to_purl_map
+        importer
+        improver
+
+``$ ./manage.py <subcommand> --help``
+---------------------------------------
+
+Displays help for the provided sub-command.
+
+For example::
+
+    $ ./manage.py import --help
+    usage: manage.py import [-h] [--list] [--all] [--version] [-v {0,1,2,3}]
+                            [--settings SETTINGS] [--pythonpath PYTHONPATH]
+                            [--traceback] [--no-color] [--force-color]
+                            [--skip-checks]
+                            [sources [sources ...]]
+
+    Import vulnerability data
+
+    positional arguments:
+      sources               Fully qualified importer name to run
+
+
+``$ ./manage.py import <importer-name>``
+------------------------------------------
+
+Import vulnerability data using the given importer name.
+
+Other variations:
+
+* ``--list`` List all available importers
+* ``--all`` Run all available importers
+
+``$ ./manage.py improve <improver-name>``
+------------------------------------------
+
+Improve the imported vulnerability data using the given improver name.
+
+Other variations:
+
+* ``--list`` List all available improvers
+* ``--all`` Run all available improvers

docs/source/conf.py

@@ -17,7 +17,7 @@
 
 # -- Project information -----------------------------------------------------
 
-project = "vulnerablecode"
+project = "VulnerableCode"
 copyright = "nexb Inc. and others"
 author = "nexb Inc. and others"