Failure trace is hard to process

[pombredanne]

>>> from fetchcode import package
>>> list(package.info('pkg:rubygems/file'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "fetchcode/src/fetchcode/package.py", line 315, in get_rubygems_data_from_purl
    response = get_response(api_url)
  File "fetchcode/src/fetchcode/package.py", line 47, in get_response
    return resp.json()
  File "fetchcode/tmp/lib/python3.6/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File ".pyenv/versions/3.6.10/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File ".pyenv/versions/3.6.10/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File ".pyenv/versions/3.6.10/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This failure is cryptic and it does not tell anything that can help me fix it...
In contrast if I update the code this way:

@router.route("pkg:rubygems/.*")
def get_rubygems_data_from_purl(purl):
    """
    Generate `Package` object from the `purl` string of rubygems type.
    """
    purl = PackageURL.from_string(purl)
    name = purl.name
    api_url = f"https://rubygems.org/api/v1/gems/{name}.json"
    try:
        response = get_response(api_url)
    except Exception as e:
        raise Exception(f'Failed to fetch: {api_url}') from e

    declared_license = response.get("licenses") or None
    homepage_url = response.get("homepage_uri")
    code_view_url = response.get("source_code_uri")
    bug_tracking_url = response.get("bug_tracker_uri")
    download_url = response.get("gem_uri")
    yield Package(
        homepage_url=homepage_url,
        api_url=api_url,
        bug_tracking_url=bug_tracking_url,
        code_view_url=code_view_url,
        declared_license=declared_license,
        download_url=download_url,
        **purl.to_dict(),
    )

I now have a clean and clear failure trace that is actionable:

>>> from fetchcode import package
>>> list(package.info('pkg:rubygems/file'))
Traceback (most recent call last):
  File "fetchcode/src/fetchcode/package.py", line 316, in get_rubygems_data_from_purl
    response = get_response(api_url)
  File "fetchcode/src/fetchcode/package.py", line 47, in get_response
    return resp.json()
  File "fetchcode/tmp/lib/python3.6/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File ".pyenv/versions/3.6.10/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File ".pyenv/versions/3.6.10/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File ".pyenv/versions/3.6.10/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "fetchcode/src/fetchcode/package.py", line 318, in get_rubygems_data_from_purl
    raise Exception(f'Failed to fetch: {api_url}') from e
Exception: Failed to fetch: https://rubygems.org/api/v1/gems/file.json

We should test and ensure we have informative traces in all our functions

[pombredanne]

In https://github.com/nexB/fetchcode/blob/ab65b2e645c889887227ea49eb3332d885fd0a54/src/fetchcode/package.py#L41 the code is problematic because this is trying to get the JSON before checking if the request was successful (e.g. a 200 HTTP answer)... otherwise it does not make sense to get JSON until we know the request succeeded:

def get_response(url):
    """
    Generate `Package` object for a `url` string
    """
    resp = requests.get(url)  <--- this is also problematic and we should check HTTP code
    if "json" in dir(resp):  <--- this is problematic
        return resp.json()   <--- this is also problematic

    raise Exception("Response of this URL does not contain json object")  <--- this is also problematic as we do not known why this failed

[TG1999]

closed by #73