Exceptions and ResourceWarnings on ray.init (Jupyter+offline)

[austinmw]

What is the problem?

I'm running ray in a JupyterLab notebook (AWS SageMaker), and the notebook has no internet connection (for data safety).
(instance type is ml.p3.16xlarge, so 64 vCPU and 488GiB ram).

When I simply do:

import ray
ray.init()

I get the following output:

2020-08-24 01:29:24,153	INFO resource_spec.py:212 -- Starting Ray with 323.83 GiB memory available for workers and up to 142.79 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='a' encoding='UTF-8'>
2020-08-24 01:29:24,500	INFO services.py:1148 -- View the Ray dashboard at localhost:8265
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='a' encoding='UTF-8'>
{'node_ip_address': '10.54.198.48',
 'redis_address': '10.54.198.48:45400',
 'object_store_address': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/plasma_store',
 'raylet_socket_name': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/raylet',
 'webui_url': 'localhost:8265',
 'session_dir': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504'}

Do I need to modify any init settings due to the fact that I'm running in Jupyter and/or because I'm offline?

Ray version and other system information (Python version, TensorFlow version, OS):

• Python 3.6.10
• ray 0.8.7

[rkooo567]

Is there any way what exceptions are ignored in your logs?

[edoakes]

@austinmw this is a warning that's printed when a Python file object is created and the file isn't closed before the object is destructed. In most versions of Python this won't be printed, but it will if you're using a debug build or pass -W (and maybe some other circumstances, haven't seen it though).

This is intentional behavior because we need these files to be long-lived to write logs to, so there's nothing to worry about. Unfortunately I can't think of a good way to silence this at the moment. Leaving this issue open for others that might come across it.

[cszhu]

Hello! This P3 issue has seen no activity in at least a year. It will be closed in 2 weeks as part of ongoing cleanup efforts.

Please remove the pending-cleanup label if you believe this issue should remain open.

Thanks for contributing to Ray!