Skip to content

[DOP-3203] Add FileMover#42

Merged
dolfinus merged 1 commit intodevelopfrom
feature/DOP-3203
May 24, 2023
Merged

[DOP-3203] Add FileMover#42
dolfinus merged 1 commit intodevelopfrom
feature/DOP-3203

Conversation

@dolfinus
Copy link
Copy Markdown
Member

@dolfinus dolfinus commented May 23, 2023

  • Added FileMover and MoveResult classes. Using FileDownloader as a template, but no HWM, no tmp_path, and target path is remote.
  • Added tests, using FileDownloader tests as a template.
  • Updated FileDownloader and FileUploader docs:
    • Fixed wrong examples in DownloadResult and FileSet
    • Improved examples with passing files list directly to .run using absolute & relative paths
    • Replaced double backticks with just one in exception messages
    • Add note that FileUploader does not support read strategies, like FileDownloader.
  • Improved exceptions logging in FileDownloader & FileUploader - if debugging is enabled, print original exception stack trace.

Docs: https://onetl--42.org.readthedocs.build/en/42/core/file_mover/index.html

Note: temporary added limit for typing-extensions version because of python/typing_extensions#188

@dolfinus dolfinus self-assigned this May 23, 2023
@codecov
Copy link
Copy Markdown

codecov bot commented May 23, 2023

Codecov Report

Merging #42 (e118ca5) into develop (9d39ff1) will increase coverage by 0.17%.
The diff coverage is 96.48%.

@@             Coverage Diff             @@
##           develop      #42      +/-   ##
===========================================
+ Coverage    92.61%   92.78%   +0.17%     
===========================================
  Files          109      115       +6     
  Lines         5401     5615     +214     
  Branches       941      973      +32     
===========================================
+ Hits          5002     5210     +208     
- Misses         309      311       +2     
- Partials        90       94       +4     
Impacted Files Coverage Δ
onetl/core/file_downloader/download_result.py 100.00% <ø> (ø)
onetl/core/file_result/file_set.py 100.00% <ø> (ø)
onetl/core/file_downloader/file_downloader.py 94.37% <70.00%> (-0.77%) ⬇️
onetl/core/file_uploader/file_uploader.py 94.28% <75.00%> (-1.10%) ⬇️
onetl/core/file_mover/file_mover.py 97.41% <97.41%> (ø)
onetl/base/__init__.py 100.00% <100.00%> (ø)
onetl/base/supports_rename_dir.py 100.00% <100.00%> (ø)
...netl/connection/file_connection/file_connection.py 92.66% <100.00%> (+1.54%) ⬆️
onetl/connection/file_connection/ftp.py 96.20% <100.00%> (+0.09%) ⬆️
onetl/connection/file_connection/hdfs.py 95.87% <100.00%> (+0.03%) ⬆️
... and 9 more

... and 1 file with indirect coverage changes

@dolfinus dolfinus force-pushed the feature/DOP-3203 branch 2 times, most recently from 01d115a to 6141330 Compare May 23, 2023 15:46
@dolfinus dolfinus marked this pull request as ready for review May 23, 2023 16:01
@dolfinus dolfinus force-pushed the feature/DOP-3203 branch 2 times, most recently from bebfdc6 to 0407427 Compare May 24, 2023 09:01
@dolfinus dolfinus force-pushed the feature/DOP-3203 branch from 0407427 to e118ca5 Compare May 24, 2023 13:17
@dolfinus dolfinus enabled auto-merge (rebase) May 24, 2023 13:18
@dolfinus dolfinus merged commit 4fae3a6 into develop May 24, 2023
@dolfinus dolfinus deleted the feature/DOP-3203 branch May 24, 2023 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants