numba-dppy extension

[PokhodenkoSA]

Reorganize code base and initial version of numba-dppy extension

• Move the numba.dppl package to its own repository (i.e. numba-dppy)
• All core Numba changes to be outlined as patches needed to build Numba with numba-dppy
• All patches need to be analyzed if they can be refactored or converted into plugins

PRs for cleaning unintended changes (make branch pydppl as close as possible to release0.51 branch):

• Revert whitespaces to release0.51 #84 @reazulhoque
  • Importing cpu_dispatcher modifies global state #89 @PokhodenkoSA
• Remove changes to CPUDispatcher #90 @reazulhoque @1e-to closes Importing cpu_dispatcher modifies global state #89
• Remove unused files #108 @reazulhoque
• Remove _remove_unneeded_phis() #86 @Hardcode84 @reazulhoque
• Remove unused var openclliteflags from setup.py #88 @reazulhoque
• Removed unintended changes from core/base.py #85 @DrTodd13
• Patch for codegen debugging #104 @DrTodd13 @reazulhoque - reviewed - most debug code to remove, 2 possible upstream PRs (1 as minimum).
  • Remove the misspelling of finalize_dynamic_globals numba/numba#6466
• Patch for disable emit_environment_object() for dppl context #100 @reazulhoque @diptorupd - reviewed - need to be tested
  • Remove context_name numba-dpex#6
• Remove unary_sqrt_extern() used for sqrt_impl #87 @DrTodd13 @reazulhoque - reviewed - need to test and remove
• Extending typeof() #101 @reazulhoque - reviewed - need investigation and could be removed after tests

PRs with patches (after refactoring it could be proposed to upstream, it is for starting discussion about this changes):

• Patch for with context #96 @AlexanderKalistratov @1e-to
  • Use TargetDispatcher from numba_dppy #118 @PokhodenkoSA - update for moving to numba-dppy
  • Add support for dispatcher retargeting using with-context numba/numba#6870 (or Support changing dispatcher options via a with context numba/numba#6770)
  • Use Numba dispatcher stack for implementing device context numba-dpex#278
    • Changes to the dispacther to support dispacthing based on dpctl device_context numba-dpex#271 - update numba-dppy to new solution
  • Patch for support numba-dppy with device_context numba/numba#6899
• Patch for lowering #98 @reazulhoque @AlexanderKalistratov @DrTodd13 - reviewed - redesign and upstream
  • Add attribute lower_extension to CPUContext numba/numba#6880
  • Numba patch for lower_extensions #215
  • Upstream patch https://github.com/IntelPython/numba/pull/98 #177
• Patch for change numpy functions mapping #99 @reazulhoque - reviewed - redesign and upstream - 2 separate changes
  • Access ufunc_db via context numba/numba#6467
  • Access replace_functions_map via PreParforPass instance numba/numba#6468
• Patch for addrspace #92 @DrTodd13 @reazulhoque - reviewed - discussion and upstream
  • Add address space in pointer type numba/numba#6469
• Initial support for Numpy subclasses (for dparray) - common branch on top of patched #139 (Initial support for Numpy subclasses (for dparray) #116) @DrTodd13 @PokhodenkoSA
  • Initial support for Numpy subclasses. numba/numba#6148
• Refactoring ParforDiagnostics #135 (f.k.a. Refactoring ParforDiagnostics #130) @1e-to @PokhodenkoSA - could be upstreamed
  • Refactoring ParforDiagnostics numba/numba#6545

All upstream patches has label https://github.com/numba/numba/labels/dpPy.

PRs which should me moved to numba-dppy package:

• Move tests for numba-dppy #102 @PokhodenkoSA - reviewed with @reazulhoque
• Move conda recipe to numba-dppy #109 @PokhodenkoSA - reviewed with @reazulhoque
• Move build_for_develop.sh to numba-dppy #110 @PokhodenkoSA - reviewed with @reazulhoque
• Move README.rst and HowTo.rst to numba-dppy #107 @PokhodenkoSA - reviewed with @reazulhoque
• Move support for dpNP #103 @PokhodenkoSA - reviewed with @reazulhoque
• Move dppl_config.py to numba-dppy #106 @PokhodenkoSA @reazulhoque - reviewed - remove code, design error handling in numba-dppy
  • Move numba/dppl_config.py to numba_dppy/config.py numba-dpex#14
• Extension point: numba config options #93 @PokhodenkoSA @reazulhoque - reviewed - need to be moved
  • Move variables from numba/core/config.py to numba_dppy/config.py numba-dpex#15
• Delete numba/dppl after moving to numba-dppy/numba_dppy with all history #111 @PokhodenkoSA - moving numba/dppl - delete it after all other PRs
  • Move numba/dppl to numba_dppy with all history numba-dpex#2
  • Add subtree numba-dppy #117
• Patch for parallel option "offload" #105 @DrTodd13 @AlexanderKalistratov @reazulhoque - reviewed - remove it but need work prior apply
  • All Numba-dppy tests and examples using the @njit decorator should be used with "with_context" numba-dpex#30
  • All Numba-dppy tests and examples using the @njit decorator should be used with "with_context" numba-dpex#46
• Uncomment require_global_compiler_lock() #120 @reazulhoque @DrTodd13 - need fixes in numba-dppy

Process current PRs to numba:

• Add flags to generate dbg symbols #119 @akharche - moved to Add flags to generate debug symbols numba-dpex#27
• Initial support for Numpy subclasses (for dparray) #116 @DrTodd13 @PokhodenkoSA - move part to numba-dppy
  • Add support for dpctl.dparray. numba-dpex#50

Patched version: #124

How to see our changes: git diff release0.51..pydppl

Recommended way of development now:

1. Create conda environment
2. Clone https://github.com/IntelPython/dpctl from branch master. Call: python setup.py develop.
3. Clone https://github.com/IntelPython/numba from branch patched. Call: python setup.py develop.
4. Clone https://github.com/IntelPython/numba-dppy from branch main. Call: python setup.py develop.
5. Install required conda packages if python setup.py develop shows that some packages are not installed.

Branches status

1. patched - contains release0.51 branch, minimal set of patches.
2. pydppl - contains all history, code similar to patched, numba-dppy subtree.
3. clean - contains all history, all patches removed, numba-dppy subtree removed - code the same as release0.51
4. patched-history - contains clean, minimal set of patches.

You should target your PRs to patched branch, because it is simpler than pydppl but contains the same code.

[PokhodenkoSA]

Related #51 - closed by #86.

[PokhodenkoSA]

From @diptorupd:

• What if dpCtl is installed and numba-dppy is not?
• How numba-dppy will understand that IntelPython/numba is installed?

[PokhodenkoSA]

I need a way to build new package with all modifications (i.e. removing numba.dppl folder) but make it different from packages we build before. Build number I think is not visible enough. Using next version 0.51.3 loses connection to base Numba version. Using 0.51.2rc1 is also not related to upstream version. Possibly, introducing new end number 0.51.2.1 is ok. Also it is pretty close in goal to buildnumber but more visible.

[PokhodenkoSA]

a way to build new package with all modifications (i.e. removing numba.dppl folder) but make it different from packages we build before

We decided to use build number which is significantly different from previous one. I.e. use build number 20 when the last one was 12.

[reazulhoque]

Update:

PRs with patches (after refactoring it could be proposed to upstream, it is for starting discussion about this changes):

• Patch for with context #96 @AlexanderKalistratov
  • Use TargetDispatcher from numba_dppy #118 @PokhodenkoSA - update for moving to numba-dppy
  • Upstream patch https://github.com/IntelPython/numba/pull/96 #176
• Patch for lowering #98 @reazulhoque @AlexanderKalistratov @DrTodd13 - reviewed - redesign and upstream
  • Upstream patch https://github.com/IntelPython/numba/pull/98 #177
• Patch for change numpy functions mapping #99 @reazulhoque - reviewed - redesign and upstream - 2 separate changes
  • Access ufunc_db via context numba/numba#6467
    • Numba patch https://github.com/numba/numba/pull/6467 #178
  • Access replace_functions_map via PreParforPass instance numba/numba#6468
    • Numba patch https://github.com/numba/numba/pull/6468 #179
• Patch for addrspace #92 @DrTodd13 @reazulhoque - reviewed - discussion and upstream
  • Add address space in pointer type numba/numba#6469
    • Numba patch https://github.com/numba/numba/pull/6469 #180
• Initial support for Numpy subclasses (for dparray) - common branch on top of patched #139 (Initial support for Numpy subclasses (for dparray) #116) @DrTodd13 @PokhodenkoSA
  • Initial support for Numpy subclasses. numba/numba#6148
• Refactoring ParforDiagnostics #135 (f.k.a. Refactoring ParforDiagnostics #130) @1e-to @PokhodenkoSA - could be upstreamed
  • Refactoring ParforDiagnostics numba/numba#6545

[PokhodenkoSA]

@reazulhoque why did you decide to make a copy of part of description and do not modify description itself?

[reazulhoque]

I intended to update the ticket. Modifying the descripton would change the original and stops from keeping versions and updates separate.