数字管道

NumPipe 是一个 Python 软件包，它通过令人尴尬的并行执行代码并将输出缓存到 HDF5 文件，使长时间运行的任务变得更容易、更快。

特征

• 将计算和可视化代码组合到单个脚本中。仅根据请求重新运行计算
• 使用该yield语句返回一段时间内的数据，这些数据将定期缓存到文件中
• 指定缓存函数之间的依赖关系
• 类似于tqdm并行工作的进度条，以显示正在运行的任务的进度
• 一个可选的 Telegram Messenger 机器人，可以通知用户完成并发送 Matplotlib 图形和动画
• 用于重新运行任务并自动保存 Matplotlib 图形和动画的命令行参数
• 使用 Slurm Workload Manager 自动创建和提交 sbatch 文件的选项（用于许多计算集群）

安装

NumPipe 可以用 pip 安装

pip install numpipe

例子

单任务

from numpipe import scheduler, pbar
from time import sleep
import matplotlib.pyplot as plt

job = scheduler()

@job.cache
def sim():
    for i in pbar(range(100)):
        sleep(.1)   # long running task...
        yield dict(i=i, x=i**2)

@job.plots
def vis():
    var = job.load(sim)
    plt.plot(var.i, var.x)

if __name__ == '__main__':
    job.run()

并行任务

from numpipe import scheduler, pbar
from time import sleep
import matplotlib.pyplot as plt

job = scheduler()

@job.cache
def sim(power):
    for i in pbar(range(100)):
        sleep(.1)   # long running task...
        yield dict(i=i, x=i**power)

@job.plots
def vis():
    for name, var in job.load(sim):
        plt.plot(var.i, var.x)

for i in range(3):
    job.add(sim, power=i)

if __name__ == '__main__':
    job.run()

更多示例

有关更多使用示例，请参见示例文件夹

命令行参数

positional arguments:
  {display,clean,slurm}
    display             display available functions and descriptions
    clean               remove all h5files that are no longer cache functions
    slurm               run on a system with the Slurm Workload Manager

optional arguments:
  -h, --help            show this help message and exit
  -r [RERUN [RERUN ...]], --rerun [RERUN [RERUN ...]]
                        re-run specific cached functions by name
  -f, --force           force over-write any existing cached data
  -d [DELETE [DELETE ...]], --delete [DELETE [DELETE ...]]
                        delete specified cached data
  -e EXCLUDE [EXCLUDE ...], --exclude EXCLUDE [EXCLUDE ...]
                        exclude cached function from being re-run
  --at-end              only run at_end functions
  --no-at-end           don't run at_end functions
  -p [PROCESSES], --processes [PROCESSES]
                        number of processes to use in parallel execution (default: cpu_count)
  -ct CACHE_TIME, --cache_time CACHE_TIME
                        time (in seconds) until data cached data is flushed to file
  --no-deps             do not rerun functions that depend on other reran functions
  --mininterval MININTERVAL
                        time (in seconds) for progress bar mininterval argument
  --notify              send notifications without delay
  --notify-message NOTIFY_MESSAGE
                        send a custom message with other notifications
  --notify-delay NOTIFY_DELAY
                        time (in seconds) before notifications will be sent
  --theme THEME         matplotlib plot theme
  --figures FIGURES [FIGURES ...]
                        which figure numbers to display
  --save [SAVE]         save figures and animations
  --save-format SAVE_FORMAT [SAVE_FORMAT ...]
                        file format for figures
  --save-figs [SAVE_FIGS]
                        save figures
  --save-anims [SAVE_ANIMS]
                        save animations
  --debug               run in debug mode (single process)

执照

NumPipe 根据 MIT 许可条款获得许可。