Server

Key hardware specifications of the server executing this step.

OS family
Linux
Allocation
Indicates whether this server appears to be dedicated to this Metaflow step or shared with other tasks. This assessment is based on comparing system-level resource usage with task-specific resource usage, analyzing patterns in CPU, memory, GPU, and VRAM utilization against both percentage thresholds and absolute values.
Shared
vCPUs
12
Memory
64,011 MiB
Storage
3,700 GiB
GPUs
1 Quadro T1000
VRAM
4,096 MiB

Cloud

Network discovery indicates the following cloud environment was utilized for this step.

Cloud Provider
unknown
Region/Datacenter
unknown
Instance Type
unknown

Usage Statistics

Current and historical (including up-to the last five successful runs) averages, peaks and other summaries on resource usage.

CPU
0.52 avg | 0.96 hist avg | 1.09 peak
Memory
319 MiB avg | 573 MiB peak | 569 MiB hist peak
Duration
6.5 sec
GPU
0.0 avg | 0.0 hist avg | 0.0 peak
VRAM
0 MiB avg | 0 MiB peak | 0 MiB hist peak
Disk Space
2,374 GiB peak
Traffic
0.004 GB in | 0 GB out

Recommendations

Based on recent average CPU usage, historical peak memory and GPU utilization.

Recommended Resources for Next Run
The Metaflow @resources decorator is limited to specifying the number of vCPUs, memory, and number of GPUs, so e.g. no way to specify the minimum amount of VRAM.
@resources(cpu=1, memory=1024)
Automated Tuning of Resources
Learn more at sparecores.com
Cheapest Cloud Server to Run This Step
Evaluated 2000+ servers options accross AWS, GCP, Azure, Hetzner and UpCloud by filtering for the required number of vCPUs, memory, GPUs and min VRAM, then ordered descending by ondemand price, and selected the first one. The price per execution is based on the current best ondemand price of the server and the current duration of the step, and does not include any storage, network traffic, IPV4 prices, the startup time or any discounts. If interested in more advanced recommendations, please get in touch!
  1. DEV-1xCPU-1GB-10GB for $0.000009/execution

CPU Usage

CPU usage for both the system and specific tasks is calculated by summing user+nice and system CPU times (in clock ticks), normalized by dividing by the total elapsed time and ticks per second. Task CPU usage encompasses all child processes.

Memory Usage

On Linux, the used server memory is calculated by total - free - buffers - cached, while it depends on psutil for other systems. Task memory usage is measured by summing PSS (on Linux), USS (on MacOS and Windows), or RSS rollups of all subprocesses.

Disk I/O Usage

Task-specific disk usage tracking is unreliable; therefore, it is recommended to monitor disk usage at the system level, encompassing all mounted disks.

Disk Space Usage

System-level disk space usage on all mounted disks.

Network Usage

Network usage is monitored solely at the system level across all interfaces.

GPU Usage

nvidia-smi reported ratios standardized between 0 and GPU count, proxying how many GPUs have been 100% utilized. Note that task-specific GPU usage is not as reliable as system-level GPU usage and limited up to 4 GPUs.

GPUs in Use

nvidia-smi reported number of GPUs with a utilization greater than 0. Note that task-specific GPU usage is not as reliable as system-level GPU usage and limited up to 4 GPUs.

VRAM Usage

nvidia-smi reported, summed up VRAM usage for all GPUs. Note that task-specific GPU usage is not as reliable as system-level GPU usage and limited up to 4 GPUs.