Metaflow Resource Tracking & Optimization

Thank you for your interest in Metaflow's @track_resources decorator and our high-level product roadmap!

This project was inspired by an insightful discussion with Ville Tuulos @ Outerbounds, who suggested integrating Spare Cores' cloud server data with resource monitoring of Metaflow steps. This idea aligned perfectly with our near-term vision for Spare Cores: leveraging our comprehensive cloud server database to optimize the cloud resources used by data science and machine learning workflows.

Current Features

Metaflow provides powerful workflow orchestration and some built-in monitoring features via versioned artifacts and card visualizations. However, it lacks detailed step-level resource monitoring. To address this gap, we've developed an open-source Python package and Metaflow extension that offers comprehensive step profiling and cloud resource recommendations. This allows you to monitor CPU, memory, GPU, and other resource usage at the step level, enhancing your ability to optimize and manage your workflows more effectively.

Automated Environment Discovery: Detects cloud and server configurations of the step executor.

Screenshot from the resource tracker Metaflow Card showing key hardware specifications of the server executing the task.

Screenshot from the resource tracker Metaflow Card showing the cloud vendor, region, and actual server type along it's pricing, which is executing the task.

Seamless Resource Tracking: Monitors CPU, memory, GPU, VRAM, disk usage, and network traffic for both local and remote steps on dedicated and shared servers.

Screenshot from the resource tracker Metaflow Card showing the temporal CPU usage on a line chart with a 1 second granularity.

Comprehensive Resource Analysis: Aggregates summary statistics from the most recent run, enhanced with historical data from the last five successful runs.

Screenshot from the resource tracker Metaflow Card showing aggregated statistics on the step's resource usage.

Smart @resources Recommendations: Suggests optimal parameters based on average CPU usage, peak memory reservation, and GPU utilization.
Multi-Cloud Cost Optimization: Finds the cheapest server type across clouds that meets your step's resource requirements and forecasts cloud expenses.

Screenshot from the resource tracker Metaflow Card showing the recommended number of vCPUs and memory amount to allocate for this Metaflow step.

All data collection and @resources optimization occur within your own environment, with only high-level resources metadata shared with the Spare Cores APIs for cloud server recommendations. We are committed to maintaining these features as fully open-source.

Upcoming Managed Central Service

We are also developing an advanced managed central service to extend the functionality of the resource tracker and optimization tool:

360° Dashboard: Track historical resource usage for any step, complete with anomaly annotations, future trend predictions, and tailored recommendations.
Proactive Resource Management: Receive email alerts for overprovisioning and potential job failures due to insufficient allocated resources.
Automated @resources Tuning: Focus on your code and models while our system optimizes infrastructure parameters.
Granular Resource Mapping: Extend tracking metrics to map steps line-by-line, complementing the time-based approach.
Step Optimization Assistant: Get recommendations for splitting steps when resource usage patterns change, such as separating data loading from model training.
Effortless Remote Execution: Our seamless remote executor identifies the optimal server type across clouds, handles provisioning and monitoring without vendor engagement or any infrastructure work on your end.

Screenshot from the resource tracker Metaflow Card showing the recommended number of vCPUs and memory amount to allocate for this Metaflow step.

Ready to optimize the cloud infrastructure powering your Metaflow workflows?
Share your contact details below to stay informed about new feature releases or to discuss your specific needs for cost-effective, right-sized cloud resources.