Batch Framework

Overview

The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client.

A batch file defines one job task.

Once started, MinIO starts processing the job. Time to completion depends on the resources available to the deployment.

Ordinary job failures use the retry limits in the definition. SILO 20261109 distinguishes authorization refusal from temporary pauses as described below.

The MinIO Batch Framework supports the following job types:

SILO 20261109: authorization and retries

SILO 20261109 binds local batch operations to the authenticated submitter at job creation and enforces the submitter’s data permissions throughout execution (#310, #311). Credentials in a job definition authorize remote connections only and do not grant local server authority. SILO re-evaluates IAM state before each action:

• An explicit denial, an expired, disabled, or revoked credential, or a changed credential secret, session token, or parent identity ends the job with failureReason.
• Identity metadata edits (such as descriptions or extended expiration dates) and LDAP group refreshes that keep the submitter’s credential intact do not stop the job; each check resolves current status, groups, and policies (#318–#320).
• Policy evaluation skips deleted policies that an IAM mapping still names, grants nothing from Allow statements that depend on unavailable request-time context, and refuses the job only when an applicable Deny statement requires request-time context that batch execution cannot supply (#318–#320).
• Batch jobs preserve Object Lock protection across expiration, key rotation, and pull replication, skipping retention-locked or legal-hold-protected versions instead of bypassing protection and finishing as complete-with-skips when protected or refused work remains (#310, #311).
• Multi-pool keyrotate jobs scope copy operations to the target pool so object copies in other pools remain readable (#312).

IAM startup, unavailable policy material, and an authorization plugin that cannot return a valid decision are temporary failures. Submission returns 503 when authorization cannot be decided. Expiration and key rotation pause with pausedReason, preserve completed progress, and retry in the same process with bounded exponential backoff and jitter (#313). Immediate or streamed listing failures also pause these jobs. Key rotation pauses when its selected source is in a moving pool and resumes after the conflict clears. Every retry resolves current identity and policies again. A pending or paused job is not a completed or failed report and is not removed by terminal-report cleanup. Administrator cancellation takes precedence over queued work and retries.

Push, pull, and Snowball replication also pause with pausedReason when a required authorization dependency or report read is temporarily unavailable, including an outage that begins during object processing, and restart their source scan from the beginning on retry (#324); see Batch Replication. Other mid-run replication failures follow the finite object retry limits in the definition and can end the run as failed.

Startup discovery retries failed manifest listings and storage reads with bounded backoff (#314) and resumes unfinished jobs, including running replication jobs (#324). Discovery remembers manifests already admitted during that pass, so a later scan cannot rerun a job that finished while waiting. Cancellation fences stale discovery results. Missing, malformed, or unsupported manifests are not queued; malformed and unsupported definitions remain on disk for inspection. A storage outage during discovery does not require a restart once storage recovers.

Completed jobs—including completed expire jobs (#315), legacy jobs, and jobs whose submitter credentials later expire or change—stay completed after a server restart. Unfinished jobs from releases without submitter binding require resubmission. New bound definitions and authorized progress reports use formats rejected by older executors. This fences those records on rollback, but does not make a mixed cluster safe: an old node can still accept and execute its own old-format jobs. Stop submissions and drain or cancel old jobs before changing versions, and do not downgrade with pending authorized jobs.

Upgrading and rolling back 20261109

This release deliberately introduces a batch-record format boundary: new job manifests use the SILO-BATCH v2 envelope and authorized progress reports also use v2. The 20261026 executor cannot execute or resume these records. Ordinary S3 object formats are unaffected, but batch jobs cannot follow a binary downgrade transparently.

1. Stop submissions to every node before upgrading. Inventory jobs with mcli batch list ALIAS and mcli batch status ALIAS JOBID. Drain running, pending and paused jobs, or cancel them with mcli batch cancel ALIAS JOBID; preserve their original YAML definitions and records of completed work.
2. Resume submissions only after every node is upgraded. Unfinished legacy jobs do not acquire an authorization binding automatically: resubmit with an identity that has the required data permissions. Reconcile completed work before resubmission to avoid unwanted repeated work. Completed legacy jobs need no resubmission.
3. Do not run batch jobs during a mixed-version upgrade: old nodes can still accept and execute their own old-format jobs without the new submitter authorization checks.
4. Before rollback, stop submissions again and drain or cancel all new jobs, including paused jobs. Do not expect 20261026 to resume their progress, strip the format envelope, or edit authorization records. Preserve the records and reconcile results on a compatible release before deciding whether to resubmit.

Use mcli batch status (20261109) to distinguish pauses, terminal failures, and completion with protected work left in place. infraAdmin still has other infrastructure administration capabilities; these job checks do not turn that role into a strict data-isolation boundary.

MinIO Batch CLI

• Install the MinIO Client
• Define an alias for the MinIO deployment

The mc batch commands include:

Access to mc batch

The submitting identity needs the relevant administration action and the local data permissions for the job. Credentials in the definition must separately permit its remote operations. See the 20261109 rules above.

The user executing the batch job must have the following permissions. You can alternatively restrict users from accessing these functions by blocking or limiting access to these actions:

admin:ListBatchJobs

Grants the user the ability to see batch jobs currently in process.

admin:DescribeBatchJob

Grants the user the ability to see the definition details of a batch job currently in process.

admin:StartBatchJob

Grants the user the ability to start a batch job. The job may be further restricted by the credentials the job uses to access either the source or target deployments.

admin:CancelBatchJob

Allows the user to stop a batch job currently in progress.

You can assign any of these actions to users independently or in any combination.

The built-in ConsoleAdmin policy includes sufficient access to perform all of these types of batch job actions.

Local Deployment

You run a batch job against a particular deployment by passing an alias to the mc batch command. The deployment you specify in the command becomes the local deployment within the context of that batch job.