Batch Processing with Spring Cloud Data Flow: Task Launchers and Schedules

I've been building microservices since the Netflix stack days, and one thing I've learned is that batch processing never goes away. Whether you're reconciling payments, generating nightly reports, or processing IoT sensor data, you need a robust way to run short-lived tasks and long-running jobs. Spring Cloud Data Flow (SCDF) is the orchestrator that makes this manageable.

But here's the hard truth: most teams get batch processing wrong. They either cram everything into a monolithic scheduler or over-engineer with complex workflow engines. SCDF strikes a balance by providing a unified platform for defining, deploying, and orchestrating data pipelines using Spring Boot applications.

In this article, we'll dive deep into Task Launchers and Schedules—the core components for batch processing in SCDF. You'll learn how to launch tasks on demand, schedule them with cron expressions, and handle the inevitable production failures. I'll share war stories from a fintech startup where a misconfigured schedule caused a cascade of duplicate payments, and how we fixed it.

If you're already familiar with Spring Cloud Task (check out my article on it), you're ready. Otherwise, go read that first. We'll build on those concepts with real-world examples from a payment reconciliation system. By the end, you'll know how to design batch pipelines that scale and recover gracefully.

What is Spring Cloud Data Flow?

Spring Cloud Data Flow is an orchestration platform for building composable data pipelines. It's built on top of Spring Cloud Task, Spring Cloud Stream, and Spring Batch. The core idea is that you define applications (tasks or streams) and compose them into pipelines using a DSL or UI.

For batch processing, SCDF treats each batch job as a Spring Cloud Task—a short-lived microservice that runs and exits. You can launch tasks manually, via REST API, or on a schedule. SCDF manages the lifecycle, including execution tracking, logging, and error handling.

Why use SCDF over plain cron jobs? Because cron doesn't scale. With SCDF, you get: - Dynamic scaling: launch tasks on Kubernetes or Cloud Foundry with resource limits. - Execution history: track every run, including parameters and exit codes. - Composable workflows: chain tasks together with conditional logic. - Monitoring: integrate with Micrometer and Actuator for metrics.

If you're still using shell scripts and cron, stop. SCDF gives you observability and resilience out of the box.

// A simple Spring Cloud Task that processes a batch file
@SpringBootApplication
@EnableTask
public class PaymentReconciliationTask {

    public static void main(String[] args) {
        SpringApplication.run(PaymentReconciliationTask.class, args);
    }

    @Bean
    public CommandLineRunner commandLineRunner(TaskRepository taskRepository) {
        return args -> {
            System.out.println("Processing batch file: " + args.getOptionValues("batchFile"));
            // Business logic here
            taskRepository.updateExternalExecutionId("batch-" + System.currentTimeMillis());
        };
    }
}

Task Launchers: On-Demand and Programmatic

A Task Launcher is a Spring Cloud Stream application that listens for task launch requests and executes them on a target platform (local, Cloud Foundry, Kubernetes). SCDF provides a default Task Launcher for each platform.

You can launch a task in three ways: 1. SCDF Shell: task launch myTask --arguments "--batchFile=/data/file.csv" 2. REST API: POST to /tasks/definitions/{name} with parameters. 3. Dashboard UI: Click 'Launch' on a task definition.

For production, you'll likely use the REST API from another service. For example, when a new file lands in S3, an event triggers a task launch. This is where the Task Launcher shines—it decouples the trigger from the execution.

Here's the gotcha: Task Launchers are stateful. If the Task Launcher application crashes, pending launches are lost. To mitigate, use a message broker (like RabbitMQ or Kafka) to buffer requests. SCDF's default Task Launcher uses a database-backed queue, but for high throughput, switch to a broker.

Another pitfall: resource contention. If you launch 100 tasks simultaneously without limits, you'll overwhelm your cluster. Always configure spring.cloud.task.max-concurrent-tasks in the Task Launcher properties.

@RestController
@RequestMapping("/api")
public class TaskLaunchController {

    @Autowired
    private TaskLauncher taskLauncher;

    @PostMapping("/launch/{taskName}")
    public String launchTask(@PathVariable String taskName, @RequestParam String batchFile) {
        TaskLaunchRequest request = new TaskLaunchRequest();
        request.setName(taskName);
        request.setArguments(new String[]{"--batchFile=" + batchFile});
        taskLauncher.launch(request);
        return "Task launched";
    }
}

Scheduling Tasks with Cron Expressions

SCDF integrates with Spring Cloud Task and uses a scheduler (like Spring's TaskScheduler or a platform-specific one like Kubernetes CronJob) to trigger tasks on a cron schedule.

To create a schedule, you define a cron expression and associate it with a task definition. For example: `` dataflow:>schedule create --name dailyReconciliation --definition "reconcile" --expression "0 0 2 ?" ` This runs the reconcile` task daily at 2 AM.

Cron expressions in SCDF follow the Spring cron format (6 fields: seconds, minutes, hours, day-of-month, month, day-of-week). The ? means 'no specific value' and is used for day-of-month or day-of-week when the other is specified.

Here's where most teams mess up: timezone. By default, cron expressions are evaluated in the server's timezone (usually UTC). If your business operates in EST, you need to set the timezone explicitly. In SCDF, you can configure spring.cloud.dataflow.scheduler.timezone globally or per schedule.

Another common mistake: not handling daylight saving time. A cron expression like 0 0 2 ? will run at 2 AM regardless of DST changes. In spring-forward, it runs at 3 AM; in fall-back, it runs twice. To avoid this, use cron expressions that are timezone-aware or schedule tasks at times that don't fall in the DST gap.

@Configuration
public class ScheduleConfig {

    @Bean
    public ScheduleInfoRepository scheduleInfoRepository(DataSource dataSource) {
        return new ScheduleInfoRepository(dataSource);
    }

    @Bean
    public Scheduler scheduler(ScheduleInfoRepository repository, TaskLauncher taskLauncher) {
        return new CronScheduler(repository, taskLauncher);
    }
}

Advanced Scheduling: Event-Driven and Chained Tasks

Cron is great for fixed intervals, but not all batch processing is time-based. Sometimes you need to launch a task when a file arrives, or after another task completes. SCDF supports this through event-driven scheduling and task chaining.

For event-driven scheduling, you can use Spring Cloud Stream to listen for events (e.g., file uploads) and launch a task. SCDF's Task Launcher can be bound to a message channel. When a message arrives, it launches the task with the message payload as arguments.

Task chaining is more complex. SCDF doesn't have a built-in DAG scheduler like Airflow, but you can simulate it using task execution listeners. For example, after Task A completes, its exit message can trigger Task B. You do this by setting spring.cloud.task.closecontext.enable=true and using a listener to send a message to a channel that launches the next task.

I once worked on a data pipeline that required three sequential tasks: extract, transform, load. We used SCDF with a custom listener that checked the exit status of the previous task and launched the next one. It worked, but error handling was painful. If you need complex workflows, consider using a dedicated workflow engine like Spring Cloud Data Flow's own 'composed tasks' or even Apache Airflow.

@Component
public class TaskChainingListener {

    @Autowired
    private TaskLauncher taskLauncher;

    @EventListener
    public void onTaskEnd(TaskExecutionEndEvent event) {
        String taskName = event.getTaskName();
        if ("extract".equals(taskName)) {
            TaskLaunchRequest request = new TaskLaunchRequest();
            request.setName("transform");
            request.setArguments(new String[]{"--inputFile=" + event.getExitMessage()});
            taskLauncher.launch(request);
        }
    }
}

What the Official Docs Won't Tell You

I've read the Spring Cloud Data Flow reference docs cover to cover. They're decent, but they gloss over several production realities.

1. Task Launcher High Availability: The docs say you can run multiple Task Launcher instances for HA. What they don't say is that without a shared database or message broker, you'll get duplicate launches. Each launcher has its own queue. You need to configure a shared TaskRepository and a distributed lock to prevent races.
2. Schedule Persistence: By default, schedules are stored in the SCDF database. But if you're using Kubernetes, the schedule might be implemented as a CronJob. If the SCDF server restarts, it re-creates the CronJobs from the database. However, if the database is lost, schedules are gone. Always back up the SCDF database.
3. Task Timeouts: There's no built-in timeout for tasks. If a task hangs (e.g., due to a deadlock), it will run forever, consuming resources. You must implement a timeout in your task code or use the platform's resource limits (e.g., Kubernetes activeDeadlineSeconds).
4. Error Handling in Schedules: When a scheduled task fails, SCDF logs the failure but does not automatically retry. You need to implement retry logic in your task or use a wrapper that catches exceptions and relaunches.
5. Monitoring: SCDF exposes metrics via Micrometer, but the docs don't tell you that you need to configure a metrics exporter (like Prometheus) manually. Also, task execution logs are not aggregated by default—you need a centralized logging solution.

// Example: Setting a timeout for a task using Spring Cloud Task properties
// application.properties
spring.cloud.task.timeout=PT30M  # 30 minutes timeout

Production Best Practices for Task Launchers and Schedules

Based on years of experience, here are my non-negotiable best practices:

1. Idempotency: Every task must be idempotent. Use a unique batch ID (e.g., based on date and file hash) and check for duplicates before processing. This prevents double processing when schedules run unexpectedly.
2. Resource Limits: Always set CPU and memory limits for tasks. In Kubernetes, use resource requests and limits. Without them, a rogue task can starve other tasks.
3. Logging and Monitoring: Use structured logging (JSON) and aggregate logs in a central system (ELK, Splunk). Set up alerts for task failures and long-running tasks.
4. Graceful Shutdown: Implement a shutdown hook to clean up resources (close DB connections, release locks) when a task is interrupted.
5. Testing Schedules: Test cron expressions in a staging environment with the same timezone. Use SCDF's dry-run feature if available.
6. Versioning: When you update a task definition, create a new version and update the schedule to use it. Don't overwrite existing definitions—this can cause running tasks to fail.
7. Security: Use Spring Security to protect SCDF endpoints. Don't expose the task launch REST API without authentication.

@Component
public class IdempotentTask {

    @Autowired
    private BatchRepository batchRepository;

    public void process(String batchId) {
        if (batchRepository.existsByBatchId(batchId)) {
            System.out.println("Batch already processed: " + batchId);
            return;
        }
        // process batch
        batchRepository.save(new BatchRecord(batchId));
    }
}