Thump and Flow: Conceptual Workflow Logic for Scheduled vs. On-Demand Clinical Integration

Every clinical integration project eventually hits a fork: do you let data flow on a fixed schedule, or do you pull it only when someone needs it? The answer seems simple until you factor in shift changes, lab result urgency, EHR update windows, and the fact that no two clinical departments tolerate latency the same way. This guide walks through the conceptual workflow logic behind scheduled and on-demand integration, helping you pick the right pattern for your context.

We write as editorial guides, not vendors or consultants with a single methodology. Our goal is to give you a framework you can adapt, not a prescription. By the end, you should be able to map your own data sources, latency requirements, and operational constraints to one of three integration approaches—or a hybrid that borrows from both.

Who Must Choose and by When

The decision between scheduled and on-demand integration rarely lands on a single person's desk. It emerges from conversations between clinical informaticists, IT architects, nursing informatics leads, and sometimes the lab or pharmacy directors who depend on timely data. If you are reading this, you are likely part of that group—or preparing to join it.

The timing of the choice matters. Early in a project, when interface specifications are still being drafted, you have the most flexibility. Later, after connection points are locked and data models are agreed, switching from scheduled to on-demand (or vice versa) becomes costly. We have seen teams commit to a batch schedule for a real-time clinical alerting feed, then spend months retrofitting middleware to reduce latency. The opposite mistake—building an on-demand interface for a nightly reporting dump—wastes resources on unnecessary real-time infrastructure.

So when must you decide? Ideally, during the requirements-gathering phase, before any code is written. But even if you are already mid-implementation, this framework can help you identify mismatches and plan incremental changes. The key is to recognize that the choice is not binary: many clinical workflows benefit from a hybrid approach, where some data flows on a schedule and other data is pulled on demand.

Consider a typical hospital scenario: the EHR sends a nightly batch of updated patient demographics to the pharmacy system. That works fine for most purposes, but when a patient is admitted through the ED with an unknown medication history, the pharmacy needs on-demand access to the latest medication list. In this case, a scheduled bulk transfer plus a real-time query endpoint covers both needs. The decision, then, is not just about technology—it is about understanding which data elements are time-sensitive and which are not.

We recommend teams create a simple matrix: list each data flow, its typical update frequency, the maximum acceptable delay, and the consequence of stale data. This exercise alone often reveals that 60–70% of flows can stay on a schedule, while the remainder require on-demand capability. The challenge is designing an architecture that supports both without doubling the maintenance burden.

Option Landscape: Three Approaches to Clinical Integration

When we talk about integration patterns in clinical settings, three broad approaches dominate: scheduled batch, real-time on-demand, and hybrid. Each has variations, but understanding the core logic helps you evaluate trade-offs without getting lost in vendor terminology.

Scheduled Batch Integration

In a scheduled batch model, data is collected, transformed, and transmitted at fixed intervals—every hour, every shift, overnight. This is the oldest and most predictable pattern. It works well for large volumes of non-urgent data: daily lab result archives, billing feeds, population health datasets. The advantages are straightforward: system load is predictable, error handling is simpler (you can retry the whole batch), and infrastructure costs are lower because you can schedule transfers during off-peak hours.

The downside is latency. If a clinician needs a result that was generated five minutes after the last batch, they wait until the next transfer. In fast-moving clinical environments—ED, ICU, surgical suites—that delay can affect decisions. Batch also struggles with data that changes frequently; each batch transfer sends the entire set, not just the changes, unless you implement delta logic.

Real-Time On-Demand Integration

On-demand integration means data is pulled or pushed as events occur, or when a user or system requests it. This pattern is essential for time-sensitive workflows: medication administration checks, lab result notifications, allergy alerts. Technologies like HL7 FHIR subscriptions, RESTful APIs, and message queues enable this pattern. The benefit is obvious: data arrives when it matters, not when a timer fires.

The costs are less obvious but real. Real-time systems require more robust infrastructure: load balancers, redundant message brokers, monitoring for latency spikes. Error handling is more complex because a single failed message can break a workflow, and retries must be carefully managed to avoid duplicates. Many clinical IT teams underestimate the operational overhead of maintaining real-time interfaces, especially when multiple source systems have different uptime guarantees.

Hybrid Integration

Most mature clinical integration environments end up hybrid. They use scheduled batches for high-volume, low-urgency data and on-demand patterns for time-sensitive transactions. The trick is defining the boundary clearly. A common pattern is to use a scheduled nightly batch to populate a data warehouse, while exposing FHIR APIs for real-time queries against the source EHR. Another hybrid approach is to use a message queue that buffers events and delivers them in near-real-time, with a scheduled fallback batch for any messages that failed to deliver.

Hybrid architectures require careful design to avoid data inconsistency. If a patient's allergy status is updated after the nightly batch, the on-demand query must return the latest value, not the batch snapshot. This often means maintaining a cache with a short TTL or routing queries directly to the source system for recent updates. The complexity is higher, but the payoff is a system that meets both operational and clinical needs without over-engineering every flow.

We have seen teams succeed with hybrid by starting simple: identify the top three time-sensitive workflows and implement on-demand for those, leaving everything else on a schedule. Over time, they add more on-demand flows as they gain confidence in the infrastructure. The opposite approach—building a full real-time platform from day one—often leads to cost overruns and underutilized capacity.

Criteria for Choosing Your Integration Pattern

How do you decide which approach fits a given data flow? We have found five criteria that consistently separate scheduled from on-demand use cases. Use these as a checklist during requirements review.

Latency Tolerance

Ask: how old can this data be before it causes a problem? For a billing feed, a 24-hour delay is usually acceptable. For a medication interaction check, even a 5-minute delay is risky. Define latency tolerance in business terms, not technical ones. Involve clinical stakeholders in this conversation; they often have a clearer sense of urgency than IT assumes.

Data Volume and Velocity

Scheduled batch handles high volume efficiently because you can compress and transfer data in bulk. On-demand systems struggle with sudden spikes—if every clinician refreshes a patient list at shift change, the API load can overwhelm the source system. Measure your peak data volume and consider whether the source system can handle real-time queries without degrading clinical performance.

Error Recovery Complexity

Batch errors are easier to manage: you detect a failed transfer, fix the issue, and rerun the batch. On-demand errors are trickier because a failed message might be part of a sequence—if a lab result notification is lost, the clinician may never know it existed. Consider your team's ability to build and monitor retry logic, dead-letter queues, and alerting for missing messages.

Source System Constraints

Not all clinical systems are built for real-time queries. Older EHRs, lab systems, and radiology information systems may have limited API capabilities or may impose licensing costs per API call. Scheduled extracts, on the other hand, are often free. Check with each system owner about their preferred integration method and any performance guarantees.

Regulatory and Audit Requirements

Some data flows require an audit trail showing exactly when data was accessed and by whom. On-demand systems can log each query, but the volume of logs can be large. Batch transfers produce fewer, larger log entries. Consider whether your compliance team needs per-transaction visibility or aggregate reporting.

We recommend scoring each data flow against these five criteria on a simple 1–5 scale. Flows that score high on latency tolerance and volume, and low on error complexity, are good candidates for scheduled batch. Flows with low latency tolerance and moderate volume are better suited for on-demand. Hybrid is appropriate when a flow has mixed characteristics—for example, high volume but also some time-sensitive subsets.

Trade-Offs at a Glance: Scheduled vs. On-Demand

To make the comparison concrete, we have structured the trade-offs in a table. This is not a complete list, but it captures the most common tensions teams face.

The table highlights a key insight: the choice is rarely about one dimension alone. A flow might have low latency tolerance (favoring on-demand) but also high volume that would overwhelm the source system (favoring batch). In that case, a hybrid approach—such as caching frequently accessed data and updating it via batch—can bridge the gap.

Another common trade-off is between error recovery and data freshness. Teams that prioritize zero data loss often lean toward batch because it is easier to verify completeness. Teams that prioritize timeliness accept the risk of occasional missed messages in exchange for near-real-time updates. Neither choice is wrong; it depends on the clinical consequence of missing data versus acting on stale data.

We have also seen teams underestimate the operational cost of on-demand interfaces. A single real-time API might seem simple, but when you have twenty such interfaces, each with different uptime requirements and error handling logic, the maintenance burden grows non-linearly. Batch interfaces, by contrast, tend to be more uniform and easier to monitor with a single dashboard.

Implementation Path After the Choice

Once you have decided which flows are scheduled and which are on-demand, the real work begins. Implementation involves more than configuring middleware; it requires changes to how your team plans, tests, and monitors integrations.

For Scheduled Batch Flows

Start by defining the batch window. Coordinate with source system owners to ensure the extract does not interfere with clinical operations. For example, avoid running a full patient demographic extract during peak clinic hours. Next, implement delta detection: instead of transferring the entire dataset each time, track changes using timestamps or change data capture (CDC) tools. This reduces transfer time and load on both systems.

Error handling for batch is straightforward: log failures, alert the integration team, and provide a manual rerun button. But do not rely solely on manual intervention. Build automated retry logic with exponential backoff, and set a maximum retry count before escalating to a human. Also, consider idempotency—if the same batch is accidentally sent twice, the receiving system should not create duplicate records.

Testing batch flows is relatively simple: compare the source and target datasets after a transfer. Automate this comparison using checksums or row counts. Schedule a daily reconciliation report to catch silent data corruption early.

For On-Demand Flows

On-demand implementation requires more upfront design. Define the API contract clearly: what data is returned, under what conditions, and with what authentication. Use standard protocols like FHIR R4 where possible, but be prepared to handle vendor-specific quirks. Many EHRs expose FHIR APIs that are not fully compliant; test edge cases like missing fields, large result sets, and timeout behavior.

Performance testing is critical. Simulate peak load—for example, all nurses refreshing a patient list at shift change—and measure response times. If the source system cannot handle the load, consider adding a cache layer that stores frequently accessed data and refreshes it via batch or near-real-time updates. This is a hybrid pattern in disguise, but it often saves the project.

Error handling for on-demand flows is more nuanced. Implement a dead-letter queue for messages that cannot be processed. Monitor queue depth and alert when it exceeds a threshold. For critical alerts (e.g., critical lab results), build a fallback mechanism: if the on-demand push fails, the system should retry, and if retries fail, escalate via pager or email. Document the expected behavior for each failure scenario.

Finally, invest in observability. On-demand systems fail in ways that batch systems do not: a single slow query can block other requests, or a misconfigured subscription can send duplicate notifications. Use distributed tracing to follow a message from source to destination, and set up dashboards that show latency percentiles, error rates, and throughput.

Risks If You Choose Wrong or Skip Steps

The consequences of a poor integration pattern choice range from annoying to dangerous. We have categorized the most common risks to help you avoid them.

Clinical Risk: Stale Data Leading to Wrong Decisions

If you use a scheduled batch for a time-sensitive clinical alert, a clinician may act on outdated information. For example, a patient's allergy status might change after the last batch, and the next batch is not due for another hour. If a medication is administered based on the stale data, the patient could have an adverse reaction. This is the most serious risk and the one that should drive your latency tolerance decisions.

Operational Risk: Overloaded Source Systems

On-demand queries, especially if not cached, can overwhelm source systems. We have seen EHRs slow to a crawl during peak hours because a new integration was polling the patient demographics table every few seconds. This affects not only the integration but also clinical users working in the EHR. Always test under load and consider rate limiting or caching.

Financial Risk: Unnecessary Infrastructure Costs

Building a full real-time platform for data that could be batch-transferred wastes money on middleware licenses, additional servers, and staff time for maintenance. Conversely, choosing batch for a flow that truly needs real-time updates can lead to costly rework later. The matrix we described earlier helps avoid both extremes.

Integration Risk: Data Inconsistency in Hybrid Setups

Hybrid architectures introduce the risk that different parts of the system see different versions of the same data. For example, a scheduled batch might update the data warehouse at midnight, while an on-demand query returns a newer value from the source system. If a report uses the warehouse and a clinical decision support tool uses the API, they might disagree. Mitigate this by clearly documenting which system is the source of truth for each data element, and by using version stamps or timestamps to detect conflicts.

Team Risk: Burnout from Complex Error Handling

On-demand systems require more vigilant monitoring and faster response times. Teams that are not prepared for this often experience burnout, leading to missed alerts and degraded reliability. Before committing to a real-time architecture, ensure your team has the tools, training, and on-call rotation to support it. If not, start with a smaller set of on-demand flows and expand gradually.

We have seen teams skip the requirements matrix and jump straight to implementation, only to discover six months later that half their on-demand interfaces are rarely used. The sunk cost makes it hard to revert. Avoid this by investing time upfront in understanding which flows truly need real-time updates.

Mini-FAQ: Common Doubts About Scheduled vs. On-Demand Integration

Over the course of many projects, certain questions surface repeatedly. Here we address the most frequent ones.

Can we use scheduled batch for everything and add on-demand later?

Yes, but with a caveat. Starting with batch is a common and sensible approach, especially for new integrations where you are still learning the data. However, if you know from the start that some flows will eventually need on-demand, design the batch pipeline to support incremental extraction and include a unique identifier for each record. This makes it easier to later add a real-time API that references the same data model. Retrofitting a batch-only system for on-demand can be expensive if the data model does not support point queries.

Does FHIR force us to use on-demand?

No. FHIR supports both patterns. You can use FHIR for scheduled bulk data export (the $export operation) and for real-time queries and subscriptions. The standard is flexible; the choice depends on your workflow, not the protocol. Many teams use FHIR APIs for on-demand queries while still running nightly batch exports for reporting.

How do we handle data that changes frequently, like patient location?

Frequently changing data is a classic candidate for on-demand or near-real-time updates. If you try to batch it, you will either miss updates or overwhelm the system with large deltas. A common pattern is to use a message queue that publishes location changes as events, and then have downstream systems subscribe to those events. This is more complex than batch, but it keeps data fresh without polling.

What about cloud-based EHRs—do they change the calculus?

Cloud EHRs often have better API performance and scalability, which makes on-demand more feasible. However, they also may charge per API call, so cost becomes a factor. Additionally, cloud EHRs may have rate limits that affect real-time workloads. The same criteria apply, but you need to factor in API pricing and limits. We recommend running a cost estimate based on expected query volume before committing to on-demand.

Is there a middle ground between batch and real-time?

Yes, several. Near-real-time patterns like micro-batch (e.g., every 30 seconds) or streaming with a short buffer offer lower latency than batch without the full complexity of real-time. These are often a good starting point for teams new to on-demand integration. You can always tighten the latency over time as you gain confidence.

Recommendation Recap Without Hype

No single integration pattern fits every clinical workflow. The right choice depends on latency tolerance, data volume, error recovery complexity, source system constraints, and regulatory requirements. We recommend the following next steps:

1. Map your data flows. List every integration point, its update frequency, and the clinical impact of stale data. Use the five criteria to score each flow.
2. Start with batch for the majority. Most clinical data flows are not time-sensitive. Use scheduled batch for reporting, billing, population health, and historical analysis. This keeps infrastructure simple and costs low.
3. Identify the critical few flows that need on-demand. These are typically clinical alerts, medication checks, and real-time dashboards. Implement on-demand only for these, using standard protocols like FHIR or message queues.
4. Design for hybrid from the start. Even if you start with batch, plan for the possibility that some flows will migrate to on-demand later. Use unique identifiers and timestamps in your data model to support both patterns.
5. Invest in monitoring and error handling. On-demand flows require more operational maturity. Build the tools and processes before you need them, not after a failure.
6. Review and adjust quarterly. Clinical workflows change. What was non-urgent last year may become urgent as new services are added. Schedule regular reviews of your integration patterns to ensure they still match operational reality.

This framework is not a one-time decision. It is a practice of continuously aligning your integration architecture with the clinical workflows it supports. Start with the matrix, build incrementally, and let the clinical context guide your choices—not the latest technology trend.