free counter

The uncomfortable truth about operational data pipelines

Data infrastructure


Were you struggling to attend Transform 2022? Have a look at all the summit sessions inside our on-demand library now! Watch here.

The planet is filled up with situations where one size will not fit all shoes, healthcare, the amount of desired sprinkles on a fudge sundae, to mention a few. You can include data pipelines to the list.

Traditionally, a data pipeline handles the connectivity to business applications, controls the requests and flow of data into new data environments, and manages the steps had a need to cleanse, organize and present a refined data product to consumers, inside or beyond your business walls. These results have grown to be indispensable in assisting decision-makers drive their business forward.

Lessons from Big Data

Many people are acquainted with the Big Data success stories: How companies like Netflix build pipelines that manage greater than a petabyte of data each day, or how Meta analyzes over 300 petabytes of clickstream data inside its analytics platforms. Its an easy task to assume that weve already solved all of the hard problems once weve reached this scale.

Unfortunately, its not that easy. Just ask anyone who works together with pipelines for operational data they’ll be the first ever to tell you that certain size definitely will not fit all.


MetaBeat 2022

MetaBeat provides together thought leaders to provide help with how metaverse technology will transform just how all industries communicate and conduct business on October 4 in SAN FRANCISCO BAY AREA, CA.

Register Here

For operational data, that is the info that underpins the core elements of a small business like financials, supply chain, and HR, organizations routinely neglect to deliver value from analytics pipelines. Thats true even though these were designed in a manner that resembles Big Data environments.

Why? Because they’re attempting to solve a fundamentally different data challenge with basically the same approach, also it doesnt work.

The problem isnt how big is the info, but how complex it really is.

Leading social or digital streaming platforms often store large datasets as some simple, ordered events. One row of data gets captured in a data pipeline for a user watching a Television show, and another records each Like button that gets clicked on a social media marketing profile. All of this data gets processed through data pipelines at tremendous speed and scale using cloud technology.

The datasets themselves are large, and thats fine as the underlying data is incredibly well-ordered and were able to start out with. The highly organized structure of clickstream data implies that billions upon vast amounts of records could be analyzed very quickly.

Data pipelines and ERP platforms

For operational systems, such as for example enterprise resource planning (ERP) platforms that a lot of organizations use to perform their essential day-to-day processes, however, its an extremely different data landscape.

Since their introduction in the 1970s, ERP systems have evolved to optimize every ounce of performance for capturing raw transactions from the business enterprise environment. Every sales order, financial ledger entry, and item of supply chain inventory needs to be captured and processed as quickly as possible.

To do this performance, ERP systems evolved to control thousands of individual database tables that track business data elements and much more relationships between those objects. This data architecture works well at ensuring a person or suppliers records are consistent as time passes.

But, since it works out, whats ideal for transaction speed within that business process typically isnt so wonderful for analytics performance. Rather than clean, straightforward, and well-organized tables that modern online applications create, there exists a spaghetti-like mess of data, spread across a complex, real-time, mission-critical application.

For example, analyzing an individual financial transaction to a companys books may need data from upward of 50 distinct tables in the backend ERP database, often with multiple lookups and calculations.

To answer questions that span a huge selection of tables and relationships, business analysts must write increasingly complex queries that often take hours to come back results. Unfortunately, these queries simply never return answers with time and leave the business enterprise flying blind at a crucial moment throughout their decision-making.

To resolve this, organizations try to further engineer the look of these data pipelines with the purpose of routing data into increasingly simplified business views that minimize the complexity of varied queries to create them better to run.

This may work theoretically, nonetheless it comes as the expense of oversimplifying the info itself. Instead of enabling analysts to ask and answer any question with data, this process frequently summarizes or reshapes the info to improve performance. This means that analysts will get fast answers to predefined questions and wait longer for the rest.

With inflexible data pipelines, asking new questions means heading back to the foundation system, that is time-consuming and becomes expensive quickly. If anything changes within the ERP application, the pipeline breaks completely.

Instead of applying a static pipeline model that cant respond effectively to data that’s more interconnected, its vital that you design this degree of connection right away.

Instead of making pipelines ever smaller to split up the problem, the look should encompass those connections instead. Used, this means addressing the essential cause of the pipeline itself: Making data accessible to users minus the time and cost connected with expensive analytical queries.

Every connected table in a complex analysis puts additional pressure on both underlying platform and the ones tasked with maintaining business performance through tuning and optimizing these queries. To reimagine the approach, one must look at how everything is optimized once the data is loaded but, importantly, before any queries run. That is generally known as query acceleration also it offers a useful shortcut.

This query acceleration approach delivers many multiples of performance in comparison to traditional data analysis. It achieves this without needing the info to prepare yourself or modeled beforehand. By scanning the complete dataset and preparing that data before queries are run, you can find fewer limitations on what questions could be answered. This improves the usefulness of the query by delivering the entire scope of the raw business data that’s available for exploration.

By questioning the essential assumptions in how exactly we acquire, process and analyze our operational data, its likely to simplify and streamline the steps had a need to move from high-cost, fragile data pipelines to faster business decisions. Remember: One size will not fit all.

Nick Jewell may be the senior director of product marketing at Incorta.


Welcome to the VentureBeat community!

DataDecisionMakers is where experts, like the technical people doing data work, can share data-related insights and innovation.

If you need to find out about cutting-edge ideas and up-to-date information, guidelines, and the continuing future of data and data tech, join us at DataDecisionMakers.

You may even considercontributing articlesof your!

Read More From DataDecisionMakers

Read More

Related Articles

Leave a Reply

Your email address will not be published.

Back to top button

Adblock Detected

Please consider supporting us by disabling your ad blocker