Avsnitt
-
Hey there,
You might already know that we’re doubling down on our latest campaign, Let’s End The CDP Battle — check out the campaign trailer in case you missed it.
Our goal here is to clear the air and make the CDP space a little less divided. And we hope to do so by bringing people together who believe that the Composable vs Packaged CDP battle is pointless.
In today’s episode, I was joined by Luke and Glenn who work with vendors from both camps and have a deep understanding of what it takes for organizations to implement a CDP-like solution successfully.
Luke was at MessageGears and is now at Snowflake, and Glenn runs Human37 where they implement CDPs of all shapes for companies of all sizes.
They both offered some valuable insights based on their experience working with CDP vendors as well as CDP customers. And to keep things fun, I also had Luke respond to the following statements with his very personal opinion:
* The Composable CDP will beat the Packaged CDP
* Composable CDP is largely a marketing term propagated by Reverse ETL vendors
* Snowflake and the other cloud providers will make ETL and Reverse ETL obsolete in the next 5 years
* There’s an opportunity for both CDP camps to come together and solve more pressing problems related to data governance and privacy compliance
All in all, here’s what’s been established so far:
Without organizational context — needs, goals, and priorities, as well as resources, culture, and philosophy — one cannot decide which approach between Composable and Packaged is better, cheaper, or faster to implement.
Moreover, knowing which of the two approaches is more suitable requires organizations to look inwards and assess what might work best for them.
We had a lot of fun recording this conversation, hope you enjoy watching it. It’s only 13 mins and I’m sure if nothing else, it’ll leave you entertained!
You can tune in on Apple, Spotify, Google, or YouTube or watch the full thing on LinkedIn and share your thoughts with us.
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
What good is data if go-to-market or GTM teams are unable to put data to work or operationalize data in their day-to-day workflows?
Operationalizing data entails going beyond deriving insights from data by leveraging it to improve everyday operations — from delivering better content and communication to identifying customers that might churn in the near future and taking appropriate action to prevent that from happening.
In this super insightful episode, Matthew Brandt, an expert data practitioner and ace content creator, shares his experience and insights on putting data to work inside modern organizations.
In his past roles, Matthew has implemented several internal data products (and given them cool names) that are only now being productized and made available as plug-and-play solutions.
Don't miss the conversation, especially if you wish to operationalize data without a fancy 4-layer data stack!
Listen now on Apple, Spotify, Google, or YouTube.
Key takeaways from this conversation:
Q. What are the prerequisites to operationalize data?
Matthew (02:08):
If you don't understand the problems you aim to solve with data, you're just going to come in guns slinging, and you'll just be shooting wild, and you won't actually hit your target.
Q. Is it non-negotiable to have a data warehouse if you want to operationalize your data?
Matthew (03:26):
In the long term, yes. But for companies just starting out, it's certainly feasible to say, oh, I'm going to take my Salesforce data and I'm going to push it into Intercom. There's nothing wrong with that because you're helping solve a day-to-day issue that the business has. No fancy analytics, no fancy machine learning, no four-layer data stack involved, just one sync between two different tools.
There’s certainly nothing wrong with a point-to-point integration like the one Matthew describes above.
When I led growth at Integromat (Make), we were growing very fast (adding 1k users every day) but we didn’t have a stack of data tools — we didn’t even have a data team. So I built many point-to-point integrations and it got the job done just fine.
In the early stages, you can either optimize for efficient growth or slow things down in favor of a future state where data is pristine but it might never get used. Here’s a fascinating read on the current state of big data that states that a big chunk of data that is used (queried) is less than 24 hours old.
Q. How can operationalizing data help sales teams in their day-to-day?
Matthew (05:37):
We had a process that fired off an alert into a Slack channel when a customer was actively on the billing page and looking at trying to change their subscription. And sales would proactively reach out to them and ask them how they're doing and if they wanted to adjust their subscription. And customers basically just felt, oh, that's crazy timing.
Matthew is referring to a project titled Slow Turtle that he implemented four years ago using Google Tag Manager to capture events and fire Slack alerts using webhooks.
This has now been productized by Product-led Sales tools proving that most SaaS tools in the market have been built inside many organizations — I find it fascinating when I hear about internal tools like Slow Turtle.
Q. How can growth teams go beyond consuming insights and use data to engage and activate users and customers?
Matthew (08:19):
You need to obviously activate the lead that you're bringing in and part of that activation means understanding what the needs of the specific users are.
One of the things that I've worked on in the past is using demographic data that we get from other sources, and tying that into the product usage data. So we could see, for example, this lead comes in, they're probably between 30 and 40, they’re from this location, they're probably working in this type of industry, if we can pull any kind of social data for them, then we understand their needs a little bit more, and then they can funnel them into different types of messaging.
So we can show them in-app messaging in the product that's slightly different for their use case. If they come from the pharmaceutical industry, they're going to use the product in a different way than they're when they're coming from the finance industry, right?
100%. Understanding the needs and priorities of different sets of users is the only way to deliver truly personalized experiences — not only in-app but across every audience touchpoint.
Matthew mentions combining demographic data with product usage data — two subtypes of first-party data. Brands can go a step further by throwing in zero-party data into the mix — data that brands collect explicitly from their audiences by asking rather.
Here’s a post that covers the key differences between first-party and zero-party data.
Q. What's the one piece of advice you have for companies that are early on their data journey?
Matthew (09:37):
Please, if you're at a company that doesn't have an established data function yet, or is just getting started, please focus on the people and the processes. Do not focus on the tool stack. There are tools available for everything. So it isn't a question of, what tool can do the job or if a tool can do the job. It's a question of, what tool can do the job for you. But you can only do that once you've established what the job actually is.
You need to really solidify the processes that you have, and you need to really hire the right people who have the mindset to understand the problem before moving forward on huge contracts with vendors. Because once you've gone down that route, it's not so easy to just flip vendors. That's unfortunately a reality — the lock-in is real.
Connect with Matthew:
* Personal website
* LinkedIn
* Twitter
* YouTube
Thanks for reading/listening — let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
Saknas det avsnitt?
-
→ Link to the series
Stream processing is another key component of real-time or streaming data infrastructure.
Zander Matheson is building Bytewax, an open source Python framework to build real-time apps using streaming data.
In this episode, Zander explains stream processing in simple terms, touches upon the difference between stateful and stateless stream processing, and describes some of its common use cases and benefits.
This episode concludes the series on real-time analytics.
Let’s dive in:
Q. In simple terms, what is stream processing?
Stream processing very generally is the ability to process over a bounded or unbounded stream of data, so you process one piece of data at a time which generally happens in real time.
Q. Can you briefly explain what Bytewax does and how it works?
Bytewax is an open source framework — a stateful stream processor that allows you to easily build on top of streaming data.
Think of it as a tool to build applications that leverage streaming data. It could be something advanced like an online machine learning algorithm for anomaly detection or it could be something simple where you are just transforming that data in real time. So Bytewax gives you the pieces to connect to streams of data, manipulate them, and then connect to downstream data systems.
Q. What are the key differences between stateful and stateless stream processing?
One of the hard things about processing streaming data is maintaining a picture of what's going on so that you can do more advanced things with it, and that's basically stateful.
Let's say I want to know all the things that happened for a certain user over a period of time, that's when I’d bring in state because I want to have a window of time and then aggregate data about a user, so I have to maintain this information about the user over time.
On the other hand, stateless stream processing only lets you act on one piece of data at a time and it very much simplifies the problem because you don't need to know what else is going on and you can scale it more easily.
Q. How can a tool like Bytewax be used in conjunction with an open source OLAP datastore such as Apache Pinot?
So you can use various systems downstream of Bytewax; what you use depends on either how internally other people are going to interact with the data or how you're going to serve it.
Bytewax could be used for a transformation layer before an OLAP database if you have a bunch of dashboards running on top of real-time data. So with Bytewax, you could do transformations and adjustments to the data, maybe add in third-party data sources and then write it out to Pinot for your system downstream of that to run queries on top of that data and produce dashboards.
Q. What are the top to-use cases of stream processing technologies?
The most common use cases today for stream processing are applications for anomaly detection — things like fraud detection for credit card transactions, or in cybersecurity, detecting when there's some anomalous behavior.
I'm not sure what is the most common but those are two that involve anomaly detection which is a pretty good use case for streaming processing.
Q. Can you explain how stream processing can be used in e-commerce to build better shopping experiences?
Stream processing has really great use cases for personalization in e-commerce. Like Amazon, even small e-commerce sites can leverage stream processing technology to offer personalized recommendations to every shopper.
Q. Which industries benefit the most from stream processing tech? Are there industries where stream processing is non-negotiable?
To answer this question it's best to zoom out.
Stream processing adds complexity to your infrastructure and so it's important that there's an ROI associated with the change that you make.
If you want to leverage stream processing, you need to make sure that that increase in complexity is going to be worth it in the end.
Industries where the closer you can get to real-time for making decisions or improving the user experience are the best industries to adopt real-time tech.
In terms of being non-negotiable, there are plenty of instances in IoT where it's a non-negotiable to have stream processing for connected devices.
You can think of connected cars or situations where you need to make a decision for the user in real time — like Uber and Lyft when they're trying to match the right driver to the right person. Ultimately, people will open Uber and Lyft, request a driver, and go with whichever is faster.
Therefore, for companies like Uber that cannot really exist without real-time technology, it makes sense to increase the complexity of their infrastructure to provide a better experience and ultimately increase revenue.
Q. Last question — what's the one piece of advice you have for companies that are evaluating stream processing or other real-time technologies?
Coming back to what we were just talking about, it's like a double down on that.
Can you increase the revenue generated by the product or decrease your costs? Can you affect the company's bottom line with the adoption of real-time tech?
That's basically it. Because there is an increase in the complexity of the code that you write and the systems you maintain.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
Exploring data tools?
Let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
There's so much great content about building data infrastructure and adopting modern data technologies — this show too has been focused on this aspect of data.
But once companies have the infra, the data, and everything in between, they have to start measuring the return on available data or ROAD.
So how can companies go about measuring and increasing ROAD?
Ruben Ugarte, a decision-making expert who has helped companies of all sizes increase their return on data, has some answers.
Let’s dive in:
Q. What is return on available data (ROAD) and why should companies care about measuring it?
The big focus for a long time has been how to collect more and more data, which is effectively a solved problem — even the smallest companies can collect millions of data points relatively easily, and data collection will continue to get even easier.
Data teams and companies, therefore, need to focus their efforts on deriving more insights from their data. And that's not a question of volume — it's really a measure of quality.
When we talk about return on available data, it's essentially the ROI on money, talent, and time spent on data efforts.
It's a simple calculation that sales, marketing, and other parts of the business go through; data needs to go through a similar process to understand if it’s profitable or if it’s just not worth the current investment.
Q. Besides infrastructure and tools, what else do companies need to measure and increase the ROAD?
Primarily, a better process for understanding the impact of data.
There's a spectrum when it comes to using data. On one end, there are companies that have data because every company has data but they're simply not using it — no one knows what data is available and how to access it.
On the other end, there are companies with sophisticated tools and mature data teams that are very good at using data. However, even within that spectrum, if you ask executives, "What is the value of all this data? How much money are you getting back from everything you're putting in?" Often, there’s no clear answer.
Having clarity on how a company makes decisions and where data plays what role has been very helpful. After all, the whole point of having data is to make better decisions.
All the reports, and dashboards, and tools, and infrastructure — all of it must help a team or an individual make better decisions and improve the outcome of their efforts.
A company I work with was all over the place when it came to decision-making — sometimes they used data, sometimes they did not, sometimes the connection between the report and the action they took was clear, but often, it wasn't. So, they couldn't say, "Well, you know, clearly, data is significant for us."
Therefore, to measure the impact of data, there has to be a concrete and consistent decision-making process.
Q. Is there a connection between a company's data culture and its return on available data?
Oh yes! I think there are levels to this connection.
The first level is very technology-driven. So many companies I’ve spoken to are very, very focused on technology — they want the best technology, which is fine, but it can also become a bit of an obsession.
"Is this really the best technology? Should we replace this with that?"
Eventually, companies realize, "Well, we have good technology so let's now focus on making sure everyone has data."
This is the second level where terms like data empowerment, data enablement, and data democratization come into play.
Companies begin to realize that they need good technology, but they need even better processes to get data into the hands of people.
In the third level, once there’s a good data culture in place, companies need to invest in training and education for teams to understand and use available data in their day-to-day, as well as to convert those efforts into something tangible for the business.
Q. We’ve established that technology is not enough, but how important is the role of a data warehouse here? Is having one non-negotiable?
No, I don’t think a data warehouse is mandatory — it’s a useful tool but I also know companies, especially if they're not as technical or sophisticated, that might benefit more from other kinds of tools.
I've always seen two types of approaches to dashboarding:
* Centralize everything into a data warehouse: Companies focus on bringing their entire universe of data into one place, and the warehouse powers data visualization.
* Use purpose-built tools but never centralize the data: This approach works for companies with fewer technical resources where they might have a really good product analytics tool, and a really good marketing attribution tool, both of which are then connected directly to a dashboarding tool.
Therefore, a data warehouse is helpful but it's not the only way to build a data culture and increase the return on data.
Q. You work with a wide range of companies — what do you generally see done well at companies with a high return on available data?
Three things:
* First, having a flexible mindset and understanding that in some cases, data will help, and in other cases, it will not. It sounds very obvious, but people get stuck and say, "Now that we have all this data, everything has to be data-driven — if you have an opinion that's not backed by data, don't even bring it up." Clearly, there's grace to this, it's not black or white.
* Second, having a really good decision-making process.
This goes back to one of the original points of having decision playbooks, being very bold about how the company makes decisions, and constantly improving those.
* Third, having a data culture.
Working towards building that culture, building the proficiency of people inside the company with data, and having the right technology of course, but really putting it all together to empower people to use the available data.
Q. Is there a specific role early-stage companies need to hire for to increase their return on data?
I always thought that a data analyst should really fit this role. However, I’ve found at many companies that analysts become too focused on the data — on creating reports and dashboards instead of focusing on the value to the business.
However, some companies have done a good job accidentally and sometimes on purpose by hiring someone who is technical enough to understand data — they might not even be an engineer, let alone a data engineer, but they understand statistics, and probability, and are generally much better at the business side of things.
This is the person saying, "We have all this data, this is what it means for the business, this is why it's worth it, and this is the ROI we’d get if we go down this path."
In theory, this person is a data analyst but in practice, the role has all kinds of names. I’ve seen people in operations and even executives who do this really well.
The kind of person I would look for is someone who is completely obsessed with how data can help the business and is able to cut all the other noise.
Q. Last question — what's your one piece of advice for companies that have a lot of data and are looking to increase their ROAD?
Companies need to become obsessed with insights.
They need to bring the same level of focus and obsession they’ve had with technology for such a long time — the obsession that has resulted in these amazing data stacks with all kinds of things like Reverse ETL, and warehouses, and this, and that. It's incredible!
Companies must divert this energy and obsession with technology toward insights.
Only then can they start asking questions like:
* How many insights are we generating every week?
* How do we generate more insights?
* What's the quality of the insights?
These questions ought to move companies closer to being data-driven, which is really the goal here at the end of the day.
So that’s my number one advice to companies — just become more obsessed with generating insights from data.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
If you’re a data practitioner who wants to become obsessed with how data can help the business, has some tips for you:
Help beat the gap between data people and non-data people — share this post with a friend!
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
What exactly is data knowledge management or DKM? What kind of companies need it and why? And how is it different from setting up a data catalog?
Nicholas Freund who is building a DKM solution at Workstream has some answers.
This episode is hosted by Kunaal Naik, a community member, data science coach, and YouTuber.
Can you please take out 2 minutes to share your preferences and make Data Beats more valuable for everyone? I’d really appreciate the help!🙏
Let’s dive in:
Q. What is Data knowledge Management or DKM?
From my perspective, data knowledge management allows people throughout a business or a company to get more out of data analytics and reporting assets.
It’s a single place to access the assets, consume training and knowledge, and ultimately collaborate with others throughout the business.
Q. Do companies need both a DKM solution and a data catalog?
Data knowledge Management is fundamentally different for a couple of reasons:
* DKM fundamentally integrates the data consumption layer of the stack, whereas a data catalog is fundamentally built around the data warehouse.
* With regards to jobs to be done and the users who use a DKM solution, it's much more about empowering business users by letting them utilize the analytics assets that have been already created.
Therefore, the majority of DKM users live within your CX/product/sales org rather than folks building out analytics for themselves.
Q. Can you briefly describe how one can maximize their value in using a DKM solution?
When it comes to the tipping point for when an organization might want to introduce a DKM solution into their environment, there are two vectors:
* First is the number of data assets and analytics assets that are out there within an environment that lead to questions like, “What should we be looking at and what should we trust?”
* Secondly, there are more and more tools that teams are using to consume data and it's no longer just a BI tool or one-off spreadsheets. Data is being pushed into all sorts of solutions such as product analytics tools, engagement tools, and operational systems.
And with more systems there's more sprawl, right?
And as organizations get through that entropy tipping point, that’s when folks would say, “We're really experiencing some acute problems here, so let's introduce a Data Knowledge Management solution”.
Q. Are there any prerequisites in terms of the data stack for companies to derive value from a DKM solution?
You can think of the value in terms of the ROI of your team's time, which translates into money and opportunity cost, and a DKM can help collapse the time spent training and enabling the organization.
With regards to the prerequisite, a DKM solution is more of a year-two type of purchase.
You're not going to purchase a DKM solution right after setting up Snowflake, for instance — the best time to introduce it is after the organization has really adopted the data stack it and that transition to entropy is starting to take place.
Absolutely!
Q. So which industries will need a DKM faster than any other industry?
We see a fit amongst the early adopters of the modern data stack — technology companies, retail, media, lots of direct-to-consumer companies, and generally speaking, companies that have achieved some level of complexity and scale
Q. Last question — what's your advice for companies evaluating a data cataloging or knowledge management solution?
I’d just think about the most acute problem that the organization is experiencing — which relates both to what the business is and how the analytics function has been set up.
If an organization is focused on the true pain points it’s experiencing, and implements a solution to solve those pain points, they’re going to be in great shape.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
What would you like to see next from Data Beats?
Data Beats provides high-quality educational content for free and while that’s not going to change, I want you to derive more value from what we do.
So let me know your preferences and help the community grow! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
→ Link to the series.
There's a lot to like about warehouse-native apps, but can they easily replace their traditional counterparts? Not really!
When it comes to email engagement, a warehouse-native email tool isn't always suitable — they're especially not a good fit for real-time use cases.
This episode is hosted by Luke Ambrosetti, a community member, former guest, and leading expert in the Warehouse-native space, who works with companies building apps on the Snowflake Data Cloud.
Our guest, Chris Hexton from Vero, an email marketing tool that also offers a warehouse-native version, explains what one needs to make a warehouse-native app work, when it's not the right choice, and why sometimes it makes sense to use both.
Let’s dive in:
Q. How would you define a Warehouse-native App or what Snowflake calls a Connected App?
I would define it as a software application that does not have its own data backend.
Instead of syncing and then storing a copy of our customer’s data, Vero connects directly to the customer's data warehouse and uses that as the data backend.
Thereon, all data read and write is from and back to the customer's data backend.
So users of Warehouse-native Apps don't need to sync or store customer data, eliminating data integration work.
Does that come with a caveat? Are there any limitations?
As you said, this can really reduce the costs as the data is not being stored twice, it just stays in one place — the customer’s data warehouse. And one doesn’t have to set up new data pipelines or new libraries to sync data to the email marketing tool.
Not having to sync data doesn’t mean this isn’t any work to be done. You obviously need a data warehouse and there needs to be data in there that can be used to create audiences and personalize emails. That’s the caveat.
Q. Okay but what are the specific steps that need to be taken in the data warehouse for a Warehouse-native App to work?
The two main steps are data collection and transformation.
To bring the data into the warehouse, one needs to instrument their product (telemetry) using CDI tools like Snowplow, RudderStack or Segment. Additionally, one might need to ingest data from third-party apps such as Stripe into the warehouse for which ELT tools like Fivetran or Airbyte can be used.
The second step, data transformation, is making sense of that data — joining it, stitching it, etc.
For email marketing use cases, one needs to build audiences using the data made available in the warehouse. The stitching of data, therefore, entails taking data from disparate sources and creating one row or record per user so that the marketer has all the data she needs to build a specific audience for activation purposes.
This second step of cleaning, and organizing the data is a critical step, for which most of our customers use dbt.
That makes perfect sense.
So a lot of the traditional email marketing tools offer a visual segmentation layer, making it easy for non-data folks to iterate and build workflows quickly.
Q. Do warehouse-native email tools also offer that?
One of the key selling points of any marketing automation or customer engagement tool is to enable marketers to do their job without relying on others, as efficiently as possible — not all marketers are super familiar with SQL and they’re definitely not familiar with the raw data floating around in the data warehouse.
At Vero, we've launched a very raw interface where one has to use SQL to build audiences or segments. That said, we’ll soon release a visual editor (visual segmentation), similar to what traditional SaaS tools offer.
So to answer your question, the visual segmentation or querying capability of a warehouse-native app works differently than that of its traditional counterpart, and there are two possibilities here:
* Data engineers can set up certain tables and views in the data warehouse and we can then build a traditional, visual drag and drop condition builder on top of those tables and views.
* The other way is to have data engineers build SQL queries with variables that a less technical user can select and play around with, without really seeing the underlying SQL.
These are the two main schools of thought and I believe both bring it up to a level that’s very approachable for a less technical person.
Luke: I agree with those two approaches. You don't have to fiddle with APIs and can do all the transformation work within the data warehouse, which you're probably already doing anyway.
Yeah I've seen you talking on LinkedIn about those traditional tools that promise a no-code or low-code set up, but one still needs data in those tools and that's often where things get stuck, right?
Maybe the data you actually need as a marketer never even makes it because the support from the rest of the org just isn’t available.
Hence, even in the early days with the less than a user-friendly interface that, we still see marketers who are super pumped because they have real confidence in the underlying data in a way they didn't before.
Vero offers two products — Vero Cloud, the traditional email marketing tool and Vero Connect, the warehouse-native component.
Q. What are the pros of each and when should each of those be used?
Vero Cloud, our core product, has been around for a while whereas we launched Vero Connect about 12 months ago. The key thing you need to use Vero Connect is a data warehouse with some (usable) data in it.
If you don't have a data warehouse in place, or you don’t have the resources to wrangle and transform the data in it, then perhaps a traditional email tool (Vero Cloud) is a better fit.
For relatively small or new organizations that are maybe graduating from something like MailChimp, Vero Cloud is a great fit 'cause in essence, we're spinning up a managed data warehouse for the customer behind the scenes and giving them the tools — such as forms to capture blog subscribers — to get relatively simple data in there.
As organizations mature, they graduate to the “connected way” of doing things — that's at least where we're at the moment.
Do you have customers using both products? If one uses the traditional version, why would they want the warehouse-native version?
Yes, we've definitely got a couple of customers using both.
Building on my answer before, we’re seeing that there are several different use cases for email marketing in an organization.
The killer use case for the warehouse-native version is product (lifecycle) marketing that includes engagement that takes place after a user signs up — product updates, announcements, or automated messages to nudge customers toward activation.
On the other hand, for top of the funnel use cases such as newsletters, customers usually don't have requisite data in their warehouse, and that's where they prefer using a traditional SaaS email marketing solution.
Why might someone move from one to the other?
As a company scales, they collect and wrangle more data and eventually have to keep millions of rows in sync across several different tools which can be very costly in terms of time, and one might fear that the data isn’t actually syncing as expected, (resulting in data quality issues).
Therefore, as the org matures and hires their first data engineer (or someone puts that hat on), they end up with a data warehouse, leading to more trust in the data as compared to the data lying across different cloud solutions (such as Vero Cloud).
Email marketing/engagement is such a wide category wih teams using multiple email tools.
Q. Which use cases are not fulfilled by warehouse-native solutions today?
Another area where a warehouse-native app is not a good fit now, and may never be, would be transactional email.
Emails that need to be sent milliseconds after someone requests a password change, for example, is not a good fit for the warehouse-native version because generally speaking, people rebuild the models and the views in the warehouse on a periodic basis that's definitely not going be milliseconds.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
If you enjoyed this episode hosted by Luke, you should check out the one where he’s the guest:
Thanks for reading — let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
→ Link to the series
In part 1 of this series, we covered how streaming data analytics is going mainstream.
Therefore, it’s useful to understand the role of change data capture or CDC in real-time analytics since CDC is a key component of streaming data infrastructure.
In the episode, John Kutay from Striim, a real-time data integration solution, answers some fundamental questions about CDC like:
* What is the role of CDC in analytics?
* Why should companies care about capturing changes to data in real time?
* How is CDC-enabled real-time ETL is different from batch ETL?
And real-time analytics isn't just for data folks to understand.
John also explains how data-adjacent teams like Product and Growth can leverage real-time data to supercharge their day-to-day workflows.
Let’s dive in:
Q. What is the simplest definition of Change Data Capture or CDC?
In broad terms, change data capture is simply tracking changes in a system.
You can compare it to a change log — if you go into Google Sheets or Google Docs, you'll see a log of changes that have taken place (version history).
Data systems have similar concepts — databases have write-ahead logs which journal all the changes that take place inside a database, and change data capture is the process of tracking and collecting those changes in a usable manner.
Q. What is the role of CDC in analytics?
There's a long history of CDC — it was initially built for the recovery processes of databases.
When databases were first being built and rolled out into enterprises, the write-ahead logs from the databases were used as a disaster recovery mechanism. If a database was shut off in the middle of an operation, enterprises could use the logs to take the database back to a normal state.
However, the role of CDC is also very applicable to analytics because you can use that same process of mining the changes from the write-ahead log to feed the data into operational analytics systems such as data warehouses (like Snowflake and BigQuery) where you’d run your analytics and reporting.
Q. What are the prerequisites in terms of the data stack to enable change data capture?
Initially, change data capture projects will start where there's some use case for the analytics team to pull data out of the operational database.
Let's say your dev team uses like MongoDB or Postgres as your backend database where you're tracking customer payments or signups, and your analytics team wants to build reports on top of that data.
In order to do that, you need to make sure that it's a cross-functional effort where engineering and analytics teams are working together to say, "Hey this is how we're gonna get the data, make sure it's secure, make sure it's efficient" because you don't want to create a client that's running more queries on top of a production engineering database.
You want to enable change data capture which is efficiently mining the logs. For instance, if you're running AWS RDS Postgres or Aurora, you have to enable write-ahead logging and think about the file rollover timeframes, etc.
CDC is definitely cross-functional with the engineering teams that own the database and the analytics teams that wish to leverage that data.
Q. Why should companies care about capturing changes in real time?
So there's always the classic batch ELT which is just taking changes from one system and applying it to another.
However, capturing changes in real time can be a real competitive advantage in terms of building real-time customer experiences.
Think about your everyday apps such as Uber — it basically connects you to a driver who's in your area now, not one who was in your area 30 minutes ago, and Uber uses real-time data infrastructure to do that.
Q. And typically how big or small are data team companies that successfully implement a CDC infrastructure?
It's across the board. Bigger teams might invest in rolling their own CDC infrastructure, but since they have more responsibilities, they might end up with an out-of-the-box product like Striim.
That said, small data teams can also use change data capture because ultimately, it's about collecting data from the cloud database and pushing it to the analytics system — essentially, teams of all sizes can implement a CDC infrastructure.
Q. Can you briefly explain how CDC-enabled real-time ETL is different from Batch ETL?
Batch ETL is inherently built on batch processing systems. Whenever a tool has terminology in it like "transform jobs" or "sync jobs", it's essentially collecting and processing data in batches which will always introduce latency somewhere in the pipe.
Even if I'm doing change data capture from the database, if the transform job and load job are on a batch frequency, that's going to add at least 15 to 30 minutes or an hour of latency in the process.
CDC with streaming will actually enable streaming ETL where you can capture and load the data as soon as it's available.
Q. Can you describe the top two use cases for real-time data streaming?
Depending on the industry there are tons of popular use cases for CDC.
A major airline I work with leverages CDC to send maintenance data from aircrafts to the ops teams in real time to cut down the cycles where people spend time waiting on the airplane for maintenance.
Another healthcare company I work with is able to take health records and put them into a smart analytics system to centralize it for their care teams in real time as well.
But a very generic horizontal use case is simply moving data from operational systems to analytical systems in an event-driven format without copying it.
Essentially, CDC eliminates waste and opt imizes performance and costs in the process of moving the data.
As you know, the goal here with this show is to enable less technical people or even non-data people to learn more about this stuff.
Q. So how can Product and Growth people — folks who work in data-adjacent teams — use real-time data in their day-to-day?
I run the growth team here at Striim and I'm very familiar with taking data from databases and actioning it for RevOps, Sales, and Marketing use cases.
For example, activating customer data during instances such as, "Okay this customer's usage has spiked in the last 30 minutes, so we should assign a support engineer right now to make sure that the customer isn’t running into any issues or incurring unforeseen costs which may upset them".
Or building real-time customer experiences by ensuring that the inventory that customers see when shopping on an e-commerce site is real-time and not stale."
Imagine this: You visit an e-commerce site, an item you wanted is in stock and ships tomorrow, you buy it, and then suddenly you find out that the items is actually out of stock (the inventory data was stale). This is not an experience you’d want your customers to have.
These are a couple of common examples but there are many more use cases for data-adjacent teams to leverage real-time data in their workflows.
Last question — what should companies look for when evaluating CDC vendors?
There's been so much innovation in the modern data stack and cloud products that have made it easy for people of all skill levels to do analytics — I believe people should double down on that strategy when looking for CDC vendors.
Look for a product that runs fully in the cloud and handles all the edge cases out of the box.
Especially if you have non-technical users who wish to leverage real-time data, and you want very low maintenance in terms of CDC.
So I’d definitely recommend an out-of-the-box solution for teams that wish to do very good operational analytics with both non-technical and technical people working together.
If your team is purely technical and you have like a very large engineering organization, you can consider stitching together a bunch of tools, especially if you must build things in-house.
Whether you go with a build-your-own or a fully managed solution like Striim for change data capture, you must guarantee to your business users that you're meeting the data SLAs and SLOs.
The product should deliver the data within the timeframe that your stakeholders expect it, and no matter what, it should make it very easy for you to have visibility into your CDC workflows.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
If you found this useful, check out part 1 of this series on real-time analytics infrastructure:
Exploring data tools?
Let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
Scott Brinker, the editor of chiefmartec.com and the VP of platform ecosystem at HubSpot, is the leading authority on marketing technology.
Scott has spent almost 15 years dissecting the martech industry and in this episode, he explains how martech is becoming data-first and what's causing this shift.
He also answers some questions about data integration technologies — ETL/ELT, Reverse ETL, iPaaS, and CDP — and shares his thoughts on warehouse-native apps.
This episode is packed with insights that'll certainly help you make more informed decisions in regard to buying data and marketing tools.
P.S. If you can’t spare 15 minutes, 9:05 - 9:40 is my favorite part. If you’re reading, here you go.
Let’s dive right in:
Q. How is Martech becoming data first and what is causing this shift?
Obviously, data is embedded in all Martech. All Martech is essentially digital and everything digital is driven by data.
But I think for the first 10 years or so, a lot of companies were thinking less about the data component of it and more of just the actual functionality — how do we learn these new channels? How do we run these new programs? How do we run these new campaigns and there was all this data involved in that?
But now that the services we're using are becoming more mature, we're understanding the opportunity to get more leverage out of those services by getting better with the kind of data that we are feeding into them.
There’s an infinite amount of data inside our companies and available through partners in the second-party data world. We don't even have to go to third-party data; we have more data than we could possibly use at this moment. And so I think it's a really exciting time for Martech professionals to be upleveling how they think about what data they can use and how they can use it effectively.
Q. Do you think Martech tools will soon become data tools and the ones that don't will become irrelevant?
Yeah at some level every Martech tool has to be a data tool.
But I think it's fair to say that some have had relatively basic data capabilities. A lot of Martech tools were very siloed when they were first created.
They had their own little database of — here's what's happening inside this tool. But they weren't necessarily as aware of — oh well what are the other signals that are happening outside of our tool that we should be aware of? As we have engagements and activities within our tool, are we making sure we're passing those signals and information further downstream to others?
I think you've seen now that the innovation happening in Martech is really happening around having better data-in and data-out capabilities.
And then of course, as you get more data in, how do you have better abilities to apply the data inside the tool?
So I think you're right — if a Martech tool doesn't keep up with this movement, they do seriously risk falling into irrelevancy.
As the VP of platform ecosystem at HubSpot, you have an insider view of the most pressing needs of GTM teams.
Q. Do you think GTM teams generally care about the technologies that are used to solve their problems?
One of the things we've seen across marketing, sales, customer success, partner management, and partner ecosystems, is all of these teams have Ops professionals — some of whom are now being aggregated into these Revenue Ops or RevOps teams.
And since leveraging digital capabilities and data has become so fundamental to everything our go-to-market teams want to do, these Ops teams have really taken on the massive responsibility of delivering on what GTM teams need.
Now, inside the Ops team, I think you'll have people who are very savvy about the tools they're using — a big part of their job is to manage the existing toolset, understand how it's evolving, as well as evaluate new capabilities that they can bring into the stack.
As you get further away from the inner workings of the Ops team, the rest of the organization is less concerned about the specific technology and more about, “hey can you deliver the actual program, capability, campaign, that I need to deliver my results (and improve the outcomes of my efforts)”.
Q. Since you talked about Ops teams, what are your thoughts on the gap between data teams and non-data teams? Are Ops teams suited to beat this gap?
Yeah, and this is one of the things where there's a wide spectrum inside the Ops profession right now.
I would say all of them are relatively data-savvy — that's a core part of what Ops is about.
But the truth is, when you're talking about how you're harnessing this data, it's not just about being savvy about data. There's increasingly these capabilities that involve data engineering — how are we piping these things and how are we managing data not just within these specific application silos but across the enterprise more broadly?
And as you get deeper and deeper into that, I mean this is some highly sophisticated work. Speaking of the Ops profession, there's a whole universe of what we would call Data Ops — people who are specializing in this (data pipelining).
So I would say most Marketing Ops and Revenue Ops people probably aren't that savvy — they're not Data Ops people, they're not data engineers but they know enough about what they need to do with the data and they can wrangle it from the business side to service the bridge between data specialists and the rest of the business users (non-data people) who just want see data applied (put in action).
Agree totally! So the rise of data warehousing has completely changed the Martech landscape by introducing new technologies like Reverse ETL and Warehouse-native apps.
Q. Do you think these have the potential to completely replace the way Martech tools are being built and sold?
Absolutely!
I've been writing a bit about this here over the past couple of years where, in many ways, the Martech stack grew up in its isolation, and the reason was very logical — digital marketing was moving faster than the rest of digital transformation inside an organization.
So marketers — this is part of why they built Marketing Tech and Ops teams — had to implement more capabilities ahead of what the rest of the organization was willing to do. Well, fast forward here a decade, and now every organization is going through — it's become cliché to even call it — digital transformation.
These digital capabilities are now being baked into the entire organization. As a result, the mission is changing where it's no longer about, “this nice, isolated tower of the Martech stack.” It's about how our marketing activities and capabilities interface with the broader technology infrastructure that the company now has specifically around data.
So this has been an exciting space as you see the Data Warehouse layer is increasingly becoming like a foundation which every department inside an organization is both able to contribute to and derive value from.
So yeah, integrating the Martech stack with the enterprise-wide data layer is probably the most exciting thing happening in Martech right now.
Got it, yeah. Now let's talk about Data Integration technologies. We've got ETL/ELT, Reverse ETL, iPaaS, and CDP.
Q. Do you see a future where companies keep paying for multiple Data Integration tools and technologies?
We've got this challenge throughout the Martech stack where there is significant overlap across different products and categories, and given the nature of how companies expand, I expect the overlap to grow further.
In some theoretical world, you would say, “Oh I don't want to pay for any overlap. I want, this one thing to do this and there's a very nice neat boundary, and then this other thing that'll do the other thing.”
That is an aspirational goal that we can head towards but today, inside most organizations, you have overlap. And that overlap is actually useful because depending on the particular way you're wanting to do something it might be easier, faster, and cheaper, to do it through one product versus the other. Having a little bit of optionality and flexibility can be beneficial.
👍👍
That said, kind of like Occam's Razor of the Martech Stack — you want your stack to be as simple as it possibly can be, but no simpler.
So do you really need an ETL, a Reverse ETL, an iPaaS, and a CDP? Probably not. But again, it really does depend on the specific structure of what's happening inside your organization.
🤔 Have questions for Scott?
Many organizations today continue to use iPaaS tool for ETL and Reverse ETL workflows.
Q. Few people understand iPaaS better than you do — can iPaaS fully be replaced by warehouse-centric data integration technologies like ETL/ELT and Reverse ETL?
No, and I think that's a good example because leading iPaaS players cater to a range of use cases and supporting data pipelining — ETL and Reverse ETL — is one of those use cases.
Another place where they promote more of their functionality is the ability to create business workflows at the application layer crossing different apps and departmental boundaries, which is something that ETL and Reverse ETL really don’t even have in the picture? I think that's an example where you typically see some people using an all-in-one iPaaS for both their ETL work and their business workflow management.
Then I think more often now, you're probably seeing cases where the data pipelining starts to be a fairly specialist capability where a lot of innovation and change is happening.
And the workflow side is still a massive opportunity — we have barely scratched all the possibilities there. I wouldn't at all be surprised for a lot of companies to have both ETL and that iPaaS workflowish capability in their stack.
I think so too. Now, you've made a ton of predictions over the last few years.
Q. If you had to make a prediction about Reverse ETL or Warehouse-native apps, what would that be?
I'm always cautious in predictions just because — was that the Yogi Berra — predictions are hard, especially about the future.
That said, I think the sophistication of the Data Warehouse layer is advancing incredibly rapidly. There was a time when the Data Warehouse was really nothing but dumb storage. It was basically like, “Yep, we just put it there and then if we actually wanna do anything with it, oh, okay well then we have to like pipe it out into different apps.” And the process of piping was slow and expensive and you ended up duplicating data all over the universe.
As this new generation of cloud warehouses continue to advance, it's more and more about saying, “Well hey, there's probably a whole bunch of activities of things you wanna do with your data that you don't actually have to pipe the data out of the warehouse to do.”
That we can keep the data in place but now we're able to actually conduct activities on it — do manipulations to it almost in a kind of virtualization of things that we had to shove elsewhere previously — which is an incredibly powerful direction.
There's a ton of innovation but it also raises a ton of questions regarding complexity. Some of the hardest work associated with data isn't the mechanics of the data — it's the agreement on the models and on the governance.
So in some ways, having all that stuff inside the one core warehouse probably makes the tech mechanics a little bit cleaner but also raises the bar on, “Well, what are the operational processes we need to put in place to, effectively do that without it just turning into chaos.”
Q. Last question — do you have any tips for companies evaluating data integration technologies that we just discussed?
Yes, if at all possible, actually get a free trial pilot program — plug it in and make it work.
The canned demo is nice, but so much of this comes down to like how easy is it for a particular technology to work in your environment?
And so I think just being really insistent on proving that out as part of the procurement process is generally sound advice.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
🧠 Want more of Martech? Scott and my friend, Juan who shares hard-to-find insights via The Martech Weekly recently launched a podcast that you don't want to miss.
🎶Playlist of the week: Mark Knopfler wants to remind us to be here now!⌛
Thanks for reading/listening/watching — let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
Jacques Corby-Tuech, a revenue ops practitioner and a keen observer of the data technology landscape, uses both a CDP and a Reverse ETL tool.
In this episode, he answers some fundamental questions about the two technologies and explains how they solve very different problems.
He also explains where a Reverse ETL tool shines, and why positioning one as a CDP replacement is "frustrating marketing that's only causing confusion in the CDP market."
If you're looking to adopt a CDP or a Reverse ETL tool, Jacque's answers will certainly bring much-needed clarity and ease the evaluation process for you.
And if you’re not, well, you’ll hopefully derive some hard-to-find insights!
Let’s dive right in:
Q. Tell us a bit about your work. What makes you so passionate about data technologies?
I work as head of revenue operations at CyberSmart, a cybersecurity startup.
My role is to look at data process and technology for the marketing, sales, and customer experience teams — I need to find technology to enable all these teams to do the work that they need to do.
It's really important to me that I find technology and I find ways of helping those teams achieve the results they want in as efficient and scalable way as possible — so that's my angle on technology.
I'm not interested in technology for technology's sake, I'm interested in technology for the outcome that it provides me.
You don't shy away from voicing your opinions!
Q. What’s the deal with Reverse ETL tools sometimes being labeled as CDPs?
It's frustrating marketing to me, really.
I've got a background in marketing and I can understand why they're doing it, but all it's doing is causing confusion in CDP market. You know, it's frustrating!
They've got a great product that has so many merits that I think causing that frustration in the market doesn't necessarily do them any favors.
Reverse ETL, in my view, is one of the many components of a ready-made CDP.
Q. What are your thoughts here?
I think that's exactly what it is.
Reverse ETL is one of the small — probably one of the smallest parts — of what a CDP is capable of doing.
I think CDPs obviously can do significantly more and Reverse ETL can do other things well, which are perhaps more interesting.
And I believe CDPs aren't going anywhere anytime soon.
Q. Even if lots of organizations adopt Reverse ETL tools and data warehouses, will these tools eventually replace a ready-made CDP?
I don't think that moment is ever going to happen.
The barrier to entry for many businesses to get a data warehouse is very significant.
I think most businesses or a lot of businesses will still be using a CRM as their warehouse, right?
So I don't think it's necessarily reasonable that everyone will be needing or wanting Reverse ETL, much in the same way that most businesses won't necessarily need or want a CDP. They'll probably want a CRM or a CDP or some sort of Reverse ETL, but not necessarily all of them. Or they may want a combination of them depending on what they're trying to do.
Q. Do you think there's room for a CDP and a Reverse ETL to co-exist?
Absolutely! That's something that I'm actually exploring right now.
We use a CDP and that's not going anywhere, but we are looking at Reverse ETL to replace an iPaaS product.
And actually, it's performing incredibly in that area — I've really enjoyed working in it and testing it, and really the comparison against iPaaS is night and day. It's a fantastic product!
Let's talk about iPaaS.
Q. Do you think Reverse ETL can replace iPaaS at some point?
Reverse ETL is doing one thing really well — getting the data from the warehouse to the destination.
One thing that iPaaS does exceptionally well is doing it the other way, right?
So your CRM, your marketing platform, whatever might end up then being the source of data, sending data somewhere else. So I think that Reverse ETL captures one component particularly, but not necessarily everything else.
And there's a lot of maturity in iPaaS that's built up in terms of data security, integration, all of that kind of stuff that is actually very important and meaningful in this market.
And let's talk about CDP implementation.
Q. What do companies really need to implement a CDP successfully and typically how long does it take?
It really depends on the CDP and what the organization is trying to get out of it. If I think to my kind of my experience with CDPs, I've primarily used Segment and where Segment really shines is in their instrumentation libraries.
And so my view is — you need engineers to help you implement that instrumentation, which, depending on the complexity of your product, could take a few days or a lot longer.
From an operational perspective though, CDPs can be really quick to set up because all you need to do is kind of connect your CRM data or connect your email platform data and suddenly you've got a data flow going into your CDP.
It really depends what you're trying to do — you can get a very quick, easy implementation and then build upon it through engineering resource.
100% agree with you on that.
Q. And what do companies need to succeed with a Reverse ETL tool?
So I've been testing out Hightouch to replace an iPaaS platform and it's been really nice and easy.
So long as I can connect to my warehouse — I spoke to my CTO and he gave me connection users to test it out — I can query the data using SQL or some other means, and then actually building out that integration is just really nice and easy.
So I think it's a great product in that respect!
🤔 Have questions for Jacques?
Q. Should non-data teams or GTM teams care whether data is made available to them via a CDP or Reverse ETL?
Absolutely not. A GTM team should only really care about what the tool is enabling them to do, not how they're getting data.
There are situations where that may change though. For example, if they want a real-time data stream, there are things that they're going to have to consider to get that.
In that respect, a CDP is better to provide that real-time data stream than a Reverse ETL tool.
It really depends on what outcome they're after, and what trade offs they're able to, and willing to make in terms of costs and resources.
We've recently seen some CDPs become Reverse ETL-like.
Q. Do you think at some point popular CDP vendors will build or buy Reverse ETL capabilities?
Oh, yeah! I'm sure if you were to look inside those companies, they were 100% building those products already — it's a no brainer for them.
Q. And what are your thoughts on ETL/ELT vendors like Fivetran building or buying Reverse ETL capabilities?
It makes sense if you look at Airbyte — they've purchased a Reverse ETL company (Grouparoo), and they're looking to do the full thing.
It makes sense for anyone in this market to be looking to expand, not just uni-directionally but bi-directionally. Again, it just seems like a bit of a no-brainer. It seems like if you can do all of your integration in a single tool, that's what you should be looking to do.
I wasn't gonna ask you this question, but now I'm gonna ask you anyway.
Q. Do you think companies would want to eventually just pay for one data integration tool? Or would they be happy to pay separately for ETL, Reverse ETL, CDP, iPaaS, etc?
I'm in two minds about this 'cause there's that constant cycle of bundling versus unbundling, which is really interesting to follow.
But there is a significant amount of value in having a single tool that does these things. The trade-off, of course, is that actually that single tool may not be as good as dedicated point solutions. So it's just balancing those trade-offs versus what your actual requirements are.
I think there's space for both to exist.
With the money that was sloshing around in the data space at the tail end of last year, start of this year, these companies are 100% gonna be investing in purchasing companies that broaden them into more of a bundled suite or they'll be looking to be acquired by the big players to create more bundled solutions.
So there's room for both, but it's gonna be interesting to see what happens in the market as consolidation probably happens.
Q. With the rise of warehouse-native apps, should Reverse ETL vendors should try to go beyond moving data and diversify their offerings?
Can they somehow even enable traditional SaaS companies to become warehouse-native?
That's a really interesting angle. You know what, I'm not 100% sure where they can go to be honest.
The reality is that not every tool in the future is gonna be built to integrate directly with the warehouse.
Your Salesforce Marketing Clouds, your Marketos, these tools are very unlikely to be changing their entire infrastructure in the near future, right?
There is 100% a market that is going to remain on the actual app-hosted platforms for a very long time — whether or not that market ends up shrinking or growing, I'm not sure. The market right now for warehouse-native tools is minuscule, I'd be surprised if it was even 1% of the total market.
So there are definitely things that will happen, but I can't get out a crystal ball and see what that's gonna be in 10-plus years — if we'll see any kind of significant movement there.
Q. Last question — what's your advice for companies that are contemplating whether to go with a ready-made CDP or a Reverse ETL?
Don't look at technology, look at your use case.
Look at what you're trying to achieve, and then score the technology against that. Focus on outcomes.
The most important thing is to just focus on the outcome that you're trying to achieve, don't focus on technology.
I’d also add that think about the needs of your teams and give them the tools they're comfortable using.
Q. Because technology is no good if nobody uses it, right?
Yeah!
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
🧠 If Jacques’ thoughts resonate, you’ll definitely enjoy his article on marketing and the modern data stack.
Thanks for reading/listening/watching — let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
→ Link to the series
Streaming data infrastructure is going mainstream.
Technologies built to cater to the needs of large-scale organizations like LinkedIn and Uber can now be utilized by startups to deliver hyper-personalized product experiences in real time.
However, making the case for streaming data infra and figuring out where to get started is not trivial. Thankfully, one of the best minds in the space, Dunith Dhanushka is here to help by answering some fundamental questions.
P.S. If you'd like to dig deeper into streaming and real-time analytics, Dunith has curated some great content on the DB forum and is happy to answer follow-up questions too.
P.P.S. The folks behind Startree built the open source real-time OLAP data store, Apache Pinot at LinkedIn. Every time you click on the "who's viewed your profile" button on LinkedIn, it is Pinot that runs a complex query to present that data to you in real time.
Let’s dive in:
Q. In simple terms, what is streaming data infrastructure?
First, I’d like to introduce you to events. Events come before streaming and represent facts about what has happened in the past. When we sequence these events into a stream, we call it streaming data — the infrastructure needed to capture, process, and make sense of events in real time.
Q. What are the prerequisites in terms of the data stack to set up streaming or real-time data pipelines?
Once you look at streaming architecture or the landscape from a high level, you can identify so many components. So we can categorize them based on their role and their responsibility.
* The first step is to produce events from your existing applications or operational systems which is done via SDKs or middleware tools.
* Secondly, you need a scalable medium to store these events and this is where real-time streaming platforms like Kafka and Pulsar come in.
* The third step is what we call “massaging” the data.
We have event producers and then the events are ingested into an event streaming platform, post which the events landing there need to go through some transformation because this is raw data we’re talking about.
It could contain some unwanted information and you often need to mask PII data, or sometimes map JSON into XML, or join two streams together and produce an enriched view, etc. This is what we mean by data massaging.
* In the fourth step, you need a serving layer to present or serve this aggregated or processed information to your end users — internal customers like analysts or decision makers inside your organization, or external users of your product.
In the case of external users, you need a real-time OLAP database or a read-optimized store to deliver real-time experiences.
Those are the four critical components that you need to set up your streaming infrastructure, but this could certainly vary based on the complexity of your use cases.
Q. So what exactly is Apache Pinot and what does it do?
Apache Pinot is a real-time OLAP database.
There are two things to note here — Real-time and OLAP (Online Analytical Processing).
Real-time indicates that Pinot can ingest data from streaming data sources like Apache Kafka, Kinesis, and Pulsar, and make that data queryable within a few seconds.
More importantly, Pinot makes it very fast to run complex aggregated OLAP queries, like the queries that scan multiple batches of data and run complex aggregations with sub-second latency and consistency, tuned for user-facing analytics.
Q. Can you give us a common example of user-facing analytics?
Yes. On LinkedIn, you'll get a notification, in real-time each time someone views your profile, right? That feature is called, "Who viewed your profile" and Pinot is what powers it.
That feature might seem simple but there are lots of complicated things going on to make it happen.
Pinot has to ingest real-time click or profile visits from all the front-end processes, store the data in scalable storage, and then run queries in real-time to answer lots of concurrent questions.
If you look at the scale of those queries, there can be multiple hundreds of thousands of queries executing on the database simultaneously.
🤔 Have questions for Dunith?
Q. What are the benefits of a real-time OLAP data store like Pinot over a regular data warehouse?
There are two main factors that differentiate Pinot from a data warehouse — Latency and Freshness of data.
Pinot can consistently produce queries over sub-second latencies, usually milliseconds. On the other hand, since data warehouses are tuned for internal use cases like exploratory analysis and BI, they produce single-digit latencies most of the time — seconds basically.
Secondly, with Pinot, the freshness of data can be maintained by ingesting from data sources in real-time, whereas with warehouses, one has to employ scripts or ELT tools to batch and load data into the warehouse periodically (on a schedule).
Q. Typically how big or small are data teams at companies that successfully implement streaming or real-time data infrastructure?
It actually depends on the complexity and the velocity of your data infrastructure — how fast you want to process your data and how complex your ecosystem is.
Let's say, you want to build a real-time dashboard for some product for which you can start with one data engineer assuming that all the ecosystem components are available as managed cloud services.
Then as your data velocity grows and your requirements grow, your product grows as well — you can then horizontally scale your team where some people can work on the serving layer, some can work on the stream processing, and others can work on data ingestion, and so on.
Q. How do data adjacent teams like Product and Growth, utilize steaming data that is usually stored in something like Pinot?
We see many use cases related to growth analytics and product metrics — especially at SaaS companies and product-led companies that use Pinot to capture and store their product and engagement metrics.
For example, the first step is to instrument the product using an SDK to emit data points as events into a certain streaming data platform, post which, we can configure Pinot to ingest from that data platform.
Once we have a fresh set of user engagement metrics such as button clicks and page views, we can utilize this vast set of data to understand user behavior.
Product teams can run analyses to derive metrics like DAUs (daily active users), or perform funnel analysis to understand points of friction and to calculate conversion rates. They can plug this data into a BI tool to build real-time reports while Pinot ensures that data is fresh and relevant.
Pinot can essentially help run these analyses really fast, while the data is still fresh and relevant — that’s what Pinot really excels at.
Q. Last question — what’s your one piece of advice for companies just getting started on their real-time data journey?
Real-time analytics is about processing data as soon as it's available.
Which means you need to put many things into consideration. For example, if you’re processing data pipelines with millions of events coming in per second, you need scalable and reliable computing and storage platforms — or infrastructure to process and make sense of those events.
You need to keep in mind that you’re dealing with complicated machinery when it comes to real-time analytics.
Real-time technologies today are getting very cheap with several managed services that make it viable to implement a streaming infrastructure without a lot of resources or technical know-how for simpler use cases too.
But you must first identify your real-time use cases.
If all you need is to populate a dashboard on a daily basis, you can easily get that done with either a data lake or a warehouse with a simple ETL job.
But then again, there can be some complex use cases such as anomaly detection, real-time recommendations, and real-time dashboards that require careful planning in terms of storage, computing, and analytics infrastructure.
To summarize, know your use case better and think of how much complexity and budget you can spend on that use case.
You can also tune in to the episode on Spotify or Apple Podcasts.
Prefer watching the interview?
Ready to keep learning about real-time analytics? Next up — Change Data Capture, a critical component of streaming data infrastructure. 👇
Exploring data tools?
Let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
As the former PM of data tooling at Uber, Kyle Kirwan has a lot of good insights to share about the data observability stack and how it's evolving. I certainly learned some new stuff from Kyle’s answers — especially about the role of lineage in data observability — hope you do too!
Let’s dive in:
Q. What is the simplest definition of data observability?
I think the simplest way to describe it would be to say that you understand what's happening inside your data pipelines, not what is happening about the pipeline infrastructure, which I think maybe we'll talk about later, but can you understand what's happening inside your pipelines?
Do you understand the state of your data in particular for most people, do you understand when something is wrong with your data, and can you pinpoint where and why something might be wrong with it?
Q. Are the terms data observability and data monitoring interchangeable?
Not quite. They're definitely closely related. So the difference is that data observability is a state, which we want to get to, right? So if I say, for example, across all my pipelines, I have data observability, or I have achieved data observability. Then what it means is that I have all of the monitoring in place so that I understand what is happening inside my pipelines.
Monitoring is a step or a tool on the path to data observability. So if you have the ability to observe what's going on inside all your pipelines, monitoring is the interface between you and being able to consume that observability information. So if we have, for example, a whole bunch of metrics and logs about what's going on inside our data pipeline, but we don't have any monitoring in place, it's very difficult for a human being to consume any information and benefit from that observability.
So data observability and data monitoring are not exactly the same thing, they're not quite interchangeable, but they're definitely very closely linked.
There's a fair bit of confusion in the data observability space. Let's change that by defining some common terminology. Q. What is data infrastructure monitoring?
Yes, so data infrastructure monitoring would be understanding what's happening with the machinery that's processing the data in each step of your pipeline.
For example, is my data warehouse up right now? What is the response time to a given query? Are my jobs and airflow running? Did a job have a failure? If a job has a repeated failure, that might cause a problem to the data that's flowing through the pipeline, because we're not processing it, right?
So data infrastructure monitoring would tell you, is airflow working? Are the jobs running? Was there an error? How is this sort of physical machinery working?
If we were to think about an oil pipeline, I know oil's a common analogy for data, we'd be thinking about the pipe itself, motors that are pumping and moving the oil down the pipe, not about the oil, which is what data observability or data monitoring would be focused on.
Q. That's a nice analogy. So what is data pipeline monitoring?
So “data pipeline” is a really interesting term. I was just talking to somebody about this recently. I think when most people say a pipeline, they have an idea of what they mean, but it's actually a very vague term, right?
It generally means what is the series of steps that the data goes through from origin to wherever it's ending up.
So maybe it's going into an analytics dashboard or maybe it's data that's being served by a predictive model. But when we look at an overall system, 'where is the pipeline' is actually a difficult question, because a lot of times we have data from a whole bunch of different places.
It's getting crisscrossed, it's getting merged. And at least in any company that's been around for a while, the graph of your "pipelines" is really, really messy or interconnected, right? And that's a good thing, 'cause you have data coming from a lot of places, going to a lot of places. It's not necessarily bad.
But if we were to talk about monitoring a pipeline, what we'd need to be able to do is trace the flow of data up and down from one specific point of interest.
So if I'm looking at a particular table somewhere in my data environment, if I wanna look at the "pipeline" of data that's related to that table, we need to look at all the upstream dependencies where it comes from, and then we'd also need to look at all the downstream children where that table eventually flows too.
And that might ultimately go into user queries, SQL queries, might go into a reverse ETL tool and out into HubSpot or Salesforce. It might end up in a Tableau dashboard. It might end up in DynamoDB where it's getting served by a model.
So when we talk about "pipeline" monitoring, I think what most people are describing is some form of the ability to isolate a particular trace in their lineage graph and then overlay on that trace various attributes of importance like what is the freshness, and do we have a particular step in that pipeline where that step is not running on time or the data that we see is not as fresh as it should be? Or do we see the volume of rows moving through a pipeline suddenly drop off in an unexpected way?
So when I think pipeline monitoring, I think about all those questions about freshness, volume, row count, nulls, and duplicates, but layered on top of the lineage graph and then isolated to a specific path that is important to the user.
Q. You mentioned ‘lineage graph’ twice. Can you briefly describe what is data lineage?
Yeah so lineage is probably a word everybody's heard plenty of times at this point, but it is that path from some upstream point. You could trace lineage starting from raw data that lands in the warehouse — obviously, that comes from somewhere else. So we could go further upstream outside of the warehouse and trace it up to a transactional database, could be Kafka topics that are getting loaded into Snowflake. It could be all the way up to what is emitting messages into those Kafka topics. So what are the message producers that are on the other end of that Pub/Sub?
Lineage is the map of where is the data emitted from? Where is it flowing through? What are all the steps that it goes through and then all the way down into some final destination if you will?
As I mentioned with reverse ETL, that might be pushing it down into some target system like Salesforce, which might in turn also be the other end of the start of the lineage graph. So it may often be cyclical, right?
But it is that map of what are all the different interconnects, what tables get joined with what other tables, and ideally it tells you information about, "Hey, here, the data is messages in a Kafka topic here, the data is rows in a table in the raw layer in Snowflake, here, it is data that's been aggregated by a dbt job, and it's being able to trace the data through each of those steps."
Q. And where does lineage fit into the observability stack? Is it like an inbuilt feature of observability tools or do companies use an external tool for data lineage to view the graph?
I was the PM of the data operations tooling team at Uber, and back there lineage was not a product, or it wasn't really part of data observability in an explicit way, it was a metadata service that would collect and crawl the lineage graph by parsing queries.
And then it would expose that graph via API to other products and services that needed to consume the lineage graph. So the data catalog could consume the lineage graph to display the relationships when you were looking at a table in the catalog. The data quality system could look up what the upstream dependencies were.
So if we flagged a problem on a particular table, we could say, "There is a problem in this table, but there are also problems in the parent table." So the problem may actually reside further upstream than the table you're looking at. In that case, the lineage was not necessarily part of the catalog, it wasn't part of quality, it was this distinct metadata service with no interface on top, no UI or anything, but then other products could consume the graph out of it.
What's interesting in the modern data stack is each individual product sort of needs some form of lineage built into it.
So someone may have lineage being collected from their catalog, and then they may also be using Bigeye, for example, for their observability, and they may be getting a second lineage graph from that tool.
So I think there's actually an interesting challenge here where lineage could be part of observability. It could definitely be part of your catalog as well. And if you're using multiple tools that each include lineage, you're gonna have to figure out how to reconcile those.
🤔 Questions?
So there are some observability tools that also have some lightweight cataloging features.
Q. Do you think eventually there will be a single data quality solution that caters to all of these different use cases, or do you think a best-of-breed purpose-built solution is the way to go?
I think that a catalog in its most basic form is kind of a design pattern, you're probably gonna end up finding many different tools in this data operations space, right? What tables do I have? What schemes are they in? What columns are there? What are their types? These are common challenges for any interface where you need to understand what's going on inside a data environment.
Now a fully purpose-built catalog with documentation with, for example, user comments or things like that, that is a product in and of itself. But the catalog, for example, we have a "catalog" in Bigeye, but it is really not designed to be a governance tool or a discovery tool. It is a catalog insofar as it assists with navigation and understanding of what's going on inside the system.
To your question about best-of-breed, I'm a big believer in the Unix toolchain approach to things, which is that I've got a bunch of different pieces, each one has a specific function, but they interconnect so I can arrange them as needed for a particular workflow or for a particular environment.
I would argue that a catalog and observability system, lineage as an underlying metadata service, access controls, et cetera, I think that these really ought to be distinct components.
Now, whether that means that those components must come from different vendors or not, I think is a different question. I think in an ideal world, there's a vendor who you can do business with, and that vendor provides each of these components and you can pick and choose the ones that make sense for your particular situation and combine them. That way, you're not managing 15 different SaaS vendor contracts. I think that would be the ideal scenario. But to need to procure one giant heavyweight tool that tries to do all of it, I think is also not something that a lot of teams are interested in.
Q. What about data testing? Where does that fit in?
Testing is actually something we started at Uber, right? So we said, "Hey, something goes wrong. Some person sees data that's clearly not correct in an analytics dashboard, or the data is just missing from the dashboard, what do we do about it?"
The first place we went was a test harness. And so the idea was that the data engineer or a data scientist can write some conditions about the data — must have the same number of rows as the parent table, it needs to be reloaded every six hours, things like that. We'd write these bodies of rules and then we'd run those on a schedule, and you would get a pass/fail.
Now, I think that those are very powerful, especially when you wanna do certain things like, if I made a mistake and now I have some explosion in the number of rows in a table — because I did a full join or a cross join, or whatever it is, in those conditions, where now I've got a ton of duplicate primary keys — stop the pipeline.
I know that that is a bad condition, it's a condition I can anticipate upfront. And I know I wanna put a very hard rule in place that tells the system what I want to happen.
That's a great place to use a test or is a warning during development for things that you can anticipate and know would go wrong. So tests are useful, and I recommend them to practically every company that Bigeye works with.
Now, where we ran into a challenge with tests, and where I think a lot of other folks are running into it or will, is that you can't write a rule proactively for every possible thing that's gonna go wrong in most of the data environments that most people find themselves working in, right?
I've run into tables that are 850 columns wide. I've run into environments where there are 10,000 plus tables in Snowflake. It's just not practical to ask a data engineering team to sit down and think about every single rule that they would wanna construct, right? It's not time efficient. It's not what people wanna do.
So observability helps with that long tail if we just harvest metrics from every single column in every single table, and if we know the relationship between the tables, etc, and we can just crawl all this information with a machine, then we can do signal processing on that, and we can identify things that look interesting.
I think that some of those conditions might then be good candidates to put a rule in place to put a test in place for, but it allows you to have this sort of long tail safety blanket or dragnet, whatever you wanna call it, for all those conditions that you would not think to test for, and which really aren't efficient to have human beings spend their time trying to anticipate.
Q. Do data observability tools also monitor the data at rest — data that is stored in the warehouse?
Yeah. I think that the majority of the tools today primarily function on data at rest, right? So as opposed to, "Hey, let's validate the data that's in a data frame while it's being processed before we write the results from a data frame back into the warehouse." That's at least as far as I can tell, that's a very uncommon, if not completely absent technique.
Most of the tools that you'll find in the market today, Bigeye included, query the data at rest, now that could be in a staging layer before it gets promoted into production, but it is still materialized at some point and stored inside the warehouse.
And so what that means is that we can query it. And from that query, we can produce those aggregated statistics, and then we can do our anomaly detection on top of that.
Now, if you are able to speak to a streaming source, for example, if you have a Kafka consumer, you could potentially do reads and things like that and do this anomaly detection on the Kafka stream before the data lands in the warehouse. So that's definitely something I think a lot of folks are interested in. But most tools today do query the data at rest.
Q. Last question — what’s the one piece of advice you have for companies looking to get started with data observability?
A. I think the main thing is that there's no magic wand in pretty much anything in data. A great tool can do a lot. It can create a lot of leverage. It can make it easier for people to work together. It can automate a ton of manual tasks, right?
So these are the things that we work on building every day at Bigeye. What it can't do is the organizational process bits. So in particular, I think what's super important to my comment just a minute ago about the impact to the business, the data team, or whoever it is that's thinking about observability needs to understand where is data in the health of the business?
Are we talking about analytics dashboards for internal stakeholders, execs, VPs, whoever are we talking about, data that's flowing to operational processes that are used by the sales team for example, or are we talking about data that's used in an actual model in production 24x7 that's in the app which is customer facing?
Understanding where is data being used in ways that are valuable to the business, but could also be a risk if they're broken.
Having an inventory or a deep understanding of those is the most important step in being successful with applying data observability because the whole point is to make those applications reliable so that the business can use data in these high-leverage scenarios without worrying that it's gonna break.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
If you’d like to hear more perspectives on the Data Observability landscape, check out:
* Part 1 with Kevin Hu, CEO of Metaplane
* Part 2 with Mona Rakibe, CEO of Telmai
Thanks for reading — let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
Data Observability, an established category in the modern data stack, includes tools that don't necessarily have the same capabilities or even solve the same problems.
What is data infrastructure monitoring? What is data pipeline monitoring? What is data testing? And what is it to monitor data at rest?
Mona Rakibe’s answers made me realize that data observability should not be an afterthought — it’s like a switch that needs to be turned on.
Let’s dive in:
Q. What is data observability and why is it important?
So, let's start with what is observability, right? The term comes from control theory where you can look at external signals and predict what's going on inside the system.
A similar concept now, observability has been used in various different tech companies or tech systems, be it API observability, infrastructure observability, and so on. So, data observability is no different.
You can look at data and you can start gathering a lot of different signals about your data, and systematically predict if there's something wrong with the health of the data. So, there's no downstream impact or before there is any downstream impact, you can figure out if something's going on with your data. So, that in a nutshell is data observability.
There's a fair bit of confusion in the observability space as there are many different tools with varying levels of features and capabilities. So let's try to demystify the data observability stack.
Q. What is Data Infrastructure monitoring?
So, to me, it really boils down to whom are you observing and whom are you monitoring, right? The moment you said data infrastructure observability, you are talking about the infrastructure that hosts your data. These are your storage devices, this is your transformation devices. These are the, so you talk about your dbt, you talk about your S3, Delta lakes, snowflake, and so on. So, typically a data engineer is also looking for, that are these systems performant? Are these systems cost optimized?
The queries that are generated are running at the optimal pace and so on. So, when you do all of that and you start observing if these things are working as expected that's monitoring the infrastructure itself, that is the infrastructure used to the right potential. Then there's the data itself is your hero, the data itself. And when I say data itself, this is the data that is used to fuel your business and analytics and machine learning the actual data itself.
And you're observing if this data actually has any signals to indicate that there is garbage in this data. So, then you are observing the data itself and that becomes data observability. Telmai sits in the data observability space not so much on the infrastructure observability space but today a lot of tools are either doing one better than the others or doing both of it.
Q. What about monitoring the actual data in the data warehouse?
A lot of tools today are observing the data warehouse and if you look at a typical data pipeline, the data warehouse is on the right side of the pipeline. This is where it's almost ready for insights. It's almost ready for analytics, right? We look at it as the last stage or last step of the pipeline. It's a very crucial step of the pipeline because it's already been consumed. So, monitoring that data warehouse is important but this is not where typically issues arise.
The issues arise at the left of the pipeline that is source system and data by nature is transformational, right? So throughout your pipeline, something is going wrong and it could lead and first be seen in the data warehouse. Our philosophy is that you have to monitor the entire pipeline, not just your data warehouse, and protect your data warehouse which is extremely important. So, you can monitor the data warehouse but often time it's not the source of issues. You have to monitor the sources of the issues for any kind of outliers and trends.
Q. Can you explain data pipeline monitoring?
When you look at the entire data pipeline, often it's shown as linear, but in a nutshell, it's, in reality, it's actually a graph, right? Like you have sources that pump in data, it gets ingested, you might be using different tool for ingestion, transformation. It enters your data sources and then it's consumed by your downstream, right? Now, when you look at it, you need to a metric monitoring system like Telmai has to monitor every step in the pipeline in order to make sure there is no issues that get ingested by that specific step of the pipeline. The advantage of full pipeline monitoring is you get to the root cause or you get to the actual issue very quickly because closest to the source of the issue.
Q. Do you think there should be or there can be a single data quality solution that addresses all of these different use cases?
Depends. But I don't think we are ready right away for that because data quality to be honest is so, so wide because if you think about it, governance comes into that, cataloging comes into that, master data management comes into that. So, if you look at data quality, it's definitely, there is gonna be a lot of room for a lot of different tools especially specialized tool. And the problem statement is so big that you may need pieces of these puzzles to work together.
So, lineage is one such example, right? Like you can do a little bit of lineage within your data warehouse, but if you want to do the full pipeline lineage and tie that back into who has access and governance and all certain tools to do that better than others. So, there's definitely gonna be room for lot of specialized tools given the space. But data observability, I feel in itself is a huge category. And there's a very big problem to solve there.
I can quote one user of Telmai and he said that, "if you folks solve data observability properly, that's a big enough problem, right? Like I have tools for lineage, I have tools for other stuff but let's make sure that we solve this problem."
🤔 Have questions?
Q. So where does data testing fit into the observability stack?
So, I would break down testing into two types, right? If you look at the previous generation data quality tools, they were heavily designed for policies, compliance, business policies, rule-based, and stuff. They are still relevant. Business rules are still relevant and no matter how much statistical analysis, ML stuff you do, there will always be room to do a little bit of rule-based stuff. The goal should be to reduce the dependencies on rules because they just don't scale.
They don't scale in today's ecosystem. It's a lot of overhead on your data team, so, you have to look at it very systematically to reduce the rules. And you can reduce them through ML-based tools, and observability too. Now there are other types of unit testing or DQ testing tools, like great expectations, and dbt has some rule-based approaches including AWS, Deequ, etc.
So you may use that as well, but keep that in mind. So they are, they're used to test your data and you can check completeness, null and other metrics but they come with an heavy upfront cost, right? Of implementing it, maintaining it, and even ongoing cost of maintaining it and managing it. So, keep that in mind and in philosophy, I suggest that reduce those rules, don't bet heavily on that because that will, what we call is death by rules that will get you into a trap of rule-based approach.
Q. And at what stage should companies invest in an observability solution? Are there any prerequisites in terms of the data stack?
So, first thing I would tell everybody is, don't fear data observability. It's very easy to implement, get started quickly. It's a switch you can turn on very quickly. So, when I say that, that means that if it's, if it is easy to use then you should get started as soon as possible. The way I look at data observability — it’s hygiene, it’s the right thing to do. It should be foundational. So, get started as early as possible.
Even if you're just getting started with re-architecting your data pipelines or when you're adding new sources also, data observability will completely make that journey much easier. So, get started as soon as possible. Don't have that fear in mind and just turn on that switch of data observability right away.
Q. Last question — what's the one piece of advice you have for companies looking to adopt an observability solution?
The main thing is to make data observability a strategic priority. So today, a lot of companies know they have issues. They know that there are issues with data quality that are directly impacting business.
My advice would be to make it a strategic priority, put KPIs around data observability, and data quality monitoring, and initiate it as a very well-designed, corporate initiative, right? There's no other way that you can improve the reliability of the data set that's important for you unless you make it a priority and whatever tools you may use but make it a priority.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
If you’d like to hear other perspectives, check out the other parts of the series on Data Observability:
* Part 1 with Kevin Hu, CEO of Metaplane
* Part 3 with Kyle Kirwan, CEO of Bigeye
🎁 If you found this useful, do spread the word!
🧠 Subscribe to receive new episodes in your inbox!
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
Data Observability is an established category but the tools that fall under this category don't necessarily have the same capabilities or even solve the same problems.
There are infrastructure monitoring tools, pipeline monitoring tools, and tools to monitor the actual data that rests in a database/warehouse/lake. And then there are data testing tools and tools to understand data lineage.
In this episode, Kevin Hu makes it sound all too simple, and he does it with a big smile.
But that's not it — Kevin is a brilliant mind so we also got him to share some advice for companies looking to invest in data observability efforts.
Let’s dive in:
Q. Please tell us what exactly is data observability.
Data observability is the degree of visibility you have into your data systems. And this visibility helps address many use cases from detecting data issues to understanding the impact of those issues or diagnosing the root cause of those issues.
There's a fair bit of confusion in the data observability space as there are many tools with varying capabilities. So let's try to address that.
Q. Can you first describe what is data infrastructure monitoring?
A. Infrastructure monitoring is a space that emerged decades ago, but really came to the fore around 10 years ago with the emergence of the cloud, like Amazon Web Services. So tools like Datadog and Splunk and New Relic help you understand whether your infrastructure is healthy. For example, how much free storage you have in your database. What are the median response times of your API or the RAM usage of an EC2 instance. And this is really critical for software teams, especially as they deploy more and more of their resources into the cloud.
Q. And can you explain what is data pipeline monitoring?
A. Pipelines, to put it simply, take A, turn it into B, and put it into C. And this is used across the data system, whether it's using airflow to pull data from a first-party system into your data warehouse or to transform data within a data warehouse or even prepare features for machine learning. And data pipeline monitoring, on the first level, is trying to understand, are my jobs running? This is a surprisingly hard question to answer sometimes. But the level two question is, are my jobs running correctly? As I take A, turn into B, and put it into C, is A what I expect, is B what I expect, and was it loaded into C correctly?
You make it sound so simple! Q. What about monitoring the actual data in the warehouse? How would you describe that?
So cloud data warehouses, like Snowflake, Redshift, and BigQuery are increasingly the center of gravity of data within companies. To put it more simply, it's where you put everything. And a lot of applications, whether it's a BI tool like Looker or reverse-ETL tool, a machine learning model, are kind of mounted on top of the warehouse. So data warehouse monitoring tries to understand whether the data within the warehouse that is used for all these systems is correct.
Q. Some observability tools also offer data cataloging and data lineage capabilities. Can you explain those briefly?
Data cataloging tries to address the problem, what does this data mean? And there is a gap between how data is represented in a technical system to how it represents business objects. So a data catalog is an easy way to attach semantic meaning to the objects within your data system. Here's how a metric is derived. Here is how a table is derived. So when the VP of Data asks you about this revenue metric, you point them towards a data catalog as opposed to having to type out the answer.
Data lineage solves the problem of understanding how data within your system relate to each other. If you trace data all the way back to the source, either a machine created it or a human put it in, but rarely do the end users of data use that raw data. This is in some ways, the job of a data team is to turn that raw data into an analytics-ready form that can be used for many different purposes. Data lineage tracks it all the way from the source down to where it's used and sometimes beyond.
Q. So where does data testing fit into the observability stack? We've talked about all of these different capabilities, so tell us about data testing.
Just to be clear that data quality and data observability are two different things. Specifically, data quality is a problem. People wake up in the world saying, shoot, I have to fix this data quality problem. Data observability is a technology that can help address the problem, but isn't a silver bullet. And it's similar to software where if any tool says that they're gonna fix all of your software bugs, they're lying. And the same thing is true for data observability tools.
We can help you build better processes to measure your data quality and to help you identify issues, prevent them, and resolve them. But we can't do everything for you. So testing fits in the picture because data quality is one of the core use cases that data observability addresses, and that testing is one particularly good way to catch issues.
🤔 Have questions?
Q. Do you think there should be a single data observability solution that addresses all of these different issues? Or do you think a purpose-built best-of-breed solution is a better approach for companies?
The classic bundling unbundling question!
The most important thing that I care about is that we solve real problems for people. And the best way that we've seen teams introduce data observability into their data stack is to start small and to start simple. And typically that means having a very focused set of features with a very well-defined goal, and then introducing that. And when it works, expand from there.
So I don't have too much of an opinion on if you should bring on one all-in-one tool or several best-of-breed tools. What I care about is that you bring on observability correctly and that you focus on very specific problems.
Q. Why is now a good time for companies to invest in a data observability solution and what are the downsides of not doing so?
At Metaplane, our customers kind of fall in the two buckets. Half of them say, okay, I'm building a data team from scratch, and I wanna get ahead of data observability. The other half of them, it's no fault of theirs, but something has happened internally. And now they're reacting to the issue. Now is the best time to bring on a data observability tool in the same way that now is the right time to bring on a software observability tool.
Where 10 years ago, you might wait for your API to go down and your customers to complain before stepping back, having a postmortem, and saying, okay we should bring on Datadog. But nowadays, Datadog is one of the first things that a backend team installs. The reason is that when an issue occurs, you have maximum historical context to both detect and resolve the issue. And that when an issue occurs, you don't have two problems, which is that you both have to fix the issue and you have to bring on a tool.
Q. Who is the typical user of a data observability solution and who are the beneficiaries?
A typical user is a data engineer or an analytics engineer. It depends on the size of company. We have some customers with like 20-person teams and their first data hire brings on Metaplane. Other customers are like 10,000-person enterprises and their head of data governance brings us on. The user of Metaplane is usually someone who is held responsible for data quality issues. Who gets the pings when a dashboard is delayed, but also has a bit of power to impact data quality issues. However, that's a very separate answer than who benefits from data quality.
Ultimately, data is not created or consumed by the data team. It's created by upstream teams, the engineering team migrating a schema in their transactional database, product and growth teams using Segment to track usage analytics events, a go-to-market (GTM) team inputting order forms. And it's used by those same teams. So when data quality improves, the entire company benefits, even though they might not call it a data quality issue. They might say, hey, this number looks wrong. Or this dashboard is delayed, which is very important to speak the language of your users when you're on your data team.
Q. Yeah, that resonates a lot because I say this all the time. Good data infrastructure should ultimately benefit the entire business. And it shouldn't just be for data teams to be more productive.
So moving on, what are the prerequisites in terms of the data stack for a company to derive value from a data observability solution?
If you have a database, you could get value out of it today. It doesn't have to be a cloud analytics data store. It could be Postgres or your transactional database, something that you want to gain greater awareness, for example, about schema changes, or if a row count drops, or if some constraint is violated. You don't necessarily have to be on a data team either. But if you have that database, you can get value from it today.
Q. Last question — what is the one piece of advice you have for companies that are evaluating a data observability solution today?
We mentioned it before — start simple. You're busy. Everyone is busy. Data teams are especially busy, 'cause people are knocking down their doors and you're having to grow your team. So data observability shouldn't be a big headache to bring on. You have many other things to do.
So start simple, bring on a tool in, for example, 10 minutes, create some simple tests like row count and freshness and schema tests across a broad swath of tables. And maybe if you have the time, dig in a little bit deeper on the most important tables based on usage and lineage, but just see how it works. Sometimes that might be enough.
Sometimes you might want an 80/20 it and go a little bit deeper and say, okay, I know that data observability works. So let's dedicate a bit more time to instrumenting more tests or adding more lineage, but don't try and eat a watermelon all in one bite, just bite by bite!
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
If you’d like to hear other perspectives, check out the other parts of the series on Data Observability:
* Part 2 with Mona Rakibe, CEO of Telmai
* Part 3 with Kyle Kirwan, CEO of Bigeye
🧠 Want to stay apprised of new data technologies and broader trends in SaaS?
🎁 Feeling generous?
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
David Jayatillake has a ton of experience building data infra as well as figuring out how to get folks to use data in their day-to-day. In this episode, David answers some fundamental questions like:
* What are the core components of a well-executed data infrastructure?
* What are the prerequisites in terms of the tech stack to set up a basic data infrastructure?
* How do data-adjacent teams like Product and Growth make use of and derive value from good data infra?
And he also offers some advice for companies getting started on their data journey.
🤔 Have questions?
Let’s dive in:
Q. Please tell us what it means to build data infrastructure in the context of the modern data landscape.
I think there's a typical stack that's understood as infrastructure in terms of being able to ELT a data warehouse, a BI tool on top, and then now increasingly, there's additional non-core pieces like reverse ETL, observability, CDPs, streaming tools as well that are being added to this infrastructure.
Q. So what are the core components of a well-executed data infrastructure?
So I think, the difference between well-executed, I think it's about things that ensure quality, things that ensure reliability, especially around the development process. So being able to use version control, CI/CD in your development process, that's gonna really enable what I think most people will consider well-executed looks like.
And that's where frameworks like dbt come in, which have been enabling development on top of data pipelines flowing from the data ware, into the data warehouse and on the data warehouse that's enabled. We're looking for more tools like that to spread further out, and dbt to take a bigger footprint as well to push that quality outwards.
Q. If we talk about building a basic data infrastructure — a minimum viable data stack — what would be the two or three tools that would comprise a minimum viable data stack?
Sure, so it depends on your context. So for some companies, if you have to work with a lot of third-party tools, you definitely need an ELT tool like Fivetran, Airbyte, Gravity Data. You need one of those kinds of tools to help you get data from those third party systems into your data warehouse. Obviously then, you need a data warehouse.
I personally think, for a minimal viable data stack you want a data warehouse that's quite easy to use, and that scales without much-needed thought and planning. You don't want to need to have a DBA. So I think Snowflake and BigQuery are the two easiest to use. BigQuery is possibly even easier for a smaller startup. It's basically a "use it and forget about it," nothing to do.
Q. Why has there been an explosion of data infrastructure tooling over the last couple of years?
I think it's because of a hangover from the big data era. So in the big data era, in order to do any amount of data engineering, you'd have to hire a huge amount of very expensive people. So I've been at a company where we were on SQL server when I joined the company as an analytics stack. And they planned to do a big migration to Hadoop on Hortonworks and it took years. And they hired a data engineering team of 50 people paying them a huge amount of money, and it actually failed. They didn't even succeed in this data project.
So what we've realized, and venture capitalists have realized is that the data engineering space is ripe for automation and for SaaS tooling, and so that's more or less achieved now. If you think of Fivetran, especially from a batch point of view, you've got big companies out there now, like Fivetran, Airbyte, who've enabled most of the connectors you'd need in the space. And you've got some VCs even that identify as like Hadoop refugees. They know about the pain from that era, and that's why they've invested in removing some of that pain.
Q. Building infrastructure is one thing and probably a fun thing for those building it, but how do organizations get various teams to actually use the infrastructure and benefit from it?
This is about communication and access. So I think, in the past you've seen because of inelasticity of how compute scaled on those systems, their access to those systems has been closely guarded. Purely data teams would have access to data warehouses and BI tools and things like that. And with the proliferation of cloud and cloud data warehouses, that then got pushed outwards, and at the last company I worked at, Lyst, everyone had access to Looker.
And Looker who was sitting on Snowflake, it never had any like capacity issues. So that's how this data infra becomes accessible to everyone, not just those building it, but the whole organization. So I think that separation of compute from storage was a really key piece of how that became possible.
Q. Do you have any specific thoughts on how data-adjacent teams like product teams and growth teams can actually derive value from a good data infrastructure?
Yeah, and I think this is really interesting because actually those teams are often what I call data domain owners. So if you've got a product team that's building a feature, let's say it's some kind of a feed of products on an e-commerce, for example, they're generating potentially many data points from their product feature, and then every time they iterate on it, it generates more of a different kind.
So actually planning how that tracking is done and planning to make it done well to avoid future analytics engineering work or that work that's not even possible, that, I think is almost like they are self enabled. If they make good decisions when they're planning their engineering work, testing it well, making sure the tracking is good, that enables them to actually get value from the data.
🤔 Have questions?
Q. How do you suggest data teams find a balance between building infrastructure and supporting the day-to-day needs of the organization?
Yeah, this is something I've had to manage, and sometimes you can do it structurally. So I ran a hub and spoke model. So I had a central team of analytics, engineers and a handful of analysts, and then we had distributed analysts and analytics engineers as well.
And the central team would often do that building infrastructure, building new ways of doing things or core pieces of the stack that other teams could then build on top of. And then the day-to-day needs were often met by the team members on the spokes who sat with those commercial parts of the organization that needed support.
Now that helps, but I think apart from structurally, you also need to think about how do we do this from a work philosophy way. We believe in dealing with tech that we believe in spending time on infrastructure to make it good and to be a force multiplier for future execution. That has got to be in there as well.
Q. There are folks who believe that the modern data stack is a fad, and that organizations should refrain from stitching together half dozen tools. What are your thoughts here?
The thing is I come from a background of having done that stitching and got value from it. And I've managed to deliver with a handful of people, and that stitch together stack more than I've seen that big data era or previous era teams able to achieve. And so the modern data stack is not a fad.
Could it be better interoperable? Yes, absolutely. Could there be some bonding? Sure. Half a dozen tools sounds like a lot, I don't think it's that much. But you're seeing now fragmentation of up to a dozen tools, and I think, yes, we're getting to a point where who wants to manage that many vendors? And I've seen new startups actually whose pure focus is stitching the tools together so that any customer has like one Okta entry point and has access and choice to many of these tools, that get automatically stitched together. So you can see that there is need, but I do believe that modern data stack is valuable.
Q. What are the biggest pros of infrastructure comprising best-of-breed purpose-built tools over an all-in-one does-it-all solution?
So I think, if you think about the most typical all-in-one solution, you'd probably think of Microsoft Power BI and like Synapse SQL server type setup. Now, the interoperability on that system is very good. That's one thing that users like about it. The vendor management's very easy. You get all of your tools bundled together at one price, all of your organization has access to it as part of their typical licensing. So in terms of cost and managing the tools, it's easier. There are the pros of that all-in-one stack.
Cons are, fundamentally, they're not focused on any specific area of that tooling. And you've seen this, how Snowflake's beaten Redshift. They've just invested the time and the thought to just focus on this one thing and do it extremely well, and they're reaping the rewards of that. And you've seen Fivetran do the same in ELT. Microsoft's ELT is not as easy to use, and not as comprehensive.
And so what you end up doing is, when you have that all-in-one solution, yes it's cheaper and it's stuck together nicely, but no single piece of it does anything to the very best of it in its class, whereas when you have the best-in-breed solutions, you get things that are really good.
So how do you get the best-in-breed to work together? That's interoperability, and I think with frameworks like dbt app in existence, that's allowing for those tools to interoperate better than before, and I think you'll see in the next couple of years, interoperability as good as a bundled, like an all-in-one stack.
Q. Last question — what is the one piece of advice you have for companies that are just getting started on the data journey?
I think, especially if they're a B2C company, get your tracking right. Just start with your tracking. Put in a CDP, there are many cheap or open-source ones like Rudderstack, Snowplow out there. Put that in right away, put something into government like Avo, and then get that piping to a data warehouse. Don't worry about what you have to do after that, but that's fine.
Don't worry about making some complex data lake pipe to the data warehouse stack. Just get it from your app to your data warehouse reliably, consistently, and completely. That's just a fantastic starting point. And then even if you don't have the people to analyze that or the time to analyze it or get value from it when you do finally get around to it, you've got this wealth of free, good data to use later.
So I think that's one piece of advice I'd give to a company starting on their data journey. And also if you are struggling, maybe it's worth getting consultants in to help you. If you don't feel like you've got the time or the focus to build a data team or to set things up properly but you know you need to, I think, spend the equivalent money on consultants who will just get it done for you, and you'd probably be surprised at how much you'd get for your money because of how much building an internal team and how long it takes to scale costs.
🤔 Have questions?
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
🎁 Feeling generous?
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
→ Link to the collection.
Once again, what exactly is a warehouse-native app?
This time, hear it from George Xing who helped build the analytics function at Lyft and deeply understands the priorities and constraints of data teams.
In this episode of Data Beats, George explains how the warehouse-native architecture is superior to its predecessor and what benefits organizations can reap by adopting tools that are built on top of the data warehouse.
George also has some advice for organizations looking to get started with a warehouse-native app.
But first, read our guide on warehouse-native apps
Let’s dive in:
Q. George, why don't you tell us what is a warehouse-native app?
Warehouse-native apps are business applications that run on top of the customer's cloud data warehouse. So something like a Snowflake or Redshift or BigQuery and relies on that piece of infrastructure as the source of truth for customer data. This is different than managed applications or the traditional way of building SaaS where the vendor will store a copy of the customer data on their own system and manage it that way. And some of the other advantages of this is you don't have to be tied to a fixed schema. So the business relationships that the customer has already defined in their cloud data warehouse, you can leverage. Warehouse-native apps are cloud... Warehouse-native apps are schema-agnostic and they also remove the need for pipelines. So they move data in and out of the cloud data warehouse without additional ETL tools.
Q. What is leading to such a shift in the way SaaS tools are being built? You mentioned some of the benefits which sort of answers the question, but what else?
One of the things that we see is that more and more so SaaS products are relying on data as a core differentiator. So in our space, in marketing, marketing ROI is really driven by the use of customer data. And the other big trend is that customer data is getting centralized in cloud data warehouses because that's where you can see all your touch points, that's where your source of truth is. And so in order to connect those to the obvious kind of architecture is to move the software into the data that is housed in the source of truth.
With warehouse-native apps, you don't have to replicate your data which obviously brings cost savings.
Q. What are the other core benefits of warehouse-native apps over a traditional-managed app?
There are so many but I think some of the ones that stand out, one is just speed to set up. One of the biggest challenges of working with a data-intensive application today, is that you have to first send all your customer data to the vendor in order to just get started or have any value, whereas in a warehouse-native app, your data's already there and you just connect the application directly to your data via a standard database connection.
And so it just simplifies and speeds up the cost of, or the time to getting set up from sometimes months to days. The other piece of this is just having more access for personalization, as I mentioned. If your warehouse is the source of truth for all your data, then you also have more data, richer data there. You can leverage a lot more of those data points for personalization. In the case of Supergrain, that would be targeting for emails.
That would be more personalized messages. And obviously, that drives more business ROI. And then I think the third piece of this gets at the pipelines. Normally you have to move data from your vendor back into your warehouse for analysis and reporting and BI tools. Warehouse-native apps simplify that because they write directly back to the warehouse, meaning the reporting is both more complete, it's faster, and it's completely managed. So you don't need to set up a separate pipeline, which also introduces costs.
Q. What are the prerequisites in terms of the data stack to adopt a warehouse-native app?
Obviously, having a warehouse is prerequisite number one, but the other piece of a data warehouse is data. So you need some way of getting data in there. And, you know, we talk to a lot of customers that have various ways, they use various ETL tools to load data in.
They also have other tools for doing transformation on top of the warehouse. And typically what that means is there's also somebody who's managing the data warehouse or either a data team or an engineering team of some sort. They're kind of responsible for the governance of the data that's in the warehouse.
And there's a nice kind of handoff between the work that they're doing and the application side, which is what Supergrain does. We're essentially making a lot of the work that they already do more accessible to downstream go-to-market teams.
🤔 Have questions?
Q. Since warehouse-native apps don't store any custom data, won't marketing campaigns break if the data changes at source or if there's an issue connecting to the customer's data warehouse?
Yeah, so I think the way that I would think about this is not, you know, warehouse-native apps versus traditional apps in terms of data storage. I would say that marketing campaigns today break a lot more often than people expect or would think because source data breaks and it breaks regardless of where the data is actually stored. You know, there's just a lot of dependencies between various different systems.
The data might not be formatted in the right way. And once it lands in a destination, then it might not render. And we've all seen those examples where emails go out with the unfilled merch tags with the curly braces. The question is, how do we solve that problem? And in our view, it's really through better data governance. So you need to have full visibility over the pipeline.
You need better lineage, you need better observability. And the problem is today with the model of moving and copying your data into a third-party system, you lose all visibility and the ability to trace the dependencies along that pipeline.
One of the benefits of warehouse-native apps is that we rely on the existing tooling in the data warehouse ecosystem today and just all the tools available to data teams to actually manage that pipeline. So it's definitely a big problem, but we feel that data warehouse apps actually are at a much better advantage when it comes to solving some of these problems.
Q. Can you briefly explain the segmentation capabilities of a warehouse-native engagement tool like Supergrain? Are there any limitations here?
So, what Supergrain offers in our application is a no-code visual builder for marketing and growth teams to create customer segments. And so you can filter for various criteria, people who have done certain actions or have certain properties. And essentially what we do is we compile that into SQL under the hood and we execute that against the data warehouse.
What that means is we can also support all kinds of joins. We can support complex account hierarchies. We can support multiple entities, funnel criteria, and just a lot more complex sophisticated segments and business logic that you normally wouldn't be able to do in a traditional segment builder type of product that doesn't run on a data warehouse.
Q. Do the visual segmentation capabilities replace the need to build data models in SQL?
It doesn't. As we talked about earlier, you know, part of the thing that you still need, you still need the data models and the data teams who are managing those data models to define things like key business metric definitions, key dimensions, key business entities, like, you know, what a user is. That's something that is very business specific that Supergrain isn't going to know out of the box. And so, essentially what we do is we expose a UI on top of the work that the data team's already doing to enable the marketing team and the growth teams here to self-serve with our visual builder.
Q. Do warehouse-native apps make reverse ETL workflows redundant?
You know, our view is no, you know, and the reason I say that is there will be plenty of applications that will be built on top of, you know, the data warehouse and, you know, as data warehouse native apps, but there will also be plenty of applications that don't, that need data from the warehouse.
I think, you know, we see this a lot where there's a number of systems and tools that don't really need the benefits that we talked about, but still need some data from the warehouse. Reverse ETL tools are a really, really good compliment in that scenario.
So we see kind of a future where a lot of the core business applications like customer engagement, which is the category that Supergrain is building in, will have warehouse-native apps, but there's a very, very long tail of applications that still serve important business use cases where reverse ETL tools fulfill like a very, very important need.
Q. Last question — what's the one piece of advice you have for companies that are looking to get started with a warehouse-native app?
I think the biggest piece of advice is actually less from a technology standpoint and more from a mindset perspective!
Obviously, you need a data warehouse. You wanna make sure that you check that box, but the other one is that as a company you should have made a concerted investment in the data warehouse as the single source of truth for your data. And you wanna turn that into a data platform. And as you've made those investments, whether it's in data or the teams, and kind of aligned the downstream customers of data to that mindset, then it becomes a lot easier to adopt a warehouse-native application like Supergrain.
You can also tune in to the episode on Spotify or Apple Podcasts.
Prefer watching the interview?
If you’d like to hear other perspectives, check out the other parts of the series on Warehouse-native apps:
Thanks for reading — let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
→ Link to the series.
What is a warehouse-native app and what are its benefits over a managed app?
Luke Ambrosetti (who was at MessageGears and is now at Snowflake) not only has first-hand experience but is also deeply knowledgeable and passionate about the warehouse-native architecture. I learned a bunch from his answers and am pretty sure you will too.
Watch the interview:
You can also tune in to the episode on Spotify or Apple Podcasts.
Prefer reading? We’ve got professionally edited transcripts for you:
Q. What is a Warehouse-native app?
So a Warehouse-native app is effectively a SaaS application that connects directly to the data warehouse and is doing a data-intensive process for you.
Q. Are connected apps or data apps the same as warehouse-native apps?
Kind of, it depends on who you ask, right? The definition of a connected app or a data app could be very different, to people. And in my view, I'm seeing connected app as kind of a good definition of a warehouse-native SaaS app. Whereas data app could be it could be a SAS, it could be that thing, but maybe data app is actually a larger term that incorporates, maybe an internal app that you create at your, at the company you work for, could be a data app. Right? It doesn't have to be some sort of external tool that you, you buy and or use
Q. What is leading to such a paradigm shift in the way B2B SaaS tools are being built?
It's very exciting, right? Lots of new companies are taking this new approach. Like this, it's called warehouse-native, warehouse-centric, warehouse first right approach. Lots of terms for it.
But it honestly, it comes from this idea of the separation of compute and storage. Right? And then specifically kind of in the industry that I work in, which is more marketing, is this idea of that I've seen, it explained very well, is this idea of the the separation of the system of engagement. Right? Of how you're reaching out to, in this case the marketing's case, your customers, right? And the system of record of, and where what is the state of that customer?
You know what they're doing, where they are. the Customer 360 as it's called. So with those two things separated, or traditionally, I should say you've had to get your data to your system, whatever your system of engagement is. Right? And sometimes, companies like Salesforce have traditionally tried to be both the system of record and the system of engagement. So, now it's with a separation of compute and storage. Right? you can now have those two things, the system of engagement and the system of record separated as well.
Questions? Check out our primer on warehouse-native apps
Q. So besides cost savings, what are some key benefits of using a warehouse-native engagement tool over a traditional one?
So yeah, and again the cost savings aspect right here is the idea that you don't have to sync that data right to whatever, your other platform is of where you want to, do either that that in whether it's, if it's marketing, it's engagement, maybe it's analytics, you don't have to sync that data there either whatever it might be. Right?
So that's where the cost savings comes from, but there are so many other, benefits as well right? So it's, you have few data silos, you do it this way. Right, instead of shipping your data out to 10 different SaaS companies. If all 10 could connect your data warehouse and use that system of record, you don't have to duplicate data, I guess, is you know what I'm trying to say, right?
So it's single source of truth, of course, as well. And that's gonna be better, especially for like things like data absorbability or analytics, right? Being able to attribute different events that happen, back to again where you have all of your data setting, instead of it, again, being in a silo you have to then extract with an ETL tool. You get to bring many of these warehouse-native cases you get to bring your own data model. So instead of having to force your data model into kind of how they view the world or how they think your data should be modeled. You get to bring your own, which is great.
Also, it's onboarding time, right? Instead of having, again, the same thing with the bring your own data model you don't have to spend days, or not days, months honestly, sometimes years, wrapping out how the data should be modeled in that destination system. Again, on the time saving, you have speed of change, right? You wanted to add a specific field, right? To whatever you're trying to do. It's easy. It shows up in, in the data warehouse or the database it's right there for you to use. And then lastly, from a security and compliance point of view your data is safe in the data warehouse. Right? You're not having to rely on these task providers to keep your data safe. So if you have, PII data, especially on customers, I come from the marketing space. That data is much safer at home.
Q. So are warehouse-native apps making reverse ETL workflows redundant?
Yes. Kind of, yeah. I mean, so the answer of course is yes and no. Right? I would say it breaks reverse ETL redundant and ETL. Because again, you're not having to kind of sync data in between the source system and the destination system, right? Your source system is your system of record. That's what you get to use. So both the reverse ETL and the EL or ETL tools can be, get replaced. That said, I think it's, is that gonna replace them forever?
No, I don't think so. I think that, this approach is really cool and great, but there are, some companies like when I think again, in marketing, you think about like like Facebook conversions API, or Google ads. Right? I don't think they have any reason to take this approach. Right? They're, you're gonna, you're still gonna have to use their APIs cause that's what they need, right? To do their business. So I don't think, those reverse ETL and El tools are gonna be completely gone.
Q. Can you tell us about the segmentation capabilities of a warehouse-native engagement tool and are there any limitations here?
Yeah. And so again, I work at MessageGears and MessageGears is traditionally an email service provider and that's kind of been our core feature is connecting directly to the data warehouse. Right. So, with this type of architecture or design, the limitation is that you do have to have a data model, right?
In marketing many companies rely on their ESPs or sometimes their traditional CDPs to do that for them. Right? So you need to have a data model because and you can't just take these tools like data ingestion tools, like a snowplow, rudderstack and and just throw data into a data warehouse right? You need those resources, you need those people. Right? To go through and intelligently design models for your, customers or marketing in this case, maybe sales to use, right?
So, when the data team does that for their end customers — the marketing team — it gives those data-adjacent teams the ability to use that data and kind of do that last mile transformation that they need to do to run what they need to run in their programs.
So it's for lower or no-code tools, right? It's gonna be very much a challenge to take this approach. Right? Because again, you have to basically say, okay bring your, here's your data model, bring your data model and now make it work with the solution we have the big reasons, some of those, Cloud SaaS are so popular is because they make it really easy, and they provide that data model for those those low code tools to use.
Q. Can the visual segmentation capability eventually replace the need to build data models in SQL?
Yeah, that's, that's a good question. Maybe. I think, we'll get there, but right now you still need some base SQL, right? Or some sort of base model to work from.
again, those data-adjacent users can go through and use those tools to do segmentation and do some really cool segmentation, honestly, at least in in our platform to do some very complex segments and in groups and labeling to go through and target their customers or their users.
At the same time, it's again, it's more of that last mile approach where they're kind of adding in the things that they need, but they need- For now, they need to start from somewhere, right? They need to have a, what I would call like a base model to work from. And from there they can go through and then create kind of, do some, some kind of, I guess additional modeling on top of that.
Q. If warehouse-native apps don't store any customer data, won't marketing campaigns sometimes break when there's an issue connecting to the customer's warehouse?
Yeah, I've got this question a couple times now from some more of the, technical users or, prospects. And the answer is yes, but at worst it's the same as or better than your typical SaaS model. Right? So with your typical SaaS model, you have to have pipelines and again, and now reverse ETL makes it a lot easier in getting your data synced.
But let's say you wanna send the email campaign to all the users who subscribed yesterday. If that pipeline breaks, then you could, if you don't if your SaaS company isn't looking at the freshness of that data. Every day, you could be sending that message to the exact same users again. Right? And the same kind of thing works here whereas, in this method, if it breaks, nothing's gonna be going out, right? Whereas if your SaaS company is not looking at data freshness, right? Something could break and then you're gonna be sending, you're giving a bad experience to customers because you're sending them the exact same content twice.
That's really interesting. I never thought of that so thanks for sharing.
Q. Last question — what is the one piece of advice you have for companies that are looking to just get started with a data warehouse?
Yeah. I would say start slow and don't try and just jump in, head first. There are so many tools. The modern data stack is so cool. It's a lot of functionality, and lots of cool features that people are doing today. At the same time, you have to start slow. It's a crawl, walk, run approach.
Right to this shutting with one of our enterprise clients who was talking about their move into BigQuery and took them three years. Three years, I think from Teradata. Kind of, on-prem. And so it's, it can take a long time. So you have to have — before you start doing the really cool ML stuff — you have to have your ingestion, your modeling, your transformations figured out. So yeah. Start slow and yeah. Have fun with it!
Thanks for reading — let’s beat the gap! 🤝
Check out our series on Warehouse-native Apps:
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
→ Link to the series.
What is a connected app or a warehouse-native app? How is it different from a managed app? And what is leading to such a paradigm shift in the way B2B SaaS tools are built?
Omer Singer has an insider view of how the connected app paradigm is taking shape and is here to answer those questions.
We also got him to share some tips on how to evaluate vendors of connected apps.
Read the primer on warehouse-native apps if you haven’t already:
Let’s dive in:
Q. What exactly is a connected app or a warehouse-native app?
A connected app is a SaaS solution that lets the customer bring their data platform of choice, and so the vendor is bringing the work to the data.
Q. Why are connected apps also being referred to as warehouse-native apps?
It's talking about the same thing, and I think it's this idea that there's been so much progress in data warehouse technology that what's possible from the SaaS integration perspective has changed, and from a Snowflake perspective, we're calling it connected applications as to differentiate from managed applications. Traditional SaaS solutions always cared about data.
If you look at the traditional, maybe the first really famous SaaS solution being Salesforce, Salesforce has a very significant database under the hood and all that data about customer opportunities and all that's stored within Salesforce. That model, we call the managed application.
Now to differentiate from that, we have the connected application and that's where the SaaS solution says, "Look, we're gonna focus on the app, and for customers that want to, they can connect us to their existing data warehouse, data platform and we will use that," and it's an exciting space. We're seeing more and more companies embracing that, and for our customers too, this is becoming a direction of choice. Customers are really preferring this model and it's helping them to be more successful.
Thanks for explaining the difference between the connected app paradigm and its traditional counterpart, managed apps.
Q. What is leading to such a paradigm shift in the way B2B tools are built?
I think this shift couldn't have happened a few years ago before all the progress in cloud data platform technology. It used to be that applications needed a backend that would handle the data and different structures, et cetera, and they had, of course, demands on how reliable it would be, how powerful it would be, so they needed to own it and to be responsible for that back end, end to end. They couldn't count on whatever database technology the customer was using 'cause maybe customers don't have the same power in their data platform and the same reliability in their data platform. I focus on the cybersecurity space and security teams often would use technologies like Elasticsearch to collect a bunch of log data to it, and a lot of vendors use Elasticsearch under the hood of their application too, but the vendor couldn't count on the customer's Elasticsearch cluster to be 24/7 available, to be scalable, and fast enough to handle large amounts of data, so then the vendor had to own the data on their side and maybe they exposed it to the customer through an API so the customer could maybe get access to some of their data through that API. With advances in the cloud data platform, I think what Snowflake has really pioneered is this cloud data platform that is very robust and consistently powerful and reliable, basically putting the vendors and their customers on the same footing. Now for the first time, the customer can bring their SaaS solution to the data platform, and in that way, they avoid a silo, avoid having the vendor own the data and then trying to get, the customer getting the data through the APIs.
Q. Can vendors of connected apps also cater to customers that don't really have a data warehouse in place?
Yeah, actually the vast majority of connected app vendors that I work with also have a managed app option. It's not either-or. We even wrote a post at Snowflake about which deployment model is better, and it turns out, it depends on the customer, and there are customers that, for them, a connected app is gonna be better, and for others, yeah, maybe if they're a smaller shop and they don't yet have their own cloud data platform and they'd rather the vendor just own the data, then the vendors have that option. It's definitely not exclusive and the nice thing is once the app is built to support Snowflake or whichever cloud data platform, data warehouse, it's gonna be consistent. You build the application once and then you can support both options.
Q. What are some key benefits for companies to adopt connected apps over managed apps?
Yeah, oh my God. Bunch of benefits. I see customers seeing so much success with this. I think the more that data is at the core of everything that that team is doing, I'm speaking to security teams and they're tasked with protecting the enterprise, reducing risk, detecting threats.
It's all about data, and unfortunately, it's been the case that security teams have been working with a very fragmented data landscape. Their data's been all over the place and a lot of their effort goes into trying to piece together what's happening across these different data silos.
The connected app model means that the silos are avoided and the security team doesn't need to go through the effort of piecing together these disparate data sets because their solutions are all pointing to the same single source of truth. That's been this kind of mythical thing, the single source of truth for the security team. It never existed before this model, at least for the security teams that I spoke to, and we do have customers now. I'll give you an example, they're speaking at our user conference next week, TripActions.
They have a great product. I use it whenever I go on a business trip. I use TripActions and their threat detection product is Hunters. The Hunters solution points to the TripActions Snowflake and it's the same Snowflake they had before the security team started using it. This is this trend where the security team is joining the rest of the company on this unified data platform.
Hunters enables the threat detection response, what used to be covered by SIM solutions. Now they also wanna do compliance automation. Great. The compliance team's not starting from scratch. They deployed their tool of choice, anecdotes, and that product points to the same single source of truth where a lot of the data is already residing, and so very quickly they can get to compliance automation. I think it's a huge benefit for the security team. It also gives 'em flexibility. If they have custom use cases, they can implement them because they own all the data. They have no disadvantage to the vendor, in the fact that they have access to all the data in an analytics platform so they can analyze it the way they want and they can share insights across the enterprise because the enterprise BI tool of choice, if they're using Tableau or Power BI, whatever they're using, that's already pointing to their Snowflake, so any kinda team in the company can get access to the insights that they need.
Q. What's in it for vendors to embrace the connected app paradigm? Doesn't it, in some ways, limit their revenue potential?
No, I don't think it limits their revenue potential. In fact, the more that this trend picks up pace and the more that teams, like security teams but also marketing teams and sales teams, see the importance in having a data-driven strategy, the more that they are going to embrace vendors that make it possible and that take an open approach and give them the freedom to own their data and to do with it whatever they want.
What we're seeing is vendors that are adopting the connected application model and support it are in a position to disrupt their entrenched competitors by saying, "Look, here's a new way. These big legacy vendors, they don't support this. They're gonna require you to send your data out to them and they'll hold onto it and they'll make you jump through hoops just to get access to some of it.
We'll meet you where you are," and that's very attractive, and so we are seeing our connected app partners growing very quickly. Snowflake Ventures has actually invested in a few of these partners, so we do have that perspective of seeing their growth and I can tell you that it is very substantial. I think more vendors are gonna see the benefits there and lean into it.
Q. Snowflake offers a program for vendors looking to embrace the connected app paradigm or when they're looking to build connected apps. Can you briefly explain what that entails?
Yeah, yeah, I'm very proud of this program. It's called Powered by Snowflake and it was created within our partner organization from an understanding that when somebody builds their product to run on Snowflake, they have different needs than the traditional enterprise that uses Snowflake maybe as a data warehouse, and that might include solution architect guidance for getting started.
How do you design? How do you build this? How do you think about your multi-tenancy model in a connected app world?
There are all sorts of interesting questions around that. How do you take advantage of unique features like Snowflake data sharing, Snowflake support for Python in Snowpark? All these different things that Snowflake's doing, ideally our partners are taking advantage of that. They're getting the guidance. They're learning from the experience that we've had in helping others to build within this model, and then also when the thing is ready, how do we help them to sell into our customers? How do we do launching the GTM and co-marketing and co-selling, helping our sellers to know what's out there?
They want to understand what they're introducing their customers to, so it's really a great program. It also includes additional support 'cause we wanna make sure that if there is an issue, that we have very fast support and that support is familiar with the application, what it's supposed to do and all that. It's a great program and I think it is easier than ever for vendors to add connected app support if they want to.
Q. What are your thoughts on the impact of connected apps on CDP and reverse ETL vendors?
Yeah, so you know my focus is on security, but I do see the connected app model taking off across the board. It just makes sense, it just makes sense, and I think there will continue to be a need for reverse ETL as long as many of the most important sources don't support the connected app model. There needs to be a way to get the data into the data warehouse and then to take action on it and to make changes in other systems based on what you see in the cloud data platform, but I think it'll evolve.
I think the role of reverse ETL vendors will evolve and I'm excited to see how they accelerate connected applications and how they actually make it more and more possible. There's one startup that I'm excited about. I've been talking to their founding team and I think it's really cool what they're working on.
They're called patch.tech and what Patch are doing is they're building kind of a foundation layer for warehouse-native applications, and I think it makes a lotta sense because why start from scratch? And I think that could be the direction for reverse ETL so that it becomes a project of hours or days instead of weeks or months for a SaaS vendor to add support for the connected application deployment model.
Q. Last question — can you share some tips on what companies should look for when evaluating vendors of connected apps?
That's a good one. I haven't been asked that before. It's a good question, and the more that we see leaders at Snowflake's customers really starting from the position of, "This is gonna be the home for our data, now which vendors will support that?"
I think it is important to have this guidance out there, so I'm glad you asked it. I think any customer evaluating a connected app vendor should get an understanding of how full is the connected app integration. Because this is a new category, you may see certain vendors implementing it partially and maybe they'll make some of the data available in your ownership as the customer while they'll still force you to pull other data from them through an API, that kind of thing.
You really ask the vendor about what is included in the connected application and what is not, and also ask about discounts, because by taking on the role of owning the data, storing it, paying for the compute of analyzing it, you as the customer, you're offsetting quite a bit of COGS for that vendor. They should be giving you a discount, right?
Otherwise, they're benefiting from it twice, once with you as a customer, second, when you're paying for the compute power that their application requires, so definitely ask for that discount and see if you can get a nice one, and then I'd say the last thing is ask if they have content that you can take as a starting place when you're analyzing their data in your data platform, because what the connected application model says is you have all the vendor's data that it would, it's actually your data, but it's traditionally the vendor's data and it's in your control.
Maybe they have some views or some models that you can use to get value from it faster and to report on it faster and self-service BI, et cetera, so ask about that, the data schemas and any content that they can give you so that you get to value faster from having this connected application deployed.
You can also tune in to the episode on Spotify or Apple Podcasts.
Prefer watching the interview?
Continue learning about warehouse-native apps:
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
What is data automation?
How is it different from iPaaS and workflow automation?
What is data unification and how is that different from identity resolution?
Nick Bonfiglio from Syncari is here to tell us and answer some related questions.
Let’s dive in:
Q. What exactly is Data Automation?
Originally, data automation was a data science concept to describe processing, normalizing, handling of data with automated techniques. So until Syncari, multi-directional data automation was only available to, well, first of all, it's available to anyone. But if you were trying to do it, it would be a lot of coders and data science that were trying to do that. So Syncari is actually bringing these capabilities, plus some MDM and CDP capabilities that we borrowed, to essentially business users with a no-code platform. In our case, it's targeted at go-to-market engines that go all the way from leads to billings inside of your go-to-market engine.
Q. Is data automation the future of iPaaS?
So the way I would answer that is, point-blank, data automation is not integration. And, rather, it's a way to generate a 360-degree unified view from all your systems, and then orchestrate the go-to-market processes across your business. Again, from leads to billings. The difference with Syncari is this distributed 360 view of your data is not centralized and it allows you to curate and share that with all your teams. And, you know, and that's a very unique position versus just simple point-to-point integrations that are moving data back and forth, whether they happen to be ETL point to points, or automation point to points, or whatever they happen to be data point to points. It's very different. The other big difference that Syncari has over iPaaS is these are stateful, multi-directional synchronizations. Now, everybody uses the word sync these days, but, truly, sync is to have data be identical in more than one place at exactly the same time. So, you know, point-to-point connectivity solutions, like iPaas, just can't touch the data at the level that we do.
Q. How is data automation different from workflow automation?
If you think about how workflow automations today work in most systems, they are triggered and have access to transitory data across two endpoints. And that truly minimizes your flexibility, whether it's an ETL, whether it's a integration, or whatever it happens to be, it's transitory data and it minimizes your flexibility in being able to create automations. And so what data automation, and especially Syncari, does is we take this data model view, a unified data model view, of all the connected systems. And you can use all of your data to orchestrate cross-system and cross-object automation.
The best way to think of it is if I wanna look up what's going on with an account based on a contact or lead that just came in, it's incredibly difficult or impossible to do with today's, you know, workflow automation platforms. So the largest difference is it can also transform data, normalize it, enrich it, calculate it, dedupe it, all centrally, and then distribute that end result to all the connected systems at the same time. And so, this ability to keep all your systems in sync and near real-time, you know, across all touch points is really what Syncari does.
Q. Would you say data automation is a team sport since it involves so many different teams?
That's a great question. We run into this all the time. The reality is you can think of it as either a team sport when you're using it for a few systems. The example here would be, like, let's say I'm a sales ops person. I'm just trying to get my sales systems to work more cohesively and be unified, and have the proper, you know, merge, dedupe, enrichment, and automations across those.
But, however, to truly align your entire business and to get business alignment across all your go-to-market teams, you have to, eventually, see this as a team sport. And if you think about this even further, the rev ops organizations in businesses today, have really emerged from the fact that they're trying to get alignment of their businesses from leads all the way through billings as a cohesive thing. So, in that case, it becomes a team sport over time, but it certainly, in many cases, starts out as an individual sport, if that makes sense.
Q. Which teams benefit the most from well-executed data automation workflows?
In our particular case, it's anyone that touches go-to-market. So whether it's product data activation from your product, if you're selling SAS, or whether you're trying to automate a fully congruent PLG motion journey from beginning to end, whether you're trying to get, you know, leads, contacts, accounts, you know, all of that out to, you know, billings and accounting systems congruent, you know, it requires, essentially, everyone to help out there, right?
🤔 Have questions?
Q. Where does data modeling take place when building these data automation workflows?
Yeah, so we try to take the pain outta data modeling out completely from the end user. If you look at Syncari, it's not about connecting two endpoints. It's about what does my data model look like as a result of running this stack that I'm using. And, so, we present the user with a unified data model of those systems. You know, there's a lot of moves in the marketplace right now to try and define a data model that's standard. That's never gonna happen. Every SAS company that is, you know, believes to be the source of truth for the business logic that they support, is gonna always have their data model.
So we took the approaches, like, we help you stick together the data model that you wanna see based on the systems that you decided to use. And for many of these systems, our connectors are super smart. They go all the way down to the schema level, and they not only sync data, but they sync schema from all of these systems. And we know how to map that schema across all these different systems. So if it's customer over here, account over there, you know, whatever, person, lead, contact, all of that is pulled together into one common view for the customer. And then they operate on top of that data model to perform the operations and automations that they want to do.
Q. What exactly is data unification? Is it the same as identity resolution?
No, no. It's very different. So the best way to think about this is in that data model that we just talked about, many systems can sync into the account object, let's say. Or, let's take the contact object at the moment. And so, I could have Salesforce, Marketo, Zendesk, NetSuite, et cetera, all feeding into my contact object, right? And I can normalize all of those into one canonical place. The ability to stitch that data together from all those systems, and to say that, "Hey, this is Arpit." When I say, "Arpit," and I change Arpit in one system, I know exactly the same record that I'm touching in all of the other systems.
Where in today's iPaaS technologies, let's say, I said, "Oh, well, I'm just gonna send over this email address as the thing, the key to sort of, the ID to send over to this other system." Well, guess what, if there's duplicates, and let's say, I did a find by email ID. I get one. Which one do I update? And this is why you see a lot of discrepancies in data across companies, because as their databases grow, more duplicates come into the picture, these iPaaS systems end up clobbering data all over the place. And so, what we've done is said, "Hey, as part of syncing, you have to make sure you're touching the right record for every object in your data model across all the connected systems." So we have borrowed some learnings from prior MDM stuff. So we have a Syncari ID and then we map every record from every system. We attach it to that Syncari ID across all the systems that we connect to, so that we know empirically which record to touch in each one of these systems based on the record that got touched in any particular system, if that makes sense. And that allows you to maintain congruency.
So if you think about it, if I don't have unified records across all these systems and I wanted to do global merge and dedupe, how do I know which merge and dedupe operations are running those systems, if I didn't have unification of that data across those systems? It's very important to do that all in one place. And what we see happening all the time in a lot of our customers is Salesforce is running their own merge and dedupe, Marketo's running their own merge and dedupe. Two different tools clobbering their data, then trying to synchronize this. It just becomes this huge mess. And then that also gets made worse by the fact that people wanna enrich the data through data providers. And where do I enrich the data? And when do I enrich the data? And then these things start ping ponging back and forth on updates. And so what we said is like, "Look, we have a canonical view of the data model. A canonical data set. You're gonna operate on top of that data set. And any operations that you do on that data set is gonna end up at the end systems in exactly the right way that you wanted it to be there."
Q. Since you earlier mentioned CDP. How would you define CDP and does Syncari, in some ways, replace a CDP?
You know, I guess I would answer that by saying that CDP was another attempt to centralize source of truth, right, from different endpoints. And try to get the data of your CDP back into the connected systems, not so trivial, right? So the best way to do it is we employ some CDP techniques as far as how we persist data and move data around, but our linkage back to the end systems is a thing that most of these CDPs can't do very well. Even the ones that have added, like, a persistency layer to their backend, they still can't get the unification done across all the connected systems to know that they're touching the right data.
So this is the thing we wanted to fix from traditional CDPs and iPaaSes that are in the market today. But in many cases, you used to think of us as being able to also write alongside CDP because they've got telemetry into systems that do a lot of different things, and we can actually read from a CDP and make that part of the overall unified data, right?
Q. Last question for you — I heard that you have a slightly different opinion about data warehouses being the central source of truth for data. Can you please walk me through it?
The reality is it's pretty straightforward. And every go-to-market system is really the source of truth. Again, for the business logic that it provides. CRM, market automation, accounting, whatever may be, that system has business logic and creates the source of truth. So just moving all that data, without consideration to how it needs to get across other systems, into a data warehouse doesn't solve the problem of how do I get congruency across the connected systems.
And if you can't get the right truth distributed to the right systems at the right time to take action with the business logic in each one of those systems, then you've really defeated the purpose of why the heck you were trying to organize that data centrally to begin with. And so what it turned out to be, for many companies, is a glorified reporting system off to the side. And it's like, okay, so I've got a reporting system off to the side, but I need to take action on some of the data that I've got in the data warehouse. What do I do? So of course, you know, we've created reverse ELTs, but those don't solve the problem because you haven't gotten unification across all the different systems. And so you can't get the right truth to the right place at the right time.
And that's the thing that's very different between centralized sources of truth and Syncari's view of the distributed source of truth across the connected systems.
🤔 Have questions?
You can also tune in to the episode on Spotify or Apple Podcasts.
Prefer watching the interview?
Thanks for reading — let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
What is entity resolution? What is identity resolution? How are they related and why getting them right is a hard problem?
Sonal Goyal from Zingg is at the forefront of solving this problem and she has some concrete answers.
Let’s dive in:
Q. What exactly is Entity Resolution?
Sounds like a very, you know, a lot of jargon, entity resolution, but at the heart of it it's actually saying that multiple records in your warehouse or on your data lake belong to the same real-world entity. And this entity could be a customer, it could be a supplier, it could be location. Just any noun that the business deals with.
Q. So identity resolution is a subset of entity resolution, right? Can you please explain?
Yeah, so when we talk about identity, right, it's actually who you are. And entity resolution at a broader level is establishing what an entity is. So entity and identity in fact are very closely tied but when we talk about identity it's more related to the person, so like a customer or a citizen. Those are identity resolution, technically that's what's identity resolution.
Q. Considering entity resolution is so important, why is it still a largely unsolved problem?
It's a very important problem but to realize the problem, I think there are some building blocks that need to be there. First of all, the enterprise has to be ready with all the data in one place, being ready with their analytics, pipelines, their ETL, for them to figure out that, you know, now they need to establish the linkages and they're ready for analysis. And that is the problem. That is the time at which, you know, when they realize that you know, there are five records belonging to one customer and they're not able to tie it together. So for entity resolution to really happen some of the building blocks need to be in place. And that is what we are seeing emerging more and more, that, you know, the base tech for companies is ready and that's why the need for entity resolution is growing.
Q. So why did you choose to work on entity resolution? Like why is it exciting?
A. So I chose to work on it primarily because I failed, you know, to solve it in my first goal. I was working as a data consultant. I was tasked with building a data lake and we had to resolve some entities from multiple databases. And when we got to solving it, like, we really, really had a tough time. And that's where it hit me. And I saw this problem again and again as part of my consulting. And I felt that, you know, one. this is a tough problem to solve. Second is that it is a problem, if solved in a very domain agnostic way can actually serve multiple industries, multiple data sets. So that's what excites me very much.
You’re building Zingg, an open-source project to solve this problem.
Q. Can you give us a high-level overview of how Zingg works?
So Zingg as an open-source project, what it does is a very simple workflow. What you say is that here is my, you know, here are my records, here are the attributes on which I want to match. Some of these attributes I am okay to have variations, some of these attributes I want them to exactly match. So what we call is fuzzy and exact matching. And that's what you can configure in the system. And you tell where the data is residing. And then Zingg starts showing you some peers, asks you to tell it whether, you know, whether there are matches or non-matches according to your business logic. And pretty much you run a few rounds like that. The AI models behind Zingg start getting refined. And pretty much after, you know, a few rounds of labeling you are kind of set with your entity resolution models.
Q. Can you briefly share some light on the difference between using Zingg versus building one's models using SQL?
So see, entity resolution can be, you know, as simple as maybe in some cases, you have a user email already and then you know that these two records with this email belong to the same individual. Or you have, you know, guest checkouts and if the user finally checks in and logs in and then you capture that email, you know how to associate the anonymous activity with the logged-in activity and thus user ID.
If it is as simple as one or two systems and, you know, very deterministic attributes on which you can actually link and match I think it is okay to go with the SQL and it's a fair choice to make. But in most cases what happens is there are multiple sources from which the data is coming. They all have variations across name, across age, across address, across telephone numbers, across multiple emails.
And then with the growing size of data as well as the variation in the attributes this becomes an increasingly tougher problem to solve. Because like it's a classic join problem, right? We all talk about tuning your joins in a database, but then if you don't have unique identifiers really what are you going to join on? So it gets very complex very quickly.
And I think that's where you have to differentiate understanding your data and differentiate whether a simplistic SQL model is good enough for you or a more advanced solution or across multiple attributes, also fuzzy and deterministic matching is something that your data set needs.
🤔 Have questions?
Q. Can you tell us how this approach differs from the identity resolution capabilities offered by CDP vendors?
So CDP vendors, so one is like Zingg is entity resolution, which is like, you know, a much broader problem statement across different languages, across scale, across entities. But just coming to identity resolution per se, most CDPs, they're like a third party system, right? They're not working directly on your warehouse. So your data warehouse still has, you know, these unresolved entities.
The CDP data is actually with the CDP vendor, which you have to then get back into your systems. Zingg on the other hand works natively on the warehouse or the data lake. So you have control over your matching process, the frequency at which you're running, over your data model. So the CDPs define their own data models. Zingg is very flexible about, you know, you defining your own data model. And I think the whole approach to Zingg is favoring different use cases and various varieties of data. Which the CDP in a very minimal sense does, but obviously, I mean, Zingg is a very focused product for entity resolution compared to the offerings by CDPs.
Q. Since you mentioned use cases besides the classic ID resolution use case, what are some other important use cases of entity resolution?
Yeah, so I think in the web world, right, we talk about CDPs and we like, we know identity resolution. But identity resolution in the web world is a very marketing or a sales defined need. But when you look at traditional industries like banking or healthcare, so there a lot of, you know, compliance, there's a lot of know your customer, anti-money laundering, GDPR, healthcare provider data.
So in healthcare you have like Sunshine Act where you have to be, as a healthcare company, you have to declare what is your affiliation with healthcare providers? So those are things where, in all those cases you actually need to resolve those entities with the various touch points which you've had with these entities. So beyond the classic CDP identity resolution, entity resolution by itself has a lot more use case.
Q. Can you explain how e-commerce merchants can benefit from entity resolution?
So that's interesting because the first use case that is going into production with Zingg is actually an e-commerce use case. We have a user who is building a review website for products and so they have, the team has scraped product data from multiple websites which they are resolving, the product data they're resolving through Zingg and then putting in the reviews. S
o that is one very item matching use case, product catalog item matching. Then the other is the guest checkout and then realizing that, you know, somebody has checked out so the user journey, the customer 360 for the user. So those are broadly two use cases in the e-commerce space.
Q. Last question for you — what's your advice for companies looking to get started with entity resolution in general and identity resolution in particular?
I would say that know your data. I think just like, just with any data project, right, know your data. Start small with one or two use cases or one or two data sets. Don't underestimate your solution.
You can also tune in to the episode on Spotify or Apple Podcasts.
Prefer watching the interview?
Identity resolution in the Data Warehouse vs in a CDP? If you’re in the process of figuring out which one’s right for you, check out this guide.
🤔 Have follow-up questions?
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe -
What is CDI, what is CDP, and what's the difference?
Can one exist without the other?
Do you need one if you have a data warehouse?
Michael Katz from mParticle is here to tell us and answer some related questions.
But first, check out our guide on this topic:
Let’s dive in:
Q. What exactly is a CDI?
So let me step back before we step into what is a CDI, what's the CDP, all that stuff. So just to table set, there's more data being created and ultimately consumed than ever before. Throw privacy requirements into the fold, plus all the change from Apple, and Google, got third-party cookies, changes to iOS14, and things have gotten complex pretty quickly. That, combined with the fact that I think the walled gardens used to be a little bit of this easy button for brands to go out and do a bunch of user acquisition that was cost-effective and highly scalable, all that's changed, right? And now what's happening, especially with the change in the economic conditions, everybody needs to do more with less. And really what that starts with is data, building a strong data foundation.
So customer data infrastructure was really built by us, and it was designed to solve kind of like the three core tenants of customer data, which is data quality, data governance, and data connectivity. So I think about it in pretty general terms, like there's the part of the iceberg that you can see, and then there's the part beneath the surface. The part that you can see is the activation of customer data. That's audience segmentation, it's audience orchestration, audience insights.
It's like the, I think what a lot of people think of as kind of core CDP functionality, and it's like the cool kind of fun, sexy stuff that's oriented towards marketers. But there's this massive, the vast majority of the challenge happens beneath the surface, right? And those are the things that ultimately need to get accounted for so that by the time you get to the top of the iceberg, or the tip of the iceberg, those things are in good order. Now, the problem is that people, a lot of CDP vendors will lead you to believe like, "Oh, well, those problems aren't as bad as other folks like us may lead people to believe," or that if they have certain solutions, it's easy to kind of back into different solutions. Here's the thing. You cannot disconnect the operational pain from having to deal with an ecosystem that is in a constant state of flux with the execution of your digital strategies.
They are one in the same, and so the bottom of the iceberg is still part of the iceberg, much like the tip of the iceberg is still part of the iceberg. CDI helps you address that bottom part of the iceberg, which is, it really amounts to the data chaos that teams face internally as a result of everything changing, right? And as everything changes, ultimately everything breaks, right? So new vendors come to market. You may wanna try them out.
APIs change, new laws are enacted which create new restrictions. Again, the change is dictated upon everybody from Apple and Google. Has everything in a kind of steady state of flux. New landing pages are added, or screens are added or removed and optimized, tracking plans changed, ownership of the tracking plans change, and so what breaks? Your data breaks, your data pipelines break, your data schemas break, the APIs break, the customer experience. Your analytics break too, right? And so trying to treat that operational complexity and that data chaos as separate from audience insights and audience segmentation is, I think, a fundamental flaw that I think a lot of teams have ultimately had to learn the hard way.
Well, that made a lot of sense. Like I say, without infrastructure, there cannot be a platform.
Q. What is a CDP then?
CDP is a tool. It's an application used by marketing teams to be able to orchestrate customer data to their different marketing partners. And I'm not saying that that's not valuable. It is valuable. It's incredibly valuable, but only insofar as making sure that you've protected data quality, because the activation the orchestration of data for marketing purposes, even for analytics, but, CDPs typically focus on the marketing use case. The output is only gonna be as good as the inputs, right?
Garbage in, garbage out. And then secondly, from a data governance standpoint, if that's not completely integrated into your compliance systems, and you end up breaking laws, the fines are real.
There's been well over a billion dollars of GDPR fines since the introduction of the regulation a couple years ago. So these aren't just somebody else's problems that works down the hall, or is in a different function, or whatever it may be. It's all one in the same. Nothing is somebody else's problem.
Q. Can a CDI exist without a CDP?
The CDI, in my view, has CDP capabilities. The CDP is for marketers. The CDI is for the full business, including marketing. What a lot of CDPs don't do is they don't stream event-level data out to different analytics and measurement services, right?
They don't do native event collection. They don't help you build a data strategy. They can't, because a lot of the times they're ingesting data from previously deployed tools or systems.
They don't give you that layer of governance. But in either case, the value is created when the data leaves the system, right? So whether it's CDI, or whether it's CDP, you still have to land the data in the downstream applications that need to consume the data. The CDP, therefore, is a subset of functionality that the CDI has offered for a few years. And CDI vendors, it's really, mParticle and Segment like it's always been. It's been the two companies that invented the CDP space.
Q. So where do identity resolution and data governance sit? From what I'm hearing, they seem to be tightly coupled with the CDI, right?
They have to be, because they have to be completely integrated into the point of data collection. If you don't get data collection and data quality done properly at the first mile, you can't solve for that stuff after the fact, right? The sequence of steps matters. The order of operations matter. And so identity resolution where that comes into play, that is part of data quality, as far as I'm concerned, because ultimately, data quality is not just about the format and structure, but it's like, are you merging the right data into the right customer profiles, right?
It's the organization of information, which has a number of impacts downstream, like how fast can the data get accessed? What are the SLAs around audience creation? So on, so forth, right? And so if you're not doing that at the point of collection, you're also exposing yourself to potential liability from a governance and compliance standpoint, because ultimately, if somebody has opted out of certain use cases, or they just don't want any of their information to be tracked, but you're still grabbing it all, and then you're saying like, "Well, I'll just kind of like figure it out later," doesn't work that way.
That delta between doing it right at the first mile, and figuring it out later, that's where liability kind of hides in plain sight. And so people are delusional to try to say like, "Well, you can unbundle everything," or again, like, "This is somebody else's problem. I can integrate with them after the fact." It doesn't work that way. It just doesn't.
Q. Should CDI vendors store a copy of the data that their customers collect, and why? Why should they do or not do that?
Well, they don't have to, right? It depends on the use case right? If it's a use case where it's really just about effectively ETLing and the data out from the point of origin, and you can treat customer data in an ephemeral nature, where it's like, you don't need to create a copy of it, yeah, sure.
You don't need to, but, and we do have customers that do it that way. We have plenty, but if you want the safety of redundancy, so to be able to create a copy of the data for the purpose of historical data replay so that if any of your vendor APIs go down, you can replay historical data into their systems, you don't have gaps in coverage, or you wanna hydrate new systems with historical data, yeah, the CDI should create a copy of that data. If you wanna build audiences that take into account, not just real-time information but historical lookbacks, then you have to create a copy of the data.
Q. And if I'm storing my own data in a data warehouse, do I still need a CDP?
Or can I do whatever I would do in a CDP, in my warehouse, and then sync the data back to downstream systems using a reverse ETL tool?
You can do, you can execute very basic marketing strategies utilizing the data warehouse, and a reverse ETL tool, right? It's not an all-or-nothing thing. I look at that construct as a perfect setup for really immature companies, right? But when you start getting into more sophisticated marketing that may require real-time personalization, sequential logic of how users are added to or removed from audiences.
Like today, we just announced an audience journey product which is this WYSIWYG editor that allows you to dynamically decide how and when certain users are added or removed from audiences based on things that they do or don't do in certain orders. And just dumping data to a data warehouse, and then spinning up a two-dimensional audience builder, it doesn't account for that, right?
Because you also need to be able to get data back in from the downstream tools. There's a real-time element to it. You may wanna create multivariate tests. Those types of things are not possible via a data warehouse-based setup. And then what you also lose is the integrated compliance and governance piece, right? So, and I can go kind of on and on.
I think it's a good starting point. I think the tension in the space right now, the reason you have both sides talking past one another, you have a lot of data engineers that just fundamentally don't understand go-to-market dynamics, and you have marketers who aren't technical enough to appreciate a lot of the work that's being done within the data engineering group. People need to come together. We've championed this idea of data as a team sport for the past few years, and it has to be an inclusive thing, not an exclusive thing.
Yeah, absolutely! This is something I talk about all the time, right? I believe there's this growing divide between GTM teams and data teams, and there needs to be a bridge that enables both of these teams to understand each other's priorities and constraints better, find a middle ground, and work together.
Q. How are ETL pipelines different from CDI pipelines?
Well, yeah, ETL pipelines typically will pull data from previously deployed internal or external systems right? An example, a very basic example would be like extracting data from Salesforce, and getting it into, I don't know, a marketing tool like Braze, for example. CDI isn't just about the extraction of data from previously deployed tools or systems. It's about building a strong data strategy, and implementing that data strategy properly from the point of data collection to be able to have as much control and transparency into the entire journey of a single bit of data, and throughout it, enrich the data, protect the data, cleanse the data, govern the data, all those things that you can't get in kind of a dump pipe.
Q. When you say data collection in terms of CDI, what is the data source here you're talking about?
Yeah, primarily, but not exclusively, primarily, it's digital properties, websites, apps, point of sale systems, connect to TV apps, those types of things, and ETL just doesn't work that way.
Q. Data Activation is pretty hot right now — how would you define it?
Yeah, well, data activation picks up where data analysis leaves off, right? So the problem with a lot of tools is, especially in the BI and analytic space is they provide great dashboards, and rich insights, but then you have the now what problem. Oh, wow!
That was a really insightful learning that we just were able to derive from that chart. Now what? What should we do about it? So data activation simplifies the process, or is designed to simplify the process of moving from insight to action, right? Otherwise, you're at this dead end.
Now, data activation has been around for forever, right? Data activation is any action that you take on data. What people are talking about now, and I think what you're probably mainly referring to is the attempted rebrand that's happening within like the reverse ETL space, because I think that they realized that was a bit too narrow, but it still is what it is.
Data activation is getting that data in motion and downstream to systems of activation, systems of engagement, right? And I think within most stacks, there's system of record. Truth, be told, across organizations, there's usually hundreds of systems of record. There's never just one. So there's a system of record, there's a system of intelligence, or systems of intelligence, and then there's systems of engagement. Engagement is marketing, it's paid media, and it's retention marketing, kind of simply put.
Q. Last question for you — what’s the one piece of advice you have for companies getting started on their CDI journey?
Yeah, it all starts with having a good data strategy. The thing that I've seen, certainly over the course of the past eight, nine years of running the business is that building a strong first-party data foundation is not easy.
It's why people and teams were relying on third-party cookies, and data enrichment, and outsourcing their results to middlemen and ad networks, and weren't thinking about building the structural capacity to be able to establish and optimize relationships with their customers. That really starts with building a strong data foundation, which starts with understanding what is the data that I need to capture to have an informed relationship dialogue, conversation, whatever, with my customer? So start there.
Get off the reactive hamster wheel of thinking tools first, or solutions first. Put your customer first. Decide and determine what does good look like? And then you can work backwards from there. It's actually not all that hard once you kind of know where to start. And the thing that we advise all of our customers on is you don't wanna capture everything.
All of this advice — just be data hoarders — is the worst thing that you can do, 'cause ultimately, you flood the system with noise versus signal. You actually wanna capture as little as possible to create the most amount of impact. So start with what are your KPIs, and what data you need to capture to calculate those KPIs? What does the customer journey look like?
How do you think about audience segmentation? What data is needed to power the stack today, how might that stack evolve, right? You don't need to solve all of it at once, but you should have a pretty good understanding of structure, format, organization, and application of data, and that's at least the makings of a lean data strategy.
You can also tune in on Spotify or Apple Podcasts.
Prefer watching the interview?
Let’s beat the gap! 🤝
Get full access to databeats newsletter 🥁 at newsletters.databeats.community/subscribe - Visa fler
