Timothy E.

🎯 Snowflake just dropped two new search modes for the 𝘚𝘌𝘈𝘙𝘊𝘏 function that are an absolute game-changer for anyone working with unstructured text: 𝘗𝘏𝘙𝘈𝘚𝘌 and 𝘌𝘟𝘈𝘊𝘛. This is the precision we've been waiting for. 1️⃣ 𝗣𝗛𝗥𝗔𝗦𝗘 𝗠𝗼𝗱𝗲: This is what you've always wanted from a text search. It finds a match only if all the tokens from your search string appear in the same order and are adjacent. Searching for '𝘦𝘳𝘳𝘰𝘳 𝘱𝘳𝘰𝘤𝘦𝘴𝘴𝘪𝘯𝘨 𝘱𝘢𝘺𝘮𝘦𝘯𝘵' will now only match that specific sequence, not documents that happen to contain the words '𝘦𝘳𝘳𝘰𝘳', '𝘱𝘳𝘰𝘤𝘦𝘴𝘴𝘪𝘯𝘨', '𝘱𝘢𝘺𝘮𝘦𝘯𝘵' scattered around. 2️⃣ 𝗘𝗫𝗔𝗖𝗧 𝗠𝗼𝗱𝗲: This takes it a step further. It does everything 𝘗𝘏𝘙𝘈𝘚𝘌 does, but it also requires the delimiters (spaces, punctuation, etc.) between the tokens to match perfectly. It's for when you need surgical precision. Think about how this simplifies things. No more complex regex or multi-level 𝘓𝘐𝘒𝘌 statements. Just clean, intuitive, and powerful phrase searching, natively in your data platform. ✅ This provides so much more flexibility than the existing OR (disjunctive) and AND (conjunctive) search modes. Imagine you're searching application logs: 𝘚𝘌𝘓𝘌𝘊𝘛 * 𝘍𝘙𝘖𝘔 𝘭𝘰𝘨𝘴 𝘞𝘏𝘌𝘙𝘌 𝘚𝘌𝘈𝘙𝘊𝘏(𝘭𝘰𝘨_𝘮𝘦𝘴𝘴𝘢𝘨𝘦, '𝘗𝘏𝘙𝘈𝘚𝘌', '𝘧𝘢𝘪𝘭𝘦𝘥 𝘵𝘰 𝘤𝘰𝘯𝘯𝘦𝘤𝘵 𝘵𝘰 𝘶𝘱𝘴𝘵𝘳𝘦𝘢𝘮 𝘴𝘦𝘳𝘷𝘪𝘤𝘦'); 💥That query is now laser-focused. It will only return the logs with that exact error string, massively speeding up debugging and root cause analysis. This is a huge win for log analytics, document search, and any use case where the order and proximity of words matter. ⁉️ What's the first thing you'll use this for? Share your ideas below! 👇 Stephen Abdul Tim Changbin Chris Alejandro Ismail Hossein Tuğba #Snowflake #DataEngineering #SQL #TextSearch #Developer #Analytics

620

19 Comments

When data agents fail, they often fail silently - giving confident-sounding answers that are wrong, and it can be hard to figure out what caused the failure. "Building and Evaluating Data Agents" is a new short course created with Snowflake and taught by Anupam Datta and Josh Reini. This course teaches you to build data agents with comprehensive evaluation built in. Skills you'll gain: - Build reliable LLM data agents using the Goal-Plan-Action framework and runtime evaluations that catch failures mid-execution - Use OpenTelemetry tracing and evaluation infrastructure to diagnose exactly where agents fail and systematically improve performance - Orchestrate multi-step workflows across web search, SQL, and document retrieval in LangGraph-based agents The result: visibility into every step of your agent's reasoning, so if something breaks, you have a systematic approach to fix it. Sign up to get started: https://lnkd.in/gVwKuCCQ

6,505

221 Comments

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 Comments

I created a video covering 5 Scenario based Interview questions related to Deletion Vectors in Databricks (a really hot topic for interviews) Question 1) You are managing a large Delta table with billions of rows.  Deleting even a few thousand rows used to be very expensive because it required rewriting files. How do deletion vectors, help solve this problem? Question 2) Imagine two cases: Case A: Deleting 100 rows Case B: Deleting 100 million rows In which case do deletion vectors bring the most benefit, and when would you consider a file rewrite instead? Question 3) Your company must comply with the GDPR (right to be forgotton). Since deletion vectors only mark rows as deleted, the data still exists in storage. How would you design a process to ensure deleted records are physically removed for compliance? Question 4) Over time, analysts complain that queries on the Delta table are becoming slower after frequent deletes. How would you confirm whether deletion vectors are the cause, and what steps can you take to restore performance? Question 5) After months of deletes and updates, you notice storage costs have gone up and queries are slower, even though deletion vectors were meant to save space and time. Why does this happen, and what Databricks operations would you apply to balance storage, performance, and compliance? Link to the video in Comments! If you know the answers, then do mention here. I am sure this video will help you a lot. This is a really hot topic for Interviews these days! Do support by liking, commenting & sharing if you truly find it valuable :) #databricks

241

9 Comments

I've talked extensively about why data contracts need to be placed upstream, the emphasis for shift left approaches in data, and why we need to start addressing data quality at the code level. With the book in its final stages (out for print soon), I can finally go back to writing on Substack, where I can detail the patterns I've observed to be effective for data contract implementations... and the various traps I've seen. In short, implementing data contracts yourself often follows this perilous journey: 1. Feel the pain of unexpected data changes. 2. Implement transformation logic as a stopgap measure. 3. Stopgap measures turn into tech debt that amplifies the original issue. 4. Get buy-in for data contracts and implement them quickly downstream. 5. See a quick win as issues are flagged before reaching business users. 6. Data changes still mount, and now your data contract is a bottleneck. 7. Realize you need to actually prevent the upstream changes. 8. Struggle with upstream teams not adopting your contract because you never accounted for them in your requirements. 9. Realize that for upstream eng teams to adopt data contracts, you need A) leadership buy-in for a top-down directive to adopt, and B) fit data contracts in existing dev workflows to maintain adoption. Steps 1-8 often take data teams over a year to go through, and by step 9, they realize their requirements are completely off for the implementation. I've seen it over and over again. I hope my future writing on these topics will help teams skip to step 9 so they can get value from data contracts as quickly as possible. To help, my co-author, Mark, and I just released this article: https://lnkd.in/g6B2JGNx Good luck!

178

12 Comments

 Interviewing for a Data Engineering Role at Databricks? Databricks is the home of Apache Spark and the Lakehouse Platform. Engineers here focus on real-time data, ML pipelines, and large-scale analytics. 10 questions you should be ready to answer: 1️⃣ Explain how Delta Lake improves reliability over a traditional data lake. 2️⃣ How would you design a streaming pipeline with Structured Streaming in Spark? 3️⃣ Walk me through optimizing a Spark job handling terabytes of data. 4️⃣ How would you enable ACID transactions in a big data pipeline? 5️⃣ Explain how you’d manage schema evolution with Delta Lake. 6️⃣ What’s your strategy for scaling ML model training pipelines in Databricks? 7️⃣ How would you integrate Databricks with external BI tools like Tableau or Power BI? 8️⃣ Describe how you’d secure a data lakehouse in a multi-tenant environment. 9️⃣ How do you debug performance bottlenecks in Spark? 🔟 Tell me about a time you balanced speed vs. cost in large-scale data jobs. 📌 Tip: Databricks values innovation, scalability, and machine learning integration. #Databricks #Spark #Lakehouse #DataEngineering

49

2 Comments