At Big Data London, one thing was abundantly clear: success with AI is now firmly linked to having access to clean, trusted, and timely data. 

Michael Ferguson, the Big Data London Conference Chairman, kicked off the event with some stats from the 2024 Wavestone Data and AI Executive Leadership Survey. 89.6% of organizations say they are increasing investment in genAI. But only 4.7% have put anything into production.

What are organizations focusing on to close the gap? 87.9% of those same organizations say that, while they are increasing genAI investment, their top priority is investment in data and analytics.

Big Data is back!

With over 20,000 attendees, 400+ speakers, jam-packed halls, and great sessions and booths, Big Data London reflected the importance of data at this critical time.

The Prophecy team was out in force at the event, focusing on a very important part of the big data equation: breaking through the data transformation bottleneck.

The three Vs of Big Data - volume, velocity, and variety (first coined way back in 2001 by Gartner analyst Doug Laney) defined the way the growth of data would upend traditional data management. Fast forward to today and we’ve gone from a largely relational and SQL-only world to adding noSQL, scaling out with Hadoop, adopting data lakes, and adding data lakehouses and open table formats for even more flexibility.

Data cloud platforms combine many of these approaches and, leveraging the elasticity and scale of the cloud, are allowing organizations to bring together all their data, overcoming the challenges of the 3Vs and delivering organizational-wide data platforms.

Modernizing wholesale banking with self-service

But getting access to clean, trusted, and timely data in those data cloud platforms is still a challenge for most organizations. We were able to share how one organization overcame this in our Big Data London session “How a Major Bank modernized wholesale banking to deliver self-service with Prophecy”.

Wholesale banking involves delivering services and solutions to financial institutions. Unlike retail banking with set products, wholesale banking is all about developing deep relationships with customers and collaboratively working together to deliver innovative solutions.

Data is key to delivering this collaboration. For this bank, the challenge was tapping the massive, 19PBs of customer and transactional data they have under management. Creating the data assets to improve customer relationships was taking too much time and was too costly. This is because, with the modern data cloud tools, the highest levels of resources were required to code the transformations and it took multiple steps across multiple types of tools to define, design, and test data assets.

With Prophecy Data Transformation Copilot, the data team was able to streamline the process and bring more people into the process. Prophecy’s visual pipeline development enabled the data team and business teams to use the same tool, speeding data asset design and testing. And with embedded AI, the data team was able to use a wider range of resources, greatly reducing the dependency on the highest level experts while still delivering data engineering quality code. The result is an overall reduction by 50% of data asset creation costs, reduction in pipeline development times of 55% and, overall, 5-10x more data delivered to the business.

Catch the talk recording here.

Data people, data strategy and … data puppets?

With this kind of impact on one of the biggest challenges in (big) data, it was no wonder that the Prophecy booth was action packed. We juggled customer visits and demos and meeting with partners and even found time to squeeze in conversations with data trailblazers including Sanjeev Mohan, Joe Reiss, and Ravit Jain. 

We also got to some great sessions, hearing about the ‘dome over the cube’ strategy to explain data platform capabilities at Boots, Sainsbury’s customer impacting data strategy, and even caught up with the Databricks team for their night out at the Design Museum. 

And delivering much needed comic relief was Scott Taylor’s Data Puppets. Voiced by leaders from across the data world and acted out by, well, puppets, put the upcoming Journey to the Center of the Single Version of the Truth on your watchlist - and catch the preview here.

Thanks - see you next year!

It was great to see that Big Data is back … and the Prophecy team will for sure be back again next year!