Prophecy presented a complete Low-Code Data Engineering product at the Data + AI Summit 2021. We had a session for Low-Code Spark, one for Low-Code Airflow, and Column-level lineage - where hundreds attended. We conducted polls in these sessions and want to share the results:

Enabling more teams on Spark

Spark has a steep learning curve and we found via polls and via ad hoc conversations with participants that about a third of the teams are fully enabled on Spark. We find that smaller organizations, especially startups are often more enabled. We also found many companies where first one or two teams are enabled, but the rest are not - and they're looking to get more team on-board. Finally, there are still a large number of companies who are new to Spark and are looking at ways to adopt it - this is a signal that Spark continue to gather interest and reinforces its primacy in data processing.

Here is a presentation showing how easily a user who wants to only do drag-and-drop and write a few SQL expressions can be enabled on Spark - for development and testing. It also points to how organizations can roll out their own frameworks for standardization.

Low-code Airflow has a lot of demand

There is a lot of hunger for making Airflow simpler to develop and deploy. There is no product in the cloud that is simple & powerful for scheduling. Airflow is powerful, and we're making it simple.

Here is the demo of how Low Code Airflow works!

Lineage is missing for most & they want it

Prophecy provides column-level lineage across all the workflows and datasets. We do see that less than 20% of the respondents had column-level lineage - whereas 75% say that they need it.

Here is a link to the part of the video that demos column-level lineage

We're excited to wrap the the summit and look forward to seeing you at the next one, hopefully in person as Covid recedes. Meanwhile, remember Prophecy is super easy to use and free to try out.

Sign up for your free Account!