We at Prophecy believe that data engineers do the amazing work of enabling analytics and machine learning by providing them with high quality and up to date data. This is critical to every business and data engineers deserve the best tools.

Apache Spark is preferred for data processing since it is powerful, general purpose and open source. However, there are multiple challenges in succeeding with Spark:

• Spark has a high learning curve, and is not accessible to many of the users in an organization. Visual ETL developers have to learn to write Spark code - in a large organization this can be a few hundred users. Then there are data analysts, machine learning engineers, metadata architects who all need to access and process the same data.
• Code quality is the next issue. As users start building workflows (or moving them from legacy ETL products such as Informatica or AbInitio), they are learning Spark while writing lots of Spark code. This leads to low performing, non-standard and un-maintainable code mess. Then we've seen a push in many organizations to build the dreaded internal framework.
• Best practices take time to learn - development is just one piece of the puzzle. Teams want to move to agile best practices, and learning how to test, how to go from dev -> test -> production, how to schedule workflows in production - all of this takes time to learn and create discipline.

What is Prophecy?

Prophecy is a low-code product and you don't need to know Spark to get started. If you can use visual drag-and-drop and can write some SQL expressions - that's all you need.

Prophecy uniquely combines visual development & code development -you can edit the same workflow in either editor, however, you are always producing high-quality Apache Spark code on Git that is 100% open and runs everywhere. You can develop 10x more productively in Prophecy than any other way - with agile best practices - code, tests, continuous integration and continuous deployment.

Announcing Prophecy SaaS

We're releasing our SaaS product today and making it free for 14 days. It runs with Databricks Spark on AWS, Azure and GCP. We'll release support for other Spark providers over the next few months. For more users, you can purchase additional licenses.

Our existing Enterprise product - that has a stand alone install inside the customer network, runs with most on-premise and cloud Spark providers.

What's included in the SaaS product?

• Create new Spark workflows using visual drag-and-drop and SQL expressions
• Connect to your Spark cluster and interactively run the workflow
• Workflow code is in Scala or PySpark - connect to your Github/Bitbucket
• Create tests for workflows
• Schedule workflows using Apache Airflow

You can get setup in 3 minutes, and build your first workflow in 5 minutes.

Sign up for your account! Free trial for 14 days.

For detailed introduction to Prophecy SaaS, the following resources are available

• Getting Started Video Course
• Developer Documentation
• Community Slack Channel

We're excited to contribute toward democratizing data engineering on Spark, making it easier for smaller companies and smaller teams to succeed with an Enterprises grade product. Please use the product, and send us your feedback - feedback@prophecy.io.