Which Development Setup Should I Use to Learn Spark?

Published on Feb 6, 2024

That’s one essential choice to make and really is a preference of yours. However, there are some things to keep in mind when making your choice.

An image showing the title 'Which development setup should I choose to learn Spark?' alongside the logos of IntelliJ IDEA, Spark, Scala, Python and databricks.

Learn to set it up yourself

I’d suggest that you set up an IDE and the Spark dependencies locally to get used to a proper development environment. After all, being a Pro means to be proficient in using an IDE and knowing how to manage dependencies.

The easy path

On the other hand, you can use a platform like databricks for a smooth start. Everything is set up for you already, and you can start writing code right away. But keep in mind that you’ll have to learn how to include dependencies, package applications into jars, and deploy applications yourself in any case down the road.

My preference and suggestion for you

I personally don’t like to code in a browser. In the end, as developers we are editing text all day long, so we should be Pros in doing so. An IDE adds a vast number of very helpful features and we should start utilizing them. Therefore, I suggest everyone learning Spark to start with an IDE right away. Setting up the development environment is a great opportunity to learn and get used to finding information.

Go all-in to become a Pro — pick the environment, which makes you feel most comfortable and gives you the most control.