Databricks and End-to-End Processes Demo Links & Help Alexander Brown Databricks and End-to-End Processes Demo Links & Help

Alex Brown Senior Consultant, Neudesic @ I Wear Sports Coats For a Reason I have been asked multiple times since I moved here to Arizona recently what high school I attend and I was told wearing a sports coat makes me look at least 18 Bad Actuary Alex Brown I personally consider myself a bad actuary. I graduated from the University of Wisconsin-Eau Claire in 2016 with a degree in Actuarial Science, but didn’t know about the tests until after graduation Senior Consultant, Neudesic Credentials https://www.linkedin.com/in/alexander-brown-56736b90/ Helped lead the migration of on premise to cloud solutions for one of the largest natural goods company in the Midwest on the Azure platform @ Alexander.brown@Neudesic.com

Getting Started with Databricks Databricks Platform – Free Trial Unlimited clusters that can scale to any size Job scheduler to execute jobs for production pipelines Fully interactive notebook with collaboration, dashboards, REST APIs Advanced security, role-based access controls, and audit logs Single Sign On Support Integration with BI tools such as Tableau, Power BI, Qlik, and Looker 14-day full feature trial (excludes cloud charges) Community Edition Single cluster limited to 6GB and no worker nodes Basica notebook without collaboration Limited to 3 max users Public environment to share your work https://databricks.com/try-databricks

Databricks: Everything you need to know to get started

Databricks Documentation There are some great resources out there to get started with Databricks. Everything that I covered in my presentation can be found or rebuilt through this Databricks documentation. These documents have code that can be copied and pasted right into a notebook you are working with. Azure Databricks Other Cloud Platforms Databricks https://docs.azuredatabricks.net/ https://docs.databricks.com/

Databricks Connections to Datasets Azure Blob Storage (Scala) Azure Data Lake Storage (Python) configs = {"dfs.adls.oauth2.access.token.provider.type": "ClientCredential", "dfs.adls.oauth2.client.id": "<your-service-client-id>", "dfs.adls.oauth2.credential": dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>"), "dfs.adls.oauth2.refresh.url": "https://login.microsoftonline.com/<your-directory-id>/oauth2/token"} # Optionally, you can add <your-directory-name> to the source URI of your mount point. dbutils.fs.mount( source = "adl://<your-data-lake-store-account-name>.azuredatalakestore.net/<your-directory-name>", mount_point = "/mnt/<mount-name>", extra_configs = configs) dbutils.fs.mount( source = "wasbs://<your-container-name>@<your-storage-account-name>.blob.core.windows.net/<your-directory-name>", mountPoint = "/mnt/<mount-name>", Azure Blob Storage (Python) dbutils.fs.mount( source = "wasbs://<your-container-name>@<your-storage-account-name>.blob.core.windows.net", mount_point = "/mnt/<mount-name>", extra_configs = {"<conf-key>":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})

Example Notebooks

Downloadable Notebooks to get started These links take you directly to premade notebooks that you can import into your databricks instance and modify to fit your dataset Complex and Nested Data Notebook https://docs.azuredatabricks.net/_static/notebooks/complex-nested-structured.html Aggregators Notebook https://docs.azuredatabricks.net/_static/notebooks/dataset-aggregator.html Structured Streaming Notebook https://docs.azuredatabricks.net/_static/notebooks/structured-streaming-python.html

Formal Databricks Trainings

Databricks Academy Here is the official place you can go to find everything from ‘Self-Paced Trainings’ to ‘Public Trainings’ and ‘Certifications’ in Databricks. Each course comes with a plethora of notebooks and videos to help get you up and off the ground in Databricks Topics that are covered include: Getting Started with Apache Spark DataFrames Databricks Deltas Structured Streaming Getting Started with Apache Spark SQL Introduction to Data Science and Machine Learning ETL Part 1: Data Extraction Tuning, Troubleshooting and Best practices for Databricks ETL Part 2: Transformations and Loads ETL Part 3: Production https://academy.databricks.com/

If you have any other questions at all, please feel free to reach out to me at any time!

About Neudesic DNA of a Successful O365 Migration What that experience has taught us…

Founded in 2002 Privately held firm headquartered in Irvine, CA Deep Technical Partnerships Global Reach

Global Reach 12 4 United States India Neudesic has 12 locations in the US and 4 India based development centers that offers a broad range of strategic and technology services 4 India

Creative Solutions WHAT WE DO? Consulting Services Managed Services Leverages a global team of consultants to deliver a unique blend of digital business strategy and trusted technical expertise to accelerate enterprise digital transformation. Creative Solutions Managed Services Neudesic’s Managed Cloud and Application Support Services help organizations establish more efficient, stable and scalable IT operations through 24x7 technical support, performance monitoring, workload right-sizing and more.  Products Neudesic’s software products are designed to fill specific business needs quickly and cost effectively, offering powerful capabilities that help your organization become more efficient and productive.

Questions?