Open source platform for the machine learning lifecycle
-
Updated
Jul 8, 2025 - Python
#
apache-spark
Apache Spark is an open source distributed general-purpose cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.
Here are 1,979 public repositories matching this topic...
Simple and Distributed Machine Learning
microsoft http opencv data-science machine-learning scala big-data ai spark apache-spark deep-learning azure ml pyspark lightgbm cognitive-services databricks synapse model-deployment onnx
-
Updated
Jul 7, 2025 - Scala
lakeFS - Data version control for your data lake | Git for data
go golang apache-spark aws-s3 google-cloud-storage data-engineering data-lake azure-storage data-version-control object-storage datalake hadoop-filesystem data-quality data-versioning azure-blob-storage apache-sparksql git-for-data lakefs datalakes
-
Updated
Jul 8, 2025 - Go
酷玩 Spark: Spark 源代码解析、Spark 类库等
-
Updated
May 18, 2022 - Scala
Interactive and Reactive Data Science using Scala and Spark.
-
Updated
May 16, 2023 - JavaScript
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
kubernetes spark apache-spark kubernetes-operator kubernetes-controller kubernetes-crd google-cloud-dataproc
-
Updated
Jul 8, 2025 - Go
BigDL: Distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray
python scala apache-spark pytorch keras-tensorflow bigdl distributed-deep-learning deep-neural-network analytics-zoo
-
Updated
Jun 17, 2025 - Jupyter Notebook
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
microsoft emr machine-learning streaming spark apache-spark csharp fsharp analytics dotnet azure bigdata spark-streaming dotnet-core dotnet-standard databricks spark-sql tpcds hdinsight tpch
-
Updated
Jul 4, 2025 - C#
Apache Spark docker image
-
Updated
Apr 21, 2023 - Shell
Feathr – A scalable, unified data and AI engineering platform for enterprise
data-science machine-learning apache-spark azure artificial-intelligence data-engineering feature-engineering data-quality mlops feature-store feature-management feature-marketplace feature-governance feature-metadata feature-platform
-
Updated
Apr 4, 2024 - Scala
A curated list of awesome Apache Spark packages and resources.
-
Updated
Oct 24, 2024 - Shell
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
-
Updated
Aug 16, 2021 - Java
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
mysql python postgres sql apache-spark sqlite postgresql challenges pyspark mysql-database data-analysis exercises tableau sql-queries pgadmin mysqlworkbench mysql-notes digital-music-store sql-data-analysis
-
Updated
Jul 18, 2022 - Jupyter Notebook
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework apache-airflow emr-cluster etl-pipeline etl-job data-engineering-pipeline airflow-dag goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
-
Updated
Jan 28, 2025 - Scala
PySpark + Scikit-learn = Sparkit-learn
-
Updated
Dec 31, 2020 - Python
(Deprecated) Scikit-learn integration package for Apache Spark
-
Updated
Dec 3, 2019 - Python
MapReduce, Spark, Java, and Scala for Data Algorithms Book
python java machine-learning scala apache-spark distributed-computing design-patterns pyspark mapreduce reducers partitioning hadoop-mapreduce distributed-algorithms mappers data-algorithms apache-hadoop
-
Updated
Oct 14, 2024 - Java
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
big-data spark apache-spark graphs pyspark networks dataframe dataframes network-motifs connected-components network-motif
-
Updated
Jul 4, 2025 - Scala
Created by Matei Zaharia
Released May 26, 2014
- Followers
- 428 followers
- Repository
- apache/spark
- Website
- github.com/topics/spark
- Wikipedia
- Wikipedia