As an organization embarks on leveraging AI / ML at enterprise scale, it is important to establish a flexible technology platform that caters well to different needs of data scientists and the engineering teams supporting them. Technology platform here includes hardware architecture and software framework that allows ML algorithms to run at scale.
Before getting into software stack directly used by data scientists, lets understand the hardware and software components required to enable machine learning.
- Hardware layer: x86 based servers (typically intel) with acceleration using GPUs (typically nvidia)
- Operating Systems: Linux (typically redhat)
- Enterprise Data Lake (EDL): Hadoop based repository like Cloudera or MapR, along with supporting stacks for data processing:
- Batch ingestion & processing: example – Apache Spark
- Stream ingestion & processing: example – Apache Spark Streaming
- Serving: example – Apache Drill
- Search & browsing: example – Splunk
Once necessary hardware and data platforms setup, the focus is on providing an effective end user computing experience to data scientists:
- Notebook framework for data manipulation and visualization: like Jupyter Notebooks or Apache Zeppelin, which support most commonly used programming languages for ML like Python and R.
- Data collection & visualization: like Elastic Stack and Tableau.
- An integrated application and data-optimized platform like IBM Spectrum makes it simple for enterprises by addressing all the needs listed above (components include enterprise grid orchestrator along with a Notebook framework and Elastic Stack).
- Machine Learning platforms: specialized platforms like DataRobot, H2O, etc simplifies ML development lifecycle and lets data scientists and engineering focus on creating business value.
There are numerous other popular platforms like Tensorflow, Anaconda, RStudio and evergreen ones like IBM SPSS, MATLAB. Given the number of options available, particularly open source ones, an attempt to create a comprehensive list will be difficult. My objective is to capture the high-level components required as part of Technology platform for an enterprise to get started with AI / ML development.