Technologist

Work deeply

Posted by Santh

We are taught from childhood that “no pain, no gain”, emphasizing the importance of spending time and effort to achieve results. Once we grow up, we encounter self-help books that teach us to pivot towards working “smart”, not just “hard” and present their own theories and practices to make their point. All of them are based on the author’s personal experiences and beliefs, and we can benefit by learning from them and customizing for ourselves. One such book that stuck a chord with me is “Deep Work” by Cal Newport. Just as I finished the chapter on Rule #1 – Work Deeply, I was able to connect my learnings to a challenge my son wanted help with and immediately wrote this blog post with my thoughts.

Let me start with references from the book, followed by my inferences that will explain the picture above.

  • Roy Baumeister on willpower: You have a finite amount of willpower that becomes depleted as you use it.
  • Ritualize: To make the most out of your deep work sessions, build rituals of the same level of strictness and idiosyncrasy. These rituals will minimize the friction in transition to depth, allowing us to go deep more easily and stay in that state longer.
  • 4DX: The 4 disciplines of execution (abbreviated 4DX) is based on the fundamental premise that execution is more difficult than strategizing. It helps address the gap between what needs to be done (strategy) and how to do it (execution):
    • Discipline #1: Focus on the Wildly Important – Ironically, the more you try to do by pushing too hard, the less you accomplish. So, execution should be aimed at a small number of “wildly important goals”.
    • Discipline #2: Act on the Lead Measures – There are two types of measures for success – lag measures and lead measures. Lag measures describe the thing you are ultimately trying to improve while lead measures focus on the new behaviors that will drive success on the lag measures. The problem with lag measures is that they come too late to change your behavior. So, start with acting on lead measures.
    • Discipline #3: Keep a Compelling Scoreboard – An always visible scoreboard creates a sense of competition that drives us to focus on these measures, even when other demands vie for our attention. It also provides a reinforcing source of motivation. Finally, it allows us to recalibrate expectations as required to achieve what is wildly important (like an agile burndown chart will help the team recalibrate efforts required to meet sprint goals).
    • Discipline #4: Create a Cadence of Accountability – A periodic review of scoreboard helps us to review progress towards lag measures and pivot as required in case the progress does not really converge towards producing the ultimate results expected.
  • Be Lazy: Tim Kreider in his “The Busy Trap” blog says – “I am not busy. I am the laziest ambitious person I know” and goes on to explain “Idleness is not just a vacation, an indulgence or a vice; it is as indispensable to the brain as vitamin D is to the body, and deprived of it we suffer a mental affliction as disfiguring as rickets. The space and quiet that idleness provides is a necessary condition for standing back from life and seeing it whole, for making unexpected connections and waiting for the wild summer lightning strikes of inspiration — it is, paradoxically, necessary to getting any work done“. So, periodic shutdown from work will enhance our ability to produce valuable output due to the following reasons:
    • Reason #1: Downtime aids insight – Unconscious Thought Theory (UTT) posits that the unconscious mind is capable of performing tasks outside of one’s awareness, and that unconscious thought (UT) is better at solving complex tasks. The implication of this line of research is that providing your conscious brain time to rest enables your unconscious mind to take a shift sorting through your most complex professional challenges.
    • Reason #2: Downtime helps recharge the energy needed to work deeply – Attention Restoration Theory (ART) asserts that people can concentrate better after spending time in nature, or even looking at scenes of nature. The core mechanism of this theory is the idea that you can restore your ability to direct your attention if you give this activity a rest.
    • Reason #3: The work that regular downtime replaces is usually not that important – Working in information technology field for US multinationals from India means ones evening is core work time, attending meetings and discussions during the limited time zone “overlap” available. So, I replaced “evening downtime” with “regular downtime”, which for me is typically a few hours every morning when I go for my morning run and spend time with family. The point is that your capacity for deep work in a given day is limited and by setting aside some time for yourself to relax and reenergize, you are missing out on much of importance.

Putting it all together: Going back to my son’s challenge, he got an unexpected one week holiday as his school had to reschedule classes due to second covid-19 wave. He was initially happy with this change as he could spend time as he wished (usually video games) for an extra week but the willpower he had originally corralled to excel academically had gone unutilized resulting in his initial euphoria eventually turning into guilt.

As humans, we are usually tempted to engage in shallow activities that are easily enjoyable (like playing video games or watching our favorite TV show) instead of deep work (like studying or writing this blog). But we are also ambitious and aspirational with a need for achievement to make our life purposeful. Willpower helps us handle this conflict by motivating us to engage in deep work required for purposeful activities that are usually difficult. But our willpower is finite that depletes with use and needs to be recharged for sustenance. Overutilizing willpower will lead to exhaustion, underutilizing it will lead to guilt and smartly utilizing will produce great results for our wildly important goals. Smartly utilizing our willpower means:

  • Using rituals to minimize the friction in transition to deep work and amplifying the benefits of finite willpower
  • Using the 4 disciplines of execution to measure and progress towards our vision and goals
  • Provide ourselves sufficient rest to recharge the energy needed to work deeply

With years of trial and error, I was already practicing some of these disciplines and this book provides additional structure that should make me more effective!

Hackers: Heroes of the Computer Revolution

Posted by Santh

As a software engineer for more than 20 years, I have seen how computing has evolved since the beginning of internet age. A couple of months back, I heard one of my seniors passionately speak about his computing experience from the 80s that evoked my curiosity on early computer revolution. My search for an authority on this topic ended with the best-selling book by Steven Levy about hacker culture published in 1984 – Hackers: Heroes of the Computer Revolution.

The intriguing element of the book is “Hacker Ethic” – in Steven’s words, it was a philosophy of sharing, openness, decentralization and getting your hands on machines at any cost to improve the machines and to improve the world. He narrates computer evolution from mid 1950s till 1984 from a hacker perspective, covering people and machines that might not be well known to people from the internet age.

For software engineers of current millennium who take CPU speed at gigahertz and memory at terabytes for granted in 64-bit machines, it is unfathomable that predecessors from this hacker era created wonders with a tiny fraction of these resources in 8-bit machines. Assembly language programmers were a celebrated lot and they innovated by hacking primitive microprocessor-based computers with machine language!

Steven explains how “Hacker Ethic” evolved over three decades starting from the closed community of early mainframe hackers on time share terminals at MIT, to the open community of self-made hardware hackers out of their garages at Silicon Valley, finally paving the way for game hackers. This forms the three parts of the book that I have summarized in the picture below.

I have watched several videos on personal computing revolution and particularly enjoyed Triumph of the Nerds. But reading a book is always a unique experience as it gives an opportunity for imagination. As I read this book, I felt as if I was sitting beside the hackers and watching them code. While computing has changed a lot over generations, one aspect has remained just the same – hackers always push computers to the limits and their hunger for more has driven the industry forward!

Blockchain

Posted by Santh

Blockchain is a distributed ledger that is used to record transactions of a digital asset such as bitcoin. It has been one of the most talked about technologies in the 2010s, continuing to be popular for its tamper-proof design even after bitcoin became notorious for its price vagaries.

Blockchain became a mainstream term since 2014 with applications across several industries being explored – to store data about property exchanges, stops in a supply chain, and even votes for a candidate. I have been following blockchain for many years and finally got some hands-on experience on Ethereum while completing a Pluralsight course.

In this blog, I have listed key blockchain terms / concepts with brief description and also summarised the development environment used:

  • Blockchain is a chain of digital data blocks, with each one containing:
    • Information about transactions such as date, time, amount, price, etc.
    • Unique code called a “hash” that distinguishes a block from another. This is generated by a hashing process.
  • Hashing: an algorithm performed on data to produce an output that can be used to verify that data is not modified, tampered with or corrupted:
    • Length of output always the same.
    • One-way: the hash cannot be converted back into the original key.
    • Digital fingerprint that allows verifying consistency.
  • Obfuscation and Encryption provide data security to blockchain.
  • Key features of Blockchain: Immutable, decentralized, verifiable, increased capacity, better security, faster settlement.
  • Blockchain can be public (like Ethereum) or private (like R3 Corda).
  • Top implementations: Ethereum, Hyperledger Fabric, Ripple, Quorum, R3 Corda.
  • Distributed Applications (DAPPs) architecture using Ethereum: Blockchain encapsulates shared data and logic related to transactions while client is responsible for interface, user credentials and private data.
  • Payment for transactions is made through “gas”.
  • Transactions contain information on recipient, signature, value, gasprice, startgas and message.
  • A consensus mechanism is required to confirm transactions that take place on a blockchain without the need for a third party. Proof of Work and Proof of Stake are the two models currently available to achieve this.
  • Proof of Work is based on cryptography, hence digital coins like Bitcoin and Ethereum are called cryptocurrencies. Cryptography uses mathematical equations that are so difficult that only powerful computers can solve them. No equation is ever the same, meaning that once it is solved, the network knows that the transaction is authentic. Although Proof of Work is an amazing invention, it needs significant amounts of electricity and it is also very limited in the number of transactions it can process at the same time.
  • With Proof of Stake, miners can mine or validate block transactions based on the amount of Bitcoin held by them. This way, instead of utilizing energy to answer Proof of Work puzzles, a miner is limited to mining a percentage of transactions that is reflective of ownership stake.

Development environment:

Ethereum uses a programming language called Solidity, which is an object-oriented language for writing smart contracts. Solidity plugins are available for popular IDEs. I used the following setup:

  • IDE – Eclipse with YAKINDU-Solidity Tools / Visual Studio Code
  • Node.js Package Manager – npm
  • Windows Package Manager – chocolatey (installed from Powershell as administrator)
  • Node.js Windows Build Tool
  • Local ethereum test server and emulator – Ganache / Test RPC
  • Dev & Testing framework – Truffle Suite
  • Crypto wallet & gateway – Metamask
  • My practice code – https://github.com/gsanth/experiment/tree/master/ethereum_experiment

To summarize, Blockchain’s potential as a decentralized form of record-keeping is enormous. From greater user privacy and heightened security to lower processing fees and fewer errors, blockchain technology will continue to see applications across several industries. However, it is easy to get carried away by blockchain’s apparent potential and get into technology overkill. So, it is important to understand the pros and cons of blockchain, and ensure we leverage it for the right use cases.

ProsCons
Decentralized (difficult to tamper)Technology cost for mining
Improved accuracy (no manual effort)Low transactions throughput
Reduced cost (no third party charge)Poor reputation (illicit activities)
Transparent Technology

AWS Certified Solution Architect – Associate

Posted by Santh

In today’s rapidly changing technology landscape, staying relevant as a software engineer hinges on one’s ability to continuously learn and master emerging technologies. To this end, structured technology courses and certifications help lay a solid foundation that can lead to eventual mastery with real life hands-on experience. I target one technology certification every year, finished Stanford Machine Learning Certification last year and decided to pursue a cloud certification in 2020.

My goal was to complete a comprehensive cloud learning path and had to choose between Google Cloud Platform, Microsoft Azure and Amazon Web Services. All of them offer similar services, so learning one will automatically build understanding of the others. I decided to pursue AWS certification as it is the clear industry leader with almost one third of global cloud market share. There are a dozen AWS certifications available and my choice was Solution Architect Associate, which is the most popular one as it covers the entire AWS offering.

Magic Quadrant for Cloud Infrastructure as a Service, Worldwide (2020)

Given my existing familiarity and understanding of cloud computing, I had a bit of a head start and was able to successfully complete my certification in about a month. This blog summarizes my experience and learnings through this journey.

AWS certifications require serious preparation and usually starts with identifying a MOOC platform for access to learning material. I had used Coursera for ML certification last year as it came bundled with a Stanford certificate. As this certification is directly provided by AWS, I found Pluralsight to be more prolific and flexible for my needs.

This was my first time using Pluralsight and was thoroughly impressed with the learning experience. I typically used youtube for quick reference of new technology topics but the video lectures on Pluralsight take learning experience to a completely different level. The one minor drawback is hands-on practice. While Coursera offered hands-on exercises, I had to independently create AWS Free Tier account to practice through AWS Management Console and CLI along with Pluralsight courses. Being a techie, this was perfectly fine with me and enjoyed this experience as well.

After completing the first Pluralsight learning path and some hands-on practice, I attempted the sample questions on AWS site and felt my preparation was insufficient. So, I finished a few more relevant Pluralsight courses, digital training from AWS site, read through whitepapers and FAQs before attempting practice exams at Kaplan (offered free with Pluralsight).

To summarize, I leveraged the following resources:

AWS is vast and a solution architect is expected to understand all its offerings. So, developing deep understanding across foundational service domains of compute, networking, storage and databases, along with other key topics like security, analytics, app integration, containerization, cloud native solutions, etc made it an awesome learning experience.

AWS Services: Knowledge Areas for Solution Architect – Associate Certification are in bold

After about 3 weeks of preparation, I scheduled my exam at https://www.aws.training/Certification. AWS certification exams had to be taken at an exam center in a controlled environment but with COVID situation, proctored online exam option was also available. I chose Pearson VUE exam and scheduled for a Sunday morning slot. All the caveats called out with proctored online exam is a bit scary, particularly if internet or power connection is lost in the middle of the 130 minutes exam (quite common in India during monsoons). Fortunately, my internet connection remained stable despite heavy rain during the exam and allowed me to pass the exam. So, here I am – AWS Certified Solution Architect – Associate!

Zero To One

Posted by Santh

Information and Communication Technology has been the primary driver of innovation and engineering advances during the last four decades. The dominance is to such an extent that the term technology today refers to these fields, though there are several other engineering disciplines that continue to exist! I am fortunate to have started my professional career in information technology and am enjoying being part of it for more than twenty years. Unlike any other technology domain that emerged as new hotspot since the industrial revolution 200 years back, the entry barrier for information technology is extremely low that allowed passionate technologists to launch their enterprises from garages. And combine this with the success of venture capital industry from 1970s, start-ups have been the primary source of innovation in the technology industry since the advent of personal computing with Intel 8080 processor.

While I have not worked for start-ups so far, I strongly believe that start-up lessons can help enterprises improve their ability to succeed while creating new products. In this blog, I will share my notes from “Zero to One”, one of the best books on start-up philosophy written by Peter Thiel, a successful entrepreneur himself.

“Zero to One” has an explanation for most of the new technology trends during the last twenty years coming from start-ups. From the Founding Fathers in politics to the Royal Society in science to Fairchild Semiconductor’s “traitorous eight” in business, small groups of people bound together by a sense of mission have changed the world for the better. The easiest explanation for this is negative: it’s hard to develop new things in big organizations, and it’s even harder to do it by yourself. Bureaucratic hierarchies move slowly, and entrenched interests shy away from risk. In the most dysfunctional organizations, signaling that work is being done becomes a better strategy for career advancement than actually doing work. At the other extreme, a lone genius might create a classic work of art or literature, but could never create an entire industry. Startups operate on the principle that you need to work with other people to get stuff done, but you also need to stay small enough so that you actually can. Clayton Christensen has provided similar explanation in his book “The Innovator’s Dilemma” through the concept of “disruptive innovation” and how most companies miss out on new waves of innovation. Does it mean big companies cannot develop new things? They can, as long as they enable the teams focused on building new things to operate like a start-up without burdening them with bureaucracy and creativity sapping processes.

Peter Thiel suggests that we must abandon the following four dogmas created after dot-com crash that still guide start-up business thinking today:

  1. Make incremental advances: Small increments using agile methods has far better chances of success today than waterfall world.
  2. Stay lean and flexible: Avoid massive plan and execute model. Instead, iterative development helps stay nimble and deliver through meaningful experimentation.
  3. Improve on competition: New things are invariably improvements on recognizable products already offered by successful competitors.
  4. Focus on product, not sales: Technology is primarily about product development, not distribution.

But having seen a number of projects in large enterprises, I would say that sticking to these principles by default and making exceptions only for compelling reasons is better.

When it comes to creating new software products for a market, don’t build an undifferentiated commodity business but one that’s so good at what it does that no other product can offer a close substitute. Google is a good example of a company that went from zero to one, after distancing from Microsoft and Yahoo almost 20 years back and became a monopoly. While monopolies sound draconian, the companies that get to the top create monopoly based on a unique value proposition they offer in their markets. So, don’t build new things unless there is a desire and plan to capture significant market share, if not monopoly. Every monopoly is unique, but they usually share some combination of the following characteristics:

  • Proprietary technology
  • Network effects
  • Economies of scale
  • Branding

Another interesting observation is around secrets: most people act as if there were no secrets left behind. With advances in Maths, Science and Technology, we know a lot more about the universe than previous generations but there are still numerous unknowns yet to be conquered. It helps to be conscious of the four social trends that have conspired to root out beliefs in secrets:

  1. Incrementalism: From an early age, we are taught that the right way to do things is to proceed one very small step at a time, day by day, grade by grade. However, unlocking secrets requires us to be brutally focused on the ultimate goal rather than staying satisfied with interim milestones.
  2. Risk Aversion: People are scared of secrets because they are scared of being wrong. If your goal is to never make a mistake in life, you shouldn’t look for secrets. And remember, you can’t create something new and impactful without making mistakes.
  3. Complacency: Getting into a top institute or corporation is viewed as an achievement in itself with nothing more to worry and you are set for life. This leads to complacency and no more fire to unlock secrets.
  4. Flatness: As globalization advances, people perceive the world as one homogenous, highly competitive marketplace and an assumption that someone else would have already found out secrets.

To summarise, when a start-up or an enterprise decides to create a new product, it should resist the temptation to go for a commodity one. It should be a product with clear differentiation that will help create a monopoly or significant market share at a minimum. This can happen only through hard work and dedication to unlock some secrets.

There is a lot more learnings from the book but I have only mentioned the key ones that can help us introspect and stay focused on our goals to create new products.

AI / ML in enterprises: Technology Platform

Posted by Santh

As an organization embarks on leveraging AI / ML at enterprise scale, it is important to establish a flexible technology platform that caters well to different needs of data scientists and the engineering teams supporting them. Technology platform here includes hardware architecture and software framework that allows ML algorithms to run at scale.

Before getting into software stack directly used by data scientists, lets understand the hardware and software components required to enable machine learning.

  • Hardware layer: x86 based servers (typically intel) with acceleration using GPUs (typically nvidia)
  • Operating Systems: Linux (typically redhat)
  • Enterprise Data Lake (EDL): Hadoop based repository like Cloudera or MapR, along with supporting stacks for data processing:
    • Batch ingestion & processing: example – Apache Spark
    • Stream ingestion & processing: example – Apache Spark Streaming
    • Serving: example – Apache Drill
    • Search & browsing: example – Splunk

Once necessary hardware and data platforms setup, the focus is on providing an effective end user computing experience to data scientists:

  • Notebook framework for data manipulation and visualization: like Jupyter Notebooks or Apache Zeppelin, which support most commonly used programming languages for ML like Python and R.
  • Data collection & visualization: like Elastic Stack and Tableau.
  • An integrated application and data-optimized platform like IBM Spectrum makes it simple for enterprises by addressing all the needs listed above (components include enterprise grid orchestrator along with a Notebook framework and Elastic Stack).
  • Machine Learning platforms: specialized platforms like DataRobot, H2O, etc simplifies ML development lifecycle and lets data scientists and engineering focus on creating business value.

There are numerous other popular platforms like Tensorflow, Anaconda, RStudio and evergreen ones like IBM SPSS, MATLAB. Given the number of options available, particularly open source ones, an attempt to create a comprehensive list will be difficult. My objective is to capture the high-level components required as part of Technology platform for an enterprise to get started with AI / ML development.

AI / ML in enterprises: Lifecycle & Departments

Posted by Santh

Many start-ups are based on AI / ML competence and require this expertise across the organization. In established enterprises, AI / ML is fast becoming pervasive across the organization given the disruption from start-ups and customer expectations. Depending on the size and level of regulation in their respective industries, machine learning activities might be embedded within existing technology teams or dedicated “horizontal” teams might be responsible for them.

ML activities that people readily recognize are the ones performed by data scientists, data engineers and the like. However, there are other business and technology teams that are essential to enable ML development. Given the potential bias and ethics implications with business decisions made by AI / ML, governance to ensure risk and regulatory compliance will be required too. In this blog, I will cover AI / ML lifecycle along with the functions and departments in an enterprise that are critical for successful ML adoption.

AI Model inventory: There is an increasing regulatory expectation that organizations should be aware of all AI / ML models used across the enterprise to effectively manage risks. This McKinsey article provides an overview of risk management expected in banking industry. As an organization embarks on creating AI / ML development process, a good starting point is to define what constitutes an AI model to ensure common understanding across the organization and create a comprehensive inventory.

Intake and prioritization: To avoid indiscriminate and inappropriate AI / ML development and use, it is important that any such development go through an intake process that evaluates risk, regulatory considerations and return on investment. It is a good practice to define certain org wide expectations and preferable to federate the responsibility for agility.

Data Management: Once an AI Model is approved for development, business and technology teams work together to identify required data, secure them from different data sources across the organization and convert them into feature set for model development.

  • Data Administrators manage various data sources, which typically are data lake (Apache Hadoop implementation) or warehouses (like Teradata) or RDBMS (like SQL Server / Oracle).
  • Data Engineers help with data preparation, wrangling, munging and feature engineering using a variety of tools (like Talend) and makes feature set available for model development.

Model Development: Data Scientists use AI / ML platforms (like Anaconda, H2O, jupyter) to develop AI models. While model development is federated in most enterprises, AI / ML governance requires them to adhere to defined risk and regulatory guidelines.

Model Validation: An Enterprise Risk team usually validates models before production use, particularly for ones that are external facing and deemed high risk.

Deployment & Monitoring: Technology team packages approved models with necessary controls and integrated into appropriate business systems and monitors for stability and resilience.

Enterprises strive to automate the entire lifecycle so that focus can be on adding business value effectively and efficiently. Open Source platforms like AirFlow, MLFlow and Kubeflow help automate orchestration and provide seamless end to end integration for all teams across AI / ML lifecycle.

AI / ML in enterprises: Challenges

Posted by Santh

Organizations need to keep up with the times for long term sustenance and with AI / ML becoming pervasive across business domains, every firm nowadays has teams trying to leverage machine learning algorithms to stay competitive.

In this blog, I will cover the top five challenges that they encounter after initial euphoria with proof of concepts (POC) and pilots.

  1. Lack of understanding: AI / ML has the potential to transform technology and business processes across the organization and create new revenue streams, mitigate risks or save costs. However, AI / ML is not a substitute for subject matter expertise. A discussion among novices will throw up a million possibilities and ML can appear to be an appropriate solution to all world problems. While machine learning is based on the ability of machine to learn by themselves, training the algorithms with appropriate data is an important aspect that can be done only by experts. To generate meaningful results, data scientists need to work in unison with business and technology professionals. Data scientists bring deep understanding of ML algorithms, business professionals identify meaningful features and data engineers help secure data from different sources that will eventually become feature set. As one can see, good understanding of ML across the organization is important to identify the right problems to solve and lack of it is the most important challenge that prevents enterprises from deriving benefit despite investment.
  2. Lack of IT infrastructure: As I had mentioned in my original ML post, machine learning came to prominence due to significant information technology advances in processing power and data storage. Enterprises can acquire the required compute power through cloud providers and many organizations also choose to build their own parallel processing infrastructure. The decision to leverage cloud vs. building internal infrastructure is based on a number of factors like regulations, scale and most importantly cost considerations. Either ways, without this investment, ML programs will not go too far. Some organizations invest in requisite hardware but fail to provide the software and database platforms required for data scientists and technologists to leverage this infrastructure. Most machine and deep learning platforms and tools used for development are open source. However, this open source cost advantage is offset by the numerous options available for ML development and there is no one size fits all solution. To summarize, the second challenge is to create powerful IT infrastructure required for ML development and deployment.
  3. Lack of Data: With good understanding and infrastructure, this challenge should be addressed but data is foundational for ML and I have listed this out separately to call out the nuances. Data should be available in sufficient quantity and with good quality for meaningful results. Data preparation is an important step – wrangling, munging, feature scaling, mean normalization, labeling and creating an appropriate feature set are essential disciplines. It is a challenge to identify problems that have requisite data at scale and prepare this data for machine learning algorithms to work on.
  4. Lack of talent: Going by the number of machine learning projects that fail to meet their purpose, the ability of existing teams across enterprises is questionable. Any technology is only as good as people working on them. A few technologies have managed to simplify the work expected from programmers (just drag and drop or configuration driven). However, machine learning still requires deep math skills and thorough understanding of algorithms. So, finding suitable talent is particularly difficult.
  5. Regulations & policies: In a diverse world with myriad regional nuances, decisions made by machines tend to undergo a lot more scrutiny than ones made by humans. Our societies are still in paranoia of machines taking over humans and governments all over the world have regulations that require proof of decisions made by machines to be fair and without bias. This challenge is made more complex by interpreters of regulations inside an enterprise who place unnecessary controls that might not address the regulation but impede ML development. So, it is important for policies to address regulatory concerns without derailing ML development.

Finally, enterprises are riddled with politics and it is possible to address all above challenges only when business, technology and other supporting functions work together seamlessly. Start-ups and technology organizations that are relatively new keep it simple and are more adept at solving these challenges. Large enterprises that have added layers of internal complexity over the years naturally find it more difficult to overcome differences and solve the same challenges.

AI / ML in enterprises: Relevance

Posted by Santh

Let’s explore two aspects that will provide insights into AI / ML relevance within technology and across enterprises. First, roles in technology that need to work on machine learning algorithms and second, areas within an enterprise that will benefit the most from AI / ML.

It is an incorrect assumption that all software engineers will work only on ML algorithms in future and demand for other skills will plummet. In fact, majority of current software engineering roles that do not require machine learning expertise will continue to exist in the future.

Software engineering functions that DO NOT require machine learning expertise: UI / UX development, interface / API development, rule based programming and several other client and server side components that requires structured or object oriented programming. In addition, there are others like database development and SDLC functions that are required for AI / ML technology lifecycle but don’t require deep machine learning knowledge. So, this leaves only data / feature engineering, data science and model deployment teams that absolutely require machine learning expertise. However, these are rapidly growing areas and demand for experts will continue to outpace many other areas.

Where can we leverage ML? Any use case where historical data can be used for making decisions but this data is so extensive that it is practically impossible for a human to comprehensively analyze the data and generate holistic insights will be a candidate for ML. The ML approach will be to leverage human subject matter expertise to source relevant data, determine the right data elements (features), select appropriate ML model and train the model to make predictions and propose decisions. A few examples:

  • Sales & Marketing: Use data around customer behavior and make recommendations. We see this all the time from Amazon, You Tube, Netflix and other technology platforms.
  • IT Operations: Use a variety of features to predict potential failures or outages and alert users / ops.
  • Customer Service: Chatbots that use natural language processing to answer user queries.
  • Intelligent Process Automation: Eliminate manual operations thereby optimizing labor costs and reducing operational risk.
  • Cyber Security: Detect malicious activity and stop attacks.
  • Anomaly detection: Every business domain needs to beware of anomalies and detecting them will reduce losses or accidents. It could be detecting defaults or money laundering or fraud for banks, detecting leak in a chemical plant, detecting a traffic violator, etc.

Every enterprise, large or small, is likely to have AI / ML opportunities that will result in bottom line benefits. In the next part, I will cover the typical challenges an enterprise faces during adoption.