• Understanding the impact of the machine learning/AI component that is built into most of the enterprise products and tools and its role in the implementation of the solution.
• What are the most important aspects that organizations needs to consider while planning the implementation of the AI-powered product?

• Organizations are faced with multiple challenges trying to adopt an AI solution. Challenges include data issues, ethics and compliance considerations, business process challenges, and misaligned leadership goals.
• When choosing the right product to meet business needs, organizations need to know what questions to ask vendors to ensure they fully understand the implications of buying an AI/ML product.

Info-Tech’s approach includes a framework that will guide organizations through the process of the Off-the-Shelf AI product selection.

To guarantee success of the Off-the-Shelf AI implementation and deliver value, organization should start with clear definition of the business case and an understanding of data.

Other steps include:

• Knowing what questions to ask vendors to evaluate AI-powered products.
• Measuring the impact of the project on your business and IT processes.
• Assessing impact on the organization and ensure team readiness.

DIY Toolkit

Guided Implementation

Workshop

Consulting

92.1%

91.7%

26.0%

• Data issues
• Business process challenges
• Implementation challenges and skill shortages
• Costs of tools and development
• Misaligned leadership goals
• Measuring and proving business value
• Legal and regulatory risks
• Cybersecurity
• Ethics
(Source: CIO, 2019)

“67% of companies are currently using machine learning, and 97% are using or planning to use it in the next year.” (Source: Deloitte, 2020)

Machine learning systems learn from experience and without explicit instructions. They learn patterns from data then analyze and make predictions based on past behavior and the patterns learned.

Artificial intelligence is a combination of technologies and can include machine learning. AI systems perform tasks mimicking human intelligence such as learning from experience and problem solving. Most importantly, AI is making its own decisions without human intervention.

The AI system can make assumptions, test these assumptions, and learn from the results.

“Machine learning is the study of computer algorithms that improve automatically through experience.” (Tom Mitchell, 1997)

“At its simplest form, artificial intelligence is a field, which combines computer science and robust datasets, to enable problem-solving.” (IBM, “What is artificial intelligence?”)

• AI/ML capabilities built into the product and might require training as part of the implementation.

• Off-the-Shelf ML/AI Models, pre-built, pre-trained, and pre-optimized for a particular task. For example, language models or image recognition models that can be used to speed up and simplify ML/AI systems development.

• Data Management Tools: Informatica AI-Powered Data Management Platform
• BI/Visualization: Einstein Discovery in Tableau
• Scientific Computation: Mathematica (by Wolfram Research)
• Most of the Chatbots
• AI-Powered Search Engines: SearchUnify Cognitive Platform, Coveo Predictive Search

• Cloud providers, AI Services:
  • Azure Cognitive services,
  • Amazon Rekognition (video and image analysis),
  • GCP Video Intelligence
• Open Source:
  • Ludwig (developed by Uber AI team), a “Code-free, Deep Learning Toolbox”

Why do you need AI in your toolset?

Clearly defined problem statement and business requirements for the tool or a model will help you select the right solution that will deliver business value even if it does not have all the latest bells and whistles.

• Artificial Intelligence
• Data Strategy

Do you know the data required for implementation?

Expected business outcome defines data requirements for implementation. Do you have the right data required to train and run the model?

• Data Governance
• Machine Bias

Is your organization ready for AI?

New skills and expertise are required through all phases of the implementation: design, build, deployment, support, and maintenance, as well as post-production support, scaling, and adoption.

New tool or model will impact your cloud and integration strategy. It will have to integrate with the existing infrastructure, in the cloud or on prem.

• Create an Architecture for AI

What questions do you need to ask when choosing the solution?

Do you know what model powers the AI tool? What data was used to train the tool and what data is required to run it? Ask the right questions.

• Rapid Application Selection Framework
• SoftwareReviews

Are you measuring impact on your processes?

Business processes need to be defined or updated to incorporate the output of the tool back into the business processes to deliver value.

IT governance and support processes need to accommodate the new AI-powered tool.

Realize and measure business value of your AI investment

Do you have a clear understanding of the value that AI will bring to your organization?Optimization?Increased revenue?Operational efficiency?

• Define a business problem that can be solved with either an AI-powered tool or an AI/ML pre-built model that will become part of the solution.
• Define expectations and assumptions around the value that AI can bring.
• Document business requirements for the tool or model.
• Define the scope for a prototype or POC.

• Define data requirements.
• Define data required for implementation.
• Determine if the required data can be acquired or captured/generated.
• Document internal and external sources of data.
• Validate data quality (define requirements and criteria for data quality).
• Define where and how the data is stored and will be stored. Does it have to be moved or consolidated?

• Define all stakeholders involved in the implementation and support.
• Define skills and expertise required through all phases of the implementation: design, build, deployment, support, and maintenance.
• Define skills and expertise required to grow AI practice and achieve the next level of adoption, scaling, and development of the tool or model POC.

• Define infrastructure requirements for either Cloud, Software-as-a-Service, or on-prem deployment of a tool or model.
• Define how the tool is integrated with existing systems and into existing infrastructure.
• Determine the cost to deploy and run the tool/model.

• Define processes that need to be updated to accommodate new functionality.
• Define how the outcome of the tool or a model (e.g. predictions) are incorporated back into the business processes.
• Define new business and IT processes that need to be defined around the tool (e.g. chatbot maintenance; analysis of the data generated by the tool).

• Define the business metrics and KPIs to measure success of the implementation against.

• Determine if there are requirements for a specific data format required for the tool or a model.
• Determine if there is a need to classify/label the data (supervised learning).
• Define privacy and security requirements.

• Define requirements for employee training. This can be vendor training for a tool or platform training in the case of a pre-built model or service.
• Define if ML/AI expertise is required.
• Is the organization ready for ML/AI? Conduct an AI literacy survey and understand team’s concerns, fears, and misconceptions and address them.

• Define requirements for:
  • Data migration.
  • Security.
  • AI/ML pipeline deployment and maintenance.
• Define requirements for operation and maintenance of the tool or model.
• Confirm infrastructure readiness.

• How AI and its output will be used across the organization.

Why do you need AI in your toolset? What value will AI deliver? Have a clear understanding of business benefits and the value AI delivers through the tool.

• Define a business problem that can be solved with either an AI-powered tool or AI/ML pre-built model.
• Define expectations and assumptions around the value that AI can bring.
• Document business requirements for a tool or model.
• Start with the POC or a prototype to test assumptions, architecture, and components of the solution.
• Define business metrics and KPIs to measure success of the implementation.

Info-Tech Insight

Question the value that AI adds to the tool you are evaluating. Don’t go after the tool simply because it has an AI label attached to it. AI/ML capabilities might add little value but increase implementation complexity. Define the problem you are solving and document business requirements for the tool or a model.

AAI solutions and technologies are helping organizations make faster decisions and predict future outcomes such as:

• Business process automation
• Intelligent integration
• Intelligent insights
• Operational efficiency improvement
• Increase revenue
• Improvement of existing products and services
• Product and process innovation

• Business objectives of the initiative that might drive the AI/ML use case.
• The business owner or primary stakeholder who will help to define business value and requirements.
• All stakeholders who will be involved or impacted.
• KPIs that will be used to assess the success of the POC.
• Data required for the implementation.

• Use the Business Requirements tab to document high-level requirements for a tool or model.
• These requirements will be used while defining criteria for a tool selection and to validate if the tool or model meets your business goals.
• You can use either traditional BRD format or a user story to document requirements.

Input

• Strategic plan of the organization
• Data strategy that defines target data capabilities required to support enterprise strategic goals
• Roadmap of business and data initiatives to support target state of data capabilities

Output

• Prioritized list of business use cases where an AI-powered tool or AI/ML can deliver business value
• List of high-level requirements for the selected use case

Materials

• Whiteboard/Flip Charts
• Off-the-Shelf-AI Analysis Tool, “Business Drivers” and “Business Requirements” tabs

Participants

• CIO
• Senior business and IT stakeholders
• Data owner(s)
• Data steward(s)
• Enterprise Architect
• Data Architect
• Data scientist/Data analyst

Understand data required for implementation

Do you have the right data to implement and run the AI-powered tool or AI/ML model?

Info-Tech Insight

Know your data. Determine data requirements to:

• Train the model during the implementation and development, and
• Run the model in production

• Define what data is required for implementation, e.g. customer data, financial data, product sentiment.
• If the data is not available, can it be acquired, gathered, or generated?
• Define the volume of data required for implementation and production.
• If the model has to be trained, do you have the data required for training (e.g. dictionary of terms)? Can it be created, gathered, or acquired?
• Document internal and external sources of data.

• Evaluate data quality for all data sources based on the requirements and criteria defined in the previous step.
• For datasets with data quality issues, determine if the data issues can be resolved (e.g. missing values are inferred). If not, can this issue be resolved by using other data sources?
• Engage a Data Governance organization to address any data quality concerns.

• Determine if there are requirements for a specific data format required for the tool or model.
• Determine if there is a need to classify/label or tag the data. What are the metadata requirements?
• Define whether or not the implementation team needs to aggregate or transform the data before it can be used.

• Define privacy requirements, as these might affect the availability of the data for ML/AI.
• Define data bias concerns and considerations. Do you have datasheets for datasets that will be used in this project? What datasets cannot be used to prevent bias?
• What are the security requirements and how will they affect data storage, product selection, and infrastructure requirements for the tool and overall solution?

• Define where and how the data is currently stored and will be stored.
• Does it have to be migrated or consolidated? Does it have to be moved to the cloud or between systems?
• Is a data lake or data warehouse a requirement for this implementation as defined by the solution architecture?

• Data Domain – e.g. Customer data
• Data Concept – e.g. Customer
• Data Internally Accessible – Identify datasets that are required for the implementation even if the data might not be available internally. Work on determining if the data ca be acquired externally or collected internally.
• Source System – define the primary source system for the data, e.g. Salesforce
• Target System (if applicable) – Define if the data needs to be migrated/transferred. For example, you might use a datalake or data warehouse for the AI/ML solution or migrate data to the cloud.
• Classification/Taxonomy/Ontology
• Data Steward
• Data Owner
• Data Quality – Data quality indicator
• Refresh Rate – Frequency of data refresh. Indicate if the data can be accessed in real time or near-real time

• Retention – Retention policy requirements
• Compliance Requirements – Define if data has to comply with any of the regulatory requirements, e.g. GDPR
• Privacy, Bias, and Ethics Considerations – Privacy Act, PIPEDA, etc. Identify if the dataset contains sensitive information that should be excluded from the model, such as gender, age, race etc. Indicate fairness metrics, if applicable.

Download the Off-the-Shelf AI Analysis Tool

Input

• Documented business use cases from Step 1.
• High-level business requirements from Step 1.
• Data catalog, data dictionaries, business glossary
• Data flows and data architecture

Output

• High-level data requirements
• List of data sources and datasets that can be used for the implementation
• Datasets that need to be collected or acquired externally

Materials

• Whiteboard/Flip Charts
• Off-the-Shelf AI Analysis Tool, “Data” tab

Participants

• CIO
• Business and IT stakeholders
• Data owner(s)
• Data steward(s)
• Enterprise Architect
• Data Architect
• Data scientist/Data analyst

• Implementation of the AI/ML-powered Off-the-Shelf Tool or an AI/ML model will require a team with a combination of skills through all phases of the project, from design of the solution to build, production, deployment, and support.
• Document the skillsets required and determine the skills gap. Before you start hiring, depending on the role, you might find talent within the organization to join the implementation team with little to no training.
• AI/ML resources that may be needed on your team driving AI implementation (you might consider bringing part-time resources to fill the gaps or use vendor developers) are:
  • Data Scientist
  • Machine Learning Engineer
  • Data Engineer
  • Data Architect
  • AI/ML Ops engineer
• Define training requirements. Consider vendor training for a tool or platform.
• Plan for future scaling and the growing of the solution and AI practice. Assess the need to apply AI in other business areas. Work with the team to analyze use cases and prioritize AI initiatives. As the practice grows, grow your team expertise.

• Identify the stakeholders who will be affected by the AI implementation.
• Work with them to understand and address any concerns, fears, or misconceptions around the role of AI and the consequences of bringing AI into the organization.
• Develop a communication and change management plan to educate everyone within the organization on the application and benefits of using AI and machine learning.

• What data (attributes) were used to train the model?
• Do you have datasheets for the data used?
• How was data bias mitigated?
• What are the data labeling/classification requirements for training the model?
• What data is required for production? E.g. volume; type of data, etc.

• Were there any open-source libraries used in the model? If yes, how were vulnerabilities and security concerns addressed?
• What algorithms are implemented in the tool/model?
• Can model parameters be configured?
• What is model accuracy?

• Level of customization required for the implementation to meet our requirements.
• Does the model require training? If yes, can you provide details? Can you estimate the effort required?
• Integration capabilities and requirements.
• Data migration requirements for tool operation and development.
• Administrator console – is this functionality available?
• Implementation timeframe.

• Is the model or tool deployable on premises or in the cloud? Do you support hybrid cloud and multi-cloud deployment?
• What cloud platforms are your product/model integrated with (AWS, Azure, GCP)?
• What are the infrastructure requirements?
• Is the model containerized/ scalable?
• What product support and product updates are available?

• Regulatory compliance (GDPR, PIPEDA, HIPAA, PCI DSS, CCPA, SOX, etc.)?
• How are data security risks addressed?

• Enhanced functionality, allows the power of AI without specialized skills (e.g., Mathematica – recognizing patterns in data).
• Might be a cheaper option compared to building a solution in-house (chatbot, for ex.).

Info-Tech Insight:

No need to reinvent the wheel and build the product you can buy, but be prepared to work around tool limitations, and make sure you understand the data and the model the tool is built on.

• Dependency on the service provider.
• The tool might not meet all the business requirements without customization.
• Bias can be built into the tool:
  • Work with the vendor to understand what data was used to train the model.
  • From the perspective of ethics and bias, learn what model is implemented in the tool and what data attributes the model uses.

• Lower cost and less time to development compared to creating and training models from scratch (e.g. using image recognition models or pre-trained language models like BERT).
• If the pre-trained and optimized model perfectly fits your needs, the model accuracy might be high and sufficient for your scenario.
• Off-the-Shelf AI models are useful for creating prototypes or POCs, for testing a hypothesis, and for validating ideas and requirements.
• Usage of Off-the-Shelf models shortens the development cycle and reduces investment risks.
• Language models are particularly useful if you don’t have data to train your own model (a “small data” scenario).
• Infrastructure and model training cost reduction.

• Might be a challenge to deploy and maintain the system in production.
• Lack of flexibility: you might not be able to configure input or output parameters to your requirements. For example, a pre-built sentiment analysis model might return four values (“positive,” “negative,” “neutral,” and “mixed”), but your solution will require only two or three values.
• Might be a challenge to comply with security and privacy requirements.
• Compliance with privacy and fairness requirements and considerations: what data was used to pretrain the model?
• If open-source libraries were used to create the model, how will vulnerabilities, risks, and security concerns be addressed?