Get Instant Access
to This Blueprint

Data Business Intelligence icon

Drive Business Value With Off-the-Shelf AI

A practical guide to ensure return on your off-the-shelf AI investment

  • Understanding the impact of the machine learning/AI component that is built into most of the enterprise products and tools and its role in the implementation of the solution.
  • Understanding the most important aspects that the organization needs to consider while planning the implementation of the AI-powered product.

Our Advice

Critical Insight

  • Organizations are faced with multiple challenges trying to adopt AI solutions. Challenges include data issues, ethics and compliance considerations, business process challenges, and misaligned leadership goals.
  • When choosing the right product to meet business needs, organizations need to know what questions to ask vendors to ensure they fully understand the implications of buying an AI/ML product.
  • To guarantee the success of your off-the-shelf AI implementation and ensure it delivers value, you must start with a clear definition of the business case and an understanding of your data.

Impact and Result

To guarantee success of the off-the-shelf AI implementation and deliver value, in addition to formulating a clear definition of the business case and understanding of data, organizations should also:

  • Know what questions to ask vendors while evaluating AI-powered products.
  • Measure the impact of the project on business and IT processes.

Drive Business Value With Off-the-Shelf AI Research & Tools

1. Drive Business Value With Off-the-Shelf AI Deck – A step-by-step approach that will help guarantee the success of your Off-the-Shelf AI implementation and ensure it delivers business value

Use this practical and actionable framework that will guide you through the planning of your Off-the-Shelf AI product implementation.

2. Off-the-Shelf AI Analysis – A tool that will guide the analysis and planning of the implementation

Use this analysis tool to ensure the success of the implementation.


Drive Business Value With Off-the-Shelf AI

A practical guide to ensure return on your Off-the-Shelf AI investment

Executive Summary

Your Challenge
  • Understanding the impact of the machine learning/AI component that is built into most of the enterprise products and tools and its role in the implementation of the solution.
  • What are the most important aspects that organizations needs to consider while planning the implementation of the AI-powered product?
Common Obstacles
  • Organizations are faced with multiple challenges trying to adopt an AI solution. Challenges include data issues, ethics and compliance considerations, business process challenges, and misaligned leadership goals.
  • When choosing the right product to meet business needs, organizations need to know what questions to ask vendors to ensure they fully understand the implications of buying an AI/ML product.
Info-Tech’s Approach

Info-Tech’s approach includes a framework that will guide organizations through the process of the Off-the-Shelf AI product selection.

To guarantee success of the Off-the-Shelf AI implementation and deliver value, organization should start with clear definition of the business case and an understanding of data.

Other steps include:

  • Knowing what questions to ask vendors to evaluate AI-powered products.
  • Measuring the impact of the project on your business and IT processes.
  • Assessing impact on the organization and ensure team readiness.

Info-Tech Insight

To guarantee the success of your Off-the-Shelf AI implementation and ensure it delivers value, you must start with a clear definition of the business case and an understanding of your data.

Info-Tech offers various levels of support to best suit your needs

DIY Toolkit

Guided Implementation

Workshop

Consulting

"Our team has already made this critical project a priority, and we have the time and capability, but some guidance along the way would be helpful." "Our team knows that we need to fix a process, but we need assistance to determine where to focus. Some check-ins along the way would help keep us on track." "We need to hit the ground running and get this project kicked off immediately. Our team has the ability to take this over once we get a framework and strategy in place." "Our team does not have the time or the knowledge to take this project on. We need assistance through the entirety of this project."

Diagnostics and consistent frameworks used throughout all four options

Getting value out of AI and machine learning investments

92.1%

of companies say they are achieving returns on their data and AI investments

91.7%

said they were increasing investments in data and AI

26.0%

of companies have AI systems in widespread production
However, CIO Magazine identified nine main hurdles to AI adoption based on the survey results:
  • Data issues
  • Business process challenges
  • Implementation challenges and skill shortages
  • Costs of tools and development
  • Misaligned leadership goals
  • Measuring and proving business value
  • Legal and regulatory risks
  • Cybersecurity
  • Ethics
  • (Source: CIO, 2019)
“Data and AI initiatives are becoming well established, investments are paying off, and companies are getting more economic value from AI.” (Source: NewVantage, 2022.)

67% of companies are currently using machine learning, and 97% are using or planning to use it in the next year.” (Source: Deloitte, 2020)

AI vs. ML

Machine learning systems learn from experience and without explicit instructions. They learn patterns from data then analyze and make predictions based on past behavior and the patterns learned.

Artificial intelligence is a combination of technologies and can include machine learning. AI systems perform tasks mimicking human intelligence such as learning from experience and problem solving. Most importantly, AI is making its own decisions without human intervention.

The AI system can make assumptions, test these assumptions, and learn from the results.

(Level of decision making required increases from left to right)
Statistical Reasoning
Infer relationships between variables

Statistical models are designed to find relationships between variables and the significance of those relationships.

Machine Learning:
Making accurate predictions

Machine learning is a subset of AI that discovers patterns from data without being explicitly programmed to do so.

Artificial Intelligence
Dynamic adaptation to novelty

AI systems choose the optimal combination of methods to solve a problem. They make assumptions, reassess the model, and reevaluate the data.

“Machine learning is the study of computer algorithms that improve automatically through experience.” (Tom Mitchell, 1997)

“At its simplest form, artificial intelligence is a field, which combines computer science and robust datasets, to enable problem-solving.” (IBM, “What is artificial intelligence?”)

Types of Off-the-Shelf AI products and solutions

ML/AI-Powered Products Off-the-Shelf Pre-built and Pre-trained AI/ML Models
  • AI/ML capabilities built into the product and might require training as part of the implementation.
  • Off-the-Shelf ML/AI Models, pre-built, pre-trained, and pre-optimized for a particular task. For example, language models or image recognition models that can be used to speed up and simplify ML/AI systems development.
Examples of OTS tools/products: Examples of OTS models:

The data inputs for these models are defined, the developer has to conform to the provided schema, and the data outputs are usually fixed due to the particular task the OTS model is built to solve.

Insight summary

Overarching insight:

To guarantee the success of your Off-the-Shelf AI implementation and ensure it delivers value, you must start with a clear definition of the business case and an understanding of your data.

Business Goals

Question the value that AI adds to the tool you are evaluating. Don’t go after the tool simply because it has an AI label attached to it. AI/ML capabilities might add little value but increase implementation complexity. Define the problem you are solving and document business requirements for the tool or a model.

Data

Know your data. Determine data requirements to:

  • Train the model during the implementation and development.
  • Run the model in production.

People/Skills

Define the skills required for the implementation and assemble the team that will support the project from requirements to deployment and support, through its entire lifecycle. Don’t forget about production support and maintenance.

Choosing an AI-Powered Tool

No need to reinvent the wheel and build a product you can buy, but be prepared to work around tool limitations, and make sure you understand the data and the model the tool is built on.

Choosing an AI/ML Model

Using Off-the-Shelf-AI models enables an agile approach to system development. Faster POC and validation of ideas and approaches, but the model might not be customizable for your requirements.

Guaranteeing Off-the-Shelf AI Implementation Success

Info-Tech Insight

To guarantee the success of your Off-the-Shelf AI implementation and ensure it delivers value, you must start with a clear definition of the business case and an understanding of your data.

Why do you need AI in your toolset?
Business Goals

Clearly defined problem statement and business requirements for the tool or a model will help you select the right solution that will deliver business value even if it does not have all the latest bells and whistles.

Small chevron pointing right.
Do you know the data required for implementation?
Data

Expected business outcome defines data requirements for implementation. Do you have the right data required to train and run the model?

Large chevron pointing right.
Is your organization ready for AI?
People/Team/ Skills

New skills and expertise are required through all phases of the implementation: design, build, deployment, support, and maintenance, as well as post-production support, scaling, and adoption.

Data Architecture/ Infrastructure

New tool or model will impact your cloud and integration strategy. It will have to integrate with the existing infrastructure, in the cloud or on prem.

Large chevron pointing right.
What questions do you need to ask when choosing the solution?
Product/ Tool or Model Selection

Do you know what model powers the AI tool? What data was used to train the tool and what data is required to run it? Ask the right questions.

Small chevron pointing right.
Are you measuring impact on your processes?
Business and IT Processes

Business processes need to be defined or updated to incorporate the output of the tool back into the business processes to deliver value.

IT governance and support processes need to accommodate the new AI-powered tool.

Small chevron pointing right.
Realize and measure business value of your AI investment
Value

Do you have a clear understanding of the value that AI will bring to your organization?Optimization?Increased revenue?Operational efficiency?

Introduction of Off-the-Shelf AI Requires a Strategic Approach

Business Goals and Value Data People/Team/ Skills Infrastructure Business and IT Processes
AI/ML–powered tools
  • Define a business problem that can be solved with either an AI-powered tool or an AI/ML pre-built model that will become part of the solution.
  • Define expectations and assumptions around the value that AI can bring.
  • Document business requirements for the tool or model.
  • Define the scope for a prototype or POC.
  • Define data requirements.
  • Define data required for implementation.
  • Determine if the required data can be acquired or captured/generated.
  • Document internal and external sources of data.
  • Validate data quality (define requirements and criteria for data quality).
  • Define where and how the data is stored and will be stored. Does it have to be moved or consolidated?
  • Define all stakeholders involved in the implementation and support.
  • Define skills and expertise required through all phases of the implementation: design, build, deployment, support, and maintenance.
  • Define skills and expertise required to grow AI practice and achieve the next level of adoption, scaling, and development of the tool or model POC.
  • Define infrastructure requirements for either Cloud, Software-as-a-Service, or on-prem deployment of a tool or model.
  • Define how the tool is integrated with existing systems and into existing infrastructure.
  • Determine the cost to deploy and run the tool/model.
  • Define processes that need to be updated to accommodate new functionality.
  • Define how the outcome of the tool or a model (e.g. predictions) are incorporated back into the business processes.
  • Define new business and IT processes that need to be defined around the tool (e.g. chatbot maintenance; analysis of the data generated by the tool).
Off-the-shelf AI/ML pre-built models
  • Define the business metrics and KPIs to measure success of the implementation against.
  • Determine if there are requirements for a specific data format required for the tool or a model.
  • Determine if there is a need to classify/label the data (supervised learning).
  • Define privacy and security requirements.
  • Define requirements for employee training. This can be vendor training for a tool or platform training in the case of a pre-built model or service.
  • Define if ML/AI expertise is required.
  • Is the organization ready for ML/AI? Conduct an AI literacy survey and understand team’s concerns, fears, and misconceptions and address them.
  • Define requirements for:
    • Data migration.
    • Security.
    • AI/ML pipeline deployment and maintenance.
  • Define requirements for operation and maintenance of the tool or model.
  • Confirm infrastructure readiness.
  • How AI and its output will be used across the organization.

Define Business Goals and Objectives

Why do you need AI in your toolset? What value will AI deliver? Have a clear understanding of business benefits and the value AI delivers through the tool.

  • Define a business problem that can be solved with either an AI-powered tool or AI/ML pre-built model.
  • Define expectations and assumptions around the value that AI can bring.
  • Document business requirements for a tool or model.
  • Start with the POC or a prototype to test assumptions, architecture, and components of the solution.
  • Define business metrics and KPIs to measure success of the implementation.

Info-Tech Insight

Question the value that AI adds to the tool you are evaluating. Don’t go after the tool simply because it has an AI label attached to it. AI/ML capabilities might add little value but increase implementation complexity. Define the problem you are solving and document business requirements for the tool or a model.

Venn diagram of 'Applied Artificial Intelligence (AAI)' with a larger circle at the top, 'Machine Learning (ML)', and three smaller ovals intersecting, 'Computer Vision', 'Natural Language Processing (NLP)', and 'Robotic Process Automation (RPA)'.

AAI solutions and technologies are helping organizations make faster decisions and predict future outcomes such as:

  • Business process automation
  • Intelligent integration
  • Intelligent insights
  • Operational efficiency improvement
  • Increase revenue
  • Improvement of existing products and services
  • Product and process innovation

1. Use Info-Tech’s Off-the-Shelf AI Analysis Tool to define business drivers and document business requirements

2-3 hours
Screenshot of the Off-the-Shelf AI Analysis Tool's Business Drivers tab, a table with columns 'AI/ML Tool or Model', 'Use Case', 'Business problem / goal for AI/ML use case', 'Description', 'Business Owner (Primary Stakeholder)', 'Priority', 'Stakeholder Groups Impacted', 'Requirements Defined? Yes/No', 'Related Data Domains', and 'KPIs'. Use the Business Drivers tab to document:
  • Business objectives of the initiative that might drive the AI/ML use case.
  • The business owner or primary stakeholder who will help to define business value and requirements.
  • All stakeholders who will be involved or impacted.
  • KPIs that will be used to assess the success of the POC.
  • Data required for the implementation.
  • Use the Business Requirements tab to document high-level requirements for a tool or model.
  • These requirements will be used while defining criteria for a tool selection and to validate if the tool or model meets your business goals.
  • You can use either traditional BRD format or a user story to document requirements.
Screenshot of the Off-the-Shelf AI Analysis Tool's Business Requirements tab, a table with columns 'Requirement ID', 'Requirement Description / user story', 'Requirement Category', 'Stakeholder / User Role', 'Requirement Priority', and 'Complexity (point estimates)'.

Download the Off-the-Shelf AI Analysis Tool

1. Define business drivers and document business requirements

Input

  • Strategic plan of the organization
  • Data strategy that defines target data capabilities required to support enterprise strategic goals
  • Roadmap of business and data initiatives to support target state of data capabilities

Output

  • Prioritized list of business use cases where an AI-powered tool or AI/ML can deliver business value
  • List of high-level requirements for the selected use case

Materials

  • Whiteboard/Flip Charts
  • Off-the-Shelf-AI Analysis Tool, “Business Drivers” and “Business Requirements” tabs

Participants

  • CIO
  • Senior business and IT stakeholders
  • Data owner(s)
  • Data steward(s)
  • Enterprise Architect
  • Data Architect
  • Data scientist/Data analyst

Understand data required for implementation

Do you have the right data to implement and run the AI-powered tool or AI/ML model?

Info-Tech Insight

Know your data. Determine data requirements to:

  • Train the model during the implementation and development, and
  • Run the model in production
AvailabilityArrow pointing rightQualityArrow pointing rightPreparationArrow pointing rightBias, Privacy, SecurityArrow pointing rightData Architecture
  • Define what data is required for implementation, e.g. customer data, financial data, product sentiment.
  • If the data is not available, can it be acquired, gathered, or generated?
  • Define the volume of data required for implementation and production.
  • If the model has to be trained, do you have the data required for training (e.g. dictionary of terms)? Can it be created, gathered, or acquired?
  • Document internal and external sources of data.
  • Evaluate data quality for all data sources based on the requirements and criteria defined in the previous step.
  • For datasets with data quality issues, determine if the data issues can be resolved (e.g. missing values are inferred). If not, can this issue be resolved by using other data sources?
  • Engage a Data Governance organization to address any data quality concerns.
  • Determine if there are requirements for a specific data format required for the tool or model.
  • Determine if there is a need to classify/label or tag the data. What are the metadata requirements?
  • Define whether or not the implementation team needs to aggregate or transform the data before it can be used.
  • Define privacy requirements, as these might affect the availability of the data for ML/AI.
  • Define data bias concerns and considerations. Do you have datasheets for datasets that will be used in this project? What datasets cannot be used to prevent bias?
  • What are the security requirements and how will they affect data storage, product selection, and infrastructure requirements for the tool and overall solution?
  • Define where and how the data is currently stored and will be stored.
  • Does it have to be migrated or consolidated? Does it have to be moved to the cloud or between systems?
  • Is a data lake or data warehouse a requirement for this implementation as defined by the solution architecture?

2. Use Info-Tech’s Off-the-Shelf AI Analysis Tool to document data requirements

2-3 hours

Use the Data tab to document the following for each data source or dataset:
  • Data Domain – e.g. Customer data
  • Data Concept – e.g. Customer
  • Data Internally Accessible – Identify datasets that are required for the implementation even if the data might not be available internally. Work on determining if the data ca be acquired externally or collected internally.
  • Source System – define the primary source system for the data, e.g. Salesforce
  • Target System (if applicable) – Define if the data needs to be migrated/transferred. For example, you might use a datalake or data warehouse for the AI/ML solution or migrate data to the cloud.
  • Classification/Taxonomy/Ontology
  • Data Steward
  • Data Owner
  • Data Quality – Data quality indicator
  • Refresh Rate – Frequency of data refresh. Indicate if the data can be accessed in real time or near-real time

Screenshot of the Off-the-Shelf AI Analysis Tool's Data tab, a spreadsheet table with the columns listed to the left and below.
  • Retention – Retention policy requirements
  • Compliance Requirements – Define if data has to comply with any of the regulatory requirements, e.g. GDPR
  • Privacy, Bias, and Ethics Considerations – Privacy Act, PIPEDA, etc. Identify if the dataset contains sensitive information that should be excluded from the model, such as gender, age, race etc. Indicate fairness metrics, if applicable.

Download the Off-the-Shelf AI Analysis Tool

2. Document data requirements

Input

  • Documented business use cases from Step 1.
  • High-level business requirements from Step 1.
  • Data catalog, data dictionaries, business glossary
  • Data flows and data architecture

Output

  • High-level data requirements
  • List of data sources and datasets that can be used for the implementation
  • Datasets that need to be collected or acquired externally

Materials

  • Whiteboard/Flip Charts
  • Off-the-Shelf AI Analysis Tool, “Data” tab

Participants

  • CIO
  • Business and IT stakeholders
  • Data owner(s)
  • Data steward(s)
  • Enterprise Architect
  • Data Architect
  • Data scientist/Data analyst

Is Your Organization Ready for AI?

Assess organizational readiness and define stakeholders impacted by the implementation. Build the team with the right skillset to drive the solution.

  • Implementation of the AI/ML-powered Off-the-Shelf Tool or an AI/ML model will require a team with a combination of skills through all phases of the project, from design of the solution to build, production, deployment, and support.
  • Document the skillsets required and determine the skills gap. Before you start hiring, depending on the role, you might find talent within the organization to join the implementation team with little to no training.
  • AI/ML resources that may be needed on your team driving AI implementation (you might consider bringing part-time resources to fill the gaps or use vendor developers) are:
    • Data Scientist
    • Machine Learning Engineer
    • Data Engineer
    • Data Architect
    • AI/ML Ops engineer
  • Define training requirements. Consider vendor training for a tool or platform.
  • Plan for future scaling and the growing of the solution and AI practice. Assess the need to apply AI in other business areas. Work with the team to analyze use cases and prioritize AI initiatives. As the practice grows, grow your team expertise.
  • Identify the stakeholders who will be affected by the AI implementation.
  • Work with them to understand and address any concerns, fears, or misconceptions around the role of AI and the consequences of bringing AI into the organization.
  • Develop a communication and change management plan to educate everyone within the organization on the application and benefits of using AI and machine learning.

Info-Tech Insight:

Define the skills required for the implementation and assemble the team that will support the project through its entire lifecycle. Don’t forget about production, support, and maintenance.

3. Build your implementation team

1-2 hours

Input: Solution conceptual design, Current resource availability

Output: Roles required for the implementation of the solution, Resources gap analysis, Training and hiring plan

Materials: Whiteboard/Flip charts, Off-the-Shelf AI Analysis Tool, “People and Team” tab

Participants: Project lead, HR, Enterprise Architect

  1. Review your solution conceptual design and define implementation team roles.
  2. Document requirements for each role.
  3. Review current org chart and job descriptions and identify skillset gaps. Draft an action plan to fill in the roles.
  4. Use Info-Tech’s Off-the-Shelf AI Analysis Tool's People and Team tab to document team roles for the entire implementation, including design, build/implement, deployment, support and maintenance, and future development.

Screenshot of the Off-the-Shelf AI Analysis Tool's People and Team tab, a table with columns 'Design', 'Implement', 'Deployment', 'Support and Maintenance', and 'Future Development'.

Download the Off-the-Shelf AI Analysis Tool

Cloud, SaaS or On Prem – what are my options and what is the impact?

Depending on the architecture of the solution, define the impact on the current infrastructure, including system integration, AI/ML pipeline deployment, maintenance, and data storage

  • Data Architecture: use the current data architecture to design the architecture for an AI-powered solution. Assess changes to the data architecture with the introduction of a new tool to make sure it is scalable enough to support the change.
  • Define infrastructure requirements for either Cloud, Software-as-a-Service, or on-prem deployment of a tool or model.
  • Define how the tool will be integrated with existing systems and into existing infrastructure.
  • Define requirements for:
    • Data migration and data storage
    • Security
    • AI/ML pipeline deployment, production monitoring, and maintenance
  • Define requirements for operation and maintenance of the tool or model.
  • Work with your infrastructure architect and vendor to determine the cost of deploying and running the tool/model.
  • Make a decision on the preferred architecture of the system and confirm infrastructure readiness.

Download the Create an Architecture for AI blueprint

4. Use Info-Tech’s Off-the-Shelf AI Analysis Tool to document infrastructure decisions

2-3 hours

Input: Solution conceptual design

Output: Infrastructure requirements, Infrastructure readiness assessment

Materials: Whiteboard/Flip charts, Off-the-Shelf AI Analysis Tool, “Infrastructure” tab

Participants: Infrastructure Architect, Solution Architect, Enterprise Architect, Data Architect, ML/AI Ops Engineer

  1. Work with Infrastructure, Data, Solution, and Enterprise Architects to define your conceptual solution architecture.
  2. Define integration and storage requirements.
  3. Document security requirements for the solution in general and the data specifically.
  4. Define MLOps requirements and tools required for ML/AI pipeline deployment and production monitoring.
  5. Use Info-Tech’s Off-the-Shelf AI Analysis Tool's Infrastructure tab to document requirements and decisions around Data and Infrastructure Architecture.

Screenshot of the Off-the-Shelf AI Analysis Tool's Infrastructure tab, a table with columns 'Cloud, SaaS or On-Prem', 'Data Migration Requirements', 'Data Storage Requirements', 'Security Requirements', 'Integrations Required', and 'AI/ML Pipeline Deployment and Maintenance Requirements'.

Download the Off-the-Shelf AI Analysis Tool

What questions do you need to ask vendors when choosing the solution?

Take advantage of Info-Tech’s Rapid Application Selection Framework (RASF) to guide tool selection, but ask vendors the right questions to understand implications of having AI/ML built into the tool or a model

Data Model Implementation and Integration Deployment Security and Compliance
  • What data (attributes) were used to train the model?
  • Do you have datasheets for the data used?
  • How was data bias mitigated?
  • What are the data labeling/classification requirements for training the model?
  • What data is required for production? E.g. volume; type of data, etc.
  • Were there any open-source libraries used in the model? If yes, how were vulnerabilities and security concerns addressed?
  • What algorithms are implemented in the tool/model?
  • Can model parameters be configured?
  • What is model accuracy?
  • Level of customization required for the implementation to meet our requirements.
  • Does the model require training? If yes, can you provide details? Can you estimate the effort required?
  • Integration capabilities and requirements.
  • Data migration requirements for tool operation and development.
  • Administrator console – is this functionality available?
  • Implementation timeframe.
  • Is the model or tool deployable on premises or in the cloud? Do you support hybrid cloud and multi-cloud deployment?
  • What cloud platforms are your product/model integrated with (AWS, Azure, GCP)?
  • What are the infrastructure requirements?
  • Is the model containerized/ scalable?
  • What product support and product updates are available?
  • Regulatory compliance (GDPR, PIPEDA, HIPAA, PCI DSS, CCPA, SOX, etc.)?
  • How are data security risks addressed?

Use Info-Tech’s Off-the-Shelf AI Analysis Tool, “Vendor Questionnaire” tab to track vendor responses to these questions.

Are you measuring impact on your processes?

Make sure that you understand the impact of the new technology on the existing business and IT processes.

And make sure your business processes are ready to take advantage of the benefits and new capabilities enabled by AI/ML.

Process automation, optimization, and improvement enabled by the technology and AI/ML-powered tools allow organizations to reduce manual work, streamline existing business processes, improve customer satisfaction, and get critical insights to assist decision making.

To take full advantage of the benefits and new capabilities enabled by the technology, make sure that business and IT processes reflect these changes:

  • Processes that need to be updated.
  • How the outcome of the tool or a model (e.g. predictions) is incorporated into the existing business processes and the processes that will monitor the accuracy of the outcome and monitor performance of the tool or model.
  • New business and IT processes that need to be defined for the tool (e.g. chatbot maintenance, analysis of the data generated by the tool, etc.).

5. Document the Impact on Business and IT Processes

2-3 hours

Input: Solution design, Existing business and IT processes

Output: Documented updates to the existing processes, Documented new business and IT processes

Materials: Whiteboard/Flip charts, Off-the-Shelf AI Analysis Tool, “Business and IT Processes” tab

Participants: Project lead, Business stakeholders, Business analyst

  1. Review current business processes affected by the implementation of the AI/ML- powered tool or model. Define the changes that need to be made. The changes might include simplification of the process due to automation of some of the steps. Some processes will need to be redesigned and some processes might become obsolete.
  2. Document high-level steps for any new processes that need to be defined around the AI/ML-powered tool. An example of such a process would be defining new IT and business processes to support a new chatbot.
  3. Use Info-Tech’s Off-the-Shelf AI Analysis Tool's Business and IT Processes tab, to document process changes.

Screenshot of the Off-the-Shelf AI Analysis Tool's Business and IT Processes tab, a table with columns 'Existing business process affected', 'New business process', 'Stakeholders involved', 'Changes to be made', and 'New Process High-Level Steps'.

Download the Off-the-Shelf AI Analysis Tool

AI-powered Tools – Considerations

PROS:
  • Enhanced functionality, allows the power of AI without specialized skills (e.g., Mathematica – recognizing patterns in data).
  • Might be a cheaper option compared to building a solution in-house (chatbot, for ex.).

Info-Tech Insight:

No need to reinvent the wheel and build the product you can buy, but be prepared to work around tool limitations, and make sure you understand the data and the model the tool is built on.

CONS:
  • Dependency on the service provider.
  • The tool might not meet all the business requirements without customization.
  • Bias can be built into the tool:
    • Work with the vendor to understand what data was used to train the model.
    • From the perspective of ethics and bias, learn what model is implemented in the tool and what data attributes the model uses.

Pre-built/pre-trained models – what to keep in mind when choosing

PROS:
  • Lower cost and less time to development compared to creating and training models from scratch (e.g. using image recognition models or pre-trained language models like BERT).
  • If the pre-trained and optimized model perfectly fits your needs, the model accuracy might be high and sufficient for your scenario.
  • Off-the-Shelf AI models are useful for creating prototypes or POCs, for testing a hypothesis, and for validating ideas and requirements.
  • Usage of Off-the-Shelf models shortens the development cycle and reduces investment risks.
  • Language models are particularly useful if you don’t have data to train your own model (a “small data” scenario).
  • Infrastructure and model training cost reduction.
CONS:
  • Might be a challenge to deploy and maintain the system in production.
  • Lack of flexibility: you might not be able to configure input or output parameters to your requirements. For example, a pre-built sentiment analysis model might return four values (“positive,” “negative,” “neutral,” and “mixed”), but your solution will require only two or three values.
  • Might be a challenge to comply with security and privacy requirements.
  • Compliance with privacy and fairness requirements and considerations: what data was used to pretrain the model?
  • If open-source libraries were used to create the model, how will vulnerabilities, risks, and security concerns be addressed?

Info-Tech Insight:

Using Off-the-Shelf AI models enables an agile approach to system development – faster POC and validation of ideas and approaches, but the model might not be customizable for your requirements.

Metrics

Metrics and KPIs for this project will depend on the business goals and objectives that you will identify in Step 1 of the tool selection process.

Metrics might include:

  • Reduction of time spent on a specific business process. If the tool is used to automate certain steps of a business process, this metric will measure how much time was saved, in minutes/hours, compared to the process time before the introduction of the tool.
  • Accuracy of prediction. This metric would measure the accuracy of estimations or predictions compared to the same estimations done before the implementation of the tool. It can be measured by generating the same prediction or estimation using the AI-powered tool or using any methods used before the introduction of the tool and comparing the results.
  • Accuracy of the search results. If the AI-powered tool is a search engine, compare a) how much time it would take a user to find an article or a piece of content they were searching for using new tool vs. previous techniques, b) how many steps it took the user to locate the required article in the search results, and c) the location of the correct piece of content in the search result list (at the top of the search result list or on the tenth page).
  • Time spent on manual tasks and activities. This metric will measure how much time, in minutes/hours, is spent by the employees or users on manual tasks if the tool automates some of these tasks.
  • Reduction of business process steps (if the steps are being automated). To derive this metric, create a map of the business process before the introduction of the AI-powered tool and after, and determine if the tool helped to simplify the process by reducing the number of process steps.

Bibliography

Adryan, Boris. “Is it all machine learning?” Badryan, Oct. 20, 2015. Accessed Feb. 2022.

“AI-Powered Data Management Platform.” Informatica, N.d. Accessed Feb 2022.

Amazon Rekognition. “Automate your image and video analysis with machine learning.” AWS. N.d. Accessed Feb 2022.

“Artificial Intelligence (AI).” IBM Cloud Education, 3 June 2020. Accessed Feb 2022.

“Artificial intelligence (AI) vs machine learning (ML).” Microsoft Azure Documentation. Accessed Feb. 2022.

“Avante Garde in the Realm of AI” SearchUnify Cognitive Platform. Accessed Feb 2022.

“Azure Cognitive Services.” Microsoft. N.d. Accessed Feb 2022.

“Becoming an AI-fueled organization. State of AI in the enterprise, 4th edition,” Deloitte, 2020. Accessed Feb. 2022.

“Coveo Predictive Search.” Coveo, N.d. Accessed Feb 2022.

”Data and AI Leadership. Executive Survey 2022. Executive Summary of Findings.” NewVantage Partners. Accessed Feb 2022.

“Einstein Discovery in Tableau.” Tableau, N.d. Accessed Feb 2022.

Korolov, Maria. “9 biggest hurdles to AI adoption.” CIO, Feb 26, 2019. Accessed Feb 2022.

Meel, Vidushi. “What Is Deep Learning? An Easy to Understand Guide.” visio.ai. Accessed Feb. 2022.

Mitchell, Tom. “Machine Learning,” McGraw Hill, 1997.

Stewart, Matthew. “The Actual Difference Between Statistics and Machine Learning.” Towards Data Science, Mar 24, 2019. Accessed Feb 2022.

“Sentiment analysis with Cognitive Services.” Microsoft Azure Documentation. Accessed February 2022.

“Three Principles for Designing ML-Powered Products.” Spotify Blog. Oct 2019, Accessed Feb 2022.

“Video Intelligence API.” Google Cloud Platform. N.d. Accessed Feb 2022

A practical guide to ensure return on your off-the-shelf AI investment

About Info-Tech

Info-Tech Research Group is the world’s fastest-growing information technology research and advisory company, proudly serving over 30,000 IT professionals.

We produce unbiased and highly relevant research to help CIOs and IT leaders make strategic, timely, and well-informed decisions. We partner closely with IT teams to provide everything they need, from actionable tools to analyst guidance, ensuring they deliver measurable results for their organizations.

What Is a Blueprint?

A blueprint is designed to be a roadmap, containing a methodology and the tools and templates you need to solve your IT problems.

Each blueprint can be accompanied by a Guided Implementation that provides you access to our world-class analysts to help you get through the project.

Talk to an Analyst

Our analyst calls are focused on helping our members use the research we produce, and our experts will guide you to successful project completion.

Book an Analyst Call on This Topic

You can start as early as tomorrow morning. Our analysts will explain the process during your first call.

Get Advice From a Subject Matter Expert

Each call will focus on explaining the material and helping you to plan your project, interpret and analyze the results of each project step, and set the direction for your next project step.

Unlock Sample Research

Author

Irina Sedenko

Visit our IT Cost Optimization Center
Over 100 analysts waiting to take your call right now: 1-519-432-3550 x2019