Choose the Right Tools for Big Data Development
Leverage Hadoop as your pilot project to gain organizational buy-in and build institutional learning.
This content requires an active subscription.
Access this content by logging in with your Info-Tech Research Group membership or contacting one of our representatives for assistance.
Your Challenge
- A relational database management system works great under many scenarios, but it has its limitations:
- Volume issues typically arise when there is a need to index large databases.
- In a multi-source environment, data collisions occur and resolving them can be expensive and time consuming.
- Velocity problems arise when large amounts of read/write transactions occur that are expensive to compute.
Our Advice
Critical Insight
- Begin your big data implementation with a baseline Hadoop pilot. This pilot will help build your knowledge of big data, how the Hadoop framework satisfies your use cases, and how it operates in your system. Each component in this baseline stack is well understood in the industry and documentation is readily available.
Impact and Result
- Provide a step-by-step starting point to begin the rollout of big data development based on your business and technical requirements.
- Highlight the challenges, impacts, potential, and mitigations in big data development.
- Identify the key metrics, benchmarks, and instrumentation points to measure the success of your big data rollout.
Contributors
Individuals who conducted expert interviews with us for this project:
- Martin Parrest, Foxnet Solutions
- Mehdi Bahrami, University of California, Merced
- Michael Hausenblas, MapR Technologies
- Michael Davison, Davis+Henderson
Vendors who conducted expert interviews with us for this project:
- Pentaho
- Informatica
Get the Complete Storyboard
See how all the steps you need to take come together, with tools and advice to help with each task on your list.
Download NowGet to Action
-
Assess fit and readiness for big data
Minimize the process and technology impacts of introducing big data to the organization.
-
Build the project team
Identify the roles and responsibilities of the big data project team.
-
Roll out the Hadoop pilot
Create a Hadoop stack based on business requirements and the data that needs to be mined and analyzed.
-
Roll out Hadoop in the organization
Customize the Hadoop pilot for fit in other areas of the organization based on instrumentation and pilot experiences.
Guided Implementation
This guided implementation is a three call advisory process.
-
Call #1: Assess fit and readiness for big data
Get off to a productive start: Assess your data analytics stacks to determine your readiness for big data. Info-Tech analysts will help you identify your gaps and create a list of tasks to fill these gaps.
-
Call #2: Prepare and roll out the Hadoop pilot
Build your Hadoop pilot stack: Review the roles and responsibilities for your Hadoop pilot, document your requirements, choose an installation approach, design and build your MapReduce program, select your analytics toolset, and test and validate your Hadoop data flow. Info-Tech analysts will discuss the fit of your Hadoop stack to your business requirements and assist in planning for the stack implementation.
-
Call #3: Roll out Hadoop in the organization
Monitor and optimize your Hadoop stack for deployment: Identify your instrumentation points and metrics, tweak your Hadoop stack based on your instrumentation to fit other areas of your organization, and apply lessons learned to other development projects. Info-Tech analysts will discuss the success of your big data rollout and help you optimize your big data stack.
Onsite Workshop
Module 1: Assess fit for big data
The Purpose
- Understand the current big data landscape.
- Identify the project team.
- Assess the current data analytics stack.
Key Benefits Achieved
- Understand the organization’s readiness for big data development.
| Activities: | Outputs: | |
|---|---|---|
| 1.1 | Document and assess the development process |
|
| 1.2 | Assess the data analytics stack |
|
| 1.3 | Address the gaps in the stack |
|
Module 2: Draw the big data flow
The Purpose
- Map the requirements to big data.
- Draw the big data flow.
Key Benefits Achieved
- Ensure business requirements are mapped to each component of the big data flow.
| Activities: | Outputs: | |
|---|---|---|
| 2.1 | Document the business requirements and use cases |
|
| 2.2 | Draw the top-down and bottom-up big data flows |
|
Module 3: Build the Hadoop stack
The Purpose
- Choose the appropriate installation approach.
- Import data into Hadoop.
- Develop the MapReduce program.
- Select big data analytics tools.
- Conduct end-to-end testing.
Key Benefits Achieved
- Create a baseline Hadoop stack that fits the organization’s needs.
- Understand the challenges of installing and managing the Hadoop stack.
| Activities: | Outputs: | |
|---|---|---|
| 3.1 | Select the installation approach |
|
| 3.2 | Classify the imported data |
|
| 3.3 | Select the data collection tools |
|
| 3.4 | Design the relational schema |
|
| 3.5 | Test and validate the dataflow |
|
| 3.6 | Choose analytics tools |
|
| 3.7 | Perform end-to-end testing |
|
Module 4: Roll out Hadoop in the organization
The Purpose
- Prepare the Hadoop stack for deployment.
- Gain institutional learning.
- Create an organizational rollout plan
Key Benefits Achieved
| Activities: | Outputs: | |
|---|---|---|
| 4.1 | Establish instrumentation points |
|
| 4.2 | Optimize the Hadoop stack |
|
| 4.3 | Develop an organization rollout plan |
|
| 4.4 | Establish a stakeholder communication plan |
|
Book Your Workshop
Onsite Workshops offer an easy way to accelerate your project. If you are unable to do the project yourself, and a Guided Implementation isn’t enough, we offer low-cost onsite delivery of our Project Workshops. We take you through every phase of your project and ensure that you have a road map in place to complete your project successfully.
Book NowApplication Development Map
-
1Create a Horizontally Optimized SDLC to Better Meet Business Demands
-
2Take a Holistic View to Optimize Release Management
-
3Application Development
-
4Choose the Right Development Platform for Enterprise Mobile Applications
-
5Develop Enterprise Mobile Applications With Realistic and Relevant Performance
-
6Stay on Top of Today’s and Tomorrow’s Mobile App Trends
-
7Make Development Teams Leaner and Improve Time-to-Release in Five Steps
-
8Strengthen the SSDLC for Enterprise Mobile Applications
-
9Use DevOps to Respond Faster to End Customers
-
10Modernize Your Corporate Website to Drive Business Value
-
11Implement Agile Practices That Work
-
12Enable Organization-Wide Collaboration by Scaling Agile
-
13Spread Best Practices With an Agile Center of Excellence
-
14Optimize Existing Applications for Security in an Untrusted World
-
15Choose the Right Tools for Big Data Development
-
16Optimize the Application Maintenance Estimation (AME) Process
-
17Use PaaS to Decrease Ongoing Application Maintenance Costs
-
18Develop APIs That Work Properly for the Organization
-
19Enter Into Mobile Development Without Confusion and Frustration
-
20Reduce Churn and Improve an Existing Software Development Practice
-
21Vendor Landscape: Application Development Tools
-
22Choose a Mobile Development Strategy
-
23Drive Web Traffic with Search Engine Optimization
-
24Develop a Position on Cloud Platform as a Service
-
25Vendor Landscape: Application Lifecycle Management
-
26Vendor Landscape: Mobile Enterprise Application Platforms
-
272016 Applications and Information Management Trends
-
28Develop a Fit-For-Purpose Modernization Process For Legacy Applications
-
29
Move to a Microservices Architecture to Reduce Maintenance Cycles
Search Code: 73868
Published: December 3, 2013
Last Revised: May 19, 2014
