Nousot has helped numerous companies move their analytics workloads to the cloud and expand their business and technical capabilities along the way. We have seen what works and what doesn’t. We have observed the challenges enterprises face and captured the lessons learned from overcoming them. This blog identifies the walls enterprises hit as they scale their implementations in the second and third years of their cloud journey.
Challenge #1: Unclear Responsibilities
Impact: Security and Cost Risks
The goal of cloud management is to reduce risk and increase ROI from moving to the cloud. Many aspects of this move require teams to develop new skills from those they had been using with on-prem systems. As disparate teams move to the cloud at their individual paces, management gaps and risks are created. The demand for independence and speed needs to be weighed against each team’s ability to provide long-term support for the implemented solutions.
For example, data science teams often need to take advantage of the elastic compute resources and tooling of the cloud; however, when doing so, they are also responsible for the networking and security of their implementation. Questions to be answered include:
- Should the network and security teams be involved?
- Are individual teams in charge of their own cloud spend, or does finance have a central role in setting budgets?
- Do DBAs need to manage all databases, or if the infrastructure is utilizing Databricks or Snowflake, will the data teams manage these platforms?
In general, there are a plethora of possibilities for a cloud journey. This becomes a problem when teams manage complexities for which they do not have sufficient knowledge or there are gaps in managing essential controls.
Solution: Well-defined Roles and Communication
The first step in preventing responsibility mayhem is to have well-defined roles. Determine the key decision makers, their responsibilities, and the inputs they will need ahead of time, and ensure they have the training and tools required. Ensure there are clearly defined channels of communication and whether teams are intended to operate independently or with support.
Challenge #2: Getting Surprised
Impact: Cost and Business Impact
It’s essential to avoid surprises from the cloud environment. This could be a large monthly bill or the business reporting a critical functionality is down. As with any system, mistakes and issues happen. The goal is to minimize such surprises and the impact they have.
Solution: Alerts
While an essential component of any cloud system is proactive alerts, many organizations fail to make this a priority from day one. All major cloud providers offer basic alerting, and for hybrid or multi-cloud environments, third-party vendors are also available.
To avoid an ominous surprise, every cloud implementation must have a budget or cost alert set up from day one. When such alerts are not in place, companies can incur hundreds or thousands of dollars in charges due to a single run-away process.
Challenge #3: Lack of Visibility into Cloud Activity
Impact: Security
Often managers don’t have visibility into — or knowledge of — the services and tools technology teams utilize in their cloud environments. The freedom and flexibility to use the best tool is a great benefit of the cloud, but technology sprawl can lead to security and support difficulties.
Additionally, a major concern is by whom and from where data is accessed. It is essential to manage and monitor all tooling endpoints.
Solution: Auditing
All cloud platforms generate logs for the actions taken. Capturing and reporting on these actions reduces the time needed to troubleshoot issues and concerns.
In addition to cloud logs, many tools have their own logging system. For example Redshift, Snowflake, and BigQuery all have internal schemas to capture actions taken.
Having too many logs generated is a risk to auditing cloud activities, driving up costs and creating a ton of extra data with little value. The key to overcoming this is to not capture everything, focusing on the specific logs and metrics with the highest impact.
Challenge #4: Lack of Visibility into Cost Drivers
Impact: Cost Overruns
Not all challenges with costs come as a surprise. Many times increasing costs month over month are expected and driven by a predictable increase in utilization of services. However,it can be difficult to tell why certain environments or technologies are driving up costs, and permissions to access cost drivers may make it difficult to assess what or who is responsible.
Solution: Cost Monitoring
Most clouds provide a cost monitoring mechanism, but not all users have access to it. In addition, most out-of-the-box cost reporting mechanisms only offer a high-level view of costs without an ability to drill down to the level that may be necessary to troubleshoot. Having custom reporting in place, based on the technologies a team utilizes, and how a company is organized, can save time and reduce churn.
This can be done by utilizing external reporting tools, adding tags to resources, and pulling custom data from tools. For example, BigQuery usage is available in billing dashboards, but seeing what queries are driving the cost requires reporting to be set up on the information schema.
Challenge #5: Lack of Visibility into Business Usage
Impact: Business Value Drivers
Most technologists realize the technical capabilities and value of migrating to the cloud. The cloud enables faster execution and deployment of code. The main goal though is to drive business value and create actionable insights to drive revenue or increase margin. A key question around a cloud implementation is who in the business is utilizing it, and what business objectives is it enabling?
Solution: Cloud Utilization Reports
For each cloud solution, metrics should be defined and tracked for how the business is impacted and is utilizing the new solution. This could be a monthly report showing how many people are accessing dashboards or how many AI/ML models have been productionized and integrated into business operations.
Having these reports makes it easy to see the value driven from cloud investments. If these reports show there isn’t a wide enough adoption, it may be an indication of a need for change management and deeper discussions with the business around what will truly make an impact.
Conclusion
We invite you to discuss your program plans before encountering the obstacles many find on their analytics journey. Please reach out to us at [email protected].
