Automating Database Administration is no Longer Optional

by Jan 14, 2021

Yes, the title of today’s blog post is somewhat provocative, but I think after reading what I have to say that you’ll agree with it.

There are several significant industry trends that impact data, data management, and database administration that render manual practices and procedures ineffective. We’ll take a high-level look at the situation, and I will explain why these trends render automation of DBA to be no longer optional.

So, what are these undeniable trends impacting the need to automate? These three-bullets comprise multiple industry trends that I will discuss moving forward:

  • Data Growth; No DBA Growth
  • Speedier Dev: Agile and DevOps
  • More Platforms: Heterogeneity and Cloud

Data Growth; No DBA Growth

Data management continues to rise in importance as organizations amass and use more data every year. Combining this avalanche of data with more data platforms and database management system types adds to the complexity of ensuring optimal database systems and applications. At the same time, organizations are not adding more DBAs and data management professionals to their staff. And then add in the changing dynamics and speed of application development and the challenge increases.

Although most IT professionals accept at face value that the amount of data we store, access, and manage is expanding, we can easily cite industry research to back up the claim. IDC in their Global DataSphere study shows data growth through at least the year 2025, at which point IDC predicts there will be 175 Zettabytes of data. A zettabyte is 2 to the 70th power bytes, or 1 followed by 21 zeros. One zettabyte is approximately equal to 1 billion terabytes. IDC also predicts the compound annual growth rate (CAGR) for data to be 61 percent. That’s a lot of data! And a lot of growth…

At the same time, IDC also shows the market for Database Management Systems continuing to expand. Their analysis breaks down the DBMS market into three components: the largest by far is Relational/SQL database systems and it is projected to grow at a CAGR of over 8 percent. Then there is the Dynamic DBMS segment, which is what IDC calls NoSQL; it is expected to grow at a five-year CAGR of 30.9%. And finally, there is the non-relational DBMS segment; this is the legacy stuff and it is declining, albeit slowly at around 2% annually.

The bottom line is that the overall DBMS market is growing and will continue to grow with an aggregate CAGR of over 8 percent, reaching a market size of over $45 billion by 2022.

So, you might think that as the amount of data we manage grows that organizations would hire and train more DBAs and data management professionals to oversee the data? But you would be wrong.

Industry research firm Computer Economics indicates that data management staff as a percentage of IT staff has risen only .5% over a four-year span. That’s less than 1 percent while over the same four-year span the amount of data would easily have more than doubled. And if we look at just DBAs, the numbers are not growing either. The Computer Economics report titled Database Administration Staffing Ratios indicates that DBAs as a percentage of overall IT staff has dropped to 2.8 percent, down from over 3 percent in recent years.

And things won’t be getting any better any time soon. A few years back, QuinStreet Enterprise Research estimated that by 2020 organizations will generate 50x as much data, but the overall IT staff managing that data will grow only 1.5x during the same span.

These clear industry trends equate to a demanding, complex environment for managing and administering database systems. Yet DBAs are still required to monitor, optimize, tune, backup, recover, change structures, and move data both on-premises and to the cloud. This is one of the reasons that to be successful requires intelligent automation of DBA tasks and procedures.

Speedier Dev: Agile and DevOps

Another important trend for DBAs is that the speed of application development has increased significantly over the past decade or so, primarily due to the uptake of agile development techniques and DevOps. DevOps changes the application delivery lifecycle to a continuous loop, with continuous development, continuous integration, and continuous delivery of code. I’ve written about DevOps and its impact on DBA here on the blog before and I direct you to those posts for more information (DevOps and the DBA, DevOps, SQL Quality, and Performance, DevOps and Database Design and Change Management, and Automation is Key to Effective Database DevOps).

As we automate the process of software delivery and infrastructure changes with improved communications between Dev and Ops, the goal is to deliver any change anywhere throughout the organization into production in a quick and accurate manner that is safe, predictable, scalable, and controlled. This means that software improvements are delivered in smaller chunks more frequently, instead of waiting until the end of the project. When you implement DevOps it invariably comes with agile development. Agile development, as opposed to waterfall and other methods, is ideal because it encourages a continuous improvement cycle. And when implemented correctly, with everyone bought in and participating in the process, the result is that development moves faster than ever before.

But to work for both Dev and Ops, where the DBA is part of Ops, means that the entire development pipeline must be automated. There is a robust set of application development tools for orchestration and automation, but things have been a bit slower on the database side of things.

To achieve success, a synergistic set of database administration and management tools for orchestration and automation need to be adopted and integrated into your DevOps practices and procedures.

Again, automation is not optional here.

More Platforms: Heterogeneity and Cloud

For a long time, the DBMS market could be narrowed down to just a few leading vendors that dominated the market. Even today the DBMS market is still predominantly dominated by SQL and relational, but the database landscape is more heterogeneous than ever before.

You can gauge just how heterogeneous things are by taking a look at the DB-Engines (www.db-engines.com) site, which tracks the popularity of DBMS offerings. DB-Engines tracks over 350 different database systems!

So, DBAs must know traditional SQL database systems, but also NoSQL databases, which behave and are managed quite differently than a SQL DBMS. Automation is required to enable managing such a diverse set of database systems.

And we haven’t even looked at cloud adoption, yet. According to a 2019 study from 451 Research, almost two-thirds of organizations are currently pursuing a hybrid IT strategy, with 65% adopting a hybrid cloud approach that leverages both on- and off-premises resources. The takeaway here is that cloud usage and adoption is increasing, but don’t let anybody tell you that everything will move to the cloud. The future is hybrid with work both on-premises and off-premises, preferably managed in an integrated fashion.

So, what does this mean for DBAs? Well, it means you still have to know how to manage your on-premises databases, and all that entails, but you also have to add the cloud as a platform and understand the various ways that cloud database implementation impacts data management and availability. For example, DBAs need to become knowledgeable in cloud cost, budgeting, and licensing issues because of the different models used for cloud billing.

Another way to look at it is that some tasks can become more difficult, but some tasks (like patch management and basic backups) may be covered by the cloud provider. And you need to understand how your DBA tools provide cloud-enabled data access and administration.

The Bottom Line

To sum things up, it is important for organizations to recognize that there is a management discipline called database administration. And it is required in order to plan, model, implement, and manage database systems. It comprises a long list of important and required tasks, including these (and probably more):

  • Creating the Database Environment
  • Provisioning Databases
  • Database Integration into IT Infrastructure
    • On-Premises
    • Cloud
  • Database Design
  • Application Design for Database Usage
  • Design Reviews
  • Database Change Management
  • Data Availability
  • Performance Management
  • Test Data Management
  • Data Integrity
  • Database Security
  • Backup and Recovery
  • Disaster Planning
  • Storage Management
  • Distributed Database Management
  • Data Warehouse Administration
  • Database Utility Management
  • Database Connectivity
  • Database Code Administration

DBAs are being asked to do more… with larger amounts and more types of data… being accessed more rapidly and from more sources… without any prolonged downtime permitted… using and supporting new database types and capabilities… and with fewer DBAs as a percentage of IT staff than ever before.

I think this makes my thesis statement for this blog pretty clear and undeniable: automating database administration is no longer optional!

(c) 2021, Craig S. Mullins