Technology continues to increase the efficiency of our everyday lives. Take light bulbs, for instance. In my short life, a 60W incandescent bulb has been reduced to a 9W LED bulb. Eventually, technology reaches the point of affordability, which in turn increases the demand for the more efficient product.

Efficiency & Consumption

Efficiency gains lead to more consumption of a resource, as illustrated in the graph below depicting Jevons paradox.

Figure 1: Jevons Paradox 

I see Jevons paradox at play in the size of Atlassian's customers' home directories. The often-mistaken idea that "storage is cheap" is a common excuse to forego storage diligence. "Hey, just get more storage," they say. Data hoarding (currently 2.5 quintillion bytes of data per day!) extends far beyond the realm of Jira and Confluence, which are just one of many places where we collect and store our data treasures. However, I’ve thought a lot about the business impact of storing all of that data, and most recently, I have been contemplating the environmental impact of it as well (which I will get into later).

What Is Your Data Growth Rate?

The thing about year-over-year data growth is that it can't continue to infinitely expand when it consumes finite resources, with the largest limiting factor being disk access speed. For example, we want our Jira data to be quickly accessible, but as data compiles and takes up space, disk access speed slows down. Everyone expects technology to save the day when the status quo runs out, and there are some really interesting new ideas, like storing data in DNA, for ways to store information. Regardless, the growth rate of our data-sets is out-pacing our ability to store them.

With growth, we focus on doubling periods, and you may know that a doubling period = 70/(growth rate). So, if your 401k grows at 7%, it will double in 10 years, and if it grows at 35%, it'll double in two years. This works when you're making money, but it doesn't if you're spending it. Another important thing to note is that every doubling period is greater than the sum of all previous values:

Figure 2: Doubling value is greater than the sum of all previous values

The doubling quantity is greater than the total of all of the values that came before it (23 > 22 + 21 + 20 or 8 > 4+2+1), which means that in order to continue growing, one will need to consume more than ever before with each doubling period.

How is Your Data Serving You?

In my opinion, our customers overvalue their data and you probably do too. This is a result of habit-forming applications and people valuing their work more than that of others. Stop reading for a moment and ask yourself, "What data am I storing, and what has it done for me lately?"

For example, your Jira instances have been around for longer than a few sprints and most of your issues are closed, but you still keep them anyway. Once several years pass, Jira ends up being filled with closed or abandoned issues, which requires performance tuning and even more hardware to keep scaling. Some of that performance at scale is because you have big problems to solve, but not all of your issues necessarily bring you value. (We'd be happy to help you with scaling  - difficult problems are a good use of expert consultants.)

The overwhelming majority of your issues are closed. They will never be looked at, and they will never serve you. However, they do cost you real money. Here's where you say, "But when I need to look back at that one thing, then it'll be the most important issue we have." Will it? Are stories from sprints four years ago serving you in the present? If you are not mindful of the data that you are holding onto, then things get cluttered and the quality of your data significantly diminishes. Eventually, your data becomes the proverbial needle in the haystack: the more hay you store, the less likely you are to find the needle lost within it.

You can’t foresee how future technologies will utilize old data, but that does not justify the cost of keeping data you’ll probably never use. The real costs of data-hoarding adds up quickly in the form of:

• More complex software features

• Bigger, faster, and more servers

• Need to purchase additional storage

• Expensive engineers to squeeze out ever-diminishing returns

Ultimately, our systems suffer because they’re expected to perform optimally while storing an enormous amount of old data. All of the computer power in the world will never be able to outrun the pace of exponential growth.

The Cost of Your Data

Data hoarding results in real costs both financially and environmentally. Making our data centers more efficient only drives higher consumption. Increased disk density and speed only encourages us to store more data. Only we, the human beings, who fear the ramifications of the “delete” button, can control what we store to justify the cost.

Take a look at the environmental impact that data storage can cause:

• "In its 2013 sustainability report, Facebook stated its data centers used 986 million kilowatt-hours of electricity—around the same amount consumed by Burkina Faso in 2012." All of those data stories are probably 60% pictures of people's pets and 40% comment threads of people arguing with your aunt across the country. Again, low-value stuff. 
• "A 2015 report found that data centers and their massive energy consumption are responsible for about 2 percent of global greenhouse gas emissions, putting them on par with the aviation industry." Given my claim that most of this data no longer serves a purpose in active systems (not backups or other low-power media), holding on to it is comparable to flying empty airplanes around just so people can look for the neat, fluffy line across the sky.
  
  

Marie-Kondo Your Data

A general rule of thumb says that if you search for something that you recently got rid of, then you are doing the right amount of purging. I would advocate for doing something similar with your data. If you want a softer approach, then archive old data into AWS Glacier or some other accessible and affordable storage, and set a reminder to delete it later. If you haven't looked at that data in six months, it’s likely that you’ll never need it again. Trust your gut on this one, it won't steer you in the wrong direction.

Attachments and logs usually take up the most space, and you can use the handy tool logrotate to keep your log directories lean. Explore your home and shared home directories for the worst offenders that are clogging up your storage. 

Custom integrations are another source of inefficiency in large instances. It can get so bad that the standard recommendation is to relegate REST traffic to a single Data Center node so that humans don't have to suffer the performance impact. Scripts using the REST API are notoriously inefficient and poll far too often to get a pseudo-real time user experience. Monitor your access logs and work with your team of developers to encourage them to be better consumers. Event-based architectures are more efficient and provide high-quality data.

Here are some ways that you can do a data purge in Jira and Confluence:

Confluence

Apps like ViewTracker provide insight into which content is used. With this tool, you can at least archive, better yet delete, unused and no longer relevant spaces.

Jira

Closed issues, completed projects, and anything that is not active or still "warm" (e.g. items dating back to previous reorganizations) are unlikely to have any real value and should at least be archived, better yet deleted.

Thank you for making it this far. Now, take a deep breath, and let go of your attachments.

Resources:

(Fig 1) https://en.wikipedia.org/wiki/Jevons_paradox

(1) https://www.theatlantic.com/technology/archive/2015/12/there-are-no-clean-clouds/420744/

(2) https://www.youtube.com/watch?v=O133ppiVnWY

(3) https://www.youtube.com/watch?v=F8ZJCtL6bPs

(4) http://www.mnforsustain.org/bartlett_arithmetic_presentation_long.htm

(5) https://www.mic.com/p/the-environmental-impact-of-data-storage-is-more-than-you-think-its-only-getting-worse-18017662

I wanted to give you an update on our climate response progress and evolving position. My belief is that burning oil and planting trees to offset it cannot be considered a net zero action because essentially, we increase the amount of biologically available carbon that will impact the climate. This is because it moves long-sequestered carbon into available storage that is easily accessible. I believe that in the future, humans will be remembered as the liberators of carbon that made that world possible. Perhaps, we'll even still be around to take credit.

We cannot realistically eliminate our dependence on fossil fuels today, but we can start to reduce it. Praecipio Consulting is an atom within a droplet inside the entire ocean of human influence, so what carbon we directly reduce has very little effect on the planet. Because of this, we have implemented initiatives that invite our employees and peers to change the way they think and act towards the environment and our communities. We believe that a change in behavior ensures that some carbon will never be unnecessarily liberated and that effect will multiply through the years. Our thinking is that this will have a larger impact over time, even more so than the promise of direct offsets based on $1 trees. We may fall short of our goal of becoming carbon neutral by the end of 2020, but with each passing year, we will drastically reduce the amount of carbon that we release as a company.

How Praecipio Consulting Will Multiply Carbon Reductions Over Time 

Let me use two examples to demonstrate, and please poke holes in my logic. First, the frequency of flooding has increased as a result of climate change, which has affected both communities and their resources. In order to help prevent flooding, Praecipio Consulting has contributed to land conservancy, which will "forever," at least for the long-term, protect the river-adjacent, tilled farm land that once regularly flooded. Because of our efforts, the field's riparian buffer is being restored with native shrubs and trees. These plants help to slow down flood water and provide a number of benefits. Downstream communities now suffer less flood damage, the land absorbs the water that causes the river level to go down, and the buffer receives river silt, which fertilizes the land (to the tune of 0.5 tons of carbon per acre per year "forever"). Additionally, these shrubs and trees help clean the flood water, making life easier on its resident wildlife, like our beloved trout. For this project, we spent the absurd amount of $500 per ton of carbon. However, with each passing year, we will spend $0 per ton, not to mention that we simultaneously support riparian restoration, protect native species, and reduce the effects of climate change on downstream communities.

Second, we have promoted and encouraged at-home composting, which several team members have taken up. Composting at home makes people more aware of the food that they consume and helps them produce less waste. It also keeps food out of landfills, where it does considerable harm by converting to the potent greenhouse gas, methane. In fact,  40% of of the food in America ends up in landfills. To me, this is one of the most shameful realities of our modern world. If we simply stopped wasting so much food, we would be able to address food shortage, reduce oil usage, increase national security, prevent climate change, and more. Composting is something everyone can do, and collectively, we would make a huge impact. I have also written other articles on food composting, which you can check out here:

• Food Waste: The big picture
• The case against composting

The Green Team Influence

"I'm a system administrator and love computers. Until recently, I was content to sit inside, watch TV, play video games, and worry too much about system maintenance going horribly wrong. Then I bought a horse farm with my wife and was introduced to gardening and composting, thanks to Praecipio Consulting's Green Team. Over the past year, we have put in a six-bed garden and grown various different vegetables and flowers, including tomatoes, squash, various peppers of different heat, spices (basil, thyme, etc.), kale, sunflowers, lavender, mint, etc. We’ve started composting and will be using that in the garden this spring when we start planting again. We have an improved Meyer lemon tree on the way that we are excited to see fruit harvest from in a couple of years, in addition to a new adventure in beekeeping this spring. A two hive apiary is in the works, and we look forward to seeing what it grows into. Other future projects will include an orchard with various fruits and continuing our off-grid cabin, which already has a composting toilet and will get a solar power upgrade soon. Now, I find myself outside more often and feel more relaxed and creative, so when that upgrade fails miserably it is easier to find a way to move forward."

-Kris Hall, Platform Reliability Engineer

"We all know that I'm not an outdoors person. I tried twice before this past year to grow tomatoes and it never went well.  But I have always wanted to learn new ways to reduce my carbon footprint, especially regarding food waste.  Jess did a presentation about composting and it sparked my interest in the topic.  She started talking about worms, and I was certain that this was the end of my composting dreams.  During that time, I started collecting certain scraps and throwing them into freezer bags. We turned those scraps into cooking stock, and as an avid cook, homemade stock makes an impressive difference in a risotto.  I continued researching composting and found the bokashi container, which didn't require worms and could be stored in my house.  This was a perfect way to continue my goal of reducing food waste.  For my birthday, my husband bought me a keyhole garden, which not only requires less water, but it also has a space dedicated to composting.  I could take the funky mess from the bokashi container and layer it into the keyhole.  Our garden was amazing. It was so successful that it eventually became overcrowded, and we had to draw up a plan to expand this year's garden. And I can now grow tomatoes."

-Shannon Fabert, Principal of Managed Services and Hosting

"During the fall of 2019, I really started to pay attention to the green initiatives that the company champions. Prior to this time, my family was making sure to put cans and bottles in our recycling bin, but that was about it. Now, not only are we much more aggressive with recycling, we are starting to make purchasing decisions based on packaging and its impact on the environment. I know these are baby steps, but I’m happy about where we’re headed as a family. Who knows, maybe composting, gardening or rainwater collection is our next step. For now, what I do know is that we will continue to take steps improve our carbon footprint and influence our neighbors as much as we can."

-Larry Brock, Sr Technical Architect