How to use historical stock market data to build your investing intuition, and move one step closer to becoming an investing superhero
On Monday, October 17, 1987, the S&P 500 dropped a little over 20%. Going back to 1950, this was the single worst one-day drop in the stock market. It showed and taught a lot then, and it can still teach market participants and investors today if they are willing to listen.
Historical financial data can be magical - it can help you travel into the past and see the forests from the trees. By listening to historical data, we can more easily understand that a single-day drop of 20% in an index such as the S&P 500 is possible. By knowing that a 20% stock market drop is possible -- and by seeing the number present itself to us in the data -- we can better understand the risks we face by investing and participating in the financial world.
How do we know whats or of stock market drops are possible?
What exactly does possible mean in the context of investing and considering severe market declines? Sure, we know it's "possible" for the S&P 500 to drop by this amount or that amount. But until you ground your understanding in some historical data, you're not going understand this possibility at a deep level.
Observing the S&P 500's daily price movements will help you learn what sort of severe drops are possible for your portfolio
First, a quick note on using the S&P 500: The S&P 500 is a great place to start because it's a reasonable proxy for the market. The S&P 500 surely doesn't represent the entire market of equities, financial assets, and let alone of all assets; but, it's a reasonable and easily-manipulatable proxy for "the market."
If we care about how bad things can get in a single day (eg. an extremely severe yet plausible one day decline in your portfolio), we'll want to look at daily price data. This type of data is readily available online. There's a lot of free data, but you'll have to pay to access longer time horizons or more esoteric data.
Below, we have S&P 500 data April 3, 1950 to September 6, 2019. This represents an almost 70-year time horizon, a bit less than the expected lifespan of a person today. You can see this in the screenshots below (bottom and top of the table shown; table ordered earliest to latest).
The data has three columns - one shows the date, the other the closing price of the S&P 500 on that date, and the last column is the day-over-day change int he S&P 500 stated as a percent. The day-over-day change is easy to calculate - it's merely the one day change divided by the previous day's closing price. Declared as a formula, it is: [Day 2 - Day 1]/Day 1
This is called the arithmetic return. More complex return types -- namely the geometric return -- exist, but they are outside the scope of this discussion. The simple arithmetic return above is sufficient for our purposes.
Observing financial market data can teach you a lot and help to build a bit of market-related intuition
Just observing stock market data like this can be useful. Exploring data visually without graphing it can give us some interesting and potentially-valuable preliminary insights. This is especially true for people who haven't done this sort of data analysis before. For example, we can observe that in the very beginning of our data set (the first few days of April 1950), the S&P 500 was around $18. This is in sharp contrast to the almost $3000 S&P 500 level we observe below for September 2019. This plainly shows us that there has been some dramatic growth over the last 70 years in absolute metrics.
Although we can see some things by observing the financial data, it's hard to determine summary statistics about the S&P 500 data set visually. Stats like mean, median, minimum, and maximum are hard to see because all of the data needs to be taken into account. Taking all of the data into account can't easily be done relying on the human mind - it's just not what it's made to do. The data set has almost 17,500 rows - that's simply too much to comprehend without the use of computing devices/methods (or without a considerable amount of time to devote to this endeavor). Luckily, such devices/methods are easily available for free (eg. Google Sheets) or cheap (eg. Microsoft Excel) in the form of software. More complex options are available that are both free and paid (eg. R, Matlab, Tableau, etc.). Something like Microsoft Excel would be enough for the vast majority of use cases, however.
Finding the minimum, or the worst stock market day over the last 70 years
Some relatively easy functions can be used in Excel -- the tool of choice for most finance people -- to get some stats on the data. In the screenshot below, you can see the average, the max, and the min. The min is what we care about most here - it represents the lowest day-over-day S&P 500 change; it represents the most severe single-day drop in the S&P 500 over the last 70 years.
We can see that the worst drop is -20.47%. That means that in one single day, the stock market effectively dropped by over 20%.
Black Monday - an over 20% one day drop in the stock market
We can see the drop occurred on Monday, October 19, 1987, by examing our data set in greater detail. By filtering the data from largest to smallest, we can see what date corresponds to the worst stock market drop. Once we have the date, we can observe what happened around it in the days before and after Black Monday, which is what Monday, October 19, 1987is called in the financial industry.
The image below is the copied and pasted data from around Black Monday. It's interesting and useful to observe what happened around that time. We can see in the 10 days around Black Monday, 7 out of 10 days were losing days. We can also combine the losses to see what the cumulative loss over the 10-day period would have been. We can see that it's even worse - over the 10 days, the stock market dropped over 26%.
That means you could wake up one day over the course of your investing life and see that your portfolio is down 20% in a single day. That event would be tough to deal with - you'd be in for a very rough day and rough week. It would take some time to recover from the loss, but recovery would definitely be possible. A mistake, however, would be to panic and deviate form your long term investment strategy for no real reason beyond the fact that you're freaked out.
Use this knowledge to avoid panic sales and other forms of freaking out during the next inevitable stock market disaster
We all get freaked out in investing - it's your money that's on the line, and you don't want to lose it. Even small drops can seem bad. Even times where there's no movement could be perceived as bad if you were anticipating gains. You can't let these investing difficulties make you make investing mistakes, however. You've got to do your best to maintain a long-term perspective on investing. Something that helps us do that is exercises like the one we just went through. Looking at historical market data, understanding how the markets have moved over time, and understanding how markets may tend to move int he future are all essential things that will buffer you from foolish investing mistakes made out of fear.
If you'd like to explore the Excel file form which the above screenshots originated, you can download the file here. You'll be able to copy and paste the S&P 500 data to do your own data analysis work, like finding the maximum one-day increase.
Dow Jones Industrial Average Components Over the Decades: A view of the changing face of enterprise in the United States
The Dow Jones Industrial Average is one of the longest-running stock market indexes in the world. Its components have changed since inception - they've changed 51 times since the inception of the index by Charles Dow.
Looking at the Dow Jones Industrial Average's (or simply the Dow's) components over time allows us to see how American business (and the world in general) has changed over the last century and a half.
We won't go into all 51 component changes here -- you can find them here if you'd like -- but we will focus on the most interest and relevant ones and discuss them in a bit more depth than you can find elsewhere on the internet. Instead of giving a cursory overview, we'll dig a bit deeper to see what underlying changes were the root causes of the changes and in the process, we'll gain the following benefits:
The Dow on July 3, 1884 (precursor)
The initial Dow (which wasn't properly the Dow Jones Industrial Average but was instead a creation of Dow called the Dow Transportation Average) consisted of the following:
As the original "Transportation Average" name should indicate, the original Dow components were heavily focused on transportation. We can clearly see that there are a lot of railroad companies represented in the initial Dow mix. In the 1880s, railroads had been around for a few decades, but they still represented the new and happening industry - similar to how technology is today fast growing and focused on thing in business even though computers have been around for a few decades already. Railroads represented Manifest Destiny and a new industrial era where lots of money was being made in the business of moving things from one place to another.
We see that 9 out of the initial 11 firms represented in the 1884 Dow were railroad companies - that's a very large representation and should clearly indicate the importance of transportation generally (and railroads specifically) in the pre-20th Century US economy. As the country moved westward and as more and more goods were in need of rapid transportation in the post-Industrial Revolution era, railroads were able to extract very healthy nominal and real profits.
Basically, a discussion of the early years of the Dow inherently is a discussion of railroads. The first public railways opened up in the US in 1830 using steam engine - by the 1880s, technology had improved as did ridership and a need for transporting goods in a new type of economy where self-reliance was beginning to give way to mass consumption and production.
The equivalent today in terms of industry would be seeing all tech firms dominating the Dow Jones Industrial Average - imagine seeing the Dow today composed of the likes of Google, Facebook, Microsoft, Oracle, Salesforce, Twitter, Apple, HP, Dell, Cisco, etc. An observer would think that the US economy was heavily dominated by tech. Luckily for us, today's economy is far more diverse than the industrial and transportation economy of the late 19th century - we have large industrial firms, firms involved in chemicals, firms involved in telecommunications, firms producing basic products, firms that primarily provide services (eg. consulting firms), etc. Today's economy is as diverse as any has been in human history.
May 26, 1896 (the first proper Dow Jones Industrial Average)
The first proper (non-transportation only) Dow Jones Industrial included the following firms:
This was the first real Dow Jones Industrial Average. Here we see many of the railroad companies replaced - only two of the firms (the Northern American Company and the Tennessee Coal, Iron and Railroad Company) are firms heavily involved in transportation.
We can see that the list has now moved away from transportation and is focused important necessities for late 19th Century America. Things like cotton, oil, tobacco, cattle feed, coal, iron, leather and rubber are all represented - these basic necessities were key to a life that was moving away from self-reliance on farms and into a mass-produce economy that required energy (in the form of gas, oil, and coal), straps, linens and other fabrics, heavy metal, electricity, etc.
If the Dow had existed 500 years prior in the Middle Ages, things like electricity, leather, cotton, coal, and rubber would not be there - most of life would consist of cattle feed and other types of feed.
Another interesting thing to note is that the names of these firms are quite basic - they are literally are names of what the company produces. Can there be any doubt that the Tennessee Coal, Iron, and Railroad Company is involved in the production of coal, iron, and railroads? Would you be surprised to find out that the United States Rubber Company produces rubber? These firms were the first of their kind - they are representations of commerce and big business in an era that had only recently exited the darkness of the Middle Ages via the Renaissance. The unique names we see that not only don't represent the firm's products or services but sometimes are not even traditional words that humans have used are only possible in a world that understands what firms are - world filled with people used to branding, buying things from companies instead of from friends or family, and have a lot of trust in business and capitalism in general. The ability of firms to market and brand themselves in order to educate the public about their products and services allows firms today to eschew the basic naming conventions of the past and to use innovative and obscure names such as Twitter or Exxon. A Twitter or an Exxon would be strange in the early years of the Dow - no one would have any idea what these firms produced. Without the ability to create an image of the firm through the use of advertising (which requires a lot - print ads, TV, radio, the internet, etc.), firms would who used strange names would find themselves at a deep disadvantage in the past. It was a far smarter idea to make sure people knew what your business did just by reading the name.
October 1, 1928 (Dow expanded to 30 firms)
As the index expanded to contain 30 firms (the size it's been ever since), the Dow was comprised of the following firms:
Here, the index was expanded to the 30 firms we have today. This was an interesting time in the history of the United States and especially its economic history. The Roaring Twenties were coming to a close and little did anyone knows that the Great Depression was right around the corner.
Here we can see the that we have a few automotive firms represented - we've got General Motors, American Car, Mack Trucks, and Nash Motors. Car companies have come on the market and are now some of the largest firms in the country. A car firm at this time would be similar to seeing the edition of technology and internet firms in the 2010s and 2020s - the firms came up over a few decades and finally took their place among the largest in the US by playing in a new and important industry.
We see that the names here are still those basic names that hearken back to an era before sophisticated marketing and advertising and before readily available means of communicated such as radio, TV, and mass color print.
July 3, 1956
This is the first Dow changes after the US entered WWII - the previous Dow adjustment occurred on March 4, 1939. Since we last saw the Dow above (1928), the US had plunged into a decade-long economic downturn called the Great Depression, entered WWII (which helped it recover), and saw droves of new babies being born in post-war America (the Baby Boom). Let's see how the Dow has been affected:
Here in 1956, we can say that we are in a totally different America. The last time we checked in was in 1928 - almost 30 years later the Depression-era youths fought a war abroad and came back home to have a ton of babies. Although key staples remain in the Dow, we can see the addition of many new firms.
We can see a big variety of firms represented here - car companies, companies producing basic materials, food-related companies, retailers, energy firms, and even a photography company in the form of Kodak. In 1950s America, technology has advanced far enough to make consumer products (photography, cars, retailing, toiletries, etc.) major parts of the economy. A Procter & Gamble wouldn't exist just 50 years prior - people didn't have the disposable incomes to shower often and use toiletries nor did they have a desire to in their mostly self-reliant forms of living. In 1950s America, a firm producing household necessities would make a lot of sense. In the same light, in 1950s America, big retailers, big tobacco, and big car companies all make sense - our conception of that era is of one that has now moved way past the agrarian roots of the United States and now is in the realm of post-WWII technology and sophistication. If you had told the people living through the Great Depression that a photography firm (Kodak) would be one of the biggest in the country, they would have scoffed and not understood why - photography was a luxury and the technology was not all there yet. The same can be said about many things represented above.
August 9, 1976
Jumping forward another twenty years, let's see where this journey has brought us:
In these 20 years, surprisingly little has changed. Only a 5 firms were replaced since the last time we checked in in 1956. By comparison, there over 30 changes from 1928 to 1956 (some back and forth). What you have in this period is a stable period of growth, some merging of firms, and a movement away from those classic self-descriptive names to the more unusual firm names we know of today.
Look above to see the firms that remained in the Dow but changed their names - International Nickel became Inco, Texas Incorporated became Texaco, Swift became Esemar, and Standard Oil of NJ became Exxon. These movements are away from names that clearly state what the firm produces to more esoteric and strange names that don't provide any indication whatsoever about the firm - clearly a reliance on marketing, adverting, and branding is required in order to educate the public and create a mental picture of the firm when such strange names are used. This is possible because we are no in a world of color television, radio, print magazines, and other forms of advertising.
This trend has continued today where almost all new and interesting firms have absolutely strange names that give zero indication of what the firm actually does - names such as Twitter, Facebook, Yelp, Apple, etc. If you took a person from 1850 and asked him to whet he or she tough a firm named Standard Oil or a firm named American Can do, he or she would very likely guess correctly. If you asked the same person to describe what he or she thought a firm like Exxon does (bear in mind this is Standard Oil with a name change), they would have no clue and rightly so because Exxon is a totally made up term. As stated above, such strange names can only work in a modern world filled with modern telecommunication systems and a general populace that is receptive to advertising and marketing.
March 17, 1997
About two more decades after our last stop, a lot has happened - the Cold War is over and we're at the apex of the 1990s economic boom. Here are the Dow components now:
Here we can see more name changes (eg. Chevron and AT&T), continuing the overall movement away from the simple names to the more strange and esoteric ones that require branding.
We can also see that some of the old components (eg. AT&T, Chevron, Exxon, Union Carbide, General Electric, General Motors, Minnesota Mining, Sears, and Union Carbide) still here - times have changed but these good firms have endured for a variety of reasons. Some endured because of good management (eg. General Electric), some because of early entry and the existence of various barriers to entry (eg. General Motors), and many because of luck.
It's interesting that even though we're in the heart of the proliferation of personal computing and the internet, there are few technology firms. This makes quite a bit of sense - it takes time for these new firms to grow to a size large enough that would put them in the Dow (the top 30 firms in the US). Firms like Apple, Cisco, Microsoft, and others might have been making big moves during this era, but they were still growing. IBM and HP are present because they were around longer - IBM was around for almost a century at this time.
November 1, 1999
About two more decades after our last stop, a lot has happened - the Cold War is over and we're at the apex of the 1990s economic boom. Here are the Dow components now:
With the addition of Intel, Microsoft, and SBC, we can see that in just about 2 years, the Dow has taken on some of the tech firms. These firms have grown in market cap by now (due in part to what would later be called the Tech Bubble) and had market caps large enough to allow placement within the Dow. The Dow here contains many old stalwarts but is filled with new firms that were either started within the last few decades or came to major prominence recently.
March 19, 2015
The last Dow Jones Industrial Average change happened in early 2015 - here is the current makeup of the Dow:
In today's Dow, we see the familiar firms that make up the market share and the mind share of the US economy today. We see an almost complete transition away from the simple self-descriptive business names to esoteric and strange names that require branding - compare this final list with the first list. Firms like National Lead, Tennessee Coal, United States Leather, and others are so clear in their descriptions while firms like Visa, Pfizer, Nike, and Apple would be totally obscure if not for branding and advertising.
Another interesting thing is the rise of big pharmaceutical and healthcare firms - firms such as Merck, Pfizer, and UnitedHealth have come to major prominence due to various large-scale factors. These factors include an aging population (Baby Boomers), a more wealthy economy that can spend more on healthcare, and the success of pharmaceutical research in producing new, innovative, and expensive drugs.
We also see the tech firms playing a bigger role - Cisco, Apple, Microsoft, and Verizon are all part of the overall technology economy, helping to provide hardware, software, and telecommunication services.
And now, given the rise of cyrptocurrencies and crypto assets to quasi-mainstream financial assets, we're dedicated to providing quality, relevant, and interesting material on cryptocurrencies and cryptoassets. Articles on Bitcoin, Ethereum, Ripple, Cardano, and many more cryptocurrencies and cryptoassets can be found on Pennies and Pounds - all that in addition to a plethora of information on what cryptoassets are, how the entire crypto industry came to be, blockchain/immutable ledger technology, mining, proof of work, proof of stake, and how to prudently invest in crypto if you are so inclined (based on your risk tolerance and ability to withstand the volatility that will come with a crypto portfolio).