There is no magic, and you should explicitly acknowledge the fact
Technology is the application of scientific knowledge to produce practical results. Magic is the use of mysterious or mystical forces to achieve supernatural results. In other words, they are opposites.
Yet too often in common parlance, we conflate the two. Apps promise to wow us with “magical” user experiences. Pundits talk about algorithms wielded like “magic spells” by modern-day wizards, often to much financial benefit. AI startups promise machine learning models that give “magical insights” into the inner workings of your company. This muddling of the esoteric and the scientific is often a ploy involving equal parts marketing, obfuscation and hand-waving, to make the ordinary seem extraordinary.
There is a different but related magic vs. science association that many of us are familiar with. It is the inclination to imbue matters involving complex science or math with magical powers, as if your inability to comprehend the specifics of a calculus concept or an idea in theoretical physics is similar to a slack-jawed observer's inability to explain how David Copperfield managed to walk through the Great Wall of China or make the Statue of Liberty disappear.
Even though it may not seem like it at the time, the answer usually is that there's a simple explanation. Sometimes it's truly simple. A trick. Sometimes it requires a bit more background knowledge and effort to understand. But whenever you are utterly confused by a subject or unable to even grasp the basics, it is useful to tell yourself:
There is no magic.
This mantra serves as a reminder that, all thing being equal, it is more likely that you are missing a few facts and/or skills required to understand something rather than the alternative: that it is something beyond your grasp, a domain reserved for wizards with special powers. It may seem like a small point, but explicitly acknowledging that there is no magic in something you don't understand is a great step first step to understand it.
Intelligence and Magic
There are many theories and concepts that fall under the topic of Intelligence. Two of the most important and relevant general categories are crystallized intelligence and fluid intelligence.
Crystallized intelligence, as the name suggests, refers to the collections of words, ideas, concepts and skills that are crystallized in your mind, ready to be summoned on demand. Fluid intelligence, on the other hand, refers to the process that happens in your brain when you are trying to reason and form new connections. They are separate but related. The more crystallized intelligence you have, the more raw fuel you have to power the fluid intelligence process.
So how does this relate to our “No Magic” theme? When you are faced with a hard problem or a new concept, your fluid intelligence (hopefully) kicks in and tries to figure it out. If you lack the crystallized intelligence required to make the right connections, you may be left scratching your head. This is when doubts start to creep in: “Maybe I'm not smart enough.", “This is too hard.", “It's like a different language. I just don't get it."
How much effort you want to put in to learn the necessary facts to allow your fluid intelligence to reason and understand the problem at hand is up to you. But no matter how much you struggle to understand something, there is no magic so there is no reason to doubt your capability to do so. If it is important enough and you are willing to put in the effort, you can learn it and understand it.
It is not that I’m so smart. But I stay with the questions much longer. -Albert Einstein
How this helps with things you know
Since software is a domain I have some experience with, I'm not baffled or mystified quite as often as I used to be. I generally know what's going on under the hood. But when you know something, or you assume you know something, there is a danger of becoming complacent. As a software developer there are many times when I have found the No Magic mantra helpful in forcing myself to break through the endless levels of abstraction and jargon to truly understand some of the fundamental concepts of the craft. But having this state of mind is also a good way to challenge and keep yourself honest.
Most coders rely on programming frameworks, open source projects and standard libraries provided by the programming language. In most cases, these building blocks abstract away a lot of the underlying complexity in common operations and give us an easy API to work with. Sorting algorithms are a great Computer Science topic, but if all had to implement a sort algorithm every time a website needed a list of US states in alphabetic order there wouldn't be a lot of time to do more important things. For example, creating those magical user experiences. And when you spend a lot of time relying on frameworks, without taking the time to understand the inner workings, there is a natural tendency to just take for granted that framework X is going to do its magic and everything will work, as long as you use the right magic spell and/or API call.
This is where the part about challenging yourself comes in. If you really want to master the technology you are working with, you have to believe that there is no magic. You have to understand that no matter how many layers of abstraction you penetrate, no matter how broad or narrow your focus is, you will never reach a level that you are not capable of understanding.
As a personal anecdote, when I first started studying machine learning seriously, I remember several times when I was so surprised and amazed with specific models and demonstrations that I had the tingly This Must Be Magic feeling. Add to that the dense mathematical formulas that are sprinkled throughout the machine learning literature, and it can quickly seem like an impossible task to make sense of it all.
But as is the case when the magician reveals the trick, the truth turns out to be rather mundane. Once understood, the magic is gone. In the case of a magic trick, the how is usually the most obvious, logical answer, with a little bit of slight-of-hand and misdirection. In the case of software, machine learning and AI, the how usually involves lots and lots of calculations, done really, really fast.
It takes these very simple-minded instructions - ‘Go fetch a number, add it to this number, put the result there, perceive if it's greater than this other number’ - but executes them at a rate of, let's say, 1,000,000 per second. At 1,000,000 per second, the results appear to be magic. -Steve Jobs
That doesn't mean it's easy. You still have to do the hard work of figuring out all those connections, of breaking down those complex formulas and understanding bits and pieces, coming back later with newfound knowledge to make sense of the other bits and pieces that were puzzling in the past. But this is all part of the process. And the process starts when you decide to unravel the magic.
How this helps with things you don't know
The most complex subjects, things you simple do not understand, are within your grasp since, you know, there is no magic. This is a powerful realization once you have it. Once you actually believe it. It will help you make progress in your existing area of expertise. But it is especially useful when you find yourself in unfamiliar intellectual terrain. Some easy examples are subjects like physics, finance and neuroscience. Start reading about the latest discoveries in theoretical astrophysics or trying to decipher how complex financial instruments work and you may walk away feeling like an 8-year-old, hopped up on birthday cake, who just witnessed Dmitry The Dazzling pull a bowling ball out of his pocket. His pocket, folks! How did he DO THAT?! Mind. Blown!
In a future post I plan to cover some specific techniques you can use to quickly learn complex subjects. But before you decide to learn something, you have to first be motivated and convinced that you have a chance to be successful. There is no magic applies to brain surgery, but I'm never going to learn how to do brain surgery because A. I have no motivation to ever cut open a skull, and B. I'm not convinced there's a safe and practical “Learn Brain Surgery At Home” system I can use on my own time. If you are motivated to learn something and doing so doesn't require access to a living human brain, a particle accelerator, a $135 million military fighter jet or other such things, then all you need to know is there is no magic.
Magic via obfuscation
Sometimes the sense that something is beyond your grasp happens not because you lack concrete knowledge or raw intellect, but simply because the subject matter was not explained well or intentionally opaque. In this case, the appearance of magic is an illusion created by the intentional or unintentional obfuscation of the subject. In short, some people make simple things seem really complicated.
Even a basic idea can take on mystifying levels of complexity in the hands of a bad writer. When you are learning a new subject, the last thing you need to deal with are the shortcomings of lazy, insecure or ill-intentioned writers. Lazy writers are too verbose and can't seem to find a thought or idea that is not worth exploring in great detail and to confusing ends. Insecure writers lack clarity of thought and mastery of the subject, forcing them to be avoid clear, concise and bold sentences in favor of plausible sounding generalities that don't cut to the core. Ill-intentioned writes have ulterior motives in conflict with the obvious benefits of clear communication. They try to puff up simple statements by using big words and unneeded jargon to add extra complexity and the air of authority. The end result is that these kinds of writers tend to use the biggest words they can find, in the greatest number possible, arranged in most most contorted ways possible.
Take, for example, this anecdotal story from the world of finance. Famed investor Warren Buffett, known for his well written annual shareholders letters, was asked by a former SEC chairman to take a section of text from a mutual fund prospectus and translate it into the plain English.
Maturity and duration management decisions are made in the context of an intermediate maturity orientation. The maturity structure of the portfolio is adjusted in the anticipation of cyclical interest rate changes. Such adjustments are not made in an effort to capture short-term, day to day movements in the market, but instead are implemented in anticipation of longer term, secular shifts in the levels of interest rates (i.e., shifts transcending and/or not inherent to the business cycle).
Reading the paragraph above, it is clear that it has something to do with interest rates. Maybe. Also, whoever wrote this knows a lot about finance. We should just give them our money and pay whatever fees they demand, right?
Below is Buffett’s translated version
We will try to profit by correctly predicting future interest rates. When we have no strong opinion, we will generally hold intermediate-term bonds.
When you can understand it, it doesn't seem quite so impressive or complicated. The same is true for magic tricks, math, computer science or any other subject. So keep that in mind the next time something stumps you. Science has explained much of the natural world around us. Technology has put a lot of that scientific understanding to work for us. There are still things we don't know. Nobody knows everything. There are still great discoveries left to be made. But the way things are looking at the moment, magic is not going to be among them.