Our world runs on randomly generated numbers and without them a surprising proportion of modern life would break down. So, why are they so hard to find?

The friends did their best to explain what they were looking for – a really crackly radio. You know, the kind that blares a hail of static between stations as you turn the dial. The shop salesman didn't know what to say.

"I think he thought we were quite crazy," recalls Mads Haahr. It was 1997, and he and his pals were nosing around a Radio Shack outlet in Berkeley, California. Most of the radios on sale had noise filters that cut out the crucial crackles they were looking for. After a little persuasion, though, the salesman agreed to let Haahr and his friends listen to one of the cheapest devices he had in the back of the shop.

As they slotted a couple of batteries into the small, grey, $10 radio, a wall of buzz burst forth. It was noisy as hell. "We were jumping up and down," says Haahr. That little radio was, potentially, about to make them rich. Haahr and three of his friends had been working on gambling software – digital slot machines and blackjack games that they wanted to host online. Back then, online casinos were just beginning to take off.

Haahr knew that they would need to be able to generate reliably random numbers. Endless streams of digits that would determine what slots came up when you yanked the virtual lever, or what cards got dealt in an online poker game. If these things weren't random, the digital casino wouldn't be very fair and players could even try to beat the system by looking for predictable patterns in the games.

And so the four friends were seeking a source of randomness that they could all agree was trustworthy. Something that, by definition, couldn't be biased. The idea had hit Haahr like a tonne of poker chips – the hiss of a dirt-cheap analogue radio.

The noise blurted out by such a device is actually a messy signal shaped by lightning and electromagnetic activity in the Earth's atmosphere. That's what generates the blast of unpredictable static. Haahr had planned to have a computer listen to all that gibberish, convert the ups and downs into little ones and zeroes, also known as bits, and then use that to produce strings of random numbers – something like 4107567387. Which, to take a simplified example, could determine that the next card you get dealt in your online poker game is, say, the six of spades.

(Oh, you were holding out for a five of clubs. Didn't you know? You gotta know when to fold 'em.)

There are some things that computers, for all their prowess, don't do well – and one of them is randomness. Sure, computers spit out data all the time, why not random numbers? The problem is that computers rely on internal mechanisms that are at some level predictable, meaning the outputs of computer algorithms eventually become predictable, too, which is not what you want if you're running a casino. The same issue can cause headaches for cryptographers. When you encrypt information, you want the keys to the code to be as random as possible, so that no-one can work out how you garbled the original text since that could allow them to read the secret message.

People have long sought external sources of randomness as the basis of random number generators. In this search for true randomness, they have looked practically everywhere for chaotic phenomena that can't be predicted or manipulated. They have listened to the racket of electrical storms, captured pictures of raindrops on glass, and played with the tiniest particles in the known Universe. The search is far from over.

In the end, the little grey radio didn't make Haahr and his friends rich after all. The online gaming business was too much in its infancy back then for the young entrepreneurs to make a killing out of it. But the random number generator they built was, they reasoned, still useful. So Haahr made it public at random.org, where it has been churning out random numbers ever since. It gets a lot of visitors.

"One I can mention is the San Francisco Mayor's Office," says Haahr. "They use our service to draw winners who are lucky enough to get affordable housing."

Other users include people who run local community lotteries. They choose the winning numbers every week on Haahr's website. Scientists have even used the online number generator to randomise participants in experiments. Marketing firms that give away prizes to consumers have also chosen their winners with the help of random.org. "People use it for drug screening, for example," adds Haahr. "Selecting employees randomly." The site also has a facility for choosing a random password. And yes, some online gambling services rely on random.org, too.

One man even says he turned to it to help him choose which discs from his 700-strong CD collection to put into his car each week.

All of these results, including, in some cases, life-changing outcomes of draws or selection processes, are derived from atmospheric activity picked up by a bunch of radio receivers. The original $10 radio has long been misplaced, confesses Haahr. Over the years, he and his colleagues set up more advanced equipment to yield high-quality atmospheric noise and, currently, the site relies on nine large receivers in different geographical locations.

If you never enter the lottery or have no intention of taking part in a clinical trial, you might think that random numbers don't really matter to you. But you'd be wrong.

Every time you choose a new password, even one you think of yourself, a computer adds another chunk of data to it. This scrambles the password for storage meaning that, if someone hacks a database and steals your password, they can't easily unscramble it and use it to access your account. That chunk of data added to the password is called a salt and it is derived from a random number.

"We're using them every day without really knowing," says Alan Woodward, a computer security expert at the University of Surrey.

Random numbers are essential for securing information, he points out, since they are used whenever a computer encrypts data so that it looks to any casual observer like mere gobbledegook. When you visit a website that begins "https://", for example, you're asking your computer and the server that stores the website to, among other things, generate some random numbers, exchange them and then use them to encrypt the data sent back and forth as you load the website and go on to use it. (This is an enormous simplification of the exchange that takes place in just a few milliseconds, but you get the drift.)

When sensitive data is flying around computer networks, especially those accessible by the public, it's essential to secure that information. Cloudflare, a tech firm that provides cloud security services, uses a lot of random numbers at its data centres. The company has sought some eye-catching ways of generating randomness – including a collection of lava lamps.

"It almost started as a joke and then we said, 'Wait a minute, we could actually do that,'" says John Graham-Cumming, chief technology officer. At Cloudflare's office in San Francisco, a camera on the ceiling is pointed down towards a wall of colourful lava lamps. "The motion of those lava lamps, the blobby, oily, waxy thing inside, is not predictable," explains Graham-Cumming. That means that the camera can take pictures of the blobs at intervals, which are then converted into a series of numbers – more or less based on the shape and position of the blobs. This idea, first demonstrated by a company called Silicon Graphics in the late 1990s, is one of multiple methods that Cloudflare has up its sleeves for generating random numbers to secure information.

The list of things people have turned to in the pursuit of random numbers is deliciously odd. One software engineer realised when out walking on a wet day that the raindrops falling on his glasses might provide a suitable source of randomness, so he simulated the patterns via a short piece of code as an experiment. Someone else tried capturing the activity of bubbles in their fish tank as a basis for random number generation. Another approach relied on the unpredictable behaviour of a kitten, its movements having been captured on a webcam.

One group of researchers even sought randomness by looking at the unique genetic sequences contained within DNA molecules inside all living things.

Then there are the many efforts that involve detecting the clicks of radioactive decay, picked up with Geiger counters, for example. You can't predict exactly when an atom will decay and release subatomic particles that set of the Geiger counter, meaning this is considered a good source of randomness. There's even a project that employed a banana to make a number generator – since the bendy fruit is known to be mildly radioactive.

In 2016, the New York Times mentioned the example of a young computer gamer who found a way of making a kind of random number generator in Minecraft. The gamer had constructed a sort of tournament for his friends within the world of Minecraft but he wanted to be able to trigger a trap at random intervals – to try and catch out contestants in the game. And so, he decided to build enclosures and within them enslave virtual cows called mooshrooms because they walk around in a random pattern. Whenever a mooshroom stepped on one of the pressure plates the gamer had installed in the floor of their enclosures, that would trigger a trap in the game-within-a-game that he had designed. An "ingenious" solution, as the New York Times put it.

Murdoch argues that we know enough to generate suitably random numbers but that there often isn't enough verification to ensure that a supposedly random output really is unpredictable. "The thing that I'd like to see more of is careful design and testing," he says.

But that's the really tricky part.

"You can't tell if something is random – you can only catch it out as non-random," explains Darren Hurley-Smith, an information security lecturer at Royal Holloway, University of London. Researchers have ways of identifying non-random number generators, such as statistical analyses that look for sequences where one number comes up more than it would if said sequence were truly random – too many threes, for instance. This isn't enough, though. What if your sequence is "123123123123123"? In that case, Hurley-Smith points out, you have lots of threes but they're no more frequently represented than one or two. However, there's a super obvious pattern to the sequence, an underlying structure, that gives it away as non-random.

For really big sets of numbers, statistical tests for randomness get very complicated and yet you can never prove that something is truly random, only that it is indistinguishable from random, based on your best analysis.

The consequences of getting randomness wrong, however, are serious.

When random number generators don't do their jobs properly, you can expect that malicious people might try to exploit them. In 2017, Wired reported on the case of a Russian hacker who allegedly got people to film the activity of slot machines at casinos. Based on the results of each play, he was able to predict the workings of the machines' internal random number generators and, therefore, determine when they would next pay out.

And about a decade ago, security researchers at the University of Cambridge, including Murdoch, famously realised that scammers were able to steal money from people by predicting the supposedly random numbers generated inside chip-and-pin cards. Such numbers are required to authenticate a transaction when you pay for something in a shop. But it turned out that it was possible to predict upcoming numbers and thus impersonate someone's debit card in order to spend their money without their authorisation. Such scams are still likely being perpetrated today, according to one of the researchers who discovered this flaw.

Even if you account for bad engineering, most of the random number generators mentioned above could still be a little more robust. The sources of randomness they depend on are, at best, just random to us. If we somehow knew every intricacy of the Universe, and could model it perfectly, we would likely be able to predict the detailed hubbub of atmospheric noise, or the pattern of raindrops falling on glass.

As such, some people argue that the best kind of random number generator is a quantum random number generator – that is, one that relies on quantum mechanical effects. These are, as far as we can tell, are as random as it gets. The weird behaviour, or entropy, of subatomic particles, including the timing of a single radioactive atom's decay, for example, are completely unpredictable. There's some discussion at to whether true randomness really exists anywhere but we can leave that to the theoretical physicists.

For practical purposes, things like the timing of photons – tiny particles of light – arriving at a detector is something that has been considered truly random and suitable as a basis for generating random numbers. Another approach involves counting photons emitted by a laser pulse, which in a special experimental setup will yield a randomly odd or even number of photons. By doing this again and again, you can generate a string of random bits and use that to churn out random numbers.

In principle, quantum random number generators are "intrinsically unpredictable", says Zhanet Zaharieva, co-founder of UK-based firm Quantum Dice, which is developing its own quantum-random-number-generating technology. And yet even some of these newer systems are susceptible to flaws that can bias their outputs. "What you end up having is a system that is a mixture of quantum entropy […] and classical noise," says Zaharieva.

Hurley-Smith notes, for instance, that some quantum random number generators rely on equipment that might make them less random over time. "If a photon hits the sensor, it will ever so slightly warm it up, possibly making it more or less sensitive to future strikes," he explains. In other words, you have a truly random phenomenon – but you'll end up skewing your detection of it one way or another, reducing the randomness of the output.

With the development of quantum technology, we can expect to see even more inventive and, hopefully, reliable random number generators in the future.

In the meantime, there's always random.org. Sometimes people write to Mads Haahr, complaining that they have spotted what looks like a predictable sequence of numbers in the site's output. That's really just because humans are very bad at recognising randomness, he says. We seem predisposed to see patterns in everything. Haahr enjoys taking the time to reply and explain these concepts in more detail.

He might not have set out to create a resource like random.org originally, but over time Haahr realised it was something he wanted to do. Perhaps, he suggests, it's because he is originally from Denmark – regularly ranked among the least corrupt countries in the world. Now, people use his website in order to convince their users, participants or audiences that they are tossing a fair coin. That whatever game or process they're running is an honest and just one – at least within the boundaries of randomness.

There would be, it must be noted, a rather large incentive for Haahr and his colleagues who collaborate on random.org to manipulate the output of the website if it allowed them to rig the result of a lottery draw, for example. They would never do that, Haahr insists, as it's against everything they stand for. But it's a fair question because, as he freely admits, it is possible.

When the chips are down, no matter how exquisite a random number generator is in principle, you still have to trust that the person running it hasn't lost their scruples.

“Always, I guess, some trust is necessary somewhere in the process,” says Haahr. “You have to believe that.”