It was red and ominously thin - US dollars, not the bulkier Chinese yuan? - and addressed to my youngest son.

''How can it be a bribe when we know you are leaving China?'' my friend protested, as we wrestled on the footpath outside my hotel and I tried to force him to take it back. I broke free and ran back to his black Audi sedan, and tossed the envelope down the far side of the back seat. I sprinted for my hotel as a younger colleague of my friend dived into the car and came after me in hot pursuit, red envelope in hand.

I bowled through a set of revolving doors, careered around a corner, slid across the polished-marble floor into an open lift. He was fast, however, and his aim was good. He dived towards me - sending hotel guests scurrying either side - and hurled the dreaded envelope through the closing elevator doors.

I banged the open button, just in time, and threw it back after him. It skidded Frisbee-style along the marble, directly through his legs. I slumped against the wall of the lift as it rose to the 18th floor.

But then I thought of my two other children and reached back into the bag. There, sure enough, were two more red envelopes. I returned to the ground floor to find the Audi gone and the original envelope still sitting there, untouched, in the middle of the hotel lobby - a great red memento to how much there was still to learn.

I came to China thinking I knew something about the place, having spent two years here as a child. I had warmed up by reviewing The Writing on the Wall, by economics writer Will Hutton, which argued that China's road ahead would become more difficult as the contradictions of dictatorship and market economics grew more acute. I dismissed his thesis – that the China model could not long survive without the values and institutions of the European Enlightenment – with the kind of certainty that can only come from being both young and not in China.

When Tara, two young children and I touched down at Beijing airport in July 2007, I was struck by how the air had changed. The fragrance of cooking oil and coal-fired kitchen stoves had been supplanted by a less identifiable photochemical pall.

We sped down a 10-lane airport expressway which had replaced the bumpy concrete road on which old men in faded Mao jackets had once led donkeys and trays of watermelons to market. The wheat fields around our old apartment at the Lido Hotel, where my brother and I had caught frogs and tadpoles in the ditches after the summer rains, were sprouting high-rise apartments as far as the smog enabled us to see.

It was not hard to see why the global price of iron ore had risen to levels that had not been seen before. For Australia – more than other developed nations – the abstract promise of rising China had arrived. It was my job to find out what was driving this industrial machine and where it might be heading.

I tried to stick to my Chinese lessons in those early weeks, but there was too much to explore. Beneath Beijing's shiny new facade it seemed the idealism, spirit of inquiry and sense of inexorable progress of the 1980s had been overshadowed by something bigger, heavier and more cynical. The noble spirits had been eclipsed but not crushed.

My notebook contained the names of a handful of terrific scholars who had worked closely with my economist father, Ross, since his time here as ambassador (1985-87). One had been quietly sending his students out in survey teams and discovering an uncharted universe of bribery and inequality, which he politely termed "the grey economy". Another was charting how the labour market was tightening, wages were rising, and citizens were exploiting a degree of economic empowerment outside the political system that they had not known before.

I was Fairfax's second correspondent, which afforded me the luxury of discovering the country in bite-sized pieces, while bureau chief Mary-Anne Toy chronicled the main story of China's journey to the Beijing Olympics. I sent out word that I was looking for a news assistant.

I met Maya Li, to use the pen name she later settled on, at the Stone Boat Cafe, on the edge of a pond at Ritan Park. It's fair to say she didn't meet the usual criteria. She was an overweight Muslim woman from western China, in a pork-loving patriarchal world where guanxi reigned supreme, who had no faculty for numbers, business or Politburo names. But, at the age of 29, she had fought harder for our profession than any journalist I knew.

Maya worked for a magazine in the Southern Media Group, which was owned by the Communist Party but based in one of China's most open cities, Guangzhou, across the border from Hong Kong. Its popular formula was both simple and subversive: to portray people as individuals, not props on a political stage.

Maya described her hero, the American war correspondent Ernie Pyle, who cared not for generals and their strategies, or facts and names, but for the souls of the men he sketched beside him in the trenches – where he later died.

Maya was the Ernie Pyle of Chinese journalism. She sketched the men, and particularly the women, in the trenches of China's industrial revolution along with those who were left behind. Why would she leave that noble project to work anonymously for a newspaper she had never heard of? She mimed the action of a vacuum pump which was sucking the oxygen out of the environment she had been working in. Some of her editors had been sidelined, sacked and even jailed. She would be my eyes, she said, as I navigated the new terrain. The fact that she spoke no English could only be good for my faltering Chinese.

I had a fairly simple aspiration, to tell Chinese stories through Chinese voices, and I thought the Western media did not say enough about the progress being made. When Maya spotted a story about two cousins who had survived a coalmine collapse, I thought it was just the kind of uplifting adventure story I was looking for. Instead, it proved to be the first day of my education.

It turned out that the flow of bribes was so powerful, accountability so non-existent, and official incentives so perverse that the reason the miners had spent six days under ground was because the local party chiefs had thrown their would-be rescuers in jail. If the miners survived, the illegal mine would need to be reported, which would benefit no one in this blackened web of interests. The cousins spent six days digging through 13 metres of collapsed tunnel and emerged, half-dead, only to have officials chase them out of town.

Maya was determined to restore to them some dignity. She jumped on an overnight train, and tracked the miners down at their home in Inner Mongolia. The village wake had morphed into a week-long homecoming celebration. Melbourne readers sent a considerable sum of money – which survived the postal journey after Tara sewed it into the padding of a stuffed toy – after reading that one of the miners had named his son "Tsinghua" after China's most prestigious university.

(Western journalists are protected from the rougher edges of the Chinese state but Maya could not afford the exposure. We talked about a pen name. She liked the passion of Sylvia Plath but I was anxious for her to find a namesake who had lived a little longer. That's when she began to appear as Maya Li, the deity of illusion.)

In our cosy diplomatic compound, a Pakistani neighbour, Shahid, introduced us to a warm and bright Shandong lady, Xiao Wei, to help look after the children. Our three-year-old boy fell in love with Shahid's daughter, enrolled in a bilingual preschool and developed asthma from the smog. Our one-year-old daughter modelled herself on Dora the Explorer and followed her brother into his new school and refused to leave. Tara immersed herself in the world of Chinese film. When the kids got home she would take them to catch frogs and tadpoles at the beautiful lotus ponds of Ritan Park, a short walk up the road. Our bathtub filled with amphibians and our balcony with assorted mammals, just like our Beijing apartment 20 years before.

As 2007 rolled on the swallows disappeared, the canals froze over and we enjoyed our first family Christmas in Beijing. Mum sent up her cloth-hung Christmas pudding, which we bathed in brandy and shared around at the Australian embassy in Beijing.

I ended the year being detained while reporting on an epic land dispute on the frozen black-soil plains near the Russian border. My heart was beating like a sparrow's until I learnt to follow the lead of Sanghee Liu, Mary-Anne's assistant, who sat beside me squashed up against the police car door but otherwise serene. Sanghee, an island of integrity and good judgment in a nation travelling at hyperspeed, had been in this position many times before.

Chinese history is often seen to move in 30- and 60-year cycles, in line with the zodiac calendar. After the Communist Revolution of 1949 and the beginning of the post-Mao reforms in 1978-79, the year 2008 arrived with momentous expectations and a tinge of dread.

In February, at Chinese New Year, the state-owned aluminium giant Chinalco crashed BHP Billiton's empire-building plans with a lightning raid for 9 per cent of Rio Tinto. My focus began to shift from economic progress and grassroots struggle to the mysterious workings of the party machine.

In March, urban centres of the Tibetan plateau exploded in the deadliest riots since 1989. The party-state was restrained in its immediate response but at a deeper, more primordial level, it kicked into a more unforgiving mode of self-preservation. The Department of Propaganda transformed the ethnic catastrophe into a nationalistic triumph for the Communist Party, defending China's honour against amorphous "Western hostile forces" who were conspiring to dismember China. The Beijing Olympics, just months away, morphed from a celebration of China's arrival on the global stage to something harder and more defiant.

In April 2008, Beijing-backed students battled pro-Tibet protesters and covered the lawns of Australia's Parliament House with red Chinese flags, setting a global pattern as the "sacred Olympic torch" snaked its way through London, Paris and San Francisco.

On May 12, the hanging lights at our Beijing office began to madly swing, signalling a different kind of tectonic fissure. We bolted for the airport and headed for the earthquake-raved carnage of Sichuan. In the ruins of Beichuan town we learnt to tread carefully over bodies camouflaged by dust. Some were well dressed and still intact. Others were squashed like insects by boulders pelted from mountains above. One, forever burnt into my mind, was hanging upside down, with his pelvis squashed flat between two car-sized boulders, and yet he was alive and had the strength to speak. I told him that if he was brave enough to hang on the rescue team would come for him, but that proved to be a lie.

On the highway above, battalions of People's Liberation Army soldiers were sitting in trucks, eating watermelons and occasionally acting in mock-rescues for the benefit of camera teams that were beaming propaganda footage across the country.

I arrived back home to Tara, who was pregnant with our third child, and she took me that same night to a resort with a pool down the bottom edge of China. I couldn't bring myself to talk, or look at the kids, but I could swim – as far out into Yalong Bay as I could physically go. I cried like I've never cried before or since.

Like a pitbull on a leash, Maya dragged me to places where journalists seldom go.

She travelled to Henan to cajole officials and businessmen, drinking Moutai for the first time in her life to the point that she made herself sick, until they relented and we pushed our way onto an epic migrant worker train ride to the cotton harvest in Xinjiang. We met poets, writers, artists, historians who, like Maya, refused to let the system eviscerate the individual.

Maya was brilliant, compulsive and difficult to manage. She should have been a star, illuminating the Chinese subaltern in her own language in her own name, not as my assistant. Everything in her professional and family experience had taught her a visceral contempt for authority.

Our partnership dissolved.

In August 2008 world leaders arrived to applaud the Beijing Olympics, the price of iron ore collapsed and financial crisis struck. I paused, as Tara gave birth to our third child.

In 2009 the Communist Party awed the world with its prodigious resources by reflating the Chinese economy, and Australia's too. Chinalco made an even more ambitious tilt at Rio Tinto, Canberra raised the spectre of Chinese military might in a new defence white paper, and bilateral relations plunged to the lowest point since the Tiananmen massacres of 1989.

In July I was in Xinjiang, covering an even bloodier round of race riots, when a contact rang to ask what had happened to a friend of mine, the Rio Tinto iron ore executive Stern Hu. I learnt he had been arrested and the iron ore wars had been elevated to a matter of Chinese national security. Australia was, again, the canary in China's coalmine.

Each night my head reeled with the implications of information absorbed but not fully digested. Where was China going? Was I doing justice to the story and getting the balance right? Grey hairs appeared, the skin under my eyes grew darker and I found myself jolted wide awake, bathed in sweat, in the pre-dawn hours each morning. With every road trip and each deep interview, I discovered afresh how little I'd known before.

I had seriously underestimated the extent to which the Communist Party had inoculated itself against the values and institutions of the European Enlightenment that underpinned capitalism in the West. The webs of patronage, bribery and thuggery that had so shocked me in Beijing's western hills extended deep into the political machine. The tools of coercion, co-option and censorship – so effective in revolution and keeping the party in power – were being deployed for the benefit of individuals within the elite.

Following the money in just one sector, coal, led to the mistress of a provincial governor, associates of a Politburo member, children of a premier and the son of a vice-president. I was drawn to the Yangtze River metropolis of Chongqing, where a maverick "princeling" politician, Bo Xilai, was waging a brutal war against the city's mafia by making it his own. "China is on a path towards becoming a mafia state," I wrote in a column of 2010 which, looking back, I'm a bit surprised I put in print.

When Chinese Australians with young families had their assets stripped and were put in jail, I did my best to make it harder for Canberra to pretend that these were mere "consular matters". Yang Hengjun (Henry Yang), a former diplomat whose charisma and flair for words had earned him millions of fans on the Chinese-language internet, told me that if foreign governments could not fight to protect their own citizens, by pressuring Chinese officials to uphold their own laws, then what hope could Chinese citizens have?

Yang lived with his family in Sydney but worked at the centre of a vast Chinese network of extraordinary journalists, intellectuals and activists. They were forcefully but strategically using their keyboards to lean against the state's resurgent power and challenge its monopoly on truth.

I experienced my own mini-earthquake when Yang "was disappeared", to use the passive phrasing that has evolved to evade censors on the Chinese internet. For those few days I worked furiously, desperately, and found it difficult to look my own children in the eye. Tara was relieved, for our family's sake, when he re-emerged.

China had arrived in the age of Sina Weibo. Despite severe restrictions, a virtual civil society was constructing itself where physical networks could not. My earlier grassroots explorations were now being made by hundreds of more capable Chinese including Maya Li, who rejoined the Southern Media Group.

I moved my tools to the virgin country of elite politics, where a foreign passport gave cover to investigate what others could not. The battle for China's future was rapidly spreading from the industrial trenches and Sina Weibo to the founding families of Chinese communism.

The pace slowed and venues shifted from frozen black-soil fields and coalmining tunnels to tea houses, cadre apartments and occasionally the old imperial courtyard houses of inner Beijing. It took months, and sometimes years, but gradually the "princeling" children of the communist revolution shared their histories and aspirations and introduced me to each other. Individually they shared a sense of crisis and a conviction that the system needed to change. Collectively, however, they were trapped in a cage of money, brutality, privilege and insecurity that offered no clear way out.

Bo Xilai, the great challenger, was vanquished. Xi Jinping took his place as the leader of the princelings and President of China.

Our children's ayi returned to her village, gave birth to a child and bought a car. Her replacement, Cui Qin, continued to delight in devouring the steamed white buns she could not afford as a child, even though her son had graduated from university and joined "China's Google", Baidu.

The stakes in China are getting higher, our bonds in Beijing are getting stronger, but the smog is getting worse.

I stopped in southern China to thank an official friend for sharing his knowledge and being a backstop for assistance if anything went wrong. We were genuinely close, and his princeling status gave him licence to speak freely, but his particular official status meant we were actors in a bigger game. "There is no functional difference between a foreign journalist and a foreign spy," his boss once bluntly told me.

After losing the battle of the red envelopes, and carefully recording 30 one-hundred US dollar bills, Tara laughed when she saw that the black leather satchel I had unthinkingly accepted was actually a Swiss beauty worth perhaps a thousand dollars.

The lessons and surprises keep accumulating. The cash has gone to charity. And the Bally bag, courtesy of the Chinese Communist Party, is up for auction at an illegal and lightly persecuted organisation: the Foreign Correspondents Club of China.