Average Treatment Effects (part 1)

These are my notes on Chapter 18 of Woolridge (2002).

With randomized treatment, the average treatment effect E[y1 - y0] is the same as the treatment on the treated E[y1 - y0|w =1]. Intuition: because treatment is random, the sample of treated people reflect the population.

The Mean Independence assumption (E[y|w] = E[y]):

    E[y0|w] = E[y0]

    E[y1|w] = E[y1]

The means of y are independent of w. These hold when treatment is independent of who benefits; it's random. This is slightly confusing because y0 is also the counterfactual that doesn't exist for those that are treated.

If y1 - y0 is not mean independent of w, ATT and ATE generally differ. 

From the weaker assumption of just E[y0|w] = E[y0] , a difference in means will estimate the Average Treatment on the Treated. i.e. E[y0|w=1] = E[y0|w=0]. 

Those that aren't treated would have had the same outcome w0 as those there were treated. Dubious as y0 is likely to influence w.

Ignorability of Treatment (Propensity Score Matching)

When w and (y0, y1) are allowed to correlated we need an assumption in order to identify treatment effects.

Assumption 1: w and (y0, y1) are independent, conditional on x.

Assumption 2, conditional mean independence, also known as selection on observables:

    E[y0|w, x] = E[y0|x] and E[y1|w, x] = E[y1|x]

Even though w and (y0, y1) may be correlated, they are uncorrelated once we partial out x.

Under the second assumption, conditional mean independence, ATE and ATT conditional on x (ATE(x) and ATT(x)) are identical.

E[y1-y0|x, w=1] = E[y1-y0|x]

A digression into regression methods

y = y0 if w = 0 and

y = y1 if w = 1


y = y0 -  w*y0 + w * y1 

y = y0 + w (y1 - y0)

Now, add in conditional mean independence.

E[y|x,w] = E[y0|x, w] + w * (E[y1|x, w] - E[y0|x, w])

             = E[y0|x] + w * (E[y1|x] - E[y0|x])

because we can drop the w's.

Now substituting w = 0 and w = 1

E[y|x, w = 1] - E[y|x, w = 0] = (E[y0|x] + (1) * (E[y1|x] - E[y0|x])) - (E[y0|x] + (0) * (E[y1|x] - E[y0|x]))

                                          = E[y0|x] + E[y1|x] - E[y0|x] - E[y0|x

                                          = E[y1|x] - E[y0|x

                                          = ATE(x)

Following some discussion on nonparametrically estimations of ATE or ATT from x,we move onto parametric regressions. In particular, the switching regression model:

y0 = mu0 + v0, E[v0] = 0

y1 = mu1 + v1, E[v1] = 0

 y = mu0 + (mu1 - mu0)w + v0 + w (v1 - v0) --eqn (18.10)

Under conditional mean independence (see above), E[v1|x] = E[v0|x], ATE = ATT, and:

$$    E[y|w,x] = \mu_0 + \alpha w + g_0(x)   $$

    E[y|w,x] = mu0 + alpha w + g0(x)  

where alpha is ATE, and g0(x) = E[v0|x].  

If E[v0|x] = eta0 + h0(x)beta0 for some vector function h0(x), then

   E[y|w,x] = mu0 + alpha w + eta0 + h0(x)beta0

                = gamma0 + alpha w + h0(x)beta0

and gamm0 is mu0 + eta0.

The function h0(x)beta0 is an example of a control function. When added to the regression of y on 1, w, it controls for possible self-selection bias.

Next, from conditional mean independence and (18.10), the switching regression model,

    y = mu0 + (mu1 - mu0)w + v0 + w (v1 - v0)

let's introduce g1(x) = E[v1|x] and g0(x) = E[v0|x]:

   E[y|w,x] = mu0 + alpha w +  g0(x) + w * (g1(x) - g0(x))

To operationalize the above, for example with linear parameters: eta0 + h0(x)beta0 and eta1 + h1(x)beta1:

   E[y|w,x] = mu0 + alpha w +  xbeta0 + w * (x - psi) delta

psi = E(x)

This translates to a regression with characteristics x, and interactions of x less sample means (demeaned) with w. squares or logarithms as levels or interactions are optional.

ATE(x) hat = alpha hat + (x - x bar) delta hat

[BH: treatment plus selection?]

The Organization Man - William H. Whyte, Jr

Trendspotting. Published in 1957, Whyte's is still an accurate summation of a large part of the present.

Whyte's Organization Man appears to be strong influence on the views of Venkatesh, of the Ribbon Farm blog. He discusses the book in 3 parts: part 1part 2, and 3

These are my notes and quotes from the book. 

Because they are the largest single group, the corporation-bound seniors are the most visible manifestation of their generation's values... Whatever their many differences, in one great respect they are all of a piece: more than any generation in memory, theirs will be a generation of bureaucrats.

The last thing students can be accused of now is dangerous discussion; they are not interested in the kind of big questions that stimulate heresy and whatever the subject - the corporation, govenrment, religion - sutdents grow restive if the talk tarries on the philosophical.

Perhaps contemporaryism  would be a better word than conservatism to describe their posture. The present, more than the past, is thier model; while they share the characteristics American faith in the future also, they see it as more of the same... The big questions are all settled; we know the direction, and while many minor details remain to be cleared up, we can be pretty sure of enjoying a wonderful upward rise.

Commentary on the younger generation of students has now become reality.

More than before, there is tremendous interest in techniques. Having no quarrel with society, they prefer to table the subject of ends and concentrate instead on means. Not what or why but how interests them...once they have equated the common wealth with organization... they will let the organization worry about goals. "These men do not question the system", an economics professor says of them, approvingly. "They want to get in there and lubricate and make them run better. They will be the technicians of the society, not innovators."

The attitude of men majoring in social science is particularly revealing on this score. Not so very long ago, the younger social scientist was apt to see his descipline as a vehicle for protest about society as well as the study of it. The seniors that set the fashion for him were frequently angry men...[References Robert and Helen Lynd's Middletown as an example of big studies in the twenties and thirties]

Reflecting the growing reconciliation with middle-class values that has affected all types of intellectuals, they are turning more and more to an interest in methodology, particularly the techniques of measurement. Older social scientists who have doen studies on broad social problems find that the younger men are comparatively unintested in the prolbems themselves. When discussion period comes, the questions the younger men ask are on the technical points; not the what, or why, but the how.

The urge to be a technician, a collaborate, shows most markedly in the kind of jobs seniors prefer. They want to work for somebody else.

In Chapter 17, the Bureaucratization of the Scientist, Whyte notes the shift towards group work as indicated by the number of coauthors per published journal article. This manifestation is rampant in medicine, and the public health disciplines, though it is not necessary for public health. It can be argued that the amount of funding required necessitates that teams work on certain projects.

Chapter 21, The Transients. Suburbia illuminates the rootlessness of the organization man. 

The man who leaves home is not the exception in American society but the key to it... And for all college men, incidentally, the higher the grades, the more likely they are to move. 

Could almost be a description of the present.

But perhaps the most important reason the transients can't go home is that they won't find it there if they do. It is not just the physical changes... As the young transients have left town, their opoosite nubers from other towns have come in, and in many American communities there has been wholesale displacement from positions of power of the names that once "meant" something.


Once people liked to think, at least, they were in control of their destinees, but few of the younger organization people cherish such notions. Most see themselves as objects more acted upon than acting - and their future, therefore, determined as much by the system as themselves.

To a large extent, to my way of thinking, the trends observed by Whyte are still evolving through Western society. The most convincing explanation for underlying trend is due to the replacement of religion by science (Jordan Peterson). In addition to the increase in conformity, collobaration and techniques, the culture of science pervades education and business. Even the characterisation of science is represented pop culture now. It seems to tie in with nihilism and the absence of belief, leading to conflict and war, speculated by Nietsche. There seems to be a general vacuum of morality and meaning. 

If there are parallels between the booming 1920s and the 2000s, perhaps this is the lull before the storm of WWII. See Kyle Bass, researchers from the IMF, and the complex systems institute (e.g. [3], [4], [5]), who are predicting increasing world conflict in the foreseeable future. 

[1] http://attackthesystem.com/the-nietzschean-prophecies/ 

[2] http://en.wikipedia.org/wiki/God_is_dead

[3] http://www.financialsense.com/contributors/russ-winter/predicting-global-revolutions-civil-wars-and-riots\

[4] http://www.financialsense.com/contributors/russ-winter/food-prices-riots-civil-wars

[5] Arezki and Bruckner http://www.imf.org/external/pubs/ft/wp/2011/wp1162.pdf

[6] Lagi et al. http://arxiv.org/pdf/1108.2455v1.pdf 

TFS Corporation (ASX:TFC)

Sometimes I think it'd be nice to be able to learn without paying for my mistakes. Back in 2007/2008 I bought some shares in NSX, a second tier stock exchange, which traded taxi licenses or something like that. I don't think it was profitable, liquid or had any particular growth prospects. But I was convinced that it was a decent pick because it was a monopoly and the ASX, which had recently merged with the SFE (the futures exchange), had been profitable in the lead-up to the great financial crisis.

Fast forward to 2013. At this point in time, I'm still keen on owning companies which are monopolies so I take a small position in Microsoft reasoning that it had been under-performing for a decade or so with sideways movements in the share price, and decent profitability. I jumped the gun and sold with a modest price increase, as all stocks gained in 2013. The catalyst for the larger price movement was the departure of Ballmer who had erred in lagging in mobile technology, and who appears to have been lacking the requisite vision to steer the ship away from obselescence.

The new CEO has been decisive. He announced staff cuts and agreed to ship the office software on Apple products. In short, the optionality in MSFT's share price is gone.

A bit later, I'm scouring Google Finance and looking for cheap stocks when I come across TFS Corporation. With a P/E of around 5, I'm convinced it's a value trap because it's so cheap. I watch it loosely and jump in when the price takes off.

Another digression: as a novice investor I thought that buying stocks on the decline was better as they were cheap; I was buying low, hoping to sell high.
Great in theory, but extremely difficult to time as it is difficult to pick the bottom until after the fact. Hence the phrase, don't catch a falling knife. I know now, but don't remember the source, that a rising stock price is a good sign. The caveat is that it's hard to tell if it's overpriced. Go with your friend, the trend.

Back to TFC. It's the monopoly I've been looking for, my conviction monopoly. It is a forestry company which sells sandalwood products. Indian sandalwood is on the preservation list so forestry is illegal. This, combined with increasing demand, has presumably sent the commodity price upwards.

TFS is the sole commercial producer of Sandalwood trees. Management's incentives are aligned with the founder having a significant percentage of shares. The founder left a private practice law firm to grow and sell trees, which suggests that he had a vision for the company which outweighed the opportunity cost of being a decently paid lawyer.

The company has been in existence for well over a decade, and has improved the survival rate (or is it extraction rate) of its trees. This confirms the suspected IP edge the company would have over any prospective entrants, the barrier to entry.

As a perfect storm for why this information hasn't been priced into the stock is the disaster that was Timbercorp and the forestry schemes that failed in the 2000s. These schemes allowed individuals to invest in forestry, agribusiness products with a tax advantage. Timbercorp went bankrupt in 2009, partially contributed by the government changing the taxation of these schemes. TFC was tarnished with a broad brush, having also offered these products, but not having mismanaged its financial affairs. It also had a market cap of less than $200m.

TFS increased production of sandalwood from late 2013. It then signed a JV with a pharmaceutical company to provide sandalwood to create treatments. A director recently bought a sizeable ~$170k AUD (at ~$1.70) amount of shares with the family trust.

Broker reports, which are publicly available and linked to from the website, indicate that production volumes are forecast to rise rapidly over the next 3-5 years. As Mr. Market tends to only believe only what he sees, I don't think it is priced in, and if anything, I think the share price will overshoot as volumes and earnings increase. I've been buying in with each successive price jump, and it looks to be taking a breather before the next increase. Time to get onboard. I can see the share price doubling or tripling from here. Beyond that I can conceivable see this niche monopoly company consolidating into a steady but less exciting cash cow. Comments welcome.

[BH: The day after I wrote the draft for this post, 14th August 2014, TFC jumped from $1.80 to $2.05 over two days. The catalyst was another set of presentation slides, which didn't really contain any news. After stewing on how amateurish this post is, with its lack of financials commentary, and whether the eclectic mix of posts is an issue, this post was published on 8 September. Other good news since then is that it appears from a corporate governance perspective that the company has been on the improve, with another director resignation (same one as mentioned above). This follows a resignation earlier in the year (Gilchrist I think, who in all likelihood was playing purely a marketing role). The (large) positions held by these individuals is a positive. ]

The Wealth of Nature - John Michael Greer

This is a dense book, with interesting thoughts throughout. It's not often a book has 5 stars for average review on Amazon and this book doesn't disappoint. One notable quirk is that the author is a self-described druid, not a journalist, scientist or writer. He also comments in a conversation with Chris Martenson, on his Peak Prosperity podcast, that he earnt his beard and ponytail by living on a commune for some period of time!

According to Greer, Ernst Schumacher, born in Bonn, 1911, layed the foundation for incorporating energy into economics. 

Renewable resources, he proposed, form the equivalent of income in the primary economy, while nonreneweable resources are the equivelent of capital...

He argued that energy  cannot be treated as one commity among many without reducing economcis to gibberish, because energy is the gateway resource that gives access to all other resources.

Greer goes on to list a few ecologically sane economists since Schumacher: Nicholas Georgescu-Roegen, Kenneth Boulding, Herman Daly and Robert Costanza. He somewhat pre-emptively points out Adam Smith did not have a degree in economics. 

Economics suffers from premature mathematization - paraphrasing here, economics involves so many variables taht the only way to control them all is to impose conditions so arbirary that the results have only the most distant relation to the real world. Greer concludes the only logical way out of the trap is to concentrate on economic history. note a set of conditions in the past that consistently lead to a common result, e.g. house bubble (p17).

Have you noticed that the guys who run macroeconomics based hedge funds are all trained in the humanities and have a keen interest in history? For example, Kyle Bass predicting that there may be war on the horizon.

Social critics have commented on the ease with which neoclassical economics ignores the interface between economic wealth and power... This is a pervasive problem in most human socities, and it's worth noting that those socities that survive over the long term tend to be the ones that work out ways to keep too much wealth from piling up uselessly in the hands of those with more power than others.

.. those nations that got through the last Great Depression intact did so by imposing sensisble checks and balances on concentrated wealth.

I recall a marginal income tax rates of 98% at some point. Need to find a reference here. And isn't it at this point in time which anti-trust law is enacted? There was a large correction towards protecting the ordinary in society after the destruction wrought by the Great Depression.

When wealth is widely distributed, more of it circulates in the productive economy of wages and consumer purchases; when wealth is concentrated in the hands of a few, more of it moves into the investment economy where the well-to-do keep thie wealth and the buildup of capital in the investment economy is one of the necessary preconditions for a speculative binge (p20).

This is echoed by Nick Hanauer, the friend of Bezos, who manufactures pillows and says that he only needs one pillow despite his wealth [1].

There's a rich irony here, in that the market economy portrayed by textbooks.. is a form of commons, and a great many people who claim to be advocates of free market have spent years arguing that commons should be eliminated wholesale in favor of private ownership.

All common systems, as Garrett Hardin pointed out in his famous essay, have to be managed in ways that prevent individuals from exploiting the commons for their own private benefit; otherwise they fail.  

Paraphrasing again, Elinor Olstrom demonstrated that it's entirely possible to manage a commons so that Hardin's Tragedy of the Commons doesn't happen (p27).

Energy, as E.F. Schumacher pointed out, is not simply one commodity among others; it is the ur-commodity, the foundation for all economic activity (p30).

a world that has nearly seven billion people on it anda  dwindling supply of fossial fuels can do without the assumption that putting people out of work and replacing them with machines by fossil fuels is hte way to prosperity (p39).

Liebig's law of the minimum: production is limited by the scarcest necessary resource

  • Around 75 cents of every dollar circulating was provided by natural processes rather than human labour - Costanza, Robert, p54.

The difference between the supply-limited goods of the primary and secondary economy and the demand-limited goods of the tertiary economy, among other things, a difference between kinds of feedback.... in a market economy all secondary goods are subject to negative feedback. ... Negative feedback loops of a very similar kind control the production of primary goods by the Earth's natural systems...It's when we get to the tertiary economy of financial goods that things change because the feedback loops governing tertiary goods are not negative but positive (p69).

Eventually the supply of buyers runs out, because everyone who is willing to plunge into the bubble has already done so. ...prices began to sink, and once again positive feedback came into play as the adothermostat shifted from heating an overheated house to cooling an overchilled one. p71

financial investments don't suffer from rising energy costs so individuals move their investments from the productive economy of goods and services to the paper economy of finance. Ironically, this is happening just as perpetually expanding money supply driven by mass borrowing at interest has become an anachronism unsuited to the new economic reality of energy contraction. p75

  • The Mayans responded to their agricultural criss by accelerating building programs, p.80.
  • Vico attempts to document the course nations run with classical Greece and Rome.
  • three-century-long beinge with finite supplies of cheap abundant energy spawned a second bubble of money and finance, abstract representations of wealth, p105.
  • stagflation from oil shocks in 1970s. Rising prices and wages struggling to keep pace with diminished economic activity.
  • second law of thermodynamics: energy moves from higher to lower concentrations.
  • notes the distinction between quantity of energy and concentration of energy, where the latter is more important. Diffuse energy - it doesn't take concentrated energy to heat a tank full of water. It can be done efficiently by diffuse energy sources such as sunlight, p138.

 the amount of work you get out of an energy source depends, not on the amount of energy contained by the source, but on the difference in energy concentration between the energy source and the environment.

  • Points out that Thomas Friedman's Hot, Flat and Crowded is correct in stating that we're facing global wierding, a worsening of conditions, unexpected and disruptive weather events.
  • Economics of contraction. Description of Easter Island and deforestation leading to the implosion of society. p160.

[1] http://www.politico.com/magazine/story/2014/06/the-pitchforks-are-coming-for-us-plutocrats-108014.html#.U_wQHfmSx8H

Additional Thoughts

The ongoing depletion of natural resources is a case of a potential tragedy of the commons. Individuals or companies have an interest in using up the finite resource, be it petrol or fish in the ocean, but by doing so will damage the collective group. The uncertainty involved in the trade-off between technological innovation and resource depletion is complex. For example, Richard Heinberg writes in 2004 about unconventional petroleum resources, suggesting that they are too costly to extract. Forward 10 years later, and we have a boom in shale oil production [2]. Even with the development in shale oil, he still believes its impact is overstated and will be short-lived [3].

Unconventional petroleum resources -- so-called "heavy oil," "oil sands," and "shale oil" -- are plentiful but extremely costly to extract and process, a fact that no technical innovation is likely to change much [1]

[1] Powerdown, Richard Heinberg

[2] Frackers and the Energy Revolution, Russ Roberts interviewis Gregory Zuckerman, http://www.econtalk.org/archives/2014/06/gregory_zuckerm.html

[3] Peak Prosperity Podcast with Chris Martenson

Powerdown - Richard Heinberg

Heinberg provides evidence our endless supply of non-renewable resources are disappearing fast. He outlines three ways of handling the pending crisis: competition for the remaining resources, powerdown - cooperation, conservation and sharing, and wishful thinking. 



Ozonomics - Andrew Charlton

Charlton provides some good insights into the Australian macroeconomy, but probably covers too much territory to summarize succinctly.Topics include debt, central banking, housing, manufacturing.  He summarizes the economy as productivity, jobs and equality. The premise of the book is that the labour government was responsible for a series of reforms in the 1980s which was the basis for the economic boom in the 90s and onwards. Charlton suggests the triumph of the Howard Government is taking credit for the results. 

The other point that sticks out is Charlton's thoughts that Latham threw the election by choosing to compete with the Liberal government about budget surpluses, a furphy for the reason that international debt markets mean that surpluses are not a real (imminent) issue.

Charlton stands out for having a doctorate from Oxford and being Rudd's economic advisor at around 30 years old. Presuming that an advisor to the Prime Minister would remain in politics, it surprising to see that he went on to head the liquor division of Coles after Labour lost office. 


 The essential feature of hte Australian economy has always been that we have a huge continent full of natural resources and a tiny population among which to share its bounty.

Dutch Disease arsies because high prices for our commodity exports drive up the value of Austrlian money, which then makes our manufactured goods more expensive relative to those produeced by foreigners.

The post-war commodity boom made the AUD one of the world's strongest currencies. Advantages in agriculture and mining lead to manufacturing weakenss. Charlton goes on to say that early leaders recognised the need to diversify the economy. Enter the second Prime Minister, Alfred Deakin, 'No nation ever claimed greatness which relied upon primary industries alone'.

This meant propping up manufacturing with high tariff walls to protect Australian- made goods from the competition of imported substitutes.

Charlton summarizes: farming and mining supports manufacturing businesses through tariffs and is also passed on to workers through a centralized wage-fixing system.

Manufacturing in Australia - Mancur Olson

In 1984 Mancur Olson wrote a paper arguing for the lowering of tariffs on manufacturing in Australia. He argued that Australia was in a favourable position to obtain increases in real income from increasing manufacturing, citing an abundance of raw materials and a well educated and technically sophisticated population. He advocated freer trade to increase gains to Australia, larger than what would be expected from the theory of comparative advantage. He writes in the summary:

... the level of protection is very important among small economies as an indication of their ability to export. There is a striking pattern:

if countries have high levels of protection on manufacturing, without a single exception, they export very little of their manufactures. Australia is the second most protectionist on manufactures of the devloped countries, and Australia exports only about 7.5 per cent of its manufactured output.

Olson suggested a test of his theory of the logic of collective action by looking at the manufacturing and degree of protection in small and medium sized countries.

.. the patterns of trade in manufactured goods provide better insights into the efficiency of the institutions and policies of a country than do pattern of trade in primary products, because the dependence of manufactures on teh endowments of natural resources is not as strong as in the case of primary products.

The intuition is that these countries are the ones where protection (a collective action) will have its greatest impact. That is, the mileage of barriers, as determined by the size of the jurisdictions, should be of greater significance than the height of the barriers. He touched on evidence for distributional coalitions acting as a drag on the economy: Low growth is expected in long-stable countries like the UK due to the influence of organizations for collective action. In contrast, Japan, Germany and Italy experienced rapid growth from the destruction of these organizations after world war 2. Olsen also suggested that the USA has similar historical trends, where the Northeast and Eastern Middlewest are decline, whereas the frontier areas in the West and recently turbulent South are doing better.

The results presented show a strong correlation between tariffs (1976) and 1973 percentage of manufactures exported (value of manufactured exports / value of manufactured output). Denmark (42.5%), Sweden (37.5%), and Austria (32.5%) are offered as counterexamples with low tariffs and large percentage of manufactures exported. 

In summary, Olson discounts the idea that protectionism is beneficial in the early stages of infant industries and interprets the abundance of small high exporting countries with low tariffs and the failure of medium sized protecionist coutnries to attain impressive outputs of manufaturing to indicate that protection of manufactures does not lead to increasing manufacturing output of a country in the long run.

The Decline of Manufacturing in Australia 

The manufacturing industry in Australia has been in decline for a protracted period of time, and has coincided with the growth of the services industry. In recent years, automakers Toyota and Ford have announced job cuts as well as the inevitable closure and relocation of plants to other countries, such as Thailand. The high labour costs arising from Australia's centralised wage setting institutions and the historically high Australian dollar no doubt played a large role in these changes. The strength of the Australian economy has been driven by a large one in a lifetime resources boom, where China now comprises 25% of Australia's exports.

Three decades have passed since Mancur Olson advocated that Australia decrease its manufacturing tariffs in order for Australia to develop internationally competitive manufacturing and consequently increase manufacturing output. What has happened since then, is that the tariffs have been reduced [1], albeit to differing degrees depending on product. Amazingly, in line with Olson's prediction, the percentage of exports to sales has risen to 35% [2]. 

Despite this, manufacturing has decilned as a share of all industries due to the growth of other industries and the contraction of manufacturing itself. Reducing tariffs appears to have improved, or coincided with the improvement of, the ratio of exports to total manufacturing but manufacturing itself has become less important over time. Is there a trade-off between size of manufacturing with the percentage of manufacturing exported? 

[1] Australian Manufacturing: A Brief History of Industry Policy and Trade Liberalisation, 1999-2000 http://www.aph.gov.au/About_Parliament/Parliamentary_Departments/Parliamentary_Library/pubs/rp/rp9900/2000RP07 

[2] The Australian Dollar and Manufacturing Exports, The Australian Industry Group, 2007.


[3] Why Manufacturing in Australia has a future, 16 April, 2014, Ross Gittins.


[4] Labour Market Snapshot #5, March 2014 Jeff Borland


The New New Thing - Michael Lewis

I've taken some notes from the New New Thing by Michael Lewis, which is about Jim Clark, the founder of Netscape.


Silicon Valley is to the United States what the United States is to the rest of the world

It is the capital of innovation... and of transience.


Software and Engineers

  • Veblen (1921) predicted that engineers would become an integral part of the economy, a world view which Clark shares, as a creator and engineer (though trained as physicist). [1]
  • The IT industry has become like arts and music industries. Capital costs are low now as hardware costs have decreased, and finance and legal support is easy to obtain. The value lies the efficient allocation of the finance and hence the supernormal returns to the visionary. There are many aspirants but the masses of app developer wannabees are commodities.
  • What I relate to the most about Clark is that he is technological purist. All the while, as these events are happening to his third company, he is coding the software to his hobby yacht with the majority of this time. The Indian software engineer, an employee, he consults can hardly believe a man of his importance is spending his spare time practicing coding.
  • This is where Clark and I depart in risk tolerance; he is an all-in type of fellow. During the dot-com crash, his net worth drops from a billion down to $250 million and every time his third company requires more funding, using a poker analogy, he raises his bets every time, as opposed to the venture capitals who want to hold. They call, in response to Clark, fearing embarrassment, should he succeed, not because they are believers.
  • To my mind, there is a lot of similarity with leading a technology company and being an investor, as forseeing trends appears to be paramount. In support of this idea, later on Clark invested heavily in Apple and Google during the great financial crisis with minor positions in Twitter and Facebook. Andreesen thinks that "software is eating the world", and so does Clark. Well it looks like it anyway. His investment in Apple was based on his view that Apple made programming simple with its object oriented platform for the iphone [2].

The Visionary

  • This leads into his childhood background, which shaped him. He started with nothing and had a huge chip on his shoulder. I suspect this is what separates him from the majority, who may have equal ability and ambition. Doubling down on a bet when circumstances go against you is reckless in the eyes in most. Clark doesn't even question it. 
  • It is worth noting that his ability to focus at the expense of all else had a cost on his personal relationships. 
  • As I am reading this book, it is hard to believe in his followers and the blind faith they have in their idol. After succeeding in his second venture, he is basically the ideas man, receiving a pile of cash and assembling a team, which he checks in on remotely, part-time. He has an epiphany that he is rewarded handsomely for being able to see the future.
  • My intuition reading the book was that luck played a large part in Clark's initial success such that subsequent success was unlikely. Really, Clark's conviction trumps the markets and everyone. 
  • I cannot help but see the parallels of Jim Clark at Silicon Graphics with Steve Jobs at Apple; his role as the inspirational creative leader and his struggle against corporate culture 
  • Head fake in the telecomputer / feigning one direction but moving another; reduced to a figurative head of the organization as chairman, he begins making media releases about new technology, resulting in pressure on his CEO to deliver on this project. As the project is in motion, he jumps ship to start a competing venture, which then, unplanned, becomes the Netscape browser. 
  • Is he at the right place at the right time in meeting Marc Andreesen?

[1] I think it's fair to say that Veblen wasn't a great writer. It's not worth your time reading the book, Engineers and the Price System.

[2] http://www.forbes.com/sites/ryanmac/2012/03/08/jim-clark-the-comeback-billionaire-who-bet-on-apple/ 

Movable Type on Ubuntu with Nginx and fcgiwrapper

The recent proliferation of static site generators has piqued my interest in leaving Wordpress which is sometimes unresponsive. Admittedly, this isn't hard given my short attention span. I thought, "Here it is, the reason why my obscure writings won't turn into a hugely influential blog is that the website is too slow. It needs to be static pages so it can scale to handle the the audience of the whole internet.", or something like it.

While deciding which python site generator use, I came across this discussion at HN.


The comment that caught my eye was the link to Jeff Attwood's explanation as to why he uses Movable Type.  His explanation is along the lines that he chooses the best tool for the job. What I heard was: the benefits of a static site generator, without glueing everything together and writing html and markdown. This commenced the beginning of another wild goose chase, better known as the path to yak shaving. Here are the brief working instructions to Movable Type on Ubuntu with fcgiwrapper and nginx. 

1. Download MT5...

  $ wget http://movabletype.org/stable/MT-5.0-en.zip

2. Unzip it

e.g. unzip MT-5.0-en.zip mt

3. Make some directories and then add symbolic links:

cd /var/www/
mkdir cgi-bin
mkdir html
ln -s /home/ubuntu/mt  /var/www/cgi-bin/mt
ln -s /home/ubuntu/mt-static /var/www/html/mt-static

4. Install nginx, spawn-fcgi and fcgiwrap

sudo apt-get install nginx spawn-fcgi libfcgi0ldbl
sudo apt-get install build-essential libfcgi-dev

5. Modify /etc/init.d/fcgiwrap to set the user and group to www-data or ubuntu. I know it's hacky in a bad way and security risk, but it works.

#FCGI_APP Variables
# Socket owner/group (will default to FCGI_USER/FCGI_GROUP if not defined) FCGI_SOCKET_OWNER="ubuntu" FCGI_SOCKET_GROUP="ubuntu"

 6. Create a socket file for fcgiwrap and set its permissions such that fcgiwrap can access it.

touch /var/run/fcgiwrap.socket
sudo chown ubuntu:ubuntu /var/run/fcgiwrap.socket

7. Nginx config file. This was the biggest pain point for me until I found link [4]:

server {
     server_name mt.bhoung.com;

     root /home/ubuntu/blog/;

     location / {
        index index.html index.htm;

     location ~ ^/cgi-bin/.*\.cgi$ {
     gzip off; #gzip makes scripts feel slower since they have to complete before getting gzipped
     fastcgi_pass unix:/var/run/fcgiwrap.socket;
     fastcgi_index index.cgi;
     fastcgi_param SCRIPT_FILENAME /var/www/$fastcgi_script_name;
     fastcgi_param QUERY_STRING $query_string;
     fastcgi_param REQUEST_METHOD $request_method;
     fastcgi_param CONTENT_TYPE $content_type;
     fastcgi_param CONTENT_LENGTH $content_length;
     fastcgi_param GATEWAY_INTERFACE CGI/1.1;
     fastcgi_param SERVER_SOFTWARE nginx;
     fastcgi_param SCRIPT_NAME $fastcgi_script_name;
     fastcgi_param REQUEST_URI $request_uri;
     fastcgi_param DOCUMENT_URI $document_uri;
     fastcgi_param DOCUMENT_ROOT $document_root;
     fastcgi_param SERVER_PROTOCOL $server_protocol;
     fastcgi_param REMOTE_ADDR $remote_addr;
     fastcgi_param REMOTE_PORT $remote_port;
     fastcgi_param SERVER_ADDR $server_addr;   
     fastcgi_param SERVER_PORT        $server_port;
     fastcgi_param SERVER_NAME        $server_name;
   # for static files 
   location /mt-static {
       root /var/www/html;

8. As I was importing my entries from wordpress I hit a file size limitation in nginx. 

# vi /usr/local/nginx/conf/nginx.conf
client_max_body_size 2M;

* I know I've skipped a few steps like the MySQL and the mt-config.cgi file... I'll get back it a bit later

[1] http://www.movabletype.org/documentation/installation/
[2] https://help.ubuntu.com/community/FcgiWrap

[3] http://library.linode.com/web-servers/nginx/perl-fastcgi/ubuntu-10.04-lucid

[4] http://wiki.nginx.org/SimpleCGI
[5] http://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/
[6] http://www.livejournal.com/doc/server/lj.install.perl_setup.modules.html

Google Maps Overlays with TileMill QGIS and R

Check it out: http://maps.bhoung.com/popden/index.html

I had been meaning to replicate the cool maps that Ben Boyer made when he was at the Chicago Tribune (population density map of Chicago: http://media.apps.chicagotribune.com/census-2010/census-demo/index.html) .

I'm glad to be able to say that I successfully completed his tutorial a few weeks ago, with Australian data. I finally found a few minutes to upload the modified code onto Amazon's S3 Buckets service. The main obstacle to following the tutorial for anyone interested was trying to generate the images using Mac OSX 10.8.2. As most of the pieces of software are constantly being updated, it was easiest to fire up Ubuntu and run invar, an open source python script to generate the images, rather than trying to install it on the Mac....

No surprises as to what population density looks like in Australia, but it's pretty nonetheless. Extensions would be look at other kinds of data, perhaps at the postcode level. Also, implementing the layering with dots as per some of the Chicago Tribune's other maps would be great too.

Update: So the time has come for me to replicate these maps with new data. I'll document as I go this time round.

1. Merging spatial data with shapefiles can be done more easily than using postgres with either R (http://gis.stackexchange.com/questions/6839/best-way-to-add-attribute-data-to-a-shape-file) or the mmqgis plugin in QGIS.

2.Carto to conver tilemill mml  file to an xml file

3. install pip and then invar


  • sudo apt-get install python-mapnik mapnik-utils libmapnik0.7
  • sudo apt-get install python-pip
  • sudo apt-get install python-greenlet
  • sudo pip install invar
  • ivtile -h
  • ivtile project.xml . -9.3 96.9 -43 159 10 16 -p 2