Teaching and agile management

Today a friend's son asked me to help him with his math. The magic of internet meant that could read up the topic on wikipedia while listening to the request and accept in confidence.

Here are a few hints on how to teach:

• Allow learning from mistakes: the young man's first request was to erase what he had done, as it was wrong. I stopped him and led him through the understanding of what he had done, building a knowledge of what "not to try again". This happened a few more times where I let him make his mistakes so that he could understand better how not to make these mistakes.
  
• Do not provide solutions but support the learning of decision making along the way: Learning is about associating together thoughts and methods and by doing so building a new way to do or understand something. In this case, I help him identify which of the mathematical tools he possessed already would help him for this type of problem.
  
• Help maintain a notion of goal at all times: The problem he needed to solve involved a good amount of tedious algebraic work. It was easy to get lost a this level of the task so I helped him maintain a link with his goal during the whole process. In this way, even when he made a mistake and needed to backtrack, he still knew where he was going.

Now I ask you to go and read up on Scrum or other agile process. You will see that what I have said above is how you should lead your agile teams. It does not matter what type of leader you are, people, technical or business, the art of helping people to be productive with quality is to get them to learn how to always do better.

There is one caveat to this process. As the "student" needs to know enough and be skilled enough to be "a good student", the agile team member needs to be knowledgeable and skilled enough to be a "good employee". Special attention must be put into the hiring process and the simple rule that remains the best is to hire the bright ones fresh out of school and have them work with your senior experienced people.

C++ and functional programming

What to say to C++ programmers about functional programming, especially now that many compilers have implemented C++0X features like lambda and auto. I am pretty good at C++, having started using it with the Zortech compiler V1 and the first versions of CFront (the orginal C++ to C translator). I have also written much fancy template code. Therefore thinking about this post leaves me in the dilemma that there is one part of me that likes the features of C++0X that "wrap up" loose ends, and another part of me, the functional programming part, that makes me want to rant! I will not rant today, but part of that reasons for being upset needs to be said, as is relevant. The "magic" of mature functional programming are:

• Statements, expressions, functions and types support each other
• Type inference and polymorphism automate the relation between statements, expressions, functions and types
• Monadic and other "advanced" types are available to define new execution models such as parallelism.

C++ templates can bee seen as a primitive functional language. But I would argue against going into production focusing on C++ templates that way as I have been burnt so many times by C++ compilers failing me with complex templates, not to mention the difficulty to hire for this skill So template functional style is not where the emphasis should be. The real issue is that although C++ expressions have a C++ type counterparty (think operators like + and -), there is no C++ type support for statements. As a result many of the core concepts of functional programming cannot mapped across C++, even with the new C++0X features of lambda, garbage collections, auto, etc. My recommendation is that some concepts should be understood by C++ developers but not necessarily with a functional emphasis. My favorite topic is as always monads. In the C++ terms, monadic concepts define properties on the "traces" of the code. If we say: f(...); g(...); then the execution trace of the program will first have gone through f and therefore when g is called, properties set by f may still be valid. A good C++ architect should understand enough about monads to define invariants that can be used in this type of trace oriented context. For the moment, I would forget about other "functional programming concepts" in C++, even if it is C++0X!

Personalize your containers (in F#)

Many years ago I implemented an augmented tree structure in C++. Inspired, I decided to implement the structure in F#. FYI by augmentation I mean storing extra data at each node of the tree that will accelerate the later operations on the tree. In DB terms, augmentations are typically used include secondary key conditions in your tree query. This is what I did:

1. Take a balanced tree implementation off the web. I was thinking finger tree but then though better to start with a simpler implementation. I chose an AVL tree. (I may revert the code to a red and black tree to simplify).
2. Add a node field to store the augmentation (a generic type)
3. Refactor the code by replacing the call of the node constructor by a call to a function that will first build an augmentation and then call the node constructor
4. Pass the generic augmentation "maker" function as argument to the function calls

On this last point I wasted much time. Initialy, I tried to encapsulate the "augmentation" api with abstract methods on one the the types. As all this code is polymorphic the compiler barfed with "would leave scope" and this type of error. After a bit too much time, I reverted to only define API signatures are with functions, no methods. (I understood later that this is mostly true for all languages: stick to functions/lambda constructions because they always scale). Then I wrote a generic select like operation that uses augmentations. And finally a "general" map operation: it uses the augmentations to optimize its search, it only transforms nodes that have the selected property AND it allows nodes to be deleted. There are still a few more things to do to wrap this code up but I again can only admire the speed at which one can write functional style. I'd love to tell you why you want to have these types of data structures but not everything can be for free.

Flash crash 2010 thinking

An old friend asked me if the flash crash had been caused by "spoofing" of the exchanges with very large flows of order entry/order cancel commands. My feeling was that this was not the case and a quick search on google picked up a confirmation in this article: The Microstructure of the ‘Flash Crash’: Flow Toxicity, Liquidity Crashes and the Probability of Informed Trading” This article supports the explanation that the main cause of the flash crash was simply the lack of liquidity because the market makers had become too edgy due to too much new information coming in to the market. It is an excellent article and brings back the notion that some systems are only sustainable with enough "friction", or said differently, by being under-optimal with regards to certain ways to look at them. When a group of people need to work together, they nominate a chief, or at least they give specific roles and ownerships to members of the group. These people get an "unfair" advantage as their ownership will most often provide benefits, even if they is not trying to use them for their own gain. Ideally for the group, processes are in place that limit personal advantages. And these advantages cannot be completely removed as that would take away the leverage that allows the overall process to happen AND BE SUSTAINABLE! There are real challenges with these types of "social process": If people remove all their leadership they end up in chaos, and yet people are often upset with some of the unfair advantages that the leaders have because of their role's functions. Market makers that cannot make a profit on the bid and ask spread must make a profit on "another spread". This can be a volatility spread, or more generally on an "information spread". Information spread is, for example, "I don't know and you don't know", going to "Now I know, and you don't"; Or it could be "I expect parameter P3 to change in my model, you don't". When a market maker can no longer find a profit making spread, he will limit his exposure; especially when this exposure leads to continuous losses caused by others having their own "better" information spread (often at the limit of inside trading). The sad conclusion is that market makers become less of a sure thing when rules that diminish their "advantage" are imposed by the exchanges, the SEC or the ESMA. Mathematically you can see this as moving from a continuous to a non-continuous regime. Call it a "percolation" of the market making process past a critical level. The big problem is that in the non-continuous regime, one where parts of the markets simply stop, as no prices are quoted, is "a nightmare". And many people simply ignore it as "impossible". The truth is that it is much more a possibility now than it ever was and that seems to be what the flash crash was all about.