Sachin Dixit's Blog

The toughest part of software development is human communication. If we group together words like specification, validation, experience, exposure so on they all represent communication.It is the centerpiece of your development procedures. If we dig deeper into these words the essence of why software development is not a fully controllable discipline like many other branches of engineering we realize that we have idea-communication problems.

When we sit together with a newly formed product development team this is the first thing I stress. It is your idea of what their idea of whats customers’ idea of the reality is. And often even our sense of reality itself is incomplete, leave alone its articulation and understanding. This realization makes our developers more humble( especially the ones with a hackrank high), the project plans realistic and the leadership vigilant. But how do you make sure that like many other mission statements postered on the wall, or at the end of email signatures or as messenger status this doesn’t become the most ignored and broken often statement. Read on.

The hard aspects of product engineering teams

Product development is hard. Hard in the sense of how some things are computationally hard. A typical services project assignment masks a lot of complexity and preparatory work from your team. The idea justification and realization might have happened already and you get a curated feature list. User experience studies; focused group discussions; technology stack debates, prototyping, sizing and many “other than programming” tasks might be hidden from you to a varying degrees. But one can always go back and read about them once the project starts.

Enter products. A commissioned software whether inhouse or outsourced might have a large exposure area, both in terms of users, features and run conditions. When you do products this happens to be your baseline case. Add to it the complexity of providing features to customers of your product customers. It changes the game from being a good housewife(or husband) to teaching people the art of courtship. In particular It :

1. changes your requirements from a list to a range (where wise choices need to be made)
2. Exposes you to comparison with known and unknown alternatives (some of which could even be non software !)
3. Stretches the boundaries of user behavior – run conditions and regulations by multiple sigmas (normal distribution analogy)
4. Forces you to design for evolution (no longer a design ideal)
5. Challenges your notion of S/Ws purpose ( when customization needs are considered, a specialized product can easily start looking like a general-purpose of that domain ) 
6. Slowly introduces version infidelity (a cocktail of customer pressure, delivery delays, competitive threats, and bad design can entice teams into breaking the baseline into customer-specific releases(welcome to product management, service guys 😛 ) )

and this is not a complete list but you get the idea. And we need to keep our teams ready for this ride.How?

I do it with what I call => complexity simulation.

Complexity Simulation

The first and foremost mindset item we need to put into the product team is that “programming language, data structures, frameworks & libraries are choices and you don’t have to always make all of them”. So is architecture ( but then I need to put lots of caveats, so I skipped). Between the time I started my career in products and now; great books and talks have come on this topic. So these days I can simply ask them to read books like code complete-clean coder and likes or ask a senior member to do a session on them( these days lots of code retreats that happen, also help a lot).

The main task of complexity simulation is to give them miniaturized product assignments. The idea is to give them specification that is

1. Not an imaginary app but represents a real-world usage scenario
2. Loose enough to allow choices but clear in scope ( so shared learning can happen )
3. Has at least one occurrence of each QoS [Auth, Logging, Performance and some failure scenario]
4. Is evolutionary (Prompt for a monolith and then graduate to modular or alike)
5. Has some arbitrary and conflicting specifications ( like demand the design to REST midway, demand configurability just before the project is to end )
6. Encourage lots of buzz words and latest hype (and then put requirements that expose their boundaries ).

A common thread in this approach was we would always have one external person walking in on a few occasions. He would sometimes question the approach, change some requirements and make some “suggestions” with intent to disturb their rhythm.

Many times we started with one person assignments and one of the disruptions was to then merge everyone’s code into a single repo. It was fun to see how long developers struggled to agree on common structure and design(despite all the training they went through).

In some cases, we would deliberately split the work across layers or components. This was to expose them to interfacing problems and also the problems on different in speed of individual development(and what they did with it?). In some brave cases, we even suggested code/module swap to simulate take over/maintenance.

Outcomes and learnings

The learnings were immense and broad-based on both sides. It was interesting to notice that when such projects were more functional (than purely technical ones )most developers would do some sort of domain-driven structure to their module. In technology-related assignments, there would be an obsessive overdose of patterns. At times they would suggest modular deployment in the first draft and it was fun to see the reactions (and later learnings ) when I would push for a monolith for the first version. The gradual evolution of their design, later on, made a lot of them smile.

Sanctity of Interfaces would always break! most good developers applied some sort of logging say: a log4 or at minimum console logs but forget to introduce meaningful traceability. Similarly, overlapping functional requirements like 5 types of banking accounts with their own special features and 4 types of roles that can act on them with varying authorization threw a lot of theory they had read off balance(say like REST, OOAD, proto links, schemas , reuse).

We would also schedule periodic peer reviews and I would particularly probe them on places where google’s help was saught and how they fitted in.

The biggest learnings, however, came in the experiment where we asked individual repo’s to be merged into a single code base.

The team would invariably run into chaos as to whose version was better but natural leaders would emerge and finally teams would self organize. By natural I don’t mean the first vocal person to occupy the mind space(which often turned out to be bad at teamwork, mostly the people on standby emerged as leaders who would guide the team to conclusion effortlessly). Tech jargon and buzzwords caused most frictions; it was only after we introduced a few common principles and vocabulary that consensus happened. I could go on detailing various such observations and how they changed as we started getting millennials in teams. But as an objective of complexity simulations, this helped our future teams prepared for the real.

In terms of experience, these teams could be a mix of freshers to say tech lead levels. Sometimes they would be new to the technology or be trained on languages/frameworks that we used.

In my opportunities with product teams, we have done various runs of this. We built: web-based chat applications(to demo lots of protocol and design issues), stock market apps(for RWD, grids and especially the spider web of 2 way communicating components); in some scenarios wrote our own version of MapReduce and toy file-based SQL engine an ESB engine of our own. And a few more which I can’t write lest I leak some company-specific designs. 

The main outcome

One might call this exercise as training. To be fair, much of the training in s/w is focused on teaching nuances of technology or framework and can’t afford to dilute their focus with the above aspects. Nor can hackathons (which are output-oriented as opposed to our approach of increasing the depth).

The biggest and not so obvious objective of my exercises was on the team aspects. Getting help, co-operating, keeping commitments, electing leaders, asking lots of clarifications are key skills. More importantly, managing conflicts, arriving at beneficial conclusions(as opposed to consensus) or responding to later stage changes, design rework and negotiating your way out of arbitrary requirements or time pressure are hard skills.

It is (also) this complexity that needs to be stimulated to working product teams(which cant come just by agile , methodology training). Many of my experiments ran for 2-4 weeks but with more self-aware, well-read and hands-on team members joining our teams these days, we also did shorter runs or few hours a day for a month side assignment. A few cases it would be a one-person run(High potential development,you see 🙂 ) . And there was a different version of this reserved especially for tech leads and budding architects ( more on it in some other post).

The only pitfall was personal equations, hype or self-doubt could seep in silently. But these were expected. And these were my teams so we took care to confront these later on.How? In future posts …

Do tell me what do you think and your experiences as a technology leader in charge of building product engineering teams.

Once you have got past the POC claims in IT appraisals its time to meet its sibling RnD. While the POC claims are easy to handle emotionally the RnD claims might get any techies nerve :).

Research and Development shortened as RnD is probably the most important activity that corporations do. It could the hard research that is done or an innovative solution or marked improvement in service offered by what your corporation deals in. That’s about the formal definition. Much like how innovation, strategy, vision, etc are used in a very diluted fashion we have also got used to RnD as a word used in work life. The misuse of the word is hardly the problem.

RnD claims in IT appraisals are probably the largest claim that could be made for top ratings. The cases are often presented as crisis+void vs heroism as the formula. Sample the following.

• The team was completely new to flutter development and (s)he did lots of RnD in solving the critical issues.
• As we moved from local machines to prod (say cloud) we had many unforeseen bugs to close (!!) and (s)he did lots of RnD…

At times the claims are even sweeping …

• Nobody in the company had worked in AR-VR or IoT or BlockChain and (s)he did lots of RnD …
• (s)he had no background in this technology ( say a .Net person on Java project) and (s)she did lots of RnD …
• There were many critical performance/Security/Deployment/Usability issues and (s)he did lots of RnD …

If you see the structure of these claims it follows crisis+void vs heroism. We must admit that the person in question has put in lots of effort (and must be given credit/reward for that). And that’s about a fair job done as far as performance assessment of an Individual is concerned. 

It’s the “why” of this structure that has a lot to reveal. The RnD claims made in appraisals hint at the following.

1. The team is clueless about the technology
2. The team was not offered adequate hands-on training  
3. The intended structure of knowledge flow/review/mentoring is not working
4. The estimation/Design/Team composition is not correct or clear
5. People are clueless about the complexity that enterprise-scale or products bring in (and the discovery of them is overwhelming and someone the does the RnD)
6. In cases where project takeover/maintenance is happening it might signal a lack of documentation of various forms.

Beyond this list, there are many people-related factors that might be at play: naïve but disillusioned developers, a (pretending) architect in the team, a panic driven style of project management or even a deep organizational culture issue. All of these are complicated issues to solve and appraisals are not the time and place for them to be taken head-on. But if you have some influence or control on the techno-people aspect of your teams the above list has 2 good uses. 

First is, of course, we can take this line of questioning and find out the extent to which we should honor this RnD claims. The list above serves as a simple and neutral line of probing.

Secondly is this is your TODO list. In my experience as a product architect which also was an in-charge capacity, we use these claims as inputs for 

1. Team training and Individual training ( Hands-on assignments, pairs identification, and so on)
2. Team recompositions 
3. Process improvements 
4. Documentations, sample code snippets, bootstrap code projects 
5.  and a big session of how to get help and where to reach out

This is a very broad list, in some cases where we could do deep dive into issues, we also realized that our training needed some fine-tuning. For eg. my freshers always struggle with ng-serve and exposing their apps on machine-specific IPs. In another case, few people could that fathom that their angular app and rest code can be packaged and run as single deployable (because the angular and spring boot were 2 diff training).In other cases, we realized that promises were not covered as part of training (and we needed them) and that the REST was oversold to them. But not everything can be so obvious and clear. Our rule of thumb to our team was “if you are struggling with something for more than 30 minutes please reach out to me”. In our case, I was the old man I/C of tech troubles in the team. But this 30 minutes timeout helped people a lot with burnout and avoided future RnD claims. 

The session on “How to get help and where to reach out” is obvious but not intuitive. Read on. Between the time I spent in product and services we had this new assignment of rescuing a project. Between me our PM and leads we were clear on what design we need to do and how much effort it will take. We even had a detailed plan ready but the time criticality mandated that we get it right the first time. We needed a team that was skilled with the technology and was quickly to reach out when stuck. And we changed our approach to team formation and working.

We did not take new people from a list of xls offered and then call for a “discussion”. We instead called say 50 people in batches. We gave them a very diluted version of the piece from our design and asked to code something around it. Yes, something that they could reason out. Googling was allowed and so was copy-paste. 

And then each one of us would pretend that we are secretly helping them pass the assignment and offer lots of suggestions and debate points. The focus was not on how well they could code a service or create a table-query it or write multi-threaded sorting. The focus was on to see if they got the solution construct right, could they understand when we told them to follow, the choices we prompted them to make or ignore a stackoverflow answer. The experience was enlightening.

People who did not know the technology struggled to find answers to their tasks online (in which case we would tell them to forget the task and code something and explain it(as sort of fresh start/second chance) ).People who understood technology could differentiate between applicable/relevant google /stackoverflow answers and also spot the correct but not applicable answers. People who were skilled at their craft, the ones who understood our tasks at the first sentence, then they would go to their favorite tutorial directly or could quote from a book where they saw it. All this while we were prompting them with different ideas, it was good to know that the ability to seek-receive help is a personal trait (and not related to your skill level directly).We could not only rescue the project but it also rocked (I had moved on by then time it finished). But most importantly we did not slog, no burnout and 7 years later our team members stay in touch fondly.

I am sure that these insights are not novel, there would be some study done on these aspects and a lot written. But to know that people didn’t know how to deliver code and didn’t quite get how to get help was not known as this major a problem. This also made me create a long session on “How to get help and where to reach out”. And I end my session on “if you struggle with something for 30 mins please reach out to me”.

And yes there are still many topics, techniques, and libraries where I and we as a team won’t have first-hand knowledge. We call these as exploration items in our sprints. Either experts or hands-on. The Millennials also loved this growing in into technology experience …

but none made a RnD claim and no brownie points asked for in appraisals … 

Was your appraisal experience this lucky?

One of the responsibilities that comes your way when you are a mid-career techie is annual appraisals. Some of them are your own team members and a lot where you participate as an external voice.

Your opinion as a technical person or subject expert is valued highly when it comes to top performers.

These top performers in the list have typical attributes, they have done lots of work in the project, often they have worked long hours and done “critical” work, won lots of “appreciation”, everybody(read the booses) like them and they have done lots of PoCs!

And what about skills? you may ask. Of course, there are skilled people, the ones on the floor that you have known as good with their craft, ones whose peers go to for their opinion ( and not help, which is overloaded term for outsourcing the hard mental work under guise of help ) and ones whose code is elegant and designs thoughtful. But there are not the traits that get highlighted often.

A manager representing his cases invariably talks in terms of hard work+critical tasks+star of the team that has done the work excellently (yes there is a word like that, an adjective(that I had to swallow many times) ). One might also sprinkle innovation as an adjective, which somehow again goes back to PoCs!

But first, let’s be fair to PoCs as their rightful place in software’s and later we can talk of their misuse in annual appraisals.

PoCs are great and helpful if they are aiming to validate (something). For example: In a mobile app in banking domain, can we skip showing real-time account balance and put Facebook like refresh button? By 2020 this technique had become mainstream but we first encountered it 5 years back it was a first-class case for a PoC on pull-based UI interactions for bank use cases. Another, more software-ly example, can we use actor patterns for fulfilling bill payments and will it spoil the user as well as IT expectations(read service guarantees). Such pointed question on will it work ? or how can we make it work ? are often a good candidate for PoC.

If the question is how will it feel, both in user experience or as a piece of running software are better-called prototypes. ( One might be keen to use MVP concept but the revenue and funding imperatives of startups are way different than typical IT setup ) .Ah yes, we can also call it as demonstration, plain and simple, without the decorative armor of “concept”.  

But all this is to redeem PoCs as a rightful technique in evolutionary software development. This is also a diluted non-exhaustive overview of PoCs. 

Back to appraisals, the list of PoCs that get listed as great points in favor of candidates goes like these. I did a POC on XML to JSON conversion in java (RIP Jackson ). I did a POC on predicting monthly account balance based on spending pattern using NumPy (regressive ! what have we proved here ? (anybody for twitter sentiment analysis (facepalm))). I did POC on ESLint in our CI-Pipeline (ah, not already using? ).

One can take something from docker, AWS, SHA 256, some spring integration scenario, message mapping, DB or Queue that is shiny new thing formulate a sentence on PoC we did.

Often these are the hands-on someone did upon which his boss is marveling and you are expected to put a stamp of approval on it. Given that its appraisal time and you are an external reviewer giving a blunt judgment might not help.

In my conversations with appraisal review committees, I used a gradual probing approach. That helps the representing manager see the real depth on the claim made and it also saves you from the potential pitfall of missing some finer point due to rushed judgment.In simple language the questioning goes like this :

1)    Is that an established fact in industry/community? (No I am not suggesting asking what is the benefit of the PoC, which is already a rehearsed narrative)

2)    Does it help other/newer team members as a reference-sample piece to use? ( in which case it’s a demo piece, a respectable work but outside the greatness claim)

3)    Why did we need it to be proved? ( in case the first question is not feasible, this helps nail the motivation behind the work )

And assuming that the case presented wasn’t a trivial work…

4)    What are the boundaries we have considered for this POC? ( the intention is not to ask assumptions which tend to be arbitrary, but to probe the limits chosen from available feature spectrum )

5)    What is the level of stubbing done here ( this one is a crucial criteria, often the solution which makes this far is heavily stubbed across layers and it’s not told honestly )

6)    What will it take to detail out this concept into complete version (this is the fairest chance given, a purpose and utility-driven work will give a neat list of evolutions that has to happen )

7)    Can we put it on GitHub? ( it might not a feasible but this is one shorthand question to expose petty projects, also this could be asked in any sequence(another variation: can we patent this ?) )

We could always split these lines of probe further and a better assessment of the work presented and see if it’s really something amazing that has been done or is it another case of ADD => Appraisal Driven Development.

Most often our frustration with Poc presented is not that they are some redo of well-known pattern/technique/sample repo. It is also fine and I believe its natural, for people to claim mundane as awesome which one can gently counter in appraisal discussions.

PoCs that are mentioned in IT appraisals are often a demonstration of hands-on ! which itself a such a wonderful thing in skill development that its misuse of PoC is frustrating (and hence this post ).

If you have experience with Spring or MEAN stack, where my viewpoint rests, you would identify with this feeling easily. We get team members, with a fair amount of trained freshers. They have a good general grasp of the framework, say angular or spring. But the moment you move away from mainstream coding scenarios like MVC (in respective frameworks ) the seams start to get loose (be happy if your team is better J ).

An engineered demonstration of hands-on works like wonder in such a situation (I hope your schedule/org/budget allows this, I have been lucky though ).First, it allows your team to try it first hand and be better equipped, it also helps with more reasoned discussion with talent managers with the team member choices are made.

But the real benefit is in what I call as complexity simulation. Enterprise s/w or product development is not easy of a development task to do. There are various Quality of Service criteria that apply to them. It could easily be the ability of design to evolve, the traceability requirements, cloud-native surprises, design compliance, legal requirements on privacy, security. At times one also needs to choose competing frameworks and libraries. And if you are in products we have cross-version feature movement, refactoring, tech debt payments, targeted customization, design compliance and a long list of “work” that is not obvious and throws developers and schedule off balance.

Such situations present a fantastic opportunity for the architects to give targeted demo assignments that simulate the real work complexity to the developers. One recent assignment I gave to my team was to write chat applications using Spring boot,H2 Db and Angular. Absurd? read on. The version one there were asked to keep one on one chat. Later we evolved it to one to many chats. Further, still, we made chat history to be a mandatory feature (exit H2, enter mongo or even MySQL in some cases ). And yes, all this with REST endpoints with swagger.

BY now the browser would crash so I let them put a 1-sec refresh interval. This later evolved into login –oauth –WebSockets –user registration –docker and so on. We could complete this exercise 3 weeks’ time.

Of course, the chat web app that everyone wrote was independent and thus was different in design and form. It wasn’t production-ready. But I had a team that had a more nuanced idea of development, design, and delivery. It was a good proof of capability for a team that would go on to work on chatbot projects (yes we also added coreNLP to the equation).

After these projects or assignments are done the team members got allocated to diff projects, they do well there. And before that, I made it a point to let the new booses know the project they did the exposure they had with me/my project.

It helped them get a realistic idea of their abilities. Many of the design choices, new items, etc that they worked on in my assignments flew into the new projects without getting morphed into PoC and get claimed into appraisals. (of course, all of them had better concrete claims to make in their appraisals minus the pocs : )

Was your appraisal experience this lucky?