Cooking Healthy Code

Leaky boat

Tue, 10 Aug 2021 16:00:00 +0000

Software systems mirror a leaky boat in many ways. Most folks in the IT organization are trying to reduce and remove the risk that the boat will sink and all its occupants will meet their demise (software fails to meet objectives).

I imagine this analogy has been used before in other contexts, or perhaps even this exact one, but I’m a bit too lazy to go digging through the interwebs for prior work. So for the sake of this article, let’s just assume this is the first time you’ve heard such an analogy.

After engaging in some debates and discussions with colleagues about the proper engineering levels and organizational design, I felt inspired to share my thoughts on the subject openly and invite criticism of the viewpoint.

TL;DR

Technical organizations may want to consider risk assessment as a primary driver of the capital allocation process alongside project and team design. Given a proper risk profile one can design financial guardrails which can be used to inform organizational composition and reduce the impact of risk factors.

Deeper dive

Broadly speaking, IT professionals tend to fall into 1 of 4 groups as illustrated in my highly sophisticated stick figure diagram at the top of this article. To simplify things, I’m focusing on engineering, because it is the department I have the most experience with.

While there is a ton of nuance within a particular job role, I’m going to focus on how each group typically views risk and then suggest a portfolio based approach to managing that risk.

Group 1 - IT Operations

This group is largely concerned with keeping several software systems running and applying patches and updates. These folks tend to be concerned about risks that could impact the current revenue stream of the business. This group typically favors stability and tends to view risk on a system by system basis. They will often recommend a managed process for rolling out changes to production. This is where software engineers skilled in fixing the root cause can patch the software and reduce the overall workload for the operations team.

Group 2 - Software Engineers

Engineers are typically concerned with risk associated with feature production and release. Engineers at most organizations are paid to deliver features to end users based on weekly or monthly sprint cycles. Thus, a typical engineer is concerned with shipping features quickly and reliably while introducing as few defects as possible. Software engineers tend to view risk at the project level. This point of view can sometimes lead to local optimization at the expense of the entire system. For this reason organizations often employ principal engineers to help align engineering efforts across the organization.

Group 3 - Principal Engineers

Principal engineers typically try to mitigate risk associated with a system-wide inability to change over time. Because of the multi-system level focus, these folks typically focus on interface definition and reduction of technical debt as well as the creation of reusable abstractions and frameworks. A common blind spot many principal engineers have is knowing when to pivot to a new technology given the current one still functions. The decision isn’t obvious and can have long term consequences which is why it’s useful to have a big picture leader like a CTO to offer guidance.

Group 4 - CTO

These folks tend to view risks from two primary perspectives in my experience. The first is from a personnel point of view. If staff can’t or won’t support and work on a particular technology stack, the service that stack provides will fall behind competitor solutions. The second risk a CTO is typically charged with mitigating is the risk of business model disruption via technical superiority. This type of disruption typically happens on a long time scale and is often the result of organizations failing to assimilate new technologies into existing business processes which opens the door to nimble startups.

Why talk about risk at all ?

The reason I’m choosing to talk about risk in this article is to suggest that perhaps it makes sense to think about staffing an IT organization from a risk point of view in addition to the more typical project based point of view.

In modern portfolio management one decides on target asset allocation percentages based on a risk tolerance and then rebalances the asset allocation to match that risk tolerance. Labor is an asset that shares many commonalities with other non fungible assets, thus it makes sense to think about it in similar terms.

When we look at labor as an asset allocation problem where that allocation is weighted by risk, it gives us a flexible framework for introducing organizational change to meet our target allocation without getting into the details of every project or process the asset pool must support.

Example labor asset allocation

The following is a very basic example with nice round numbers to illustrate how a company might staff their organization based on this type of asset allocation strategy. A real analysis would take more time, have more risk factors and have numbers associated with real world conditions.

factor 1 year	frequency	impact per incident	yearly cost
downtime incident	6	$100,000	$600,000
delayed feature	2.5	$200,000	$500,000
can’t change	0.5	$600,000	$300,000
disrupted	0.05	$5,000,000	$250,000

Depending on how these numbers change from year to year a company might look to rebalance its labor pool. For example, if downtime became even more expensive and frequent, you’d likely want to increase staff levels to mitigate the impact.

Given our cost analysis we might end up with the following allocation.

level	cost per staff member	staff count
IT Operations	$100,000	6
Software Engineers	$125,000	4
Principal Engineers	$150,000	2
CTO	$250,000	1

The knowledge gap

Fri, 15 Jan 2021 16:00:00 +0000

If you are creating software, business processes, or digital content for profit, then you are a member of the knowledge economy and may find this article to be worth your time.

Knowledge workers can typically be found in the following departments (sorry if I’m forgetting any):

Human resources
Information Technology
Data Science & Analytics
Executive Team
Legal
Marketing

Given the pervasiveness of knowledge workers in most post-industrial companies, all else being equal, it is reasonable to conclude that companies that hire, develop, and retain the most productive knowledge workers will see improved business metrics over the long term and may ultimately dominate their respective industries.

If you buy into this argument, then the rest of the article will likely resonate with you.

Historical knowledge is useful but insufficient in an ever-changing world. Career paths regularly pop into and out of existence. For example, when I first started working as a software developer, there was no such thing as a cloud architect, because compute clouds didn’t exist. Given a quickly changing digital world, adaptation becomes crucial for individuals, teams, and entire companies.

Learning is the process that sentient beings employ to acquire the knowledge needed to adapt to new circumstances. Since adaptation is required, learning is critical.

Given how critical learning is to success, one would think that every individual and organization participating in the knowledge economy would optimize their behavior to become the best learners they could be. In theory, this makes sense, but in practice, I have found there to be a stubbornly persistent learning gap, which ultimately leads to a knowledge gap. I believe the gaps exist because individuals are never taught how to learn effectively and organizations often don’t prioritize it.

The truth about learning

A recurring truth I’ve seen repeated in almost all the books I’ve read, as well as experienced first hand, is that learning is slow. There seems to be no getting around this simple fact. Slow learning is a cognitive sciences first principle.

It’s also a bit intuitive right? Parents regularly experience this while teaching their children how to read. If your children are anything like mine, you’d have observed that it took a minimum of 2 years to become a reasonably good reader.

The CTO of AWS Werner Vogels sums it up with the quote, “There is no compression algorithm for experience” and for years I have been telling project managers that the task they are looking to accomplish, “requires 10 years of experience and then a few weeks”.

The fact that learning can take a long time has a few important repercussions:

Knowledge transfers are largely impossible if performed at the end of knowledge acquisition.
The churn of knowledge workers is particularly expensive for companies.
Training seminars aren’t learning environments.
Small gains in the learning rate can have a huge return on investment.

I will address each point in this list in order, saving my favorite topic, learning rate improvement, for last.

Knowledge Transfers

Knowledge transfers are impossible because knowledge resides in one’s brain and is largely the product of facts and experiences. While individuals may be able to acquire some facts, they likely won’t be able to experience those facts in the same order or under the same conditions as someone else.

Poor knowledge sharing at the organizational level is a form of intellectual debt. Similar to technical, cultural, and fiscal debt, the debt tends to go unnoticed in the beginning and accrue over time. Individual research is often a high-value activity for exploring possibilities in the short term. However, failure to build a team around a promising direction can create an unmanageable intellectual debt burden that ends up costing an organization more money in the long run.

Given knowledge transfers are impossible, workers should shift the narrative and ask their managers what they really want. It’s likely the manager doesn’t actually want a transfer of knowledge, but to instead just get access to key systems and documents.

An alternative approach to knowledge transfers that can help address intellectual debt, is to size projects properly and pay down the costs of knowledge transfers on an ongoing basis. Proper sizing leads to teams of at least 3 with some overlap in skill set. For maximum return on this investment, knowledge workers should be conducting peer reviews of the work being performed.

Lastly, teams of knowledge workers should aggressively build abstractions that reduce the amount an individual needs to know in order to be productive. For marketing, this might look like a press release template. For legal this might be standard language around similar contracts and for software developers, this means leveraging pre-built libraries.

Churn is expensive

I have written at length on my blog about this subject so I won’t bore you here with all the ways that churn happens at a company and will instead focus on the intellectual cost of knowledge worker churn.

Consider that in order for a knowledge worker to be productive they must become familiar with the general business processes, including all the basics that all workers would have to learn. Then that same worker added to a new team, must acquire the knowledge to use the technologies favored by that particular team. After that, a new hire will also need to learn the standards and practices of the team. Lastly, the person may have to learn the history of the work they are being charged with bringing into the future because without a grasp of history one is doomed to repeat it.

In practice, I have found that it takes a minimum of 2 months for most knowledge workers to contribute meaningfully to a project and much more time for projects that are more complex. Learning in a complex project does not scale linearly because you have to learn about each part of the project and the network of connections those parts make. I’ll leave it up to managers to decide how much churn is acceptable and to employ strategies to reduce it, as that isn’t the central focus of this article.

In short, knowledge leaves the organization alongside employees; thus, churn makes organizations dumber.

Training isn’t learning

Training should not be confused with learning. In the knowledge professions, this tends to take the role of conferences, 3-day seminars, and lunch and learns. All three of these things serve an important purpose, but they won’t enable folks to learn meaningfully. In the book Pragmatic Thinking & Learning they call this “sheep dip training”.

I like to think about these types of events as solving the following problems.

Legal compliance
Marketing and sales opportunity for sponsors
A chance to network with colleagues
Initial exposure to new ideas
Certification pre-requisite

Given it solves a number of problems, training is still relevant and useful, it’s just insufficient to truly learn and by extension create knowledge. In order for an organization to create lasting knowledge, it must prioritize and promote learning as part of the typical workday instead of facilitating sheep dip training.

Bend the curve to close the gap

Congrats if you are still with me, we are finally going to get into some meaningful steps folks can take to help improve their learning rates. Improving the learning rate of a team or individual is one of the most productive activities any knowledge worker can engage in.

The reason should seem obvious. Increasing the learning rate allows folks to acquire knowledge more quickly which may help reduce the impact of the previous three issues associated with slow learning. Additionally, improved learning habits will benefit the individual for their entire career.

Learning is still going to be slow compared to most things in life, but it can be a bit faster if one learns how to learn.

Press pause

Folks assume that re-reading and drilling will improve their learning and their ability to retain knowledge. Research suggests that in the short term this works, but in the long term, this behavior can decrease one’s ability to retain information.

Instead of drilling, folks should take breaks of varying lengths between study sessions. The literature suggests that if one waits a while and then tries to recall what was learned before studying again, retention can be greatly improved.

I recently started using babbel.com to learn Italian and noticed they employ this technique. It seems reasonable to assume that a company devoted to teaching languages, would employ the technique, based on evidence of its efficacy.

What’s interesting is that one need not actually remember the content during the break period to see retention gains. The act of pausing and then trying to remember is actually enough to offer meaningful improvements over the long term.

Get moving

Exercise has been shown to improve one’s capacity to absorb information. In the book Spark the author makes the argument and provides compelling evidence that exercise can increase neuron growth and help forge new connections. Additionally, students who exercised before an exam performed better than those that didn’t.

For this reason, I often go for walks during times when I want to increase my capacity to learn. The quiet helps me focus and the exercise helps forge the neural connections that are necessary for long-term learning and growth. I’d encourage others to engage in some form of physical activity often.

Become a teacher

Teachers tend to be excellent learners. They understand through trial and error and formal training, how different people learn in different settings which they can use to refine their own learning style. Additionally, a teacher is almost always required to become a subject matter expert. Moving from novice to expert is a process that many folks never experience, but it is common for educators. Since becoming an expert is often a goal of learning, teaching can be instructive towards our learning goals.

In practice, most knowledge workers won’t just quit their job to become a professor, but that doesn’t mean they won’t be presented with opportunities to teach others. There are many ways we can all learn to be teachers and thus better students. Some of my favorites include parenting, mentoring colleagues, conducting lunch and learns, and writing blog articles. In short, learning to teach helps folks learn to learn.

Write it down

Writing things down can be used in two ways, and both are effective. The first is to actively take notes on a notepad. Handwritten notes may feel inefficient, but it is exactly this inefficiency that leads to better learning and retention. Handwriting notes is a technique that seems obvious, but most folks don’t do it these days because they adopted the practices of reading on an e-reader, tablet, or phone and conducting meetings via computer with screen recording enable.

Typing during a zoom meeting is distracting to all the participants and highlighting on an e-book you never go back to doesn’t offer any long term value. Instead, folks might consider taking purposeful notes in a physical notebook that can be easily referenced at a later date. One of the great things about taking notes this way is that you can quickly draw a diagram to illustrate complex connections between topics which helps reinforce one’s understanding of the subject.

The second way we can use writing to help with learning is to simply write a blog article or book report, in the same vein as a college student writing papers as part of their coursework. The practice of writing should not stop with graduation. Knowledge workers should continue to write about their most interesting projects and thoughts in order to reinforce their own knowledge and help others gain initial exposure to ideas. Ask yourself this; who do you think is going to remember more of this article, you or me? The act of writing is not just a way to share knowledge, but to acquire it as well.

Difficulty Matters

The authors of Make it Stick, make the argument that learning should be difficult, which implies that, easy learning is an oxymoron. Said another way, if it’s easy you likely haven’t learned very much. The book calls these, “desired difficulties”. One example the author highlights is how folks were able to recall more of what they learned when the text they were reading was slightly blurry or skewed.

The book also discusses how struggling to generate answers to tough test questions produced more long term retention. It turns out it’s better for teachers to give hard tests, not only in the interest of assessing what the student learned but also in the interest of enabling them to learn in the first place.

In the workplace, the test might be to apply your newly learned knowledge in a novel way to help solve business problems or field questions from senior leadership. I always advise anyone looking to advance their knowledge and their career to seek out the most difficult projects they could possibly complete successfully.

A failure to solve a difficult problem will yield value in the form of learning and if successful will yield value beyond just learning. Either way, the individual will gain more from a harder challenge than from an easier one. If managers wish for their reports to learn more, then they should work to increase the difficulty of each subsequent project the team is responsible for.

Diversify

The book Range explains, in great detail, how individuals that pursue a wide array of activities, having only minimal overlap, can often produce better learning effects than individuals who are comparatively narrowly focused. The concept flies contrary to the dogma most of us have heard our entire lives. From a very early age we are taught to focus, focus, focus and we’ll achieve our goals. It turns out that this is a false narrative that can lead to successful learning in subjects that benefit from such a style and poor learning in most other areas.

The reason it can be difficult to learn from a narrow focus is that humans use analogies and concepts from other subject areas to form general mental models. Generalized mental models tend to stick with folks more than specific models since they are more widely applicable and therefore command more of the individual’s attention.

In the book Where Good Ideas Come From, Steven Johnson makes the argument that most good ideas come from networks of people. A network of people is by definition more general than a single individual. It thus seems reasonable that one would want to create teams with diverse perspectives in order to improve both learning and by extension idea generation. Knowledge workers should look to a broad array of subjects in order to improve their learning, but also to a broad array of people. Fields that exclude groups of people will fail to learn as effectively as those that embrace diversity.

Book Recommendations

Most of the content in this article was acquired through years of trial and error, lengthy team discussions, and individual study. If you are looking to up your learning game, I would highly recommend reading the following books and corresponding chapters.

Slack by Tom DeMarco: chapter 26 - Where learning happens
Pragmatic Thinking & Learning by Andy Hunt: chapter 6 - Learn Deliberately
Make it stick by Peter C. Brown, Henry L. Roediger III, and Mark A. McDanial
Range by David Epstein: Chapter 4 - Learning Fast and Slow
Spark by John J. Ratey MD: Chapter 2 - Learning
Where Good Ideas Come From by Steven Johnson

The season for code

Tue, 24 Dec 2019 16:00:00 +0000

It’s currently Dec 24th, 2019 and Santa is getting ready to make his rounds. While presents and overeating are great, this time of year is also a time for reflection, and with that in mind, I’d like to share my thoughts about software development and empathy and hopefully make a convincing argument that having empathy will make you a better developer.

Software developers are constantly being tasked with empathizing with others whether they know it or not. In the workplace there are generally three categories of people, that developers, need to empathize with on any given day.

Themselves
Coworkers
Customers

All three of these categories have multiple people functioning in different job roles at different points in their career and existing in the current or future tense. That last point requires some clarification. You need to show empathy to these groups in the moment but also take steps that show you are considering the future when making software development decisions. This may seem obtuse at first, however, entire job functions exist for the sole purpose of aligning current activity with future goals because it can have a large impact on productivity.

Having a deep sense of empathy, and by extension, a deep sense of responsibility to the people surrounding you, is in my opinion, a pre-condition for doing great work. Additionally, empathy is foundational to maintaining a collaborative organization. Without empathy, a culture of shortcuts and blame can quickly take root. In the simplest of scenarios:

Junior manager sets aggressive deadline (a deadline lacking empathy)
Developer writes sloppy code (code lacking empathy)
Bug finds its way into production (lack of empathy for customer)
Senior manager fires Junior manager (lack of empathy for system as a whole)

If one repeats this pattern enough times, a lack of empathy will quickly devolve into a nightmare work environment.

As a developer you might be wondering about what you can do to break the sociopathic cycle and introduce more empathy into your codebase. In the spirit of Christmas, my gift to my readers, is a list of my favorite things for showing software project empathy. When I encounter a codebase that does all of the following things, that codebase just screams:

I CARE ABOUT YOU, MYSELF, AND OUR CUSTOMERS.

Test your code

In 2019, this should be obvious but any code that can’t be tested, can’t really be iterated on. Lack of automated tests is a special kind of negligence. Lack of testing is a very clear indication that you are only thinking about yourself in the present moment.

That said there are some real challenges to fully testing your code. Testing external services is usually what trips people up the most. I’ve found that for these situations it’s best to make liberal use of mocks. At my company agero we use the following languages with the following mocking libraries.

Python unittest.mock, moto
Ruby rspec double
Javascript sinon, rewiremock

Given these great testing libraries, 2020 sure is looking bright.

Static code analysis

Any modern software shop should be using a number of static code analysis tools. These tools help keep your codebase clean uniform and reduce security vulnerabilities. A clean organized codebase implies that the person working on that code is thinking about there own and others future mental health. Just as too much clutter and disorganization in the physical world causes stress, so too, does clutter and disorganization in the digital world. These are a few of the tools that have helped folks at my company get a handle on code clutter.

Document your code

This one is simple. Write comments for your code that clearly explains what it does. Additionally, for languages like python, you can use inline docstrings that can easily be rendered to html.

"""My Module"""

class Foo:
    """My Class"""

    def __init__(self):
        """My Constructor"""
        pass

    def add(self, a, b):
        """Return a + b"""
        return(a + b)

Along these same lines, use variable names for magic numbers and strings. For example, the following code requires the reader to understand what the number 25 means.

def run():
    day = calculate()
    if day == 25:
      return result

The following code is a bit longer but clearly explains through variable naming what the number twelve means.

CHRISTMAS=25

def run():
    day = calculate()
    if day == CHRISTMAS:
      return result

Write (a little) less code

This is tricky and taken to an extreme, writing less code can make your code unreadable, so don’t go too far. Less code means less to manage and less bugs for others to fix in the future which shows that you are thinking about others as you develop a solution to your current problem. Three strategies can help you write less code.

Use stable, versioned libraries.
Write idiomatic code that takes advantage of the standard library.
Use stable managed services instead of rolling your own (using stable managed services also shows empathy to operations folks).

Example of non-idiomatic ruby

data=[1,2,3,4]

def square_list(d)
  new_list = []
  for i in d do
    new_list.push(i*i)
  end
  return new_list
end

Example of idiomatic ruby

This second example is easier to understand and simpler to maintain.

data=[1,2,3,4]

def square_list(d)
  d.map { |i| i * i }
end

Write infrastructure as code

Writing your infrastructure as code instead of clicking through screens shows empathy for others, by making it clear, that you realize they are human, and might accidentally change the wrong config setting otherwise. Coding infrastructure means anyone can look at that configuration, propose changes, and reason about how those changes may impact other systems.

This practice shows that you care about the mental health of everyone that needs to have a working environment. Some tools we’ve used at Agero to great affect are:

Extract reusable libraries

No matter the size of the team, it is a good idea to identify common patterns in your codebase and pull them out of that codebase into an installable, reusable, module. Every major language comes with some form of package management in order to extend the standard library (npm, maven, rpm, pip…) so the tools for sharing are built in.

Extra points for open sourcing anything your company allows. Allowing others to reuse your work shows empathy for another persons time. By giving someone else work you may have spent weeks perfecting, you are saving them the trouble of doing the same.

Adhere to known interfaces

This is something I am a firm believer in and I believe shows great empathy for the person tasked with consuming your interface. Put another way; extend and wrap. Don’t reimplement if an existing standard is already in place.

If you extend an interface without breaking existing functionality, you remove the burden of someone using your software interface having to write special integrations into their codebase. Recently, my team wrote a common logger that is now being used across the organization.

I believe that the reason it was adopted so readily by teams is that the interface remained that of the standard logging library while only changing the output as required by our logging specification. Had we not created this shared module with a stable known interface, every team would have had to implement the interface individually wasting weeks of time or worse they spec would be ignored altogether.

As another example, my team is currently developing a solution that we can use to proxy to 3rd party API’s while extending functionality where it makes sense. The result to developers is that they will be able to use their existing tools to interface with our API and we’ll be able to add value as appropriate.

Cache like it’s your money

Caching is foundational to efficient computer systems. Efficient computer systems means customers get information faster, systems do less work, causing them to use less energy (empathy for mother nature), and often end up costing less than than uncached systems. Caches do create an added layer of complexity, that left unaddressed, will reduce your overall empathy score for the system. But, through clever cache invalidation approaches you can eliminate much of the headache.

Rails uses russian doll caching to help reduce complexities associated with cache invalidation. Fastly, uses a technique called surrogate keys to greatly improve efficiencies with cache invalidation. Lastly, simple tools like memoization can help you cache the result of a method call that would otherwise yield the same result. There is no one size fits all, but cache invalidation strategies are often worth getting right in order to make caching easier to use in your project.

Mentor based programming

If you are a more advanced programmer, you can show great empathy for your coworkers by helping them level up. Selfishly, this will help you in the long run as they will produce better more maintainable code, that you may be tasked with adding too, in the future. Spreading knowledge is a powerful display of empathy that can help others for the rest of their careers. Some habits that help cement this philosophy include:

Detailed code reviews
Pair programming
Open office hours
Lunch and learns
Book sharing
Technical Blogging ;)

Use type hinting

This is a simple coding technique that applies to Python, PHP, and perhaps a few other scripting languages. The premise is simple: Types are hinted in your code by annotating variables. IDE’s and linters can then be used to surface potential data type mismatches in your code.

Adding type hints to your code doesn’t make the code function any differently, but it does help document your code and make is more useable. Thus, given type hinting is completely optional, it really is an act of kindness to your fellow developers when you include it. The following is an example of code with and without type hinting.

As an aside, parsing data to a typed object is often very useful, I’ve found pydantic to be extremely helpful in reducing errors associated with parsing JSON documents.

With hinting

def greeting(name: str) -> str:
    return 'Hello ' + name

Without hinting

def greeting(name):
    return 'Hello ' + name

As you can see the difference is minimal, but in larger projects the impact can be huge.

Named parameters

Named parameters is similar to type hinting in that it documents your code and allows IDE’s deliver more value, which save the developer time and headaches debugging code and searching through docs. Again your code will function without named parameters but will be more difficult for future folks to maintain. Named parameters simply means that you should prefer an actual parameter if you know what it’s going to be ahead of time over an array or hash containing similar information.

In the simplest case I’ll illustrate via a python function.

With named parameters

def greeting(firstname: str, lastname: str) -> str:
    return 'Hello ' + firstname + ' ' + lastname


# call this function via

output = greeting(firstname="Jeff", lastname="Durand")
print(output)

Without named parameters

def greeting(args):
    firstname, lastname = args
    return 'Hello ' + firstname + ' ' + lastname

# this function via (no idea what the parameters represent)
output = greeting(["Jeff", "Durand"])
print(output)

A codebase that passes around a bunch of loosely understood data-structures would quickly drive other developers crazy. This craziness could end in any number of ways, but I’d like to think named parameters helped the next developer looking at it avoid a lifetime of isolation and alcoholism after slowly losing her mind.

Avoid long Regular expressions

This article says it all. Regular expressions can be great when you are looking to filter out a little text or perform some simple validation. But very quickly they get out of hand and when they do, they are special kind of hell. They are impossible to reason about and often very difficult to fully test since matching can be nondeterministic. In general pre-filtering and tokenizing strings before running them through a regex, can greatly simplify things and make your code much easier to reason about.

Use docker for development environments

I find the usage of docker for development environments to be a very empathetic choice in technologies. The idea is simple, developers can use pre-baked development environments instead of building one from scratch. But docker goes a step further than that because it is also a cross platform technology. I realize it might not be perfect in windows, but it does work, and given you can get docker working, you can generally execute all the required code for a given dev environment.

Docker makes it clear that you don’t expect another developer to have a full understanding of the entire environment required to run the project. That understanding and thoughtfulness makes others more productive including yourself.

Merry Christmas

All of these things on my list required empathy and also improve software projects measurably. So this year, think of someone else for a change!!

To go fast, go alone. To go far, go together.

Happy Coding in 2020

Musical Chairs

Fri, 09 Aug 2019 16:00:00 +0000

Most managers understand that employee turnover can be disruptive and expensive. The cost to recruit, train, and provide a competitive wage is typically one of an organizations highest costs in the knowledge economy. Failure to address employee churn and its many different causes can create a downward spiral. If left uncorrected, the churn rate keeps increasing until the organization becomes a revolving door of employees.

First let’s level set; I’m a technologist, software developer, and systems architect. So it may be valid to ask what business I have commenting on organizational & business issues at all. One might argue that these discussions are better left for the MBA folks and Accountants. Fair point, let me first lay out my reasoning around writing an article like this at all.

Conway’s law

organizations which design systems, are constrained to produce designs which are copies of the communication structures of these organizations.
Melvin Conway

Melvin was a smart person and very influential in the early years of computer science. If you buy into Conway’s law (which I do), it’s relatively easy to make the logical leap to the following axiom.

organizational and technical architectures are inseparable

While they are not the same thing, I reason that one simple can’t have a solid technical architecture without a good organizational architecture, if one is producing technologies within the constraints of an organization whose size is greater than that of a single team.

Thus, I believe that offering my perspective around organizational architecture is not only valuable, but a core part of my responsibility as an aspiring leader in the technology field.

Types of turnover

I believe their are at least two types of turnover that commonly occur at any organization.

External
Internal (primary focus of this post)

External

Most folks know what external turnover is, so I won’t spend a ton of time on it. External turnover is when a team member decides to leave the company for a number of reasons, including but not limited to:

Life changing event
Lack of upward mobility
Underpaid for level of responsibility
Burnout
Cultural mismatch leading to infighting
Feeling like work doesn’t make a difference in the world
Loss of faith in senior leadership
Internal Turnover

For example, a life changing event, might include a new child or health issue that needs tending too for example. Burnout is usually caused by unrealistic deadlines and too much routine work, mixed with unhealthy work life balance. Cultural mismatch, is typically the product of differing philosophies about the value of the product being produced, and the best way to produce it. Lastly, a loss of faith in senior leadership, can lead employees to sense the storm about to hit and trigger their flight reflex.

Entire books have been written about reducing external turnover, so I won’t bore you with my take on it. Instead, I’ll focus the rest of this article on Internal turnover, with the goal of explaining what it is, how it can damage a team, and how to reduce it.

Internal Turnover

Internal turnover is primarily attributable to chaotic internal struggles, muddy product vision, or even, too much external turnover. Both internal and external turnover are related and reinforcing. Teams are broken apart and moved around frequently, as management tries to solve a list of real or imagined problems. This movement of individuals, makes it difficult for teams to reach peak performance.

Team forming, and cohesion take a lot of time and effort. Bruck Tuckman was a famous psychologist who studied and wrote extensively about group dynamics. His most notable work, known as Tuckman’s stages of group development, identified four key stages of group development that teams typically go through in order to become productive. The following graphic provides some context around the process.

Note: you can’t skip stages, you must progress linearly. This means that teams that are constantly changing are constantly being “reset”.

In Ray Dalio’s book Principles he outlines how important it is for teams to get in sync. His core principle, states that everyone doesn’t have to agree with one another, but that everyone does need to understand what’s happening and be able to argue for or against it and eventually fall in line when a decision is made. The relevance of this insight is that if teams are continually changing they can’t get in sync, which leads to teams spending all of their time re-litigating things that have already been decided.

Both Tuckman and Dalio speak to the idea, that if teams are constantly changing, they can never reach the performance stage and will be doomed to sub-par performance. To make matters worse, a common “remedy” managers often try to apply to improve teams that have not reached the performance stage, is to try and add more humans, either from within the company, or as new hires. The result is illustrated in the following diagram.

As you can see, the addition of new team members over time could have the result of keeping the team locked in a constant state of forming. When teams are stuck in the earlier stages of development, they are unable to function on their own, and will require much more direction. If managers are unable to move on to other teams, the organization may fail to scale properly. Teams that never reach the performance stage, will always need a dedicated hands on manager. Ideally, managers would be able to manage multiple teams, but may be unable too if teams can’t progress to the performing stage.

Adding more people to a team that hasn’t reached the performance stage yet, reinforces a concept known as The Mythical Man Month. The Mythical Man Month states that; adding another person to an already late project will just make it later.

The notion that managers can add another person and the project will be completed on time, is a hold out from the industrial age. In the industrial age, adding another worker to a factory production floor would increase output and help fulfill an order that was running late. In the knowledge economy, this is no longer true, since communication is at the center of all valuable work being done by the team. Ultimately, when managers add another person, they increase the communication complexity, reset the team dynamic, and thus, slow the overall team production schedule.

My own take on this subject, is that, years of evolution has conditioned humans to view outsiders as inherently less trustworthy than insiders. Groups that are frequently having to assimilate new members, have to spend enormous amounts of time building trust, which can slow down the workflow and ultimately lead to even the top performers becoming frustrated. This frustration will likely lead to top individuals leaving the company, which has widely been documented as an high expense for companies.

To restate the core problem simply:

If teams are changing membership frequently, the team will never reach peak performance, leading to top performers jumping ship, which will doom the organization to a lifetime of substandard results.

Hopefully, the following graphic will help illustrate how all this movement can disrupt a teams ability to perform.

With luck, I’ve convinced you that high internal turnover is a real thing and when it’s high, it can cost organizations dearly in the form of increased external turnover and sub-par team performance. If you do not agree with my premise, you may stop reading here, I won’t be offended.

If you are reading this sentence, I’ll assume you’d like a few pointers on how to limit internal turnover to help your teams become more productive. This list isn’t meant to be exhaustive so please feel free to leave a comment, if you think of other ways to reduce internal turnover rates.

Tips for reducing Internal Turnover

Change Projects not teams

Often, it is better to let high performing teams take complete ownership of a new project than it is to try and stitch together a new team for a new project. Reassigning teams will lead to some members of the team having less than perfect skills for the new project. This should be viewed as a growth opportunity for those individuals and they should be required to learn the new skills. Iterative learning, is at the core of the agile development process which has been shown to be an effective way to improve team output. The strategy for success, is for teams to adjust the project requirements and their skillset over time rather adjusting team composition.

The core principle is that it’s easier and thus more cost effective to learn new skills, and assimilate them, than it is to assimilate new team members. Learning new skills is a win for everyone at the company. Conversely, changing teams, and as a result, being forced to go through the stages of group development again is costly and time consuming with no direct payoff for the employee or the organization.

Tasking a stable team with a new project is much preferred, to breaking up the team, but reassignments can also be disruptive if done too often and without solid reasoning. Moving from half finished project to half finished project can eat up large chunks of a teams time and make them feel like they are never really adding value to the organization. In my experience, most projects have periods of lesser activity; if possible, use slower periods to switch projects if possible as it will be less disruptive and deflating to team moral.

Hire learning generalists in favor of specialists

Learning generalists can complete many different tasks depending on what the project calls for. They are usually inquisitive by nature, which allows them to quickly pick up skills they might be lacking. This makes it easy to task a team of generalists with a new project, removing the need to break teams apart in the first place, as project requirements change.

Generalists will likely have to learn less overall than a team of specialists, since good enough knowledge of a subject is often good enough. Secondly, because generalists have a wide range of knowledge they will have many common interests with one and other, which will help them socialize and form tighter bonds with each other more quickly (birds of a feather flock together).

Many of today’s tasks in the workplace are tasks of integration, not invention. These types of tasks require folks to process a large amount of information, socialize it to others, and integrate it into a complete picture. Generalists tend to be much better at this, as they have a wide palette from which to paint their solutions from.

Obviously, if you are trying to solve a very specific problem, then it may be worth hiring a specialist but I would advise all hiring managers to consider if a specialist is really needed or if it makes more sense to hire a generalist that could learn to specialize in the subject area of interest.

Hire entire teams

If there is a project that demands a specific sized team, try to hire everyone for the project or form the team in one round of hiring. Interview for all positions simultaneously and try to onboard the team as a unit to avoid being reset when each new member joins.

Hiring entire teams can be a great strategy for organizations with the resources to purchase smaller, high preforming, undercapitalized startups. Often the team that makes up a startup is of much higher value than the market share of that same startup. If done correctly, large organizations can jumpstart their team building efforts by purchasing a company. It’s a nice bonus if the company can provide a symbiotic product offering, but that may not even be required if the price is right.

Be careful when purchasing a small startup to make sure a majority of people on the team are on board with the purchase and willing to work for the new company. If the new team is combative with the parent company, this strategy could be more trouble than it is worth.

Avoid mixing outsourced with in-sourced

At a very basic level, team members should have aligned incentives. Without aligned incentives it can be extremely difficult to get team members to work together cohesively. This can be hard or impossible to achieve in an environment where there are two different classes of employee, with different incentive structures, and different bosses; working on the same project.

Often in-sourced employees are more permanent than outsourced. This means that in-sourced employees tend to create solutions that not only meet today’s requirements but also allow for change and the meeting of future requirements.

Outsourced employees are often tasked with spending only the minimum amount of hours to complete the task they have been assigned, after which, they can be assigned to a new team, project, or even another company altogether. The turnover of the outsourced employees can throw off they dynamic of the in-sourced employees by causing them to constantly be adapting to new personality types and skill levels.

The tension created from this dynamic can end up leading great employees to seek teams that don’t have outsourced team members and when that’s not available they may eventually leave the organization altogether. The result is typically, increased turnover and lower overall productivity.

How to be a mentor

Wed, 05 Jun 2019 16:00:00 +0000

I use this blog to organize my thoughts about my professional work, with the intent of helping others organize their thoughts, if they find themselves in a similar position. While much of my work is related to writing code and connecting the technology dots, another large and growing part of my work is centered around technical and career mentorship.

This article will focus on the mentorship part of my life. First I’ll define what I believe the role of a mentor is in a technical setting (may be applicable to other industries). Second, I’ll attempt to impart some wisdom I’ve acquired during my years mentoring, with the hope that future mentors will have a fruitful experience with their mentees.

The role of a mentor

In my opinion the role of a mentor is to provide the following guidance - in order of importance:

Challenge the mentee’s thought process without breaking their spirit or making them feel stupid for not considering the alternative points of view.
Offer information recommendations (books and tutorials) and provide opportunities to learn new skills.
Introduce them to other people that may help shape their career.
Bring the mentee back from the metaphorical cliff’s edge just before they are about to jump (convince them not to quit).
Ask subject relevant questions, that help the mentee consider 2nd and 3rd or effects of the choices they are making.

I hope those bullet points are self explanatory, but if they are not hopefully the rest of the article will offer more practical advice.

Don’t give them the answer

Beginner mentor’s often make the mistake of thinking they are a dictionary to be used as a way for mentees to lookup the answer to a specifc question. Don’t make the mistake of acting like a far inferior version of google or stack-exchange. First you will never keep pace with the vast knowledge the internet can provide, so don’t even try. Second, even if you had a photographic memory you would be doing your mentee a disservice by giving them the answer.

I firmly believe that giving someone the answer to the question is a form of intellecutal theft. Robbing someone of the pride and joy of struggling through, and ultimately solving a hard problem, is antithetical to your role as Mentor. Try as hard as you can to avoid this trap.

What you should do instead

Guide them towards the answer by asking them questions they didn’t think to ask.

For example a mentee may ask if it’s worth writing a unit test for a specific piece of code slated for a production release. You could respond by saying, “yes you always write unit tests, no exceptions”. The outcome is likely, that a test gets written and the mentee is slightly better at writing unit tests.

By telling them yes and giving them the answer, you have actually done them a slight disservice. Giving them the answer inhibits their ability to forge the neural pathway themselves by discoving the why on their own. If instead, you asked a question that explored what would happen if they didn’t write a test for the code, the mentee will typically convince themselves of the idea and cement the reasoning required to forge the idea into a habbit and eventual practice.

Not giving the answer, takes in immense amount of patience, from both parties. It’s important that both the mentor and the mentee understand the reasoning behind taking the long way around, to avoid frustration and resentment.

The movie inception is spot on, with this concept, in my opinion. It’s very hard to convince an individual of an idea. It’s often more effective to guide a person towards believing they had the idea all on their own.

Be less available

It’s tempting to think that always being their for your mentee at all hours of the day will provide them with the best, quickest path towards growth, and enlightenment. However, I’ve found this to be completely untrue in practice. If you are always available, you run a very high risk of creating a codependent mentee, where they are unable to complete work, without your input. This is unhealthy and ultimately won’t serve to enrich the career of either party.

What you should do instead

If the question being asked is simple, and your mentee is using you like a dictionary, I advise just ignoring them for a short period. An hour or so is usually sufficient, then follow up with them and ask if they still need help. If they still need help, guide them with pointed questions, and perhaps a link to some reference material you have handy.

If the question is non-trival and involves more thought, encourage them to schedule something for the “following day” on your calendar. This will give them time to think about questions they want to ask and ponder possible solution and literally, sleep on it. This process of sleeping on it, is really important. What often ends up happening is that, the next day meeting ends up taking the tone of an informed discussion, where the mentee is just verifying their already cohesive thoughts about the orignal question, rather than trying to answer the orignal question.

This process is valuable because it offers the mentor an opportunity to learn something from the mentee and vice versa. In general, discussion is greater than lecture, in the context of your relationship with your mentee.

Discuss your history

Mentor’s often have more domain experience than mentee’s. Openly discussing this experience can provide some great direction for self guided learning. For example, if you have extensive experience with a particular technology. Covering the pros and cons, and lesson’s learned while using that technology, can be incredibly valuable to help focus an individual’s selection, in books or academic courses that can enrich their learning.

Be careful to not to impose too much of your bias on a particular topic. Statements like, technology X doesn’t do this so don’t use it are less helpful than guiding mentee’s towards the technology that would likely be adaquate.

Adaquate is key here. There may be 5 ways to solve a problem where each one is better than the next. It’s not always important that your mentee jumps right to the best solution but that they implement something that solves their problem and then reflect on what they could have done better. As with so many things in life, learning is about the journey and not the destination as long as immediate requirements are satisfied. Be okay with imperfection, but not negligence.

Don’t let them quit

Assuming they have chosen their career path and are pursuing some goal. Don’t let them quit when the going gets tough. Ultimately much of success in this world comes down to grit. How many cycles folks are willing to put into solving the problem makes a big difference in reaching a successful outcome.

Help them achieve smaller wins instead

Try to reframe success in the form of a smaller problem. For example if your mentee says they would like to create a nuclear fusion reactor in the next year with zero prior knowledge of what’s involved, it’s your job to lay out some first steps that will lead them ultimately to conclude that their original plan was unrealistic and they lacked the skills, while still keeping them on a productive path. Instead you might suggest they take an elementary physics course and meet with some experts in the field.

Small wins will keep folks engaged and reduce the likelihood of a quitting and bring folks back from the brink of collapse.

Don’t mentor an unwilling participant

Forced mentorship rarely works. Several times, my mentees have decided I wasn’t the right person for helping them achieve their goals. Additionally, I’ve had to drop certain mentees because they weren’t keeping up their end of the bargain. Missed appointements and closed mindedness are usually the triggers. If the mentee can’t be present and is unwilling to engage in discussion with the mentor the relationship will not be fruitful and it should be terminated.

In addition to not being a good fit in the first place, If you do your job correctly, your mentee will eventually outgrow you. A mentor to mentee relationship may not be forever. Evaluating if the relationship is working every once in a while will help you figure out if it’s time for a change.

That’s all I have for now, I have to give a shout-out to my current employer Agero, while we don’t yet have an official mentorship program, I’ve found that if you’re a willing mentee there are a number folks in leadership positions that openly offer guidance and support. In the cut throat tech world this can be hard to find. If you are looking for a job at a company that supports continual improvement and learning for its workforce, I would encourage you to apply for an open position.

If you wish to contribute to the conversation please leave a comment

Counting on dynamodb

Tue, 02 Apr 2019 16:00:00 +0000

Recently, some folks at my company Agero starting discussing strategies for pre-aggregating data that could be used for statistical compuation based on business metrics. Specifically, the question came up as to how we could maintain event counts.

Counting has a long history in the field of probability and statistics and is in many ways the most foundational form of data required for data science teams. A histogram is nothing more than bucketed counts and a probabilty distribution is just the continous version of the same thing. Maintaining counts is often a requirement of useful statistical computation. Thus, there is value in considering how one might count lots of things.

Having drank the serverless cool-aid a while ago, I’ve been using lambda and dynamodb heavily in recent years. I present the following as a possible solution to maintain counts in this type of environment:

Overview

First a rough architecture:

The infrastructure is pretty simple. We have a table that recieves our event data. This could be purchase data for a customer, ad impressions for a visitor or any other monotonic event source.

When an event arrives in our event table, dynamodb will automatically invoke our lambda function with the operation type “INSERT” and the payload representing the data that was placed into the event table. Based on this event data we construct a “PutItem” operation to upsert into our count table. For our use case we wanted to know, “how many jobs have been placed for a specific customer in the last 24 hours”.

For this demo I decided that minute level precision was close enough, but one could extend this solution to any level of precision. Really it comes down to how much extra one is willing to pay for additional decimal places of precision. There are trade-offs with any architecture and this architecture trades off a small bit of precision for a huge speed improvement on large datasets. Similarly there are tradeoffs in precision and cost in the real world. For example, one might use a tape measure for carpentry but a micrometer for machining.

Given we require minute level precision, we bucket all counts by both hour and minute. To illustrate why, consider the following:

Given this diagram, the problem we are solving for with different levels of granularity becomes clear. When we move to the next hour in our series, we don’t have a full hour of data. Thus, we need to look back at the historical minutes and fill them in appropriately. This design means that our code will have to fetch between [24,24 + 59] dynamodb keys for every count we wish to calculate. The upper bound of 83 keys helps make our system performance more stable than if we had to make calculations based on 0..n number of events. 83 is also a good number as it can be queried without pagination from dynamodb.

The requirement to query up to 83 keys means that if we have very few events per 24 hour period the advantage of pre-aggregating goes away and we likely wouldn’t need to pre-aggregate at all. Thus, this type of counting is best used when we have a relatively high volume of event data coming in.

Implementation Details

This article is not about counting and aggregating generally. It’s intended to help folks aggregate and count things on dynamodb specifically. In this section I’ll dive into some of the techy details. I’m using dynamodb here but this same system could be used on any key value datastore that supports similar operations (Cassandra?).

The first thing we need to do is set up the infrastructure as illustrated in the Architecture diagram above. We use cloudformation to do that. It should be clear that this is creating two tables and one lambda function. The lambda function holds the code for translating events into counts.

Infrastructure

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  ProcessDynamoDBStream:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handle.handler
      Runtime: python3.7
      Policies: 
        - AmazonDynamoDBFullAccess
        - AWSLambdaDynamoDBExecutionRole
      CodeUri: ./streamHandler
      Events:
        Stream:
          Type: DynamoDB
          Properties:
            Stream: !GetAtt DynamoDBTableEvents.StreamArn
            BatchSize: 10
            StartingPosition: TRIM_HORIZON

  DynamoDBTableEvents:
    Type: AWS::DynamoDB::Table
    Properties: 
      TableName: 'event-counter-demo-events-table'
      AttributeDefinitions: 
        - AttributeName: id
          AttributeType: S
      KeySchema: 
        - AttributeName: id
          KeyType: HASH
      ProvisionedThroughput: 
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      StreamSpecification:
        StreamViewType: NEW_IMAGE

  DynamoDBTableCounts:
    Type: AWS::DynamoDB::Table
    Properties: 
      TableName: 'event-counter-demo-counts-table'
      AttributeDefinitions: 
        - AttributeName: id
          AttributeType: S
      KeySchema: 
        - AttributeName: id
          KeyType: HASH
      ProvisionedThroughput: 
        ReadCapacityUnits: 30
        WriteCapacityUnits: 30

Code

# this is your lambda code handler.py
import boto3

table_name = 'event-counter-demo-counts-table'
dbclient = boto3.resource('dynamodb')
table = dbclient.Table(table_name)

# All timestamps are in milliseconds.  These constants make it easy
# to convert units for bucketing
MILLISECONDS_IN_AN_HOUR = 3600000
MILLISECONDS_IN_A_MINUTE= 60000
MILLISECONDS_IN_A_SECOND= 1000

def handler(event, context):
    for record in event['Records']:
        if not record['eventName'] == 'INSERT':
            next

        dynamodata = record['dynamodb']['NewImage']

        # helps us store multiple counters in one table
        counterKeyPrefix = dynamodata['counterKeyPrefix']['S']

        # this helps us set the ttl based on how long we want the 
        # count to remain valid.  Different counters may have different requirements
        retention = int(dynamodata['retention']['N'])

        # snap the event time to the following buckets
        milliseconds = int(dynamodata["insertUtcTime"]['N'])
        hours = int(milliseconds / MILLISECONDS_IN_AN_HOUR)
        minutes = int(milliseconds / MILLISECONDS_IN_A_MINUTE) 
        seconds = int(milliseconds / MILLISECONDS_IN_A_SECOND)

        hourkey = "%s_hour_%s" % (counterKeyPrefix, hours)
        minutekey = "%s_minute_%s" % (counterKeyPrefix, minutes)
        
        # The magic of upserts.  If we have no value set it to 1 otherwise increment.
        # also given Null ttl, set that so we can expire the key automatically and avoid 
        # filling our table with un-used data.
        update_args = {
            'Key': {
                'id': hourkey
            },
            'UpdateExpression': 'SET #count = if_not_exists(#count, :defaultval) + :val, #ttl = if_not_exists(#ttl, :ttl)',
            'ExpressionAttributeValues': {
                ':val': 1,
                ':defaultval': 0,
                ':ttl': seconds + retention
            },
            'ExpressionAttributeNames': {
                '#ttl': 'ttl',
                '#count': 'count'
            },
            'ReturnValues': 'UPDATED_NEW'
        }

        # update the hour key
        resp = table.update_item(**update_args)

        # update the minute key
        update_args['Key'] = { 'id': minutekey }
        resp = table.update_item(**update_args)
        print(resp)

Now that you have code and infrastructure in place you can generate some event data and see the result.

# mockdata.py

import boto3
import uuid
import time
import datetime
from random import randint

event_table_name = 'event-counter-demo-events-table'
session = boto3.session.Session(profile_name='default')
dbclient = session.resource('dynamodb')

table = dbclient.Table(event_table_name)
start_hour = 1
end_hour = 37 # in real life there is no end...

TTL_FUTURE = 90000 # 25 hours in seconds
MILLISECONDS_IN_AN_HOUR = 3600000

# we want time in milliseconds 
second, milli = str(time.time()).split('.')
if len(milli) >= 4:
    milli = milli[0:4] 
else:
    milli = milli.zfill(4)

string_time = "%s%s" % (second, milli)
start_timestamp = int(string_time)

# create a bunch of random data for a 37 hour time period
for current_hour in range(start_hour, end_hour):
    for times in range(0, randint(20, 100)):
        simulated_insert_time = start_timestamp + randint(0, MILLISECONDS_IN_AN_HOUR-1)
        print(simulated_insert_time)

        # simulates 3 counters where we are counting per customer
        provider_id = str(randint(1, 3))
        counterKeyPrefix = "job_event_provider_id_%s" % provider_id

        example_event = {
            'id': str(uuid.uuid4()),
            'insertUtcTime': simulated_insert_time,
            'counterKeyPrefix': counterKeyPrefix,
            'retention': TTL_FUTURE,
            'providerId': provider_id,
            'attribute1': 'data',
            'attribute2': 'data',
            'current_hour': current_hour
        }
        resp = table.put_item(Item=example_event)
        print(resp)

    start_timestamp = start_timestamp + MILLISECONDS_IN_AN_HOUR

Lastly, you can query the counts database for a specific time range, aggregate the counts and do as you wish with that information.

# query.py

import boto3

FIRST_EVENT_TIME = # milliseconds for first event
PROVIDER_ID = 2

HOUR = int(FIRST_EVENT_TIME / 3600000) + 27
MINUTE = int(FIRST_EVENT_TIME / 60000) + (27 * 60)
MINUTE_MINUS_DAY = MINUTE - 1440

MILLISECONDS_REMAINING = FIRST_EVENT_TIME % 3600000
MINUTES_INTO_HOUR = int(((MILLISECONDS_REMAINING / 1000) / 60))

TRAILING_MINUTES = 60 - MINUTES_INTO_HOUR 

session = boto3.session.Session(profile_name='default')
client = session.client('dynamodb')

# create a list of the hour keys we need
hour_keys = [ "job_event_provider_id_%s_hour_%s" % (PROVIDER_ID, HOUR - i) for i in range(0,24)]

# create a list of the minute keys we need
minutes_keys = [ "job_event_provider_id_%s_minute_%s" % (PROVIDER_ID, MINUTE_MINUS_DAY - i) for i in range(1, TRAILING_MINUTES)]

all_keys = hour_keys + minutes_keys

# format keys for dynamodb query
all_keys_query = [ { 'id': { 'S': k }} for k in all_keys]

print(all_keys_query)

resp = client.batch_get_item(
    RequestItems={
        'event-counter-demo-counts-table': {
            'Keys': all_keys_query,
            'AttributesToGet': [ 'count' ]
        }
    }
)
results = resp["Responses"]["event-counter-demo-counts-table"]

# aggregate all values client side
counts = [ int(result["count"]["N"]) for result in results]
total = sum(counts)
print(total)

Fin

That’s all you really need for a basic dynamodb based counting system. There are many other ways you might construct a counting system and this article aims to offer but one approach. Questions and comments are always welcome.

Who are you?

Sun, 10 Feb 2019 16:00:00 +0000

The big question right? Who are any of us? Folks reading this will probably conclude that I’m a self absorbed yuppie for making this the subject of a blog post. While that may be the case (feel free to judge me if you must), my motivation for writing this post is that I view myself as a mentor and as a mentor, I would like to encourage folks to think a little bit about who they are in their professional careers.

This article likely applies to anyone but it’s important to remember I’m speaking through the lens of being a professional software architect and technical mentor. If you are fond of thought exercises you’ll probably enjoy this article. If you struggle with the hypothetical, my guess is this article won’t appeal to you. You’ve been warned.

Why is it important to think about who you are professionally?

Who you think you are and who you want to become, will greatly shape your career path, and largely determine where you fit into an organization, as well as shape what types of organizations you are likely to be attracted too in the first place. Additionally, if you put zero thought into who you are or who you want to become, your identity and career will likely be determined for you.

In my industry, folks have titles like “engineer III”, “senior engineer”, “product manager” or other similarly generic professional names. While these names might be useful for determining compensation from an HR perspective they do little to signal to the rest of the organization what type of person they are dealing with. I would advise anyone, to do everything they can to avoid the company label trap when discussing who they are, because unintentional or intentional it is a trap that could end up controlling you by controlling your identity. To be clear you will still be labeled by the company as “engineer III” but don’t let that define your role and responsibilities and how you interact with those around you. Try not to fall into the trap of being “Just a ___” fill in the blank

Hopefully, I’ve convinced you that taking some time to think about who you are and what you represent and stand for professionally is time well spent. The rest of this article will revolve around some ideas for how you might explore your professional identity.

What is your spirit animal?

Wow! Really Jeff? Yeah I know, this is silly and bizarre and who the hell cares about anyone’s spirit animal? I had this very reaction when someone first brought this line of questioning up to me.

I was working for gazelle.com and we were conducting interviews for a new software developer we wanted to add to the team. We were trying to come up with questions to ask the interviewee and one of my colleagues launched into a story about how he went in for an interview early in his career, and the founder asked him without flinching, what his spirit animal was. We all laughed at the absurdity of it, but then something funny happened. We all went around (half jokingly) and said what our spirit animals were. This of course forced us to think about why a particular animal personified us. The exercise yielded some fun and interesting results.

In the interest of transparency and getting the ball rolling I’ll tell you my spirit animal:

The Emperor Penguin.

The reasons the Emporer Penguin is my spirit animal are as follows:

They are patient (they incubate eggs for up to 68 days in brutal cold weather).
They care about the common good.
They are grounded (they can’t fly, I like to think I’m grounded in my thinking).
They are a bit silly (on land) but it doesn’t seem to bother them.

I encourage you to think about what your spirit animal is and why. It can be a fun / silly way to have a moment of reflection.

What would have been your profession 200 years ago?

This question forces you to think about your values more than your current position. For example, if you are a salesman you might say that you’d have been either a bare knuckle boxer or maybe a fortune teller (I’ll leave the reader to determine what I think about salemen). Try to forget the fact that, 200 years ago most people were farmers, and try to come up with something creative.

To get the ball rolling I’ll go first.

I believe I would have been a bar keep or Inn Owner (truthfully this still sounds like a great option). The idea of operating a hub for discussion and ideas is incredibly appealing to me. Taverns and dinner tables are the original open source in my opinion. Get a few drinks in folks and ideas tend to flow. Taverns are where deals were struck and decisions were made in the absense of a more formal settings.

Lastly, a bar keep is often a sounding board for people’s problems which keeps him in the know about not only that individual’s issue, but also would help one put their finger on the pulse of the entire community. Getting the unfiltered story from a group by talking to them individually can often lead to solutions and ideas that would otherwise prove elusive in a more organized meeting style setting.

Parting thoughts

There are likely a number of other questions that are even better at giving you some clue about who you are or might become professionally.

In closing: My title says “Software Architect” and that is the job that I perform. However, I like to think I approach that job with penguins and bar keeps in mind. Hopefully that approach makes me more effective in my job role and more fulfilled in my career.

Please leave a comment if you have who are you question you’d like to share or if you just want to tell me about your spirit animal :)

python builder pattern

Fri, 28 Dec 2018 16:00:00 +0000

After years of ruby and javascript I’ve started doing some work with python recently after joining the team at Agero. One of the first things I noticed was that a lot of folks in the python world pass around a lot of named arguments as a convention. My first example of this is the boto3 library used to interact with AWS in python (for proof, click here)

In an effort to reduce some of this parameter passing folks might be tempted to make their own class that reduces the number of parameters one needs to pass around. I offer a simpler functional solution taken from some patterns I’ve seen used in the functional javascript community.

Enter partial function application with a builder pattern. The following code highlights how one might create reusable functions without creating a domain specific class hierarchy.

Take the boto3 library. More specifically, take the scan function for a dyanmodb table. The following is an example of how one could use this concept to produce a convenient way to iterate through all items in a dynamo collection.

# ./function_builder.py
import functools

class FunctionBuilder:

    def __init__(self, func):
       self.func = func

    def having(self, *args, **kwargs):
        self.func = functools.partial(self.func, *args, **kwargs)
        return self

    def get(self):
        return self.func

# ./dynamo_iterator.py
class DynamoIterator:
    
    def __init__(self, func):
        """Given dynamodb function that returns items and a LastEvaluatedKey iterate
    
        Keyword arguments:
        func -- function that takes LastEvaluatedKey and returns { items:  [] }
        """
        self.func = func
        self.last_evaluated_key = None
        self.first_fetch = True

    def _get_items(self, db_result):
        return db_result.get('Items', [])

    def _has_more_results(self):
        return self.last_evaluated_key and len(self.data) == 0

    def __iter__(self):
        """Required to implement the Iterator interface in python"""
        return self

    def __next__(self):
        """Required to implement the Iterator interface in python"""
        if self.first_fetch or self._has_more_results():
            self.first_fetch = False
            result = self.func(LastEvaluatedKey=self.last_evaluated_key)
            self.last_evaluated_key = result.get('LastEvaluatedKey', None)
            self.data = self._get_items(result)

        if len(self.data) <= 0:
            raise StopIteration

        return self.data.pop()

    """Python 2.7 requireds next vs 3.7 that requires __next__ hence we alias the function"""
    next=__next__

Now that we have these two simple base classes we can iterate through a collection of dynamodb objects using the following code.

# ./main.py
import boto3
from .dynamo_iterator import DynamoIterator
from .function_builder import FunctionBuilder

dynamodb = boto3.resource('dynamodb')
table = self.dynamodb.Table('my-table-name')

partial_scan_all = FunctionBuilder(table.scan).having(Limit=100, Select='ALL_ATTRIBUTES').get() 

# this will iterate through every item in a dynamodb table.  Obviously not meant for quickly returning values to a user interface.
for item in DynamoIterator(partial_scan_all):
    print(item)

Hopefully you found this code example useful. If you have any questions please leave a comment and I’ll try to help with an answer.

Static asset headers in rails 5

Wed, 09 Aug 2017 16:00:00 +0000

First things first. Conventional wisdom tells us we should be serving our static assets from something like nginx because it’s optimized for doing such a task. Generally speaking I disagree with that thinking and instead offer a different solution. One should be serving static content out of a CDN and that CDN should be caching your content at edge locations.

Given we want to cache content at edge locations we will want a mechanism for specifying in our rails application how we would like our our CDN to cache the content. As in, how long should the CDN hold onto the content before returning to our origin server to fetch it again? There are two ways that you might accomplish this.

Configure the CDN to cache the correct thing by overwriting headers produced by your application.
When a resource is requested from your origin provide the proper headers so that the CDN knows how to handle the asset.

In my opinion option 2 is much preferred. Allowing your application to specify headers gives you more control and keeps the logic of your application where it belongs. Assuming I’ve convinced you to use a CDN and configure the caching mechanism to work via headers specified at at the origin. The logical question is how do I configure the proper headers such that my CDN will cache my assets.

The following code works with the fastly CDN service but probably works well with many others.

This code configures the browser cache and your CDN to cache assets for 10 years. Since we are digesting all assets, meanings we are creating new versions if anything changes, we can cache these assets forever.

# in your config/application.rb file
config.public_file_server.headers = {
  'Surrogate-Control' => 'max-age=315360000',
  'Cache-Control' => 'maxage=315360000, public, no-check',
  'Expires' => 10.year.from_now.httpdate,
  'Date' => 4.days.ago.httpdate,
  'Last-Modified' => 10.days.ago.httpdate
}

The result of this config behind a proper CDN is that you assets will end up being fetched exactly once from your rails application and then never again until you clear the cache. Even then, the browser cache will ensure that only new visitors to your site will need to request the assets. In my opinion, Rails is perfectly well suited to serving up the occassional asset in this context and will likely have little impact on overall performance. Furthermore keeping the caching logic in the code base has obvious utility including allowing caching headers to be tested more easily via unit tests.

As always, leave a comment if you found this useful.

A gentle introduction to Machine Learning

Thu, 01 Dec 2016 16:00:00 +0000

I recently completed a great course on Machine Learning, the course is offered by coursera so check it out if you find this article interesting. Until now, I’ve only been a passive consumer of machine learning in my own application development and home life Amazon Echo. As a scientist and engineer, I always knew that some interesting algorithms were crunching data and then making complex decisions behind the scene, but I didn’t really understand the mechanics of that process. I believe writing is a great way to share information as well as solidify one’s understanding about a particular subject and so it is with that reasoning that I write the following article. If you don’t care for understanding how machine learning works but are more interested in why it’s important right now feel free to skip over the details.

The basic idea behind machine learning.

Machine learning is concerned with looking at a source of data and figuring out (fitting) a mathematical function that characterizes the behavior captured in that data. Let’s say we have a farm and that farm uses water and fertilizer we record that data every year and then associate it with a corn yield. So that table of data may look something like the following.

fertilizer(lbs)	Water (G)	Corn(lbs)
100	600	800
110	400	600
80	700	500
120	600	810

The challenge is to find a mathematical function such that given some value of water and some other value of fertilizer, we predict the amount of corn output we should see for the season. Anyone familiar with basic economic analysis will recognize this as a simple linear regression problem than can be solved quickly with a little linear algebra. Typically one would never use machine learning to solve such a simple problem as deterministic methods work just fine at this scale.

What does this have to do with machine learning?

As the number of data points gets larger and you have more and more components contributing to the “Corn” yield, the calculations required to solve the problem via deterministic methods become more and more computationally expensive and complex. At a large enough scale you could have a problem that would take an entire lifetime to solve. Thus we have to use more involved mathematical techniques that estimate the solution rather than solving exactly.

In addition to complex linear problems, we might not be trying to find linear functions at all. This leads us to explore techniques for finding more complex functions that can make better predictions given a new set of inputs (features). It turns out that machine learning techniques can fit a wide variety of functions and thus have broad appeal across domain models.

Least Squared Errors (linear models)

Let’s say I have some data and I’m looking for a function to characterize that data. A measure of how well my function does at classifying the data could be how far away from a line each data point was. This is the error between my function and the data it represents. In order to account for points below and above the line and count them equally, one might square the error. If you add up all the squared-error values you get for a particular function, you have a way to evaluate how good your function “fits” the data you have.

It’s worth noting that different problems involve different functions that measure the amount of error, but the premise remains the same. The error produced is often referred to as the cost of the model and thus the error measuring function is referred to as a Cost function.

If you have a way of measuring how much error your function has then you have an objective way of deciding if one function is better than another. Machine learning techniques offer us a way to discover the function that works the best without looking at an infinite number of possible solutions.

Gradient Descent

The next logical question is how do I quickly find the best function given an infinite number of possible functions. The answer lies in differential calculus. Calculus tells us that if we take the derivative of function and set it equal to 0 then solve the equation we will get the optimal values. Now since we don’t know the function in advance, things aren’t so simple, but we can exploit this truth. Mainly we can get a sense of the direction where the minima lies by calculating the partial derivatives given our current best function. We can then use this information to take a small step in a direction towards a better function. We do this until a specific number of iterations or until we are no longer making much progress at which point we stop and assume our solution is good enough. Here is a video illustrating the concept clearly.

This technique forms the foundation for most machine learning problems. Solving different problems is simply a matter of formulating them so that they can be solved using this technique.

Why is machine learning so exciting?

The reason machine learning is exciting is because a huge number of problems can be re-formulated as probabilistic models. Meaning sometimes we only care that an answer is close enough to the correct answer and we’d be willing to trade compute time human or computer to get an answer that’s 99.999% correct. Furthermore, some problems only make sense in terms of probabilities. For instance, if you are predicting if it’s going to rain 5 days from now it would not make sense to say there is 100% chance of anything happening in the future.

why is machine learning so much more exciting lately?

The reason is pretty simple. We now have access to massive amounts of data, the likes of which the human race has previously never recorded (the amount of data gathered per day is accelerating). Secondly, the semiconductor industry has managed to make silicon that is particularly well suited for Linear Algebra calculations (even if they didn’t, the fact that many linear algebra operations can be parallelized over multiple machines means we could just use more off the shelf computers to do our calculations anyway). Thus, we now have the compute infrastructure to make use of this ever increasing data set.

So what? how does more data improve the solutions to our machine learning problems?

It turns out that there are a huge number of problems that perform better, the more data one has. Think about what we are trying to do. We are making mathematical models that estimate the real world, many times the data we have describes the real world, the better our description of the real world the better our understanding could potential be and thus the better our prediction about what might happen in that world might be. Having more data and compute power allows computers to understand the world through data in ways that have never been possible.

How does all this relate to Artificial Intelligence?

Think of artificial intelligence as systems of machine learning models adapted for human consumption. The idea is that you want to make a system that can take some input, run it through 1..n different machine learning algorithms and other filtering processes and then spit out a useful result for a human. So machine learning algorithms are just one of a number of sub-systems that make up an artificially intelligent system. It could be that a system like Amazon’s Alexa uses hundreds of different machine learning algorithms to present the user with one fluid experience. The average consumer will never interact directly with a machine learning model but almost everyone will interact with an artificial intelligence system.

Conclusion and thoughts

Learning these concepts has been an eye-opening experience and I believe there are a lot of interesting applications that will be created as things progress in this space. Clearly, this article is a Math lite summary about machine learning but I encourage you to learn more by taking the coursera course (amazingly it’s totally free). I plan to use machine learning models to make more intelligent systems for hobby work (drones to chase the squirrels from my garden anyone?) or for profit (fraud detection, customer sentiment,product recommendations, and BI based decision making). I Can’t wait to get started.