Categories
Uncategorized

Swift Property Wrappers

A property wrapper in Swift is a simple and clean way to mix behaviors with properties via syntactic sugar. Simple means that to wrap a property you just declare it with an @ sign and the name of wrapper. Clean means that all the code for the injected behavior lives in one place. You could do this all with functions but property wrappers require less cognitive juggling.

Let’s walk through an example of using a property wrapper to mix constraints with a String property so that we can use it represent a Tic Tac Toe game board.

Here are some well known rules of Tic Tac Toe which we want any representation to conform with:

  • The board has 9 squares, no more, no less
  • The only marks a player can make on a board are noughts and crosses.

A Do-Nothing Implementation

Let’s start with a simple property wrapper that represents a Tic Tac Toe board as a String with no rules enforced…

@propertyWrapper
 struct TicTacToeBoard {
     private var state = ""
     
     var wrappedValue: String {
         get { return state }
         set { state = newValue }
     }
     
     init() {
         state = ""
     }
 }

This code handles getting, setting, and initialization of a wrapped property. Getting, Setting, and initialization are the moments in the wrapped property’s life cycle where we will want to mix-in constraints to ensure the TicTacToeBoard conforms to our rules.

To wrap a property with this wrapper we annotate it with the name of the wrapper inside a struct or class…

struct TicTacToeGame {
     @TicTacToeBoard public var gameState: String
 }

(Obviously in a real Tic Tac Toe game we would have many more properties.)

To test our wrapper we can instantiate a game object and see is we can write to and read from the game state property…

var game = TicTacToeGame()
game.gameState = "_"
print(game.gameState)
// output: _

Excellent! We’re at a good starting point. Our wrapper is working but doing nothing. It’s time to mix-in some behavior with the game state property.

Constraint 1

Let’s start by ensuring the length of a Tic Tac Toe game board, as memorialized in the game state property, is always nine squares long.

(I know that in the real world a Tic Tac Toe game board is a 3 x 3 square but from a state point of view that is an implementation detail.)

@propertyWrapper
struct TicTacToeBoard0 {
  private var state = ""
  private var length = 9
 	 	  	 
  var wrappedValue: String {
  get { // ... }
  set {
    // Constraint #1
    // ensure that the length of the newValue is not too 
    // long or too short
    if newValue.count > length {
	  	  	  	  	  	        
    // Constraint #1a	  	  	  	  	  	  
    // truncate a board that is too long
    state = newValue.prefix(length) + "" 
       // the + "" make the Substring a String
    } else {
 	  	  	  	  	  	  	  	  // Constraint #1b
 	  	  	  	  	  	  	  	  // padd a board that is too short
 	  	  	  	  	  	  	  	  state = newValue + String(repeating: "_", count: length - newValue.count)
 	  	  	  	  	  	  	  	  // cound not use String.padding() because of SIGBART
 	  	  	  	  	  	  }
 	  	  	  	  }
 	  	  }
 	 	  	 
 	  	  init() { //... }
 }

We’ve expanded the set clause to check for a state string that is too long or too short. If the state string is too long (newValue.count > length) then we truncate it. We just throw the characters beyond index 8 away. if the state string is too short (newValue.count < length) we pad the right end with underscore characters. We’re making some harsh calls here, throwing away data, which we will deal with later on.

Note that I had to do some tricks to get this simple code to work. Swift has come a long way in making strings, substrings, and characters interoperable but I still had to add an empty string to coerce the result of prefix() into a string. I also could not use the padding() method to pad the state string because doing so resulted in a crash. Maybe these are bugs or maybe I’m not reading deeply enough into Apple’s String class documentation.

When we run our updated code the result is a game board that is always exactly 9 squares in length…

var game = TicTacToeGame()
game.gameState = "_"
print(game.gameState)
// output: _________

Constraint 2

Now let’s ensure that a game board only contains legal characters. In the case of Tic Tac Toe we’ll use the following characters as symbols:

  • “_” for empty square
  • “o” for nought
  • “x” for cross

Now that we’ve defined how our game universe is symbolized it’s time to put these definitions into code..

@propertyWrapper
 struct TicTacToeBoard {
 	  	  private var state = ""
 	  	  private var length = 9
 	  	  private var legalChararacters = "_ox"
 	 	  	 
 	  	  var wrappedValue: String {
 	  	  	  	  get { return state }
 	  	  	  	  set { 	 	  	  	  	  	  	 
 	  	  	  	  	  	  // Constraint #1
 	  	  	  	  	  	  // ...

 	  	  	  	  	  	  // Constraint #2
 	  	  	  	  	  	  // ensure that the newValue only contains legal chars
 	  	  	  	  	  	  let legalizedState = state.map { legalChararacters.contains($0) ? $0 : "_" }
 	  	  	  	  	  	  state = String(legalizedState)
 	  	  	  	  }
 	  	  }
 	 	  	 
 	  	  init() { ... }
 }

For the purposes of this demo we don’t have to define the meaning of each symbol. We just have to let the game know that “_”, “o”, and “x” are the only legal characters.

We’re using a map function to map any illegal characters to an empty square (“_”). Here we are again, throwing away valuable data because it doesn’t fit our model of what a Tic Tac Toe game is. (We’ll correct this problem shortly, I promise.)

Run a test and we’ll see that illegal characters are replaced and the length of the game board remains consistent…

var game = TicTacToeGame()
game.gameState = "_ox_x_z_x_"
print(game.gameState)
// output: _ox_x___x

Retaining Data

It kind of feels like we are done. We have successfully used a property wrapper to inject our model of a Tic Tac Toe game into an ordinary String object. But I’m still nervous about throw away data. It might be important to preserve the original value of the game state before we sanitize it. Future maintainers of this Tic Tac Toe repo might need to use that data to improve the game. Lucky for us Swift’s property wrapper mechanism has a property called projectedValue which we can use to retain the original game state value.

Let’s implement a projectedValue for our Tic Tac Toe game…

@propertyWrapper
 struct TicTacToeBoard {
 	  	  private var state = ""
 	  	  private var length = 9
 	  	  private var legalChararacters = "_ox"
 	  	  var projectedValue = ""
 	 	  	 
 	  	  var wrappedValue: String {
 	  	  	  	  get { return state }
 	  	  	  	  set {
 	  	  	  	  	  	  // Save the original newValue as the projectedValue
 	  	  	  	  	  	  projectedValue = newValue
 	 	  	  	  	  	  	 
 	  	  	  	  	  	  // Constraint #1
 	  	  	  	  	  	  // ensure that the length of the newValue is not too long or too short
 	  	  	  	  	  	  if newValue.count > length {
 	  	  	  	  	  	  	  	  // Constraint #1a
 	  	  	  	  	  	  	  	  // truncate a board that is too long
 	  	  	  	  	  	  	  	  state = newValue.prefix(length) + "" // the + "" make the Substring a String
 	  	  	  	  	  	  } else {
 	  	  	  	  	  	  	  	  // Constraint #1b
 	  	  	  	  	  	  	  	  // padd a board that is too short
 	  	  	  	  	  	  	  	  state = newValue + String(repeating: "_", count: length - newValue.count)
 	  	  	  	  	  	  	  	  // cound not use String.padding() because of SIGBART
 	  	  	  	  	  	  }
 	  	  	  	  	  	  // Constraint #2
 	  	  	  	  	  	  // ensure that the newValue only contains legal chars
 	  	  	  	  	  	  let legalizedState = state.map { legalChararacters.contains($0) ? $0 : "_" }
 	  	  	  	  	  	  state = String(legalizedState)
 	  	  	  	  }
 	  	  }
 	 	  	 
 	  	  init() {
 	  	  	  	  state = "_________"
 	  	  }
 }

All we had to do was add a var called projectedValue and then assign it newValue before we started messing around with it. To access the projectedValue you use a $ in front of the wrapped property name. Let’s update our previous test to print out both the wrapped property and the projected value…

var game = TicTacToeGame()
game.gameState = "_ox_x_z_x_"
print(game.gameState, game.$gameState)
// output: _ox_x___x _ox_x_z_x_

A projectedValue, like a wrapped value, is not persistent. If we wanted to we could log both values to a file with a IOS 8601 date-time stamp.

When to use Wrapped Values

We could have implemented the behavior of our Tic Tac Toe game board in many ways with the traditional methods and properties of a class or struct. We could have made the game board a type or composed of a set of types.

Classes and structs (types) are excellent for managing complexity. As a codebase evolves over time with many maintainers the power of types helps keep the code organized, documented, and testable. But managing complexity requires its own level of complexity. Sometimes the effort involved in maintaining and extending custom types overwhelms the code itself so that we spend more time refactoring old code than we do writing new functionality.

Property wrappers, which form the basis of powerful abstractions, like @State, @Binding, @ObservedObject and @Published, are used to weave together existing types without inheritance, extension, or composition. @TicTacToeBoard is not a new type, it’s a String type that has been decorated with additional code. Property wrappers are a great way of keeping abstractions separate and independent.

Looking back on my example, I’m not sure it is a good one for the property wrapper abstraction. @State is a good as a property wrapper because it makes any property observable. @TicTacToeBoard is not an abstraction that many other properties are going need to worry about.

Going forward I would use a property wrapper for memory management, state management, storage management, logging, and DB connections. These are all behaviors that Apple should just provide–as they have done with state management via @State. But until Apple does, if they do, you’ll want to have a library of management code injectable though property wrappers for all your properties.

Categories
Uncategorized

Learning by Doing

The Wrong Way

When I learned to code, I thought I was learning the wrong way. The 1980s were the Bronze Age of the personal computer with the Apple II, Commodore 64, Atari 800, and the TRS-80 competing for mind and market share. I had gotten it into my head that every home, school, and office would soon have a PC dutifully managing info and automating tasks like the faithful robots that Isaac Asimov had written about. I felt desperate in my need to understand the inner workings of these heralds of the digital age.

Since I was a cash-strapped college student all I could afford was the deeply discounted and distinctly weird TI-99/4A. I bought the Texas Instruments home computer because its specs were impressive (16-Bit CPU) and it cost about 100 bucks. The TI did not set the world on fire. There was very little commercial software. If I wanted to do more than play a few cartridge-based games and I would need to write my own programs.

There was a very vibrant computer hobbyist magazine market in the 80s. My favorite magazine was COMPUTE! Every issue was chock full of news, speculation, and tutorials for each of the major PC systems. I read COMPUTE! from cover to cover and typed in every TI BASIC program printed each month. Since the TI was far from popular there were few articles devoted to its flavor of BASIC. A few days after I devoured a new issue of COMPUTE! I ran out of things to do with my TI.

Trial and Error

Out of desperation and desire I did something unthinkable! I attempted to adopt the Apple, Commodore, Atari, and TRS programs for the lowly TI. At first my efforts simply failed. The BASIC programming language that all these systems shared was broadly similar, but the display, sound, and file sub systems were very different. I learned, the hard way, that what we might call backend programs today were far more portable than frontend programs. The calculation code of a financial calculator could easily be written in a kind of “universal BASIC” while the charting code had to be redesigned entirely. (We have this problem day! It’s why iOS and Android apps have to be different codebases.)  

I was learning, through trial and error, some of the basic principles of software engineering. Insights that I use in the current Golden Age of cloud computing and internet connected devices. I don’t believe I would have retained, or even truly understood, these principles if I had not attempted to “just do it.” An important part of my learning experience is that I had no idea if I would succeed. It was pure R&D.

Learning by Doing Principles

Here’s what I’ve found to be the important elements of learning by doing (from someone who is learning by doing it).

  1. Limited access to expertise: If we have a lifeline, we’re going to use it. Don’t use it. Don’t Google that error message or look for the answer on Stack Overflow. Just bang your head against the wall until something shakes loose.
  2. Unlimited access to examples: This is what is truly great about open source. We have so many examples of working code. These examples might not directly solve the problem at hand, but they provide hints and inspiration.
  3. Unlimited time: For work and play its best to timebox all our activities lest we miss deadlines or overindulge. But for learning it best to take all the time we need. Learning by doing is exploratory and self-driven. If we rush it, we risk completing the project but not the learning.
  4. Simple systems: The TI-99/4A had a single CPU that executed code slowly, in 16KB of RAM, and plotted graphics on a 256×192 bitmap. There wasn’t much there to distract the learner from her self-appointed task. Lucky for us, every UNIX-based computer has a terminal app that transforms your modern supercomputer into something like a 1980’s home computer. We can even install BASIC! (Well, don’t go that far, Python or JavaScript are fine for learning to code projects.)

The Right Way

In hindsight I realize that my mix of limited resources and unlimited time were a big part of my learning succuss. I had to improvise. I had to stretch and grow. I could not Google or Stack Overflow the answers.  

Don’t take my word for it! Learning by doing (project-based learning) is a pedagogical approved alternative to teacher-led instruction. It also looks a lot like a good Agile development process! At Apple we used to say, “you don’t really know how to write code until after you have written it.” This is why the 2nd  and 3rd  generations of every software system are big improvements. It’s not the new features, it’s the lessons learned by doing that lead to safer, faster, and reliable software.

The Only Way

The core lesson I’ve learned by writing code for 30 years is that we’re still figuring it out. While we have amassed a considerable amount of knowledge the pace of technological change continuously challenges that knowledge. When it comes to coding learning by doing isn’t the wrong way – it’s the only way.

Categories
Uncategorized

Learning from First Principles

A Troublesome Student

I was a poor student in elementary school. I was unable to focus for more than a few minutes at a time. Bored and restless I treated the classroom as a kind of one-man show to entertain myself and earn the admiration of my peers. I remember bringing our family’s fancy silverware to school one morning in 4th grade. During math class I distributed the cutlery to each student at each desk as if we were about to enjoy a meal. But there was no meal and my mother and teachers were not amused.

In this day and age, I would have been diagnosed with a “learning problem.” Most likely Attention Deficit Disorder, the infamous ADD, and perhaps medicated. Luckily, I grew up in the dark ages of the late 1960s. The diagnosis was “troublemaker” and the solution was a desk in the hallway outside of the classroom. I was given only math worksheets and a pencil for entertainment. The educational system had given up on me.

Sitting alone and isolated was one of the best things that ever happened to me as a child.

With no audience and no windows in which to daydream I opened the workbook to page one and started doing math problems. There was nothing else to do and no instructor to rebel against. 

It was slow going at first but by the end of the month I was in love with math and several chapters ahead of the class. I moved up and out from 4th grade into 5th grade math. I was rehabilitated into the inside of the classroom. Eventually, I even started to pay attention to the math teachers. 

I had learned 4th grade math from first principles.

I should note that I’m not a great mathematician. Because of my loner learning style, I don’t do math like classroom-trained people do. For example, I never bothered to memorize my times tables. Instead I write out a table of three number lines, natural, even, and odd. I can then derive any integer product, dividend, or remainder by lookup and simple arithmetic. I had created a tool to avoid route memorization.

More importantly this was my rough introduction into learning how to learn.

Learning without Guidance

Learning from first principles means learning without a guide and building up from what you already know. The most famous example of first principles are the Elements of Euclid. I’m sure you’ve heard of this ancient Greek geometer. He started with a small set of fundamental observations about the behavior of points and lines on a plane and derived hundreds of conclusions that formed the basis of geometry as we practice it today. 

During the early part of my career we had to be a bit like Euclid to figure out how to develop software, deliver applications, and build teams from first principles. Back in the 1980s and 1990s we didn’t have best practices and opinionated programming languages. We had to solve problems with code without guidance.

At Apple we did object-oriented programming before there were object-oriented programming languages. We learned from first principles that function pointers and pointers to pointers (handles) could be used to encapsulate properties and methods into objects. We didn’t invent C++ or Objective-C but when these advances came around, they felt like old friends.

At DoubleClick in the 2000s we did cloud computing without containers or virtual machines. We learned from first principles how to configure hardware and software, so that every server was an exact clone. This enabled ad serving workloads to be automatically propagated through our network so that our QoS was easily maintained in the face of failures. We didn’t invent AWS but when Amazon started marketing their excess capacity, we had a similar business plan on the table. 

The power of learning from first principles is that it empowers you to figure out the future on your own with the tools you have at hand. The best practices and opinionated programming languages if today are the first principle hacks of yesterday.

A Short Guide to Guideless Learning

So, how does one learn this way? How does one start down the path of first principles?

It starts with observation.  Start like Euclid did with a set of simple, self-evident facts that a non-expert can understand. Euclid did this by observing how two points and lines interact to create angles and shapes on a flat surface.  

It helps, in my opinion, to be a non-expert in order to make these observations. I fight hard to ignore what I “know” so I can see what is really going on.

Restrict your observations to the abstract. Euclid only recorded observations are universal regardless of the environment. He focused on how points and lines interact on an ideal surface. Euclid didn’t get caught up in specific details.

Abstraction is a superpower all of us share. Abstraction is the ability to discount the non-essential and to focus on the properties that are shared by seemingly unique entities. It’s been helpful to me to think about effects — not causes — when I’m thinking abstractly.

Build up a foundation of true definitions. Euclid recorded his observations as a handful of definitions upon which he built the empire of two-dimensional geometry. These definitions were Euclid’s building blocks in the same way that the definitions of legal terms build up a contract or the line items of expenses and credits build up a budget.

I find it helpful to record each observation about the system I’m studying on an index card. These cards become my working dictionary for the problem space. As a bonus index card are cheap, easy to order and reorder, and restrict my verbosity.

Apply simple logic to the definitions to make predictions. Euclid used his definitions to build up hundreds of propositions. These propositions were not simple and not obvious, and yet he derived them simply and obviously. When Euclid was done, he could predict the number of degrees in an angle, the length of lines, and the diameter of circles.

It helps to express predictions in diagrams and to show how the logic works in with arrows and annotation. I used to start with the diagram. But if the underlying observations, abstractions, and definitions are not rigorous then the diagram is worthless. I’ve seen these worthless diagrams on whiteboards and in business docs during my entire career.

Be a Lazy 9-Year-Old

The lazy 9-year-old version of me accidently discovered learning from first principles out of a fascination with patterns and a lack of fascination with listening to others. I’ve matured and now deeply value what others, even elementary school math teachers, have to say.

A great place to discover learning from first principles is in the classroom. The student has to be receptive. The teacher has to be empowered. Parents have to be patient. Society has to be supportive.

I still use learning from first principles to this day. It’s how I figure out how to deliver more value with less budget or scale software development with hundreds of engineers. In full disclosure I do read and value all the books and blog post on best practices and prior experiences. But I view this prior art as theory and not law. There is still a lazy 9-year-old deep inside me who isn’t content with being told. I need to find out how things work for myself.

Categories
Uncategorized

Success Means Learning

Much of my success in life I attribute not to the fortunes or misfortunes of my birth (genetics, socioeconomics) but to my ability to learn and act on what I’ve learned.

I could be totally wrong. As an experiment of one all my success could be due to luck and happenstance. I have learned that all-or-nothing explanations tend to be wrong. It’s unlikely that any answer to any complex question is due to a single factor. There is almost always a collaborator lurking in the background!

In the case of learning I have found that the ability to continuously learn is much more important to living in the 20th and 21st centuries than other highly desirable abilities. A good general learner is better able to handle change, surprise, and ambiguity than the expert or the specialist. This is my personal experience of personal growth and of managing hundreds of software engineers and managers over the last 30 years.

Here is a trivial example:

  • JP is an okay coder. Not brilliant but able to reliably get the job done with help from co-workers and stack overflow. 
  • PJ is an excellent coder. Brilliant but only really into one programming paradigm. 
  • JP has no strong opinions and thus their work is all over the place in terms of coding style and techniques.
  • PJ has many very strong opinions on “best practices” and thus their code is clear, concise, and predictable.
  • It is interesting to note that a JP story point tends to be a larger but fixed value as compared to PJ. JP takes about the same amount of time and effort to get anything done whereas PJ tends to be more variable in their output. 
  • A scrum master can sleep on the job with JP but really needs to pay attention to PJ as one never quite knows when or what PJ is going to deliver.

I bet you’ve already caught on to my attempt at fooling you! Over my 30 years of coding I’ve been both JP and PJ. I started out as a PJ and gradually I’ve learned to become a JP. The time and effort it takes to become an expert and to formulate strong opinions has led to diminishing returns. Our world just continues to spin faster with more variability than purists have time to master.

Who would you rather have on your team, JP or PJ? Who would you rather manage? Who would you rather be?

We need both JPs and PJs. I’ve worked at startups and large enterprises where we have tried to hire only one type and that has led to disaster. JPs make few “clever hacks” while PJs hit few milestones. A team of mostly JPs with some PJs seems to be ideal.

The main difference between JP and PJ is how they learn and how they use what they learn.

In the following series of six blog posts I’ll look at how I’ve optimized my ability to learn based on the themes of learning from first principles, learning by doing, thinking with external representation, learning through community, learning through imitation, and the realization that ultimately all learning is self-guided. 

In full disclosure, I have invented none of these concepts and not everyone agrees that these ideas completely model the learning experience. I’m just standing on the shoulders of giants and organizing useful knowledge, in a general way, as best as I can.

I’d love to hear about your experiences with learning so that I can better refine my own ideas (which, BTW, is an example of community learning). 

Categories
Uncategorized

Hidden Bluetooth Menus in macOS Big Sur

Last night my magic keyboard developed a bad case of typing lag. As I was coding in Xcode I observed a huge delay (in seconds!) between pressing a key and its corresponding character appearing on the the screen. 😖

IT Skills Activate

To diagnose and narrow down the problem (Xcode keyboard processing? A rogue process running on my Mac? Bluetooth bug in Big Sur? The keyboard itself?) I did the usual: Googled and looked for typing latency with various apps and keyboards. I isolated and consistently reproduced the problem to the Magic Keyboard. With a wired connection to the Mac there was no lag but over a Bluetooth connection the keyboard was stuttering!

This blog post by Scott Wezey (some guy on the Internet generous enough to share his experiences) seemed to help: https://scottswezey.com/2020/04/12/mac-bluetooth-lag/

Well, in full disclosure, the problem went away all by itself before I got to try any of Scott’s suggestions. I hate when that happens! There is a kind of uncertainty principle of hardware system debugging where just closely observing the problem makes it dematerialize. I’ve found that patiently disconnecting, reconnecting and then if that doesn’t work, rebooting (DR&B) makes 99% of all hardware problems run and hide. I suspect some cache was cleared or some quantum flux was flexed. What ever it was the problem was it is now solved or hibernating and I’m happily typing this blog post with normal latency.

Option-Click Treasure

But Scott did remind me that the macOS is a treasure trove of hidden menus! Holding option while clicking on the Bluetooth icon in the Mac menubar yields additional options! These options are generally features for hardware admins and power users. For example, the clicking on the Bluetooth menu yields a list of connected devices. Command, option, and shift clicking icons (in different key combinations) reveals different sets of features.

Clicking (with the primary mouse button) shows a short list of previously and currently connected devices (with battery level), the ability to toggle Bluetooth on and off, and a link to Bluetooth system preferences.

macOS Big Sur Bluetooth icon menu.

Option-clicking reveals the list with much more diagnostic info: MAC address, firmware version, Role (client/server), and RSSI (signal strength). With this info a good hardware admin can resolve problems with distance and configuration. Well, except that modern Bluetooth devices automatically configure themselves, so really all you can do is DR&B.

macOS Big Sur Bluetooth icon option click 
menu.

Option-shift clicking reveals even more: three utility commands that manually do the work of configuration: Reset, Factory reset, and remove all. Reset is basically DR&B. Factory reset returns the device to it’s just out of the box state. Remove all disconnects all connected bluetooth devices. This last option is a great way to sweep away the cruft of poorly connected Bluetooth devices that might be interfering with each other (or spying on you).

macOS Big Sur Bluetooth icon option-shift click 
menu.

DR&B FTW

The moral of this tale is that when you’re experiencing Bluetooth issues do the option-shift click on the menubar icon if DR&B doesn’t work. You might find that a keyboard and mouse are in conflict or a ghost connection is haunting your Mac!

Oddly the Bluetooth system preferences doesn’t have the admin tools that option-shift clicking reveals. Maybe all this is in an Apple support manual. I can’t seem to find it!

I’ve started a GitHub repo to collect these hidden gems, not just for Bluetooth, but for everything that macOS provides. Please contribute with a pull request!

Categories
Uncategorized

Virus and Science

Illustration by Henrique Alvim Corrêa, from the 1906 Belgium (French language) edition of H.G. Wells' "The War of the Worlds", 1906. A scan from the book.

Like many, my life has been disrupted by this virus. Honestly, I don’t want to even acknowledge this virus. The only virtue of the Coronavirus is that should be widely apparent that we, humanity, are all in the same boat and that boat is fragile.

In the The World of the Worlds, written in 1872, HG Wells wrote about a technologically advanced species invading the earth and destroying its native inhabitants. No forces the earthlings could muster could stop the aliens and their machines. In the final hour, when all hope for the Earth was lost, the “Martians-dead!-slain by the putrefactive and disease bacteria against which their systems were unprepared; slain as the red weed was being slain; slain, after all man’s devices had failed, by the humblest things that God, in his wisdom, has put upon this earth.”

I just want to note that in the world of today we are the Martians. We are technologically advanced, bent on remaking the world, and yet somehow unprepared for the task.

I believe we are unprepared because our political, business, and cultural systems have not kept up with the advances of technical change. I do not believe we should go back to living like hunter-gatherers or the Amish (even the Amish get vaccinated these days). I do believe we should take a breath and catch up with our creations.

The Coronavirus was not created by technology (in spite of the conspiracy theories). Mother Nature is just doing what she always does, evolving her children and looking for opportunities for genetic code to succeed. This is evolution in action and we see it with antibiotic resistant bacteria and the rise of insulin resistance in modern humans. One is caused by how quickly microorganism evolve and the other is caused by how slowly macro-organisms evolve.

We have the science and technology to handle pandemics as well as antibiotic resistance and all the rest, but we have to listen to scientists and doctors. I know that sometimes, science and medicine seems to go against common sense, contradict long and deeply held personal beliefs, and has a habit of changing as new data comes in. This makes science and medicine vulnerable to ridicule, misuse, and misunderstanding.

If we start listening to scientists and doctors, instead of second guessing and villainizing them, species-level problems like pandemics, antibiotic resistance, and global warming will not go away, but we will be able to flatten their curves. If we don’t stop acting like science is just one of many sources of truth, even through we are mighty Martians, we will be felled under the weight of our own ignorance.

In The Age of Louis XIV Will and Ariel Durant wrote about the rise of science from 1648 to 1715, “Slowly the mood of Europe, for better or worse, was changing from supernaturalism to secularism, from the hopes of heaven and fears of hell to plans for the enlargement of knowledge and the improvement of human life.”

Are we stuck in the 17th century or can we move on and accept that we’re living in the 21st?

Categories
Uncategorized

XML and Immortal Docments

I just read Jeff Haung’s A Manifesto for Preserving Content on the Web. He made some good suggestions (seven of them) to help keep web content available as technical progress works hard to erase everything digital that has gone before.

I don’t know if everything published to the web deserves to be saved but much of it does and it’s a shame that we don’t have some industry standard way to preserve old websites. Jeff notes the that Wayback Machine and Archive.org  preserve some content but are subject to the same dilemma as the rest of web–eventually every tech dies of it’s native form of link rot.

For longer than I care to admit (11 years!), I’ve been posting my own thoughts to my own WordPress instance. But one day WordPress or me will depart this node of existence. I’m considering migration to a hosted solution and something like Jekyll. That may well postpone the problem but not solve it. I could archive my words on a CD of some sort. But will my decedents be able to parse WordPress or Jekyll or any contemporary file format?

While I like the idea of printing PDFs to stone tablets from a perversity stand point what is really needed is a good articulation of the problem and a crowd sourced, open source solution.

Jeff’s first suggestion is pretty good: “return to vanilla HTML/CSS.” But what is version of HTML/CSS is vanilla? The original version? The current version? Tomorrow’s version? That is the problem with living tech! It keeps evolving!

I would like to suggest XML 1.1. It’s not perfect but its stable (i.e. pretty dead, unlikely to change), most web documents can be translated into it, and most importantly we have it already.

I know that XML is complex and wordy. I would not recommend XML for your web app’s config file format or build system’s make file. But as an archiving format I think XML would be pretty good.

If all our dev tools, from IDEs to blog editors, dumped an archive version of our output as XML, future archaeologists could easily figure out how to resurrect our digital expressions.

As an added bonus, an archive standard based on XML would help services like Wayback Machine and archive.org do their jobs more easily.

Even better, it would be cool if we all chip in to create a global XML digital archive. An Esperanto for the the divergent digital world! We could keep diverging our tech with a clear conscious and this archive would be the place for web browsers and search engines to hunt for the ghosts of dead links.

Now there are all sorts of problems with this idea. Problems of veracity and fidelity. Problems of spam and abuse. We would have to make the archive uninteresting to opportunists and accept some limitations. A good way to solve these type of problems is to limit the archive to text-only written in some dead language, like Latin, where it would would be too much effort to abuse (or that abuse would rise to the level of fine art).

What about the visual and audio? Well, it could be described. Just like we (are supposed to do) for accessibility. The descriptions could be generated by machine learning (or people, I’m not prejudiced against humans). It just has to be done on the fly without out human initiation or intervention.

Perfect! Now, everything time I release an app, blog post, or video clip, an annotated text description written in Latin and structured in XML is automagically archived in the permanent collection of human output.

Categories
Uncategorized

Mac Terminal App and Special Key Mapping

Mapping key presses in Apple’s macOS Terminal app

For fun I like to write command line applications in C using VIM. It’s like rolling back the calendar to a golden age before mice and OOP ruined everything. The discipline of writing and debugging a C99 program without a modern IDE’s firehose of autocompletion suggestions is like zen meditation for me. I have to be totally focused, totally present to get anything to compile!

Apple’s Terminal app is fine. There are other options, many of them awesome, but as part of this painstakingly minimal approach I just want to stick with vanilla. Even my vim.rc file is vanilla.

So far I’ve only run into one super annoying problem with Terminal and processing key presses with C99’s ssize_t read(int fildes, void *buf, size_t nbytes)!

Apple’s Terminal doesn’t send me some of the special keys by default. Specifically <PAGE UP> and <PAGE DOWN>. And I am betting that other, like <HOME> and <END> may have been overridden as well.

I need <PAGE UP> and <PAGE DOWN> to send read() the ASCII codes "<esc>[5~" and "<esc>[6~" respectively so I can pretend it’s 1983! (The original Macintosh was put on sale to the public in 1984 and after that it’s been all mice and OOP).

But there is a cure for Terminal!

Under the Terminal menu choose Preferences and click the Keyboard tab for the profile you are going use as a pre-GUI app shell. Press the tiny + button to “add key setting”. Select your special key from the key popup and make sure modifier is set to “none” and action is set to “send text”.

If you want tom map <PAGE UP> to its historically accurate function click into the input field and hit the <ESC> key. Terminal will populate the input field an octal escape code (\033).

So far this has been the hardest part of the exercise and why I wrote this blog post for posterity. If you need to remap key codes you probably know that <ESC> is \o33. You might think the letter o is the number 0 but then you have bigger problems if you are writing a C99 program like me.

Anyway, the rest of this exercise just involves normal typing of keys that turn into letters in the expected way!

Making <PAGE UP> send <esc>[5~ to STDIN_FILENO

This is all just bad behavior for so many reasons. What makes the Terminal beautiful it that works with ASCII codes that are both Integer values, Character values, and key codes. If you were to Click Now, you’d have a first hand information from graphic designers and coders on how these terminals can be made eye-appealing. These ASCII code describe the data and the rendering of the data on the screen. If Apple, anybody’s, Terminal app diverts a key code so that that read() can’t read it–well it’s a web browser that doesn’t conform to HTML standards.

You might be thinking: “Who cares about a terminal app in this age of 5G, Mixed Reality, Machine Learning, Cloud-based, Retina Displays?”

Under all our modern fancy toys are command line apps access through terminals. Your web server, your compliers, and your operating systems, are all administered through command lines.

For decades enterprise companies, including Microsoft, have tried to make the command line and ASCII terminals obsolete. They created GUI control panels and web-based admin dashboards. But you know what? They are all failures of various magnitudes–slow, incomplete, and harder to use than typing commands into an interactive ASCII terminal. Especially during a crisis, when the servers are down, and software is crashing, and the OS is hung.

OK, back to work on my C code. I bet I could run over one million instances of this ASCII terminal app on my off-the-shelf Mac Mini!

Categories
Uncategorized

Introduction to Scrum and Management (Part 1 of 3 or 4)

I just finished reading Scrum: the Art of Doing Twice the Work in Half the Time by Jeff and JJ Sutherland . Jeff Sutherland co-created Scrum in the 90s. JJ Sutherland is the CEO of Scrum Inc and works closely with his father. 

Prior to this, I’ve read the big thick technical tomes on Scrum, mostly published in the early 00s, and more blog posts than I care to admit. I’ve also practiced Scrum, at some level and in some form, for the last 15 years. I’ve adopted Scrum and adapted Scrum leading dev teams startups and large enterprises. But I’m not a Scrum master. I’m not trained or certified in Scrum. As is clear from Sutherland’s book, I’m the person in the role that Scrum wants to replace: the engineering manager. 

Even though the father-son team that wrote this less technical and more introspective Scrum book and run Scrum Inc have little use for managers like me, I like Scrum. I’ve seen amazing results from each Scrum practice that I’ve  supported. I was part of the management team at Spotify when we developed the famous Tribes, Squads, Chapters, and Guilds strategy of scaling an engineering culture. From my perspective, when Scrum works, it works well in every dimension. Developers and stakeholders are happy, work is visible and predictable, and products better fit their purpose. 

Curiously Scrum doesn’t like me and my kind-as a manager. And Scrum’s dislike is not unfounded: Most of the resistance to Scrum comes from management. As the Sutherlands note, even after a wildly successful Scrum implementation, it’s usually the managers who “pull back” from Scrum and return an org to “command and control”. I have personally not tried to claw back Scrum but I understand and sympathize with the impulse. 

In this series of blog posts I’m going to explore the relationship of managers and management to scrum masters and Scrum. I want to explore why Scrum left the management role out of the equation and why an elegant and powerful algorithm and data structure like Scrum is so hard to implement without destroying the aspects that make it work in the first place. Finally, I will give some tips to improve the Scrum process so that managers are not the enemy but rather the friend of Scrum.

Before we go I just want to point out that while I’ve read and watched plenty of blog posts, tweets, and YouTube Videos declaring that Agile is dead and that Scrum is Not an Agile Framework neither of these sentiments are true!

Agile and Scrum have problems, mostly because both were conceived with particular aspects of work culture ignored: like managers, governance, telecommunications, and large teams. Agile and Scrum were also cooked up before today’s highly mobile, remote-mostly, co-working culture became popular/possible. That Agile and Scrum have survived these transformations mostly intact points to the strength of these methods of human collaboration.

Agile is not dead and Scrum is a flavor of Agile. Let’s help them live up to their ideals!

Click here for Part Two

Categories
Uncategorized

Big O Primer

This is what happens when you don’t know Big O

Introduction

Big O is all about saving time and saving space, two resources that computer algorithms consume to do repetitive jobs (like sorting strings, calculating sums, or finding primes in haystacks). Big O analysis is required when data points grow very numerous. If you need to sort thousands, millions, billions, or trillions of data points then Big O will save you time and/or space, help you pick the right algorithm, guide you on making the right trade-offs, and ultimately be enable you to be more responsive to users. Imagine if Google Search told users to grab a cup of office while searching for the best coffee shop… madness would break out!

You could argue that Big O analysis is not required when filling a few dozen rows with fetched data in a web view. But it’s a good idea to understand why a naive user experience can create a backend scaling nightmare. Product Managers, Designers, and Frontend Devs will want to understand and be able to discuss why fetching a fresh list of every customer from the database for every page view a bad idea. Even with caching and CDNs Big O leads to better system design choices.

Properties

There are three points of view on algorithm complexity analysis:

  • Big O: upper bounds, worst case estimate.
  • Big Ω: lower bounds, best case estimate.
  • Big Theta: lower and upper, tightest estimate.

The distinctions are literally academic and generally when we let’s do a Big O analysis of algorithm complexity we mean let’s find the tightest, most likely estimate.

There are two types of algorithm complexity analysis:

  • Time complexity: an estimation of how much time source code will take to do its job.
  • Space complexity: an estimation of how much space source code will take to do its job.

Most of the time we worry most about time complexity (performance) and we will happily trade memory (space) to save time. Generally, there is an inverse relationship between speed of execution and the amount of working space available. The faster an operation can be performed the less local space your source code has to work with to do it.

CPUs have memory registers that they can access quickly but only for small amounts of data (bytes and words). CPUs also have access to increasing slower but larger caches for storing data (L1, L2, L3). You, as a coder, usually don’t worry about CPUs, registers, and level 1, 2, or 3 caches. Your programming language environment (virtual machine, compiler) will optimize all this very low-level time and space management on your behalf.

Modern software programs, which I will call source code, live in a virtual world where code magically runs and memory is dutifully managed. As a coder you still occasionally have to step in and personally manage memory, understand what data is allocated to the stack versus the heap, and make sure the objects your source code has created have been destroyed when no longer needed.

Modern software applications, which I will also call source code, live in a virtual world where apps magically run and storage is autoscaled. As a system designer you’ll have to choose your virtual machines and storage APIs carefully-or not if you let managed app services and containerization manage all your code and data as a service.

Usage

Big O will help you help you make wise choices with program design and system design. Big O, like other dialects of mathematics, is abstract and applies to wide range of software phenomenon. Using Big O to optimize your source code means faster web pages, apps, and games that are easier to maintain, more responsive to the users, and less costly to operate.

Big O reminds me of Calculus and Probability! It’s all about estimating the rate of change in how your source code processes data over time. Will that rate of change get slower with more data? That answer is almost always yes! Big O helps you identify bottlenecks in your source code, wasted work, and needless duplication. Big O does this through a formal system of estimation, showing you what your source code will do with a given algorithm and a given set of data points.

There are even more bottlenecks that will slow down your app even if your algorithms are optimized using Big O. The speed of your CPU, the number of cores it contains, the number servers, the ability to load balance, the number of database connections, and the bandwidth and latency of your network are the usual suspects when it comes to poor performance in system design. Big O can help with these bottlenecks as well so remember to include these conditions in your circle of concern!

Big O estimated are used to:

  • Estimate how much CPU or memory source code requires, over time, to do its job given the amount of data anticipated.
  • Evaluate algorithms for bottlenecks.

Big O helps engineers find ways to improve scalability and performance of source code:

  • Cache results so they are not repeated, trading off one resource (memory) for another more limited resource (compute).
  • Use the right algorithm and/or data structure for the job! (Hash Table, Binary Tree, almost never Bubble Sort).
  • Divide the work across many servers (or cores) and filter the results into a workable data set (Map Reduce).

Notation

Big O notation is expressed using a kind of mathematical shorthand that focuses on what is important in the execution of repeated operations in a loop. The large number should be really big, thousands, millions, billions, or trillions, for Big O to help.

Big O looks like this O(n log n)

  • O( … ) tell us there is going to be a complexity estimate between the parentheses.
  • n log n is a particular formula that expresses the complexity estimate. It’s a very concise and simple set of terms that explain how much the algorithm, as written in source code, will most likely cost over time.
  • n represents the number of data points. You should represent unrelated, but significant, data points with their own letters. O(n log n + x^2) is an algorithm that works on two independent sets of data (and it’s probably a very slow algorithm).

Big O notation ignores:

  • Constants: values that don’t change over time or in space (memory).
  • Small terms: values that don’t contribute much to the accelerating amount of time or space required for processing.

Big O uses logarithms to express change over time:

  • A logarithm is a ratio-number used to express the rate of change over time for a given value.
  • An easy way to think of rate-of-change is to image a grid with an x-axis and a y-axis with the origin (x = 0, y = 0) in the lower left. If you plot a line that moves up and to the right, the rate of change is the relation between the x-value and the y-value as the line moves across the grid.
  • A logarithm expresses the ratio between x and y as a single number so you can apply that ration to each operation that processes points of data.
  • The current value of a logarithm depends on the previous value.
  • The Big O term log n means for each step in this operation the data point n will change logarithmically.
  • log n is a lot easier to write than n — (n * 1/2) for each step!
  • Graphing logarithmic series makes the efficiency of an algorithm obvious. The steeper the line the slower the performance (or the more space the process will require) over time.

Values

Common Big O values (from inefficient to efficient):

  • O(n!): Factorial time!
    Near vertical acceleration over a short period of time. This is code that slows down (or eats up a lot of space) almost immediately and continues to get slower rapidly. Our CPUs feel like they are stuck. Our memory chips feel bloated!
  • O(x^n): Exponential time!
    Similar to O(n!) only you have a bit more time or space before your source code hits the wall. (A tiny bit.)
  • O(n^x): Algebraic time!
    Creates a more gradual increase in acceleration of complexity… more gradual than O(x^n) but still very slow. Our CPUs are smoking and our memory chips are packed to the gills.
  • O(n^2): Quadratic time!
    Yields gradual increase in complexity by the square of n. We’re getting better but our CPUs are out of breath and our memory chips need to go on a diet.
  • O(n log n): Quasilinear time!
    Our first logarithm appears. The increase in complexity is a little more gradual increase but still unacceptable in the long run. The n means one operation for every data point. The log n means diminishing increments of additional work for each data point over time. n log n means the n is multiplied by log n for each step.
  • O(n): Linear time!
    Not too shabby. For every point of data we have one operation, which results in a very gradual increase in complexity, like a train starting out slow and building up steam.
  • O(n + x): Multiple Unrelated Terms
    Sometimes terms don’t have a relationship and you can’t reduce them. O(n + x) means O(n) + O(x). There will be many permutations of non-reducible terms in the real world: O(n + x!), O(n + x log x), etc….
  • O(log n): Logarithmic time!
    Very Nice. Like O(n log n) but without the n term so it a very slow buildup of complexity. The log n means the work get 50% smaller with each step.
  • O(1): Constant time!
    This is the best kind of time. No matter what the value is of n is, the time or space it takes to perform an operation remains the same. Very predictable!

Analysis (discovering terms, reducing terms, discarding terms):

  • Look for a loop! “For each n, do something” creates work or uses space for each data point. This potentially creates O(n).
  • Look for adjacent loops! “For each n, do something; For each x, do something” potentially creates O(n + x).
  • Look for nested loops! “For each n, do something for each x”. This creates layers of work for each data point. This potentially creates O(n^2)
  • Look for shifted loops! “For each decreasing value of n, do something for each x”. This creates layers of work that get 50% smaller over time as the input is divided in half with each step. This potentially creates O(log n).