Principles of Modern Software Development

The how and why of embracing change

Panu Kalliokoski
24.8.2015
CSC

Practicalities

• three-part training with a theory session (this one) and two hands-on workshops
  • the workshops put the theory into practice on specific development platforms
• more specific topics will be covered with forthcoming focused birds-of-a-feather meetings (BoFs)
  • these will be decided by public demand
• we also want to set up a permanent structure to spread and share best practices of programming

What does modern mean here?

• there are essentially only two software development methodologies:
  1. fiddle with it until it works (and if it ain't broken, don't fix)
  2. endlessly improve the process of service development
• it's the latter methodology we consider modern.
• many tools and practices for this have been invented over the last five years!

What do principles mean here?

• principles are clear-cut rules: do this, don't do that.
  • so that it really should be clear whether you are abiding by the principle or not.
  • we are trying to strike a balance between a practical, down-to-earth approach, and wide applicability
• we try to pick the most important principles
  • as they are important, they really mean if you don't follow this, you're screwed.

Course content preview

• purpose statement
• some underlying principles
• some actual principles
• open discussion of existing CSC practices
• roadmap to improvement
• other checklists

Some underlying principles

This section is to set up the goal of forthcoming principles.

In this section:

• why embrace change
• how to embrance change

Why embrace change?

because it exists, d'oh

Why embrace change (even more)?

• some aspects of updating a service are marked with fear, especially fear of breaking things
• this fear is often subconscious, preventing people from undertaking a development task
• the psychological reasons for this can be countered with e.g. refactoring low-hanging fruit
• however, the technical reasons for fear are countered with modern software development practices.

How to embrace change?

• publicly keep track of everything
• automate everything that is possible
• find problems proactively as soon as possible
• use safety nets for easily rolling back changes
• keep things as simple as possible (and not simpler)

Time for break or discussion?

Publicly keep track of everything

• everything means: known problems, plans, schedules, work, source code, build process, deployment process, testing process, existing deployments, existing tools, communication channels, software/service metrics, possibly more
• how: issue trackers, version control, automation scripts, operations databases, documentation

Automate everything that is possible

• everything means: changes to everything you keep track of, everything that takes more than one step / command
• three strikes and you automate
• how: scripts, build systems, integrations between tracking applications, periodical batch jobs
• corollary: never edit by hand anything else than source. (Use e.g. templates: m4, cpp, Jinja).

Find problems proactively as soon as possible

• problems mean: technical and semantic bugs, missing or wrong documentation, missing automation, bad performance, security bugs
• proactively means: you should be looking for problems (automatically)
• how: regression tests, test-driven development, profiling, code reviews, hallway usability testing, new integrations, unit tests

Use safety nets for easily rolling back changes

• changes mean: changes to code, changes to dependencies, changes in hardware, changes in deployment, changes in specifications, changes in API's, changes in contracts, and more
• how: version control, multiple deployments, isolated testing environments, make it configurable first, API versioning, one model, many views

The name of the game is to avoid decisions, at least the big and painful ones. Linus Torvalds

Some actual principles

Now we look at practical principles that help fulfill the aforementioned goals.

In this section: use a bug database, use source control, fix bugs before writing new code, automate tests, automate builds, automate change staging, refactor, explicitly declare and isolate dependencies, make daily builds, have someone responsible for testing, make code reviews, document your use cases, do hallway usability testing, design to distribute and scale with the process model, ensure uninterrupted work, separate stateless and stateful components.

Use a bug database

• e.g. a text file, Excel sheet, SQL database, Jira, RT, trac, roundup
• to:
  • prevent forgetting
  • enable priorisation
  • track progress
• write down everything you can't handle immediately.
• preferably use a bug database that can be updated directly from commit messages (in source control).

Use source control

• e.g. git, Mercurial, Subversion, CVS
• to:
  • never lose code
  • enable gracious rollback
  • find out exactly what changed
  • overcome psychological challenges to ditching cruft
• you should keep everything under version control that is not automatically generated.

Fix bugs before writing new code

• that is, find bugs first and prioritise things that are done badly
• technical debt builds up to the point where no progress can be made
• corollary: trying to hurry a project by taking shortcuts will make it just worse.

Automate tests

• when you know what something should do, write a program that tests it does that
  • when something goes wrong, write a test that would have caught that
• to:
  • prevent omissions in manual testing
  • have a quick way to verify changes
  • prevent remotely-caused bugs
  • have a metric of progress
• it feels stupid when you do it for the first time, but when the same test fails after change #115 to another component… you will feel the gratitude.

Automate builds

• e.g. make, ant, maven, distutils
• to:
  • prevent mistakes in manual processes
  • document the build process
  • easily try changes in the build process
  • test the effects of toolchain changes
• corollary: testing the build still works is the least you can do after changes. Make it one step (at most).

Automate change staging

• e.g. make, install scripts, puppet, ansible, dpkg, rpm
• to:
  • prevent mistakes in manual installation
  • document the installation process
  • scale up with minimal effort
  • failover with minimal effort
  • easily test dependencies of the OS platform
  • ensure different deploys correspond to each other
• ideally, it should be one command from checked-out source to working deployment on any target server / working distribution uploaded

Refactor

• refactoring is the act of changing code without changing its behaviour to make it more maintainable
• to:
  • shorten code and remove duplication
  • make forthcoming changes easier
  • improve understanding of the code
  • get familiar with code written by someone else
• refactoring is very easy if you have automated regression tests!

Explicitly declare and isolate dependencies

• e.g. virtual servers, virtualenv, chroot, lxc, docker
• you can't know your deployment automation is properly installing all dependencies unless you deploy to an isolated environment
• to:
  • document the installation process
  • find missing dependencies
  • know what to reconfigure when moving the service
  • decrease fear of changing deployment platforms

Make daily builds

• e.g. cron, Jenkins, Bamboo
• to:
  • proactively catch breakage
  • provide easily-deployable packages for internal or external use
  • generate statistics
  • measure progress
• daily build should be as inclusive as possible: e.g. also run all tests, make distributable images, run it in a testing environment, etc.

Time for break or discussion?

Have someone responsible for testing

• you should do testing, someone should specialise in it
• to:
  • proactively catch bugs and improve quality
  • ensure that tests will get written
  • ensure that API specs are understood consistently
  • try out stuff the programmer did not think about

Make code reviews

• have other people watch through the content of your commits, especially people who should be using the code
• to:
  • spread information
  • promote collective code ownership
  • get suggestions
  • improve quality

Document your use cases

• if you don't have a spec of what people are supposed to do with your software, it's hard to make it useful
• use cases can also be machine readable functional tests
• there are numerous methodologies
  • what I advocate is that use cases are at least complete sentences, tell how you can use the software, and are written using very concrete terms.

Do hallway usability testing

• for usability, the best testers are those who don't know anything about the software
• you may educate the testers about what they should achieve, but not about how the program works
• watch very closely how they go about it — every unnecessary click they make is a slight usability problem.

Design to distribute and scale with the process model

• e.g. pipes / sockets / HTTP for inter-component communication
• to:
  • decouple your components
  • make API's that are easy to debug
  • make it possible to install components on separate servers
  • enable a full N-to-M mapping between frontends and backends

Ensure uninterrupted work

• sometimes it takes very much time to get efficient
• that amount of time is wasted every time you need to switch your focus to something else

Separate stateless and stateful components

• stateful services can be handled by e.g. SQL/NoSQL databases, memcached, cloud services, IMAP or logging servers
• all application logic should be in stateless processes
• to:
  • scale up in a simple way (install more deploys of stateless services)
  • decouple model from views
  • enable multiple views (on multiple platforms)
  • simplify application logic code

Stuff to discuss

Time for break or discussion?

• existing practices in CSC
• how to apply these principles to practice?
• topics for BoFs?

Where to next?

In this section:

• other checklists
• types of testing
• other levels of improvement

Other checklists

• Security: OWASP Top 10
• Coding best practices: Deviq principle list
• Release engineering: Google checklist
• Software development: The Joel Test
• Automatically cloud-deployable web applications: The 12-factor application

Types of testing

• how to test
  • invariant testing
  • unit testing
  • functional testing
• what to test
  • usability testing
  • performance testing
  • security testing
  • integration testing

Other levels of improvement

• coding style: how to write beautiful code
  • writing self-documenting code
• three rules of optimization
• building a community
  • openness
  • licensing
• keeping management happy